Display method in screen-off mode and apparatus

Abstract

A display method in a screen-off mode and an apparatus. The method comprises: an electronic device enters a screen-off mode, and displays a screen-off display object on a display screen in the screen-off mode; on the basis of human face or human eye detection, the electronic device determines the orientation of the human face or human eyes, the orientation being associated with the relative position between the human face or human eyes and the electronic device; and, on the basis of a change of the orientation of the human face or human eyes, the electronic device dynamically changes the screen-off display object. Thus, the effect of dynamic interaction between a screen-off display object and a user is achieved.

Description

Display method and device in screen-off mode

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 29, 2023, with application number 202311112182.7 and application name “A method for interaction and electronic device in screen-off state”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 8, 2023, with application number 202311167861.4 and application name “A method and device for display in screen-off mode”, all contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to human-computer interaction technology, and more specifically, to a display method and device in screen-off mode. Background Art

[0004] Smart devices are becoming increasingly popular because they can provide rich and intelligent services. A smartphone is a type of smart device that has an independent operating system and runtime. Users can install software, games, navigation, and other applications provided by third-party service providers, and can access wireless networks through mobile communication networks.

[0005] With the popularization of smart devices, users' requirements for human-computer interaction are getting higher and higher. Therefore, how to improve human-computer interaction to enhance user experience is a technical problem that needs to be solved at present.

[0006] Summary of the Invention

[0007] The present application provides a display method and device in screen-off mode, which can enable an electronic device and a user to have a good interactive experience even in screen-off mode.

[0008] In a first aspect, the present application provides a display method in screen-off mode, which is applied to an electronic device with a display screen. The method may include the following steps: the electronic device enters screen-off mode, and displays a screen-off display object on the display screen in screen-off mode; determining the orientation of a face or eye based on face or eye detection, wherein the orientation is associated with the relative position between the face or eye and the electronic device; and dynamically changing the screen-off display object based on changes in the orientation of the face or eye.

[0009] In the above implementation, in the screen-off mode, the screen-off display object can be dynamically changed based on the change in the orientation of the face or eyes, thereby achieving a dynamic interactive effect between the screen-off display object and the user.

[0010] In one possible implementation, the screen-off display object includes a first component, which is used to represent eyes; dynamically changing the screen-off display object according to changes in the orientation of the human face or the human eyes includes: when the orientation of the human face changes, changing the first component according to the change, so that the display position or line of sight direction of the changed first component matches the orientation of the human face or the human eyes.

[0011] The above implementation method can achieve an interactive effect in which the screen-off display object changes dynamically with the changes in the orientation of the face / eyes in the screen-off mode, so that the line of sight of the screen-off display object follows the user.

[0012] Optionally, the first component includes an eyeball in the eye; and changing the first component according to the change includes: changing the display position of the eyeball in the eye according to the change.

[0013] In one possible implementation, the screen-off display object includes a second component, which is used to express emotions; the method also includes: determining the emotions expressed by the face or the eyes based on face or eye detection, and updating the second component so that the emotions expressed by the screen-off display object match the emotions expressed by the face or the eyes.

[0014] In the above implementation, in the screen-off mode, the screen-off display object changes dynamically following the changes in the emotions expressed by the face / eyes, so that the emotions expressed by the screen-off display object follow the interactive effect of the user's emotions.

[0015] Optionally, determining the emotion expressed by the face or eyes based on face or eye detection includes: utilizing artificial intelligence (AI) capabilities to perform face or eye detection to determine the emotion expressed by the face or eyes.

[0016] In one possible implementation, the change in the orientation of the human face or the human eye includes: the human face or the human eye changes from a first orientation to a second orientation; the dynamic change of the screen display object based on the change in the method of the human face or the human eye includes: playing a first animation, the first animation includes N frames, N is an integer greater than 1, and each of the N frames contains the screen display object; in the N frames, the display state of the screen display object in the first frame matches the first orientation, the display state of the screen display object in the last frame matches the second orientation, and the display state of the screen display object in the second frame to the N-1th frame matches an intermediate orientation in the transition process from the first orientation to the second orientation.

[0017] In one possible implementation, the screen-off display object includes a three-dimensional object; and dynamically changing the screen-off display object according to the change in the orientation of the human face or the human eyes includes: when the orientation of the human face or the human eyes changes, changing the orientation of the three-dimensional object accordingly, so that the orientation of the three-dimensional object after the change matches the orientation of the human face or the human eyes.

[0018] In the above implementation, in screen-off mode, the orientation of the screen-off display object dynamically changes with the orientation of the face / eyes, ensuring that the screen-off display object always faces the user. Furthermore, because the screen-off display object is three-dimensional, it can achieve a naked-eye 3D visual effect.

[0019] In one possible implementation, the screen-off display object includes a rotatable text object, the text object is used to display at least two texts, and when the text object is rotated, part of the at least two texts is displayed and / or part of the text is hidden; the screen-off display object is dynamically changed according to the change in the orientation of the face or the eyes, including: when the orientation of the face or the eyes changes, the text object rotates to present the hidden text. In some embodiments, the text before the rotation is different from the text presented after the rotation. Please refer to the example of Figure 13.

[0020] In the above implementation, in the screen-off mode, the text object can display different text content as the orientation of the face / eyes changes, thereby achieving an interactive effect with the user.

[0021] Optionally, when the orientation of the human face or the human eyes changes, the text object rotates to present the hidden text, including: when the orientation of the human face or the human eyes is a first orientation, the text object rotates in a direction opposite to the first orientation to present the hidden text.

[0022] Optionally, the text object is a three-dimensional object, wherein the first surface of the three-dimensional object displays the first text, and the second surface of the three-dimensional object displays the second text; when the orientation of the human face or the human eye changes, the text object rotates to reveal the hidden text, including: when the orientation of the human face or the human eye changes from the first orientation to the second orientation, the text object rotates from the first angle to the second angle; or when the orientation of the human face or the human eye changes from the second orientation to the first orientation, the text object rotates from the second angle to the first angle. Wherein, at the first angle, the first surface of the text object is displayed, and at the second angle, the second surface of the text object is displayed.

[0023] In one possible implementation, the text object is displayed as a hollow effect, and the hollow portion of the text object reveals a portion of the image of the desktop wallpaper of the electronic device; the method also includes: in response to an unlocking operation, switching from the screen-off mode to the screen-on mode through a one-shot visual effect, and displaying the desktop of the electronic device and the desktop wallpaper.

[0024] A possible implementation method further includes: obtaining information detected by the electronic device using a front camera or a first sensor; and performing face or eye detection based on the detected information.

[0025] In a possible implementation, the change in the orientation of the human face or the human eyes is caused by movement of the user or movement of the electronic device.

[0026] In a second aspect, the present application provides a display method in screen-off mode, which is applied to an electronic device with a display screen, including: the electronic device enters the screen-off mode; when it is detected that human eyes are looking at the display screen, a screen-off display object is displayed on the display screen in the screen-off mode; based on face or eye detection, the orientation of the face or eye is determined, and the orientation is associated with the relative position between the face or eye and the electronic device; according to the change of the orientation of the face or eye, the screen-off display object is dynamically changed.

[0027] In the above implementation, in screen-off mode, the screen-off display object is displayed only when a person's eyes are detected looking at the display screen, thereby saving power consumption of the electronic device. Furthermore, because the screen-off display object can be dynamically changed based on changes in the orientation of a person's face or eyes, a dynamic interaction effect between the screen-off display object and the user can be achieved.

[0028] Other implementation methods of the second aspect can refer to the first aspect mentioned above and will not be described in detail.

[0029] In a third aspect, a device is provided, comprising a memory, a processor, and a display device, wherein the memory is used to store a computer program, and the processor is used to read the computer program so that the device implements the method as described in any one of the first aspect or the second aspect above.

[0030] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the computer implements the method as described in any one of the first aspect or the second aspect.

[0031] In a fifth aspect, a computer program product is provided, comprising computer-readable instructions, which, when the computer program product is run on a computer, enables the computer to execute the method as described in any one of the first or second aspects above.

[0032] In a sixth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute a method as described in any one of the first or second aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a schematic diagram of an emoticon in an embodiment of the present application;

[0034] FIG2 is a schematic diagram of a naked-eye 3D object in an embodiment of the present application;

[0035] FIG3 is a schematic diagram of a text object in an embodiment of the present application;

[0036] FIG4 is a schematic diagram of enabling the screen-off display function and setting a screen-off display object in an embodiment of the present application;

[0037] FIG5 is a schematic diagram of several face orientations in an embodiment of the present application;

[0038] FIG6 is a schematic diagram of the hardware structure of an electronic device in an embodiment of the present application;

[0039] FIG7 is a schematic diagram of a software architecture of an electronic device in an embodiment of the present application;

[0040] FIG8 is a schematic diagram of an interaction in which emoticons dynamically change with user position in screen-off mode in scenario 1 of an embodiment of the present application;

[0041] FIG9 is a schematic diagram of an interaction in which emoticons dynamically change with user emotions in screen-off mode in scenario 2 of an embodiment of the present application;

[0042] FIG10 is a schematic diagram of an interaction in which emoticons dynamically change with user position and emotion in screen-off mode in scenario three of an embodiment of the present application;

[0043] FIG11 is a schematic diagram of an interaction in which a naked-eye 3D object dynamically changes with a user's expression in screen-off mode in scenario 4 of an embodiment of the present application;

[0044] FIG12 is a schematic diagram of the appearance animation of the naked-eye 3D object in scene 4 of the embodiment of the present application;

[0045] FIG13 is a schematic diagram showing a cube dynamically changing with the user's position in screen-off mode in scenario five of an embodiment of the present application, thereby displaying text content on different surfaces of the cube;

[0046] FIG14 is a schematic diagram of switching from screen-off mode to screen-on mode in scenario 5 of the application embodiment;

[0047] FIG15 is a schematic diagram of another text object dynamically changing with changes in the user's position in Scenario 5 of an embodiment of the present application;

[0048] FIG16 is a schematic flow chart of a display method in screen-off mode provided in an embodiment of the present application;

[0049] FIG17 is a schematic flow chart of another display method in screen-off mode provided in an embodiment of the present application;

[0050] FIG18 is a schematic structural diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of this application and the appended claims, the singular expressions "a", "an", "the", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one, two or more. The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0052] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0053] In order to better understand this application, some technologies and technical terms involved in this application are briefly explained below.

[0054] (1) Always on display (AOD)

[0055] AOD is a feature that allows electronic devices to display useful thumbnail information. Even when the electronic device's display screen is turned off (that is, in screen-off mode, or also known as screen-off state), basic information (such as clock, calendar, notifications, call status, etc.) can be displayed in a minimized manner, allowing users to view basic information anytime, anywhere in the most convenient way while using low power. In other words, in screen-off mode, the electronic device's display screen can only display limited information such as time and notifications.

[0056] In screen-off mode, individual pixels on the display screen of the electronic device are lit up to display information such as clocks and notifications, which can save power consumption compared to lighting up the entire display screen to display information such as clocks and notifications.

[0057] AOD technology allows an electronic device to display a certain amount of information when it is in screen-off mode, so AOD is also called screen-off display (or off-screen display).

[0058] In the embodiment of the present application, the content that can be displayed in the screen-off mode is called a screen-off display object, which may also be called an all-weather display object or an AOD object. The embodiment of the present application does not impose any restrictions on the naming of the object.

[0059] In screen-off mode, one or more screen-off display objects can be displayed. These objects can be pre-configured. Users can view the type and display method of these pre-configured screen-off display objects in the settings interface and select the screen-off display objects they wish to display in screen-off mode. Screen-off display objects can also be generated based on user-selected images or input text, allowing for more personalized screen-off display objects.

[0060] For example, in the embodiment of the present application, the type and setting method of the screen-off display object may include:

[0061] Basic information. This includes information such as time, date, text message and call reminders. Users can choose to display or not display basic information. They can also choose how to display basic information, for example, with options including touch display (displaying basic information when the screen is touched), timed display, and all-day display.

[0062] -Pictures. The pictures may include static pictures or dynamic pictures. The electronic device may have multiple pictures built in for the user to choose from. Optionally, the electronic device may also generate personalized pictures (such as personalized static pictures or personalized dynamic pictures) based on the pictures selected by the user or the editing operations on the pictures as the screen display object. Optionally, the electronic device may also allow the user to select a picture from the gallery of the electronic device and generate the screen display object based on the picture. In addition, the screen display object may also include an animation composed of multiple pictures.

[0063] -Emojis. An electronic device may have multiple built-in emojis for the user to choose from. Optionally, the electronic device may also allow the user to use emojis downloaded from the network or obtained from other sources, which is not limited in this embodiment of the present application.

[0064] An emoticon can be composed of multiple components. For example, an emoticon can include an object for representing eyes, an object for representing a mouth, and so on. Figure 1 exemplifies an emoticon, which includes components such as eyes, glasses, and a mouth. In other embodiments, the screen-off interface can also display multiple different emoticons at the same time. In this application, in order to distinguish from the screen-off display object, the components included in an screen-off display object are referred to as components. For example, for the emoticon shown in Figure 1, the object used to represent the eyes in the emoticon is referred to as an eye component or a first component, and the object used to represent the mouth in the emoticon is referred to as a mouth component or a second component, and so on.

[0065] In some embodiments of the present application, an electronic device may allow a user to create (or construct) their own personalized emoticons. For example, the electronic device may have pre-installed components for creating emoticons, and the user may select these components to create a personalized emoticon. For example, the electronic device may have pre-installed multiple styles of eyes, multiple styles of noses, and multiple styles of mouths, and the user may select a style of eyes, nose, and mouth from these, and perform some personalized settings or editing to generate a personalized emoticon.

[0066] - Avatars. Similar to emoticons, users can choose from built-in avatars, avatars obtained from other sources, or personalized avatars designed or edited by the user. A avatar can include one or more components.

[0067] It can be understood that the virtual image in the embodiment of the present application may include images of people, etc., and the embodiment of the present application does not limit this.

[0068] -Signature information. Electronic devices can have multiple signature information and display methods (such as fonts and colors) built in for users to choose from. Optionally, the electronic device can also generate a personalized signature for the user as a screen-off display object based on the signature content entered by the user and the display method selected.

[0069] Optionally, an embodiment of the present application further provides an off-screen display object, which has a naked-eye 3D display effect and is referred to herein as a naked-eye 3D object. FIG2 exemplarily shows an example of a naked-eye 3D object, which is displayed as a 3D "astronaut".

[0070] Optionally, embodiments of the present application also provide a screen-off display object called a "text object," which can display multiple texts. The displayed text can include signature information, date, weather, time, etc. It is understood that embodiments of the present application do not restrict the naming of text objects, and the following description uses text objects as an example.

[0071] The text object can be in different states, and different texts can be displayed in different states. In one state, part of the text may be displayed and part of the text may be hidden. In one possible implementation, the text object is a three-dimensional polyhedron, and different surfaces of the three-dimensional polyhedron can be used to display different text contents. Figure 3 exemplifies a schematic diagram of displaying multiple text contents through a three-dimensional cube. The figure only shows the text contents displayed on two surfaces of the three-dimensional cube, one of which is used to display the date (as shown in the figure "SUN SEP 07"), and the other surface is used to display a personalized signature (as shown in the figure "JUST LOVE LIFE").

[0072] In some other embodiments of the present application, other information may be displayed on the surface of the three-dimensional polyhedron, such as weather, etc., which is not limited in the embodiments of the present application.

[0073] In an embodiment of the present application, the electronic device allows the user to turn on or off the screen display function, and may also allow the user to set the screen display object. For example, FIG4 shows a user interface schematic diagram for a user to turn on the screen display function and set the screen display object provided in an embodiment of the present application. As shown in FIG4 , the user interface includes a screen display function option 401, and the user can turn on or off the screen display function by clicking on the function option. The user interface also provides a variety of emoticons, naked-eye 3D objects and other screen display objects (such as the content displayed in the dotted box 402 area in the figure) for the user to choose. It can be understood that the above only exemplifies several screen display objects and the setting method of the screen display objects. The embodiment of the present application does not limit the type and setting method of the screen display object.

[0074] (2) Eyes on display (EOD)

[0075] In one implementation, EOD technology means that when the electronic device detects that the user is looking at the display screen, the AOD display will be illuminated; otherwise, the AOD display will not be illuminated. In other words, when the electronic device detects that the user is looking at the display screen, the screen-off display object is displayed on the display screen. If the electronic device does not detect that the user's eyes are looking at the display screen for a set period of time (for example, when the electronic device is placed in a pocket or bag), the screen-off display is turned off to save device power consumption.

[0076] The EOD function can also be considered as a type of AOD function.

[0077] (3) Face detection

[0078] Face detection refers to the process of searching any given image using a certain strategy (or algorithm) to determine whether it contains a face. If so, the method further determines the face's position, size, and posture.

[0079] Optionally, the embodiment of the present application may adopt the following face detection method:

[0080] Knowledge-based rule-based methods: These methods use facial rules (e.g., the distribution of eyes, nose, mouth, etc.) to detect faces in input images. These methods primarily use simple rules to describe facial features and the relationships between these features.

[0081] Invariant feature-based methods: Face detection is performed on input images by looking for invariant features of the face. This type of method assumes that certain facial features (such as skin color, edges, texture, etc.) remain unchanged under different postures, viewing angles, and expressions.

[0082] Template matching-based method: The facial features of a face are described by a standard face template. When performing face detection, the correlation value between the input image subwindow and the standard face template is first calculated, and then the obtained correlation value is compared with a pre-set threshold to determine whether there is a face in the input image.

[0083] Statistical model-based methods treat facial regions as patterns, using a large amount of facial data as training samples to learn underlying rules and construct a classifier. This classifier then detects faces by identifying the attributes of all possible regional patterns in the image. This type of method can be implemented using artificial intelligence (AI), such as neural networks, for example.

[0084] It will be understood that the above only illustrates several face detection methods, and the embodiments of the present application do not limit the face detection methods.

[0085] In some embodiments of the present application, the electronic device can collect information through a device or component provided thereon, and then perform face detection based on the collected information and using the above-mentioned face detection algorithm. Optionally, the device or component can be a front-facing camera provided on the electronic device, and the electronic device can perform face detection based on the image collected by the front-facing camera. The device or component may also be other types of sensors, such as an infrared detection sensor, which is not limited in the embodiments of the present application.

[0086] In some embodiments of the present application, based on face detection, the electronic device can determine the orientation of the face, and the orientation of the face is associated with the relative position between the face and the electronic device. The association between the orientation of the face and the relative position between the face and the electronic device can be understood as: the orientation of the face can reflect or represent the relative position between the face and the electronic device, or in other words, the relative position between the face and the electronic device can be determined based on the orientation of the face.

[0087] Optionally, the face detection algorithm can be used to obtain the position of the face in the image. Specifically, the position can be the offset of the face relative to the center of the image. The orientation of the face can be determined based on the position of the face in the image. The orientation can reflect the relative position between the face and the electronic device. Exemplary possible situations include the following:

[0088] Case 1: A reference point on the face (such as the center point of the face, the position of the tip of the nose on the face, or the center point between the eyes on the face, etc.) is located in the central area of ​​the image, where the central area can be, for example, the center point of the image, or a smaller area with the center point of the image as the center and a set length as the radius. In this case, it can be determined that the face is facing the display screen of the electronic device, or that the face is not offset to the left, right, downward or upward relative to the display screen.

[0089] Case 2: The reference point on the face is located in the left area of ​​the central area of ​​the image. In this case, it can be determined that the face is facing the left side of the display screen of the electronic device, or that the face is offset to the left relative to the display screen.

[0090] Case 3: The reference point on the face is located in the right area of ​​the central area of ​​the image. In this case, it can be determined that the face is facing the right side of the display screen of the electronic device, or that the face is offset to the right relative to the display screen.

[0091] Case 4: The reference point on the face is located in the upper area of ​​the central area of ​​the image. In this case, it can be determined that the upper side of the face is facing the display screen of the electronic device, or that the face is offset upward relative to the display screen.

[0092] Case 5: The reference point on the face is located in the lower area of ​​the central area of ​​the image. In this case, it can be determined that the lower side of the face is facing the display screen of the electronic device, or that the face is offset downward relative to the display screen.

[0093] Furthermore, based on the offset of the face in the image, the offset of the face relative to the display screen of the electronic device can be further determined.

[0094] Figure 5 shows schematic diagrams of face orientation in several situations. As shown in Figure 5 (a), the user's head is located in the center relative to the electronic device display screen, and the black dot on the display screen is used to indicate the orientation of the user's face. As shown in Figure 5 (b), the user's head is located to the right of the electronic device display screen, and the black dot on the display screen is used to indicate the orientation of the user's face. As shown in Figure 5 (c), the user's head is located to the left of the electronic device display screen, and the black dot on the display screen is used to indicate the orientation of the user's face.

[0095] It is understood that the above merely illustrates several possible relative positions between a face and an electronic device, and the present application does not limit this. Based on the above principles, the position of a face in an image can be determined based on the face detection information, and based on this position, the relative position of the face with respect to the electronic device (or the display screen of the electronic device) can be determined.

[0096] In other embodiments of the present application, the rotation angle of the face can be obtained by a face detection algorithm, and accordingly, the rotation angle of the face relative to the electronic device (or the display screen of the electronic device) can be determined based on the rotation angle of the face. The rotation angle may include one or more of the following: a rotation angle in the horizontal dimension, a rotation angle in the vertical dimension, and a rotation angle in the depth dimension.

[0097] In other embodiments of the present application, the depth of field of the face can be obtained through a face detection algorithm, and accordingly, the distance of the face relative to the electronic device (or the display screen of the electronic device) can be determined based on the depth of field of the face.

[0098] In other embodiments of the present application, a facial expression can be obtained by a face detection algorithm, and the emotion expressed by the face can be further determined based on the facial expression. For example, the emotion may include: happiness, sadness, anger, etc. The face detection algorithm can determine the facial expression based on details of the face, such as the shape of the facial features (such as the opening and closing state of the eyes and mouth), the relative position of the facial features, etc.

[0099] Based on the above description of the face detection algorithm, it can be understood that in the embodiment of the present application, the face detection algorithm can determine one or more of the following: the position offset of the face relative to the display screen of the electronic device, the rotation angle of the face relative to the display screen of the electronic device, the depth of field of the face relative to the display screen of the electronic device, the expression of the face, etc.

[0100] Similar to face detection, eye detection (e.g., information such as the distance between the eyes, the center point between the eyes, and the state of the eyes) can be used to obtain information such as the position offset of the eyes relative to the electronic device display screen, the rotation angle of the eyes relative to the electronic device display screen, the depth of field of the eyes relative to the electronic device display screen, and facial expressions. In the following text of this application, face detection or eye detection is referred to as "face / eye detection."

[0101] (4) Eye tracking

[0102] Eye tracking technology primarily studies the acquisition, modeling, and simulation of eye movement information. Eye tracking can include tracking based on changes in the characteristics of the eye and its surroundings, tracking based on changes in the angle of the iris, or actively projecting infrared or other light beams onto the iris to extract features for tracking. Equipment used to implement eye tracking can include infrared devices and image acquisition devices. For example, pattern recognition algorithms such as template matching, projection, and neural networks can be used to implement eye tracking based on color or grayscale images captured by image acquisition devices.

[0103] In some embodiments of the present application, eye tracking can be used to determine whether a person's eyes are looking at the display screen of an electronic device. When it is determined that the person's eyes are looking at the display screen of the electronic device, an AOD display can be performed.

[0104] In other embodiments of the present application, the line of sight of a person's eyes, or changes in the line of sight of a person's eyes, can be determined through eye tracking.

[0105] In other embodiments of the present application, the state of a person's eyes can be obtained through eye tracking, and the expression of a person's face can be determined based on the state of the person's eyes.

[0106] In the related art, when the AOD display is turned on, the screen-off display object is displayed on the display screen of the electronic device. If the screen-off display object is a static object (such as a static picture or text), the screen-off display object is always displayed as static on the display screen. If the screen-off display object is a dynamic object (such as a dynamic picture), the screen-off display object is displayed as a static state after the dynamic display is completed, and the dynamic effect of the screen-off display object is preset. That is to say, in the related art, whether the screen-off display object is static or dynamic, it will not produce an interactive effect with the user.

[0107] To improve interactivity in screen-off mode, embodiments of the present application provide a display method in screen-off mode and related devices that can implement the method. In embodiments of the present application, in screen-off mode, the electronic device can determine the orientation of the face / eye relative to the electronic device's display screen based on face / eye detection, thereby dynamically displaying screen-off display objects based on the orientation of the face / eye relative to the electronic device's display screen, thereby achieving a user experience of dynamic interaction with the screen-off display objects based on the orientation change between the face / eye relative to the electronic device's display screen.

[0108] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0109] The method provided in the embodiment of the present application can be applied to electronic devices. The electronic device can realize human-computer interaction. For example, the electronic device can specifically be a non-folding screen mobile phone, a folding screen mobile phone, a wearable device (such as a smart watch, a smart bracelet, etc.), a tablet computer, a laptop computer, a smart screen, a car terminal, a computer, a computer, a personal computer (personal computer, PC), an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, a personal digital assistant (personal digital assistant, PDA), a virtual reality (virtual reality, VR) device / augmented reality (augmented reality, AR) device, an artificial intelligence (artificial intelligence, AI) device and other terminal devices with display functions, and the operating systems installed on the above electronic devices include but are not limited to This application does not limit the specific type of the above electronic equipment or the installed operating system.

[0110] Referring to Figure 6, a possible hardware structure of the electronic device 100 in an embodiment of the present application is shown. The electronic device can execute the method provided in the embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc.

[0111] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a display processing unit (DPU), and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110. Among them, the processor is the nerve center and command center of the electronic device 100. The processor can generate an operation control signal based on the instruction opcode and timing signal to complete the control of instruction fetching and execution. A memory can also be set in the processor 110 for storing instructions and data.

[0112] The camera 193 may include one camera or a group of cameras. The camera 193 may include a front camera and a rear camera.

[0113] The sensor module 180 may include one or more of the following: a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, a vibration sensor, and an infrared detection sensor.

[0114] The electronic device 100 can realize a display function through the GPU, the display screen 194 and the application processor AP, etc. The display screen 194 is used to display images, videos, etc.

[0115] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0116] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0117] Figure 7 is a block diagram of the software system architecture of an electronic device provided in an embodiment of the present application. Exemplarily, as shown in Figure 7, the software system architecture of the electronic device can be a layered architecture, for example, the software can be divided into several layers, and the layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer (framework, FWK), the runtime and system library, the kernel layer, and the hardware layer.

[0118] The application layer can include a series of application packages. As shown in Figure 7, the application layer can include camera, settings, skin modules, user interface (UI), third-party applications, etc. Among them, third-party applications can include wireless local area network (WLAN), music, calls, Bluetooth, video, memo, notes, etc.

[0119] In one possible implementation, applications can be developed using Java by calling the application programming interface (API) provided by the application framework layer. Developers can use the application framework layer to interact with the underlying operating system layers (e.g., the hardware layer, kernel layer, etc.) to develop their own applications. The application framework layer primarily provides a series of services and management systems for the operating system.

[0120] The application framework layer provides an application programming interface and programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 7, the application framework layer can include an activity manager, a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and so on.

[0121] The activity manager is used to manage the life cycle of each application and provide common navigation back functions, providing an interactive interface for all program windows.

[0122] The window manager manages windowed applications. It can determine the display size, determine whether a status bar is present, lock the screen, take screenshots, and more. Content providers store and retrieve data and make it accessible to applications. This data can include video, images, audio, incoming and outgoing calls, browsing history and bookmarks, and the phone book.

[0123] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0124] The phone manager is used to provide communication functions for electronic devices, such as call status management (including answering, hanging up, etc.).

[0125] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.

[0126] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0127] The runtime consists of a core library and a virtual machine. The runtime is responsible for operating system scheduling and management. The core library consists of two parts: one for the Java language's callable functions and the other for the operating system's core libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine performs functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0128] The system library may include multiple functional modules. For example: surface manager, media framework, three-dimensional graphics processing library (for example: OpenGL ES), two-dimensional graphics engine (for example: SGL), etc. The surface manager is used to manage the display subsystem and provides the fusion of two-dimensional and three-dimensional layers for multiple applications. The media framework supports a variety of commonly used audio and video format playback and recording, as well as static image files, etc. The media framework can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc. The two-dimensional graphics engine is a drawing engine for two-dimensional drawing. In some embodiments, the three-dimensional graphics processing library can be used to draw three-dimensional motion trajectory images, and the two-dimensional graphics engine can be used to draw two-dimensional motion trajectory images.

[0129] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0130] The hardware layer can include various sensors, such as accelerometers, gravity sensors, touch sensors, etc.

[0131] It should be understood that the hardware structure of the electronic device can be as shown in Figure 6, and the software system architecture can be as shown in Figure 7, wherein the software programs and / or modules corresponding to the software system architecture in the electronic device can be stored in the internal memory 121, and the processor 110 can run the software programs and applications stored in the internal memory 121 to execute the process of the display method provided in the embodiment of the present application.

[0132] In addition to the above examples, the method proposed in this application can also be applied to other types of software systems, such as Hongmeng operating system, Windows operating system, iOS operating system, etc.

[0133] In a display method in screen-off mode provided in an embodiment of the present application, after an electronic device enters screen-off mode, it displays a screen-off display object on the display screen in screen-off mode. The electronic device performs face / eye detection to determine the orientation of the face / eye, which is related to the relative position of the electronic device and the face / eye. The electronic device can dynamically change the screen-off display object based on the change in the orientation of the face / eye, thereby achieving a dynamic interactive effect.

[0134] It should be understood that the objects displayed on the off screen may be displayed all the time, or may be displayed only when it is detected that the user's eyes are looking at the screen (or a face is detected).

[0135] In order to better understand the display method in the screen-off mode provided in the embodiment of the present application, the following first describes the human-computer interaction behavior and interface changes on the electronic device side in the embodiment of the present application using several possible scenarios as examples.

[0136] Scene 1

[0137] Scenario 1 takes the emoticons shown in FIG1 as an example, and describes an interactive scenario in which the emoticons displayed on the display screen in screen-off mode dynamically change as the user's position changes.

[0138] Referring to FIG8 (a), the electronic device is currently in screen-off mode. After the front-facing camera on the electronic device captures an image, the electronic device performs face / eye detection on the image to determine that the face / eye is located at the center of the image. Since the face / eye is currently centered relative to the display screen, the positions of eyeball 801 and glasses 802 in emoticon 800 on the display screen match the current positions of the face / eye. As shown in the figure, eyeball 801 is located in the middle of the eyes, achieving the effect that emoticon 800 is looking at the user.

[0139] Referring to FIG8(b), after the user's head moves to the right, the front-facing camera of the electronic device captures an image. The electronic device then performs face / eye detection on the image, determining that the face / eye is located slightly to the right of the center of the image. Since the face / eye is now slightly to the right of the center relative to the display screen, the position of the eyeball 801 and glasses 802 in emoticon 800 on the display screen shifts to the right to match the current position of the face / eye. This creates the effect of emoticon 800's gaze following the user's position, maintaining its gaze.

[0140] As shown in Figure 8(c), after the user's head continues to move downward, the front-facing camera of the electronic device captures an image. The electronic device then performs face / eye detection on the image, determining that the face / eye is located slightly to the lower right of the center of the image. Since the face / eye is now slightly to the lower right of the center relative to the display screen, the position of the eyeball 801 and glasses 802 in emoticon 800 on the display screen shifts downward to match the current position of the face / eye, achieving the effect that emoticon 800's gaze follows the user's position, maintaining its gaze.

[0141] The user's head continues to move, and accordingly, the positions of the eyeball 801 and the glasses 802 in the emoticon 800 change along with the change of the user's position, as shown in (d) in FIG8 .

[0142] Optionally, when the positions of the eyeball 801 and the glasses 802 change with the changes in the user's position, the degree of position change of the eyeball 801 and the glasses 802 (such as displacement size and / or angle) may be different. For example, the displacement of the eyeball 801 may be greater than the displacement of the glasses 802, so as to achieve a more natural and realistic dynamic display effect. This embodiment of the present application is not limited to this.

[0143] Optionally, if no face / eye is detected based on the image captured by the front camera, the electronic device can maintain the state of the emoticon displayed last time (or the last time before the current moment), or it can also display the default state of the emoticon, for example, the state of the emoticon in (a) in Figure 8 can be set to the default state.

[0144] Scene 2

[0145] Scenario 2 takes the emoticons shown in FIG1 as an example and describes an interactive scenario in which the emoticons displayed on the screen in screen-off mode change dynamically with the user's emotions.

[0146] Referring to (a) in Figure 9, the electronic device is currently in screen-off mode. After the front camera set on the electronic device captures an image, the electronic device performs face / eye detection on the image and determines that the emotion expressed by the face is sadness. The electronic device displays the mouth 901 in the emoticon 900 as curved downward to express the emotion of sadness, so that the emotion expressed by the emoticon 900 matches the emotion expressed by the current face.

[0147] Referring to (b) in Figure 9 , after the user's emotion changes from sadness to happiness, the front camera set on the electronic device captures an image. The electronic device performs face / detection on the image and determines that the emotion expressed by the face is happy. The electronic device displays the mouth 901 in the emoticon 900 as curved upward to express a happy emotion, so that the emotion expressed by the emoticon 900 changes dynamically with the changes in the user's emotion and always matches the user's emotion, thereby establishing an emotional state connection with the user.

[0148] Figure 9 (c) and (d) show another emoticon that changes dynamically from sad to happy according to the emotion expressed by the user's face.

[0149] In another embodiment, if it is detected that the emotion expressed by the current face is sadness, the mouth 901 in the emoticon 900 can be displayed as being bent upward to express a happy emotion. That is to say, the emotion expressed by the emoticon can be different from the emotion expressed by the current face (for example, opposite), thereby achieving the effect of improving the user's mood or pleasing the user.

[0150] Optionally, if no face / eyes are detected based on the image captured by the front camera, the electronic device can maintain the state of the emoticon displayed last time (or the last time before the current moment), or it can also display the default state of the emoticon, for example, the state in which the emoticon expresses a happy emotion can be set as the default state.

[0151] Scene 3

[0152] Scenario 3 takes the emoticons shown in FIG1 as an example, and describes an interactive scenario in which the emoticons displayed on the screen in screen-off mode change dynamically with changes in the user's position and emotions.

[0153] The implementation method of scenario three can be understood as combining the implementation method of scenario one with the implementation method of scenario two. That is to say, through face / eye detection, both the orientation of the face / eye and the emotion expressed by the face can be determined. Accordingly, matching emoticons can be displayed based on the orientation and expression of the face / eye. On the one hand, the line of sight of the emoticon follows the user's position, achieving the effect of always looking at the user. On the other hand, the emotions expressed by the emoticon change dynamically with the changes in the user's emotions and always match the user's emotions.

[0154] Figure 10 illustrates an example of the dynamic changes in emoticons displayed on an electronic device's display screen as the relative position between a user and the screen changes, and the user's emotions also change. Figure 10 shows the orientation of the user's face and the facial expression at that moment, and correspondingly, the display effect of the emoticons on the display screen.

[0155] Scene 4

[0156] Scenario 4 takes the naked eye 3D object shown in Figure 2 as an example to describe an interactive scenario in which the naked eye 3D object displayed on the display screen in screen off mode changes dynamically with the user's position. The naked eye 3D object is a three-dimensional display effect.

[0157] Referring to FIG. 11 (a), the electronic device is currently in screen-off mode. After the front-facing camera on the electronic device captures an image, the electronic device performs face / eye detection on the image to determine that the face / eye is located at the center of the image. Since the face / eye is currently centered relative to the display screen, the orientation of the naked-eye 3D object 1100 on the display screen matches the current position of the face / eye. As shown in the figure, the naked-eye 3D object 1100 faces the user, creating the effect that the naked-eye 3D object 1100 is facing the user.

[0158] Referring to (b) in FIG11 , after the user's head moves to the right, the front-facing camera of the electronic device captures an image. The electronic device then performs face / eye detection on the image, determining that the face / eye is located slightly to the right of the center of the image. Since the face / eye is now slightly to the right of the center relative to the display screen, the orientation of the naked-eye 3D object 1100 on the display screen shifts to the right. This shifted orientation matches the current position of the face / eye, achieving the effect of the naked-eye 3D object 1100 shifting in orientation with the user's position, always facing the user.

[0159] Referring to (c) in FIG11 , after the user's head moves to the left, the front-facing camera of the electronic device captures an image. The electronic device then performs face / eye detection on the image, determining that the face / eye is located slightly to the left of the center of the image. Since the face / eye is now slightly to the left of the center relative to the display screen, the orientation of the naked-eye 3D object 1100 on the display screen shifts to the left. This shifted orientation matches the current position of the face / eye, achieving the effect of the naked-eye 3D object 1100 changing its orientation in response to the user's position, always facing the user.

[0160] Since the naked-eye 3D object has a three-dimensional display effect and can present images at different angles according to changes in the orientation of the human face, a naked-eye 3D visual effect can be achieved.

[0161] Figure 11 describes the change in the horizontal orientation of a human face as an example. The embodiments of the present application do not limit this. For example, when the orientation of a human face changes in other directions, the naked-eye 3D object can also rotate in that direction based on the change to achieve a dynamic change effect toward the current user.

[0162] In some embodiments of the present application, when the object displayed on the off-screen display is a naked-eye 3D object, the naked-eye 3D object can be directly displayed on the display screen when the display screen is illuminated. In other embodiments, when the display screen is illuminated, an entrance animation of the naked-eye 3D object can be first displayed to demonstrate the entrance effect of the naked-eye 3D object, thereby increasing the vividness and interest of the display effect.

[0163] Taking the naked-eye 3D object shown in FIG2 as an example, FIG12 exemplifies a schematic diagram of several frames in the appearance animation of the naked-eye 3D object. As shown in FIG12 , through this appearance animation, the effect of the naked-eye 3D object "astronaut" gradually exiting the cabin can be presented. Based on this appearance animation, when the user's eyes are fixed on the display screen of the electronic device, a hatch appears on the screen, and the "astronaut" comes out of the hatch. After that, while the user is looking at the display screen of the electronic device, he shakes the electronic device or moves his face. The current picture will change according to the position of the face, presenting the picture associated with the corresponding perspective, thereby achieving the naked-eye 3D effect.

[0164] Scene 5

[0165] Scenario 5 takes the display of text through a cube as an example, and describes a scenario in which the cube changes dynamically with the change of the user's position, thereby displaying the text content on different surfaces of the cube.

[0166] Each surface of the cube can display text, and the text may include one or more of date, time, weather, signature, etc. In this example, the display of date and signature information on three surfaces of the cube is described as an example.

[0167] Referring to Figure 13 , after the front-facing camera on the electronic device captures an image, the electronic device performs face / eye detection on the image to determine the orientation of the face / eye. When the face / eye is centered relative to the display screen, one surface of the cube on the display screen faces the user, and the text displayed on this surface is a date, specifically "SUM SEP 07" as shown in the figure.

[0168] When the user's head moves to the left, the front-facing camera on the electronic device captures an image. The electronic device then performs face / eye detection on the image, determining that the face / eye is located slightly to the left of the center of the image. Since the face / eye is now slightly to the left of the center relative to the display screen, the orientation of the cube on the display screen deflects to the left, and a portion of the other surface of the rotated cube is displayed. As the user's head continues to move to the left, the orientation of the cube on the display screen continues to deflect to the left, and the entire left surface of the rotated cube is displayed. As shown in FIG13 , the entire left surface of the cube is displayed, facing the user, and the text displayed on it is a personalized signature message, specifically "JUST LOVE LIFE" as shown in the figure.

[0169] Similarly, when the user's head moves to the right, the cube gradually rotates to the right as the user's head moves, thereby gradually revealing the right side, allowing the user to see the signature information displayed on the right side.

[0170] Optionally, as shown in Figure 13, each surface in the cube can be partially hollowed out. For example, the text part can be hollowed out. For example, the text displayed on each surface in Figure 13, the part occupied by the text is hollowed out, through which the desktop wallpaper on the lower layer can be presented, and the other parts are opaque; for another example, on one surface of the cube, the part occupied by the text is opaque, and the rest is hollowed out, through which the desktop wallpaper on the lower layer can be presented.

[0171] Figure 13 describes the change in the orientation of a human face in the horizontal direction as an example. The embodiments of the present application do not limit this. For example, when the orientation of a human face changes in other directions, the text object can also rotate in the opposite direction based on the change to display the text content on the corresponding surface.

[0172] Based on Scenario 5 above, the following effect can be achieved: In screen-off mode, when the user looks at the electronic device, a large poster with the current time and date appears on the display. The font is hollowed out, revealing the user's currently set desktop wallpaper. While looking at the device, the user shakes the device or moves their face, and the current image changes based on the position of the face, revealing hidden information corresponding to the perspective.

[0173] Optionally, when the user performs an unlocking operation and the display screen is unlocked, the electronic device exits the screen-off display mode, and the electronic device display screen lights up in full screen, displaying the desktop and desktop wallpaper. During this process, the electronic device can switch from the screen-off mode to the screen-on mode through a one-shot visual effect, and display the desktop and desktop wallpaper of the electronic device. In the embodiment of the present application, since the text objects in the screen-off mode are displayed as a hollow effect, part of the desktop wallpaper can be seen, so that part of the desktop wallpaper can also be seen in the screen-off mode, and then through the transition animation from the screen-off mode to the screen-on mode, a smooth, coherent and natural transition to the screen-on mode can be achieved, thereby achieving a "one-shot" display effect.

[0174] Referring to Figure 14, a transition animation for switching from screen-off mode to screen-on mode provided by an embodiment of the present application is exemplified to achieve a "one-shot" display effect. The effect of the transition animation is: the picture on the current display screen is gradually enlarged with the center point position, and the lower desktop wallpaper is displayed more and more until it is all displayed and switches to screen-on mode. Optionally, in the transition animation, the transparency of the non-hollowed-out part of the text object can be gradually increased to achieve a display effect of the desktop background from invisible to clear, thereby making the transition process from screen-off mode to screen-on mode smoother and more natural.

[0175] Figure 15 illustrates another dynamic display effect in scenario five. Similar to the dynamic display effect shown in Figure 13, the difference is that the style of the text object is different from that of the text object in Figure 13. The text object shown in Figure 15 is displayed in a two-dimensional effect. The text object has different states, and the text content displayed in different states is different. For example, in one state, it displays date and time information (such as the text content displayed by the text object when the user's face is in the middle orientation in the figure), in another state, it displays date, location information, and weather information (such as the text content displayed by the text object when the user's face is in the left orientation in the figure), and in another state, it displays date and personalized signature information (such as the text content displayed by the text object when the user's face is in the side orientation in the figure). Depending on the different orientations of the user's face / eyes, text objects in different states can be displayed. Figure 15 describes the example of a change in the horizontal orientation of the face, but the embodiments of the present application are not limited to this. For example, when the orientation of the face changes in other directions, the text object can also display text content in the corresponding state based on the change.

[0176] The display method in the screen off mode provided in an embodiment of the present application is described below in conjunction with Figure 16.

[0177] Referring to Figure 16, which is a flowchart of a display method in screen-off mode provided in an embodiment of the present application, the process can be executed by the electronic device shown in Figure 6 or Figure 7, or by a functional module, processor, or chip in the electronic device. The following process is described with the electronic device as the execution subject. As shown in the figure, the process may include the following steps:

[0178] Step 1601: The electronic device enters the screen-off mode, and displays a screen-off display object on the display screen in the screen-off mode.

[0179] In an embodiment of the present application, the screen-off function can be enabled in advance and the screen-off display objects to be displayed in the screen-off mode can be set. In this way, when the electronic device meets the screen-off condition (for example, the electronic device has not been used for a set time), it can automatically enter the screen-off mode and display the screen-off display objects on the display screen in the screen-off mode. An example of enabling the screen-off mode and setting the screen-off display objects to be displayed in the screen-off mode can be found in FIG. 4 and will not be repeated here.

[0180] In the embodiment of the present application, there is no restriction on the objects displayed on the screen, for example, it can be signature information, emoticons, pictures, virtual images, naked-eye 3D objects, text objects, etc.

[0181] Step 1602: The electronic device determines the orientation of the face / eye based on face / eye detection, where the orientation of the face / eye is associated with the relative position between the face / eye and the electronic device.

[0182] In one possible implementation, the front camera or first sensor provided on the electronic device can collect information and obtain detection information for face / eye detection. The "first sensor" here is used to refer to one or more types of sensors, and the information detected by the sensor can be used for face / eye detection. Exemplarily, the sensor can be an infrared detection sensor, and the infrared detection sensor can collect infrared images for face detection. Taking the front camera as an example, after the front camera collects the image, the image can be passed to the processor, and the processor executes the face / eye detection algorithm to implement face / eye detection for the image. The implementation method of face / eye detection can refer to the previous article.

[0183] Optionally, the front camera or the first sensor on the electronic device can collect information according to a set period.

[0184] As previously described, in embodiments of the present application, face / eye detection can obtain one or more of the following information: the position of the face / eye relative to the electronic device display screen, the rotation angle of the face / eye relative to the electronic device display screen, and the depth of field of the face / eye relative to the electronic device display screen. Optionally, combining multiple pieces of information to determine the orientation of the face / eye can further improve accuracy.

[0185] Step 1603: The electronic device dynamically changes the screen display object according to the change in the orientation of the face / eyes.

[0186] In the embodiments of the present application, when the electronic device is stationary, changes in the user's position (such as movement) or rotation or shaking of the user's head may cause the orientation of the face / eyes to change, or the relative position between the face / eyes and the electronic device display screen to change; when the user is stationary, changes in the position of the electronic device, such as rotation or shaking, may cause the orientation of the face / eyes to change, or the relative position between the face / eyes and the electronic device display screen to change. Regardless of the cause of the change, in the embodiments of the present application, the object displayed on the off screen will dynamically change accordingly.

[0187] Optionally, since the electronic device can periodically capture images and perform face / eye detection on the images, it can obtain changes in the orientation of the face / eye in a timely manner as the orientation of the face / eye changes, and can also obtain changes in the emotions expressed by the face / eye, so that the display of the screen-off display object can be updated in a timely manner to achieve the effect of dynamically changing the screen-off display object.

[0188] In one possible implementation, the screen-off display object includes a first component, which is used to represent an eye. In step 1603, when the orientation of the face changes, the electronic device can change the first component in the screen-off display object according to the change (such as changing its position, state or shape, etc.), so that the display position or sight direction of the changed first component matches the current orientation of the face or eye. Optionally, the above-mentioned first component includes an eyeball in the eye. Accordingly, the electronic device can change the display position of the eyeball in the eye according to the change in the orientation of the face / eye, so that the sight line of the screen-off display object changes with the change in the orientation of the user's face / eye, so that the user feels that the screen-off display object is always looking at the user.

[0189] For example, the effect of the line of sight of the screen-off display object changing with the orientation of the user's face / eyes can be made smoother by playing an animation. Taking the change of a face from a first orientation to a second orientation as an example, during this process, the electronic device can play a first animation, and the first animation includes N frames (N is an integer greater than 1). Each of the N frames contains the screen-off display object. In the N frames, the display state of the screen-off display object in the first frame matches the first orientation, the display state of the screen-off display object in the last frame matches the second orientation, and the display state of the screen-off display object in the second frame to the N-1th frame matches an intermediate orientation in the transition process from the first orientation to the second orientation. By playing the first animation, a smooth dynamic change effect can be obtained.

[0190] A possible example may be the above-mentioned scenario 1 (see FIG8 for details).

[0191] Optionally, the above-mentioned screen-off display object may further include a second component, which is used to express emotions. In step 1503, the electronic device may also determine the emotion expressed by the face or eyes based on face or eye detection, and update the second component according to the emotion (such as updating the position, state or shape of the second component), so that the emotion expressed by the screen-off display object matches the emotion expressed by the current face or eyes. A possible example may be the above-mentioned scenario 2 (see Figure 9 for details) or scenario 3 (see Figure 10 for details).

[0192] In another possible implementation, the screen-off display object includes a three-dimensional object, an example of which can be shown in Figure 2. In step 1603, when the electronic device detects a change in the orientation of a face or eye, it can change the orientation of the three-dimensional object so that the changed orientation of the three-dimensional object matches the orientation of the face or eye, thereby achieving a dynamic naked-eye 3D display effect. One possible example can be as described in Scenario 4 above (see Figure 11 for details).

[0193] In another possible implementation, the screen display object includes a text object, which is used to display at least two texts. When the text object is rotated, part of the text of at least two texts is displayed and / or part of the text is hidden. An example of the text object can be shown in Figure 3. In step 1503, when the electronic device detects that the orientation of the face or the eyes of the person has changed, the text object rotates to present the hidden text. Exemplarily, when the electronic device detects that the orientation of the face or the eyes of the person is a first orientation, the text object rotates in a direction opposite to the first orientation to present the hidden text.

[0194] Taking the text object as a three-dimensional object as an example, the first surface of the three-dimensional object displays first text, and the second surface of the three-dimensional object displays second text. When the electronic device detects that the orientation of a human face or eye changes from the first orientation to the second orientation, the text object rotates from the first angle to the second angle; or when the electronic device detects that the orientation of a human face or eye changes from the second orientation to the first orientation, the text object rotates from the second angle to the first angle. At the first angle, the first surface of the text object is displayed and the second surface is hidden, and at the second angle, the second surface of the text object is displayed and the first surface is hidden.

[0195] Optionally, the text object is displayed as a hollow effect, and the hollow portion of the text object reveals a portion of the image of the desktop wallpaper of the electronic device.

[0196] A possible example may be the above-mentioned scenario five (see FIG. 13 or FIG. 15 for details).

[0197] In an embodiment of the present application, when the electronic device satisfies the requirement to switch to the bright screen mode (for example, the user turns on the screen by touching the screen), and the display screen of the electronic device is unlocked, the electronic device switches from the off screen mode to the bright screen mode. In the bright screen mode, the desktop and desktop background are displayed on the display screen of the electronic device.

[0198] Optionally, in an embodiment of the present application, during the process of switching from screen-off mode to screen-on mode, the electronic device can play a transition animation to achieve a one-shot visual effect, switching from screen-off mode to screen-on mode to improve viewing smoothness and provide users with a better visual experience.

[0199] Taking the above-mentioned text object displayed as a hollow effect as an example, the electronic device receives the user's unlocking operation, such as the user's unlocking operation can be a touch screen operation or other types of operations, which is not limited in the embodiments of the present application. After the electronic device receives the user's unlocking operation, in response, the electronic device can switch from the screen-off mode to the screen-on mode through a one-shot visual effect, and display the desktop and desktop wallpaper of the electronic device in the screen-on mode. Since the text object has a hollow display effect, its hollow part will reveal part of the desktop wallpaper. By playing the transition animation, more parts of the desktop wallpaper can be gradually revealed until all the desktop wallpaper is revealed, so as to improve fluency. A possible example can be as described in Scenario 5 above (see Figure 14 for details).

[0200] In some embodiments of the present application, when the electronic device detects a change in the orientation of a face / eye, it can dynamically generate a screen-off display object that matches the current orientation to achieve the purpose of dynamically displaying the screen-off display object.

[0201] In one possible implementation, the electronic device can, based on AI capabilities, generate a screen-off display object that matches the current orientation based on the user-set screen-off display object and the current face / eye orientation, and refresh the display. A specific implementation of generating a screen-off display object based on AI capabilities can be: the screen-off display object set by the user and the current face / eye orientation information are input into an AI-based image generation model to generate a new screen-off display object.

[0202] In another possible implementation, the screen-off display object may include multiple layers stacked in sequence, and the screen-off display object can be displayed by stacking the multiple layers. When the electronic device detects a change in the orientation of a face or eye, the screen-off display object can be dynamically changed by changing the position of some of the multiple layers, or moving the layers.

[0203] For example, the emoji shown in Figure 1 is formed by stacking four layers from top to bottom: glasses, eyeballs, whites of the eyes, and a mouth. Based on the changes in the orientation of the face / eyes, the electronic device can calculate the direction and amount of movement of the glasses layer and the eyeball layer, respectively. The electronic device then moves the glasses and eyeball layers based on the calculation results. This allows the new emoji, formed by stacking these four layers, to match the current orientation of the face / eyes.

[0204] In another possible implementation, the screen-off display object may include multiple layers stacked in sequence, and the screen-off display object can be displayed by stacking the multiple layers. When the electronic device detects a change in the orientation of the face or eyes, the corresponding layer in the multiple layers can be replaced with a new layer, thereby dynamically changing the screen-off display object.

[0205] For example, the emoticon shown in Figure 1 is formed by stacking four layers from top to bottom: glasses, eyeballs, whites of the eyes, and a mouth. Based on the emotional changes expressed by the face / eyes, the electronic device can determine that the emotion has changed from sadness to happiness. The electronic device then replaces the layer containing the mouth with a new layer, replacing the original downward-curving mouth with an upward-curving one. This new emoticon, formed by stacking these four layers, now matches the emotion expressed by the face / eyes.

[0206] It will be understood that the above are only exemplary methods for dynamically updating the screen-off display object, and the embodiments of the present application do not limit this.

[0207] Referring to Figure 17, which is a flowchart of a display method in screen-off mode provided in an embodiment of the present application, the process can be executed by the electronic device shown in Figure 6 or Figure 7, or by a functional module, processor, or chip in the electronic device. The following process is described with the electronic device as the execution subject. As shown in the figure, the process may include the following steps:

[0208] Step 1701: The electronic device enters screen-off mode.

[0209] In an embodiment of the present application, the screen off function can be turned on in advance, and the EOD function can be turned on, so that when the electronic device meets the screen off conditions (such as not using the electronic device within a set time), it can automatically enter the screen off mode.

[0210] Step 1702: When the electronic device detects that a person's eyes are gazing at the display screen, the electronic device displays a screen-off display object on the display screen in the screen-off mode.

[0211] In an embodiment of the present application, after the electronic device enters the screen-off mode, when the electronic device detects that a person is looking at the display screen, the screen-off display object is displayed on the display screen. The setting method of the screen-off display object can be as described above.

[0212] Optionally, the electronic device can capture images through a front camera, and the electronic device can determine whether the human eye is looking at the display screen of the electronic device by executing an eye tracking algorithm on the image.

[0213] Step 1703: The electronic device determines the orientation of the face / eye based on face / eye detection, where the orientation of the face / eye is associated with the relative position between the face / eye and the electronic device.

[0214] For the specific implementation of this step, please refer to the relevant content in Figure 16.

[0215] Step 1704: The electronic device dynamically changes the screen display object according to the change in the orientation of the face / eyes.

[0216] For the specific implementation of this step, please refer to the relevant content in Figure 16.

[0217] The process shown in Figure 17 can achieve the "air triggering" dynamic change display effect of the screen-off display object. Specifically, the user's behavior of looking at the display screen can trigger the display of the screen-off display object. Subsequently, by shaking the electronic device, or by shaking or moving the head, the dynamic change of the screen-off display object can be triggered.

[0218] In some embodiments of the present application, the processing operations of face / eye detection performed in the screen-off mode and the processing operations of dynamically displaying the screen-off display object based on the changes in the orientation of the face / eye can be performed by a low-power AI processor (or low-power AI chip) to reduce the power consumption of the electronic device.

[0219] In the above-mentioned embodiments of the present application, in screen-off mode, the display objects can dynamically change based on the user's current mood, gaze direction, and the angle at which the electronic device is held. This satisfies the user's need for personalization, broadens the scope of interactive scenarios in screen-off mode, and allows users to experience rich and detailed interactive effects, providing a better visual experience. Furthermore, it effectively reduces power consumption and the risk of screen burn-in in screen-off mode.

[0220] Figure 18 is a schematic diagram of the structure of a possible device provided by an embodiment of the present application. This device can be used to implement the functions of the electronic device in the above method embodiment, and thus can also achieve the beneficial effects of the above method embodiment. In the embodiment of the present application, the device can be an electronic device as shown in Figure 6, or it can be a module (such as a chip) used in a corresponding device.

[0221] As shown in Figure 18, the apparatus 1800 includes a processing unit 1810 and a transceiver unit 1820. The apparatus 1800 is used to implement the functions of the terminal device or the sorting system in the method embodiment shown in Figure 16 or Figure 17 above.

[0222] Optionally, the transceiver unit may include a receiving unit and a sending unit. The receiving unit may receive information, such as signaling or data, and the sending unit may send information, such as signaling or data, under the control of the processing unit.

[0223] When the device 1800 is used to implement the functions of the electronic device in the method embodiment as shown in Figure 16: the processing unit 1810 is used to control the electronic device to enter the screen-off mode, and display the screen-off display object on the display screen in the screen-off mode; the processing unit 1810 determines the orientation of the face or the eye based on face or eye detection, and the orientation is associated with the relative position between the face or the eye and the electronic device; and dynamically changes the screen-off display object according to the change of the orientation of the face or the eye.

[0224] When the device 1800 is used to implement the functions of the electronic device in the method embodiment as shown in Figure 17: the processing unit 1810 is used to control the electronic device to enter the screen-off mode; when it is detected that human eyes are looking at the display screen, the screen-off display object is displayed on the display screen in the screen-off mode; the orientation of the face or the eye is determined based on face or eye detection, and the orientation is associated with the relative position between the face or the eye and the electronic device; according to the change of the orientation of the face or the eye, the screen-off display object is dynamically changed.

[0225] A more detailed description of the above-mentioned processing unit 1810 and the transceiver unit 1820 can be directly obtained by referring to the relevant description in the method embodiment shown in Figure 16 or Figure 17, and will not be repeated here.

[0226] An embodiment of the present application also provides a device comprising a processor and an interface circuit. The processor and the interface circuit are coupled to each other. It is understandable that the interface circuit may be a transceiver or an input / output interface. Optionally, the transceiver may include a receiver and a transmitter. Optionally, the device may further include a memory for storing instructions executed by the processor or storing input data required by the processor to execute instructions or storing data generated after the processor executes instructions. When the device is used to implement the method shown in Figure 16 or Figure 17, the processor is used to implement the functions of the above-mentioned processing unit, and the interface circuit is used to implement the functions of the above-mentioned transceiver unit.

[0227] When the above-mentioned device is a chip applied to an electronic device, the chip implements the functions of the electronic device in the above-mentioned method embodiment. The chip receives information from other modules in the electronic device (such as a radio frequency module or antenna), and the information is sent by the network device to the electronic device; or the chip sends information to other modules in the electronic device (such as a radio frequency module or antenna), and the information is sent by the electronic device to the network device.

[0228] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0229] In the present application, another example of a device is provided, which includes at least one processor and at least one memory, wherein the at least one processor and the at least one memory are coupled, and the at least one memory is used to store instructions. When the instructions are executed by the at least one processor, the communication device executes the method in the above embodiment.

[0230] The method steps in the embodiments of the present application can be implemented in hardware or in software instructions that can be executed by a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disk, mobile hard disk, CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. The storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal. The processor and storage medium can also exist in a network device or a terminal as discrete components.

[0231] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are performed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video disk; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or nonvolatile storage medium, or may include both volatile and nonvolatile types of storage media.

[0232] In the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0233] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of this application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula of this application, the character " / " indicates that the previous and next associated objects are in a "division" relationship. "Including at least one of A, B and C" can mean: including A; including B; including C; including A and B; including A and C; including B and C; including A, B and C.

[0234] It is understood that the various numbers used in the embodiments of this application are merely for ease of description and are not intended to limit the scope of the embodiments of this application. The order of the sequence numbers of the above-mentioned processes does not necessarily imply a specific order of execution; the order of execution of the processes should be determined by their functions and inherent logic.

Claims

1. A display method in screen-off mode, using an electronic device with a display screen, characterized in that: include: Entering a screen-off mode, and displaying a screen-off display object on the display screen in the screen-off mode; Determining the orientation of a face or eye based on face or eye detection, wherein the orientation is associated with a relative position between the face or eye and the electronic device; The screen-off display object is dynamically changed according to the change of the orientation of the human face or the human eyes.

2. A display method in screen-off mode, using an electronic device with a display screen, characterized in that: include: Enter screen off mode; When it is detected that human eyes are gazing at the display screen, displaying a screen-off display object on the display screen in the screen-off mode; Determining the orientation of a face or eye based on face or eye detection, wherein the orientation is associated with a relative position between the face or eye and the electronic device; The screen-off display object is dynamically changed according to the change of the orientation of the human face or the human eyes.

3. The method according to claim 1 or 2, characterized in that: The screen-off display object includes a first component, and the first component is used to represent eyes; The dynamically changing the screen-off display object according to the change of the orientation of the face or the eyes of the person, comprises: When the orientation of the human face or the human eye changes, the first component is changed according to the change, so that the display position or sight direction of the changed first component matches the orientation of the human face or the human eye.

4. The method according to claim 3, characterized in that The first component includes an eyeball in an eye; The step of changing the first component according to the change includes: The display position of the eyeball in the eye is changed according to the change.

5. The method according to any one of claims 3-4, characterized in that: The screen-off display object includes a second component, and the second component is used to express emotions; The method further comprises: Based on face or eye detection, the emotion expressed by the face or the eye is determined, and the second component is updated so that the emotion presented by the screen-off display object matches the emotion expressed by the face or the eye.

6. The method according to claim 5, characterized in that The determining the emotion expressed by the face or eyes based on face or eyes detection includes: Artificial intelligence (AI) capabilities are used to detect faces or eyes to determine the emotions expressed by the faces or eyes.

7. The method according to any one of claims 1 to 6, characterized in that: The change in the orientation of the face or the eyes includes: the face or the eyes change from a first orientation to a second orientation; The dynamically changing the screen display object according to the change of the method of the human face or the human eyes includes: Play a first animation, wherein the first animation includes N frames, where N is an integer greater than 1, and each of the N frames includes the screen-off display object; among the N frames, a display state of the screen-off display object in a first frame matches the first orientation, a display state of the screen-off display object in a last frame matches the second orientation, and a display state of the screen-off display object in a second frame to an N-1th frame matches an intermediate orientation in a transition process from the first orientation to the second orientation.

8. The method according to any one of claims 1 to 7, characterized in that: The screen-off display object includes a three-dimensional object; The dynamically changing the screen-off display object according to the change of the orientation of the face or the eyes of the person, comprises: When the orientation of the human face or the human eyes changes, the orientation of the three-dimensional object is changed accordingly, so that the orientation of the three-dimensional object after the change matches the orientation of the human face or the human eyes.

9. The method according to claim 1 or 2, characterized in that: The screen-off display object comprises a rotatable text object, the text object is used to display at least two texts, and when the text object is rotated, part of the at least two texts is displayed and / or part of the text is hidden; The dynamically changing the screen-off display object according to the change of the orientation of the face or the eyes of the person, comprises: When the orientation of the human face or the human eyes changes, the text object rotates to present the hidden text.

10. The method according to claim 9, characterized in that When the orientation of the human face or the human eyes changes, the text object rotates to present the hidden text, including: When the orientation of the human face or human eyes is a first orientation, the text object rotates in a direction opposite to the first orientation to present the hidden text.

11. The method according to any one of claims 9 to 10, characterized in that: The text object is a three-dimensional object, a first surface of the three-dimensional object displays a first text, and a second surface of the three-dimensional object displays a second text; When the orientation of the human face or the human eyes changes, the text object rotates to present the hidden text, including: When the orientation of the human face or the human eyes changes from a first orientation to a second orientation, the text object rotates from a first angle to a second angle; or When the orientation of the human face or the human eye changes from the second orientation to the first orientation, the text object rotates from the second angle to the first angle; Wherein, at the first angle, the first surface of the text object is displayed, and at the second angle, the second surface of the text object is displayed.

12. The method according to any one of claims 9 to 11, characterized in that: The text object is displayed as a hollowed-out effect, and the hollowed-out portion of the text object reveals a portion of the image of the desktop wallpaper of the electronic device; The method further comprises: In response to the unlocking operation, the screen is switched from the off-screen mode to the on-screen mode through a one-shot visual effect, and the desktop of the electronic device and the desktop wallpaper are displayed.

13. The method according to any one of claims 1 to 12, characterized in that: Also includes: Acquiring information detected by the electronic device using a front camera or a first sensor; Perform face or eye detection based on the detected information.

14. The method according to any one of claims 1 to 13, characterized in that: The change in the orientation of the human face or the human eyes is caused by the movement of the user or the movement of the electronic device.

15. A device, characterized in that: The device comprises a memory, a processor and a display device, wherein the memory is used to store a computer program, and the processor is used to read the computer program so that the device implements the method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the computer is caused to implement the method according to any one of claims 1 to 14.

17. A computer program product, characterized in that The computer program product comprises computer-readable instructions, and when the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 14.

18. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method according to any one of claims 1-14.

Images