Display method and device, electronic equipment and medium

Abstract

The invention relates to a display method and device, electronic equipment and a medium, the display method is applied to the electronic equipment, the display method comprises the steps that image information and / or sound information of a user are / is acquired, and the image information reflects face information and / or limb information of the user; generating an anthropomorphic image based on the image information and / or the sound information; and displaying the anthropomorphic image. The anthropomorphic image is generated based on the image information and / or the sound information of the user, and the anthropomorphic image has the personal characteristics of the user, so that the state of the user can be reflected in real time, the display content of the screen is enriched, the interestingness and vividness of display are enhanced, and the use experience of the user is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of electronic devices, and in particular to a display method, device, electronic device and medium. Background Art

[0002] At present, with the development of electronic devices, users have higher and higher requirements for visual experience when using electronic devices. Among them, the Always On Display (AOD) function is a display mode in which the entire screen is not lit up, and specific content is displayed in a part of the screen. In the prior art, after the AOD function is activated, the specified area of ​​the electronic device screen can display preset content such as dynamic and static pictures, date and time, weather conditions, notification messages, etc. The preset content is not personalized and not vivid enough, and cannot reflect the current state of the user. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a display method, device, electronic device and medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a display method is provided, which is applied to an electronic device, and the display method includes:

[0005] Acquiring image information and / or voice information of a user, wherein the image information reflects facial information and / or body information of the user;

[0006] generating an anthropomorphic image based on the image information and / or the sound information;

[0007] The anthropomorphic image is displayed.

[0008] In some exemplary embodiments of the present disclosure, the method further includes: recognizing a limb motion according to the limb information, the limb motion including a gesture;

[0009] The step of generating an anthropomorphic image based on the image information and / or the sound information includes:

[0010] When the gesture is different from the screen-lighting gesture, generating an anthropomorphic image based on the image information and / or the sound information;

[0011] When the gesture is the same as the screen-lighting gesture, the screen is turned on.

[0012] In some exemplary embodiments of the present disclosure, the gesture is different from the screen-lighting gesture, including:

[0013] The gesture is different from the screen-lighting gesture and is the same as the gesture used to trigger the display of the anthropomorphic image.

[0014] In some exemplary embodiments of the present disclosure, the method further includes:

[0015] Determine the state of the user based on the image information and / or the sound information, where the state includes at least one of the following: facial expression, emotion, and body movement;

[0016] The step of generating an anthropomorphic image based on the image information and / or the sound information includes:

[0017] The anthropomorphic image is generated based on the image information and / or the sound information, as well as the status of the character model and the user.

[0018] In some exemplary embodiments of the present disclosure, the anthropomorphic image is a dynamic image based on the image of the user, or a dynamic image based on the image of an object other than the user.

[0019] In some exemplary embodiments of the present disclosure, the state of the anthropomorphic image is the same as or different from the state of the user.

[0020] In some exemplary embodiments of the present disclosure, determining the user's state based on the image information and / or the sound information includes:

[0021] Determining the quality of the image information based on the clarity and brightness of the image information;

[0022] If the quality of the image information meets the condition, determining the user's expression based on a face recognition model and the image information, and / or determining the user's gesture based on a gesture recognition model and the image information;

[0023] If the quality of the image information does not meet the conditions, the user's emotion is determined based on the emotion recognition model and the image information and / or the sound information, and / or the user's gesture is determined based on the image recognition model and the image information.

[0024] In some exemplary embodiments of the present disclosure, determining the user's expression based on the face recognition model and the image information includes:

[0025] Based on the face recognition model and the image information, determining the positions of key points on the user's face and the recognition probabilities of the key points on the face;

[0026] When the recognition probabilities of all the facial key points are greater than or equal to a first threshold, the user's expression is determined based on the positions of the facial key points.

[0027] In some exemplary embodiments of the present disclosure, determining the user's emotion based on the emotion recognition model and the image information and / or the sound information includes:

[0028] When there is a facial key point whose recognition probability is less than the first threshold, the user's emotion is determined based on the emotion recognition model and the image information and / or the sound information.

[0029] In some exemplary embodiments of the present disclosure, determining the user's emotion based on the emotion recognition model and the sound information includes:

[0030] Preprocessing the sound information;

[0031] Based on the emotion recognition model and the preprocessed sound information, the user's emotion and emotion level are determined.

[0032] In some exemplary embodiments of the present disclosure, determining the user's gesture based on the gesture recognition model and the image information includes:

[0033] Based on the gesture recognition model and the image information, determining the positions of key points of the user's hand and the recognition probability of the key points of the hand;

[0034] When the recognition probabilities of all the hand key points are greater than or equal to a second threshold, determining the user's gesture based on the hand key points;

[0035] When there is a hand key point whose recognition probability is less than the second threshold, the user's gesture cannot be determined.

[0036] In some exemplary embodiments of the present disclosure, displaying the anthropomorphic image includes:

[0037] The screen is in a screen-off state, and the anthropomorphic image is displayed at the center of the screen.

[0038] According to a second aspect of an embodiment of the present disclosure, a display device is provided, which is applied to an electronic device, and the display device includes:

[0039] An acquisition module, used to acquire image information and / or sound information of a user, wherein the image information reflects facial information and / or body information of the user;

[0040] A generating module, used for generating an anthropomorphic image based on the image information and / or the sound information;

[0041] The display module is used to display the anthropomorphic image.

[0042] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0043] Screen;

[0044] processor;

[0045] a memory for storing processor-executable instructions;

[0046] Among them, the processor is configured to execute executable instructions in the memory to implement the display method provided in the first aspect of the present disclosure.

[0047] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which executable instructions are stored. When the executable instructions are executed by a processor, the display method provided by the first aspect of the present disclosure is implemented.

[0048] The above method disclosed in the present invention has the following beneficial effects: the electronic device generates and displays an anthropomorphic image based on image information and / or sound information. Since the anthropomorphic image is generated based on the image information and / or sound information of the user, the anthropomorphic image has the personal characteristics of the user and can reflect the user's status in real time. At the same time, it enriches the display content of the screen, enhances the fun and vividness of the display, and improves the user's experience.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0051] Figure 1 The figure is a flowchart of a display method according to an exemplary embodiment.

[0052] Figure 2 The figure is a flowchart of a display method according to an exemplary embodiment.

[0053] Figure 3 is a block diagram of a display device according to an exemplary embodiment.

[0054] Figure 4 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0056] In the prior art, after the screen-off display function is activated, a designated area of ​​the electronic device screen can display preset content such as dynamic and static pictures, date and time, weather conditions, notification messages, etc. The preset content is not personalized and is not vivid enough to reflect the user's current status.

[0057] In order to solve the above problems, the present disclosure provides a display method, which is applied to an electronic device, using a camera to obtain the user's image information, using a microphone to obtain the user's voice information, and the electronic device generates an anthropomorphic image based on the image information and / or the voice information, and displays it on the screen to achieve real-time monitoring of the user's status. Since the anthropomorphic image is generated based on the user's image information and voice information, the anthropomorphic image has the user's personal characteristics, so the anthropomorphic image is displayed in the screen-off state. On the basis of ensuring low power consumption and low memory usage, it can also enrich the display content of the screen, enhance the fun and vividness of the display, and improve the user's experience.

[0058] The exemplary embodiments of the present disclosure provide a display method, which is applied to electronic devices, specifically, smart devices such as mobile phones, tablet computers, notebooks, smart watches, etc. The display method of the present disclosure is implemented by a control device of the electronic device, which is also a control chip of the electronic device, and has a data processing function, as well as information and instruction receiving and sending functions, etc. The control device includes a display device involved in the following text.

[0059] like Figure 1 As shown, the display method shown in the present disclosure includes:

[0060] S101, obtaining image information and / or voice information of a user, where the image information reflects facial information and / or body information of the user;

[0061] S102, generating an anthropomorphic image based on the image information and / or the sound information;

[0062] S103: Display an anthropomorphic image.

[0063] In step S101, the electronic device is built with a camera module for collecting image information, for example, the camera module includes an AON (Always-ON) camera, and the control device maintains a communication connection with the camera module to obtain image information when the user uses the electronic device. The image information can reflect the user's facial information and / or body information. The specific content of the image information is related to the distance between the user and the electronic device. Since the camera has a limited viewing angle range for collecting images, if the user is close to the electronic device, the camera can only collect local information of the user, such as obtaining the user's facial information; if the user is far away from the electronic device, the camera can collect information in a larger range, such as obtaining the user's facial information and body information. The electronic device also has a built-in microphone module for collecting sound information. The control device maintains a communication connection with the microphone module to obtain sound information when the user uses the electronic device. The sound information can reflect the user's speech content, frequency, volume, etc. Among them, the camera module and the microphone module can be in a normally open state to obtain the user's image information and sound information in real time to accurately reflect the user's status; the camera module and the microphone module can also be turned on at a fixed time to obtain the user's image information and sound information at fixed intervals to reduce power consumption; the camera module and the microphone module can also be turned on alternately to obtain the user's image information or sound information at fixed intervals.

[0064] In step S102, since the image information reflects the user's facial information and / or body information, and the sound information reflects the volume and frequency of the user's speech, the control device can determine the user's facial expressions and / or body movements through the image information, determine the user's emotions through the sound information, and generate an anthropomorphic image matching the user's state for display on the screen.

[0065] If only image information is collected, the control device can determine the expression and body movement that the anthropomorphic image is to reflect by identifying facial information and body information, and generate an image that matches the user's current expression and body movement. For example, if the user is currently frowning and one of his hands is moving up and down, an anthropomorphic image with a frown and one hand waving is generated based on the collected image information. In addition, if the image information only includes the user's facial information, only one anthropomorphic image with expression changes can be generated; if the image information only includes the user's body information, only one anthropomorphic image with body movement changes can be generated. Since the image information includes the user's facial information and has a high degree of privacy, in order to improve the security of privacy, the processing of the image information including facial information can be completed in the Trusted Execution Environment (TEE). Processing and generating anthropomorphic images in TEE does not involve the privacy module and does not store user information, which can effectively protect the privacy security of users.

[0066] If only sound information is collected, the control device can identify the sound information, determine the emotion that the anthropomorphic image is to reflect, and generate an image that matches the user's current emotion. For example, the sound information reflects that the user's current speaking volume is loud and the frequency is high, reflecting that the user is in an angry state, and then an angry anthropomorphic image is generated based on the collected sound information. Since sound information can only reflect the user's emotions, in order to improve the vividness of the anthropomorphic image, the control device pre-stores expression changes and body movement changes that match the emotions, that is, after determining the user's emotions, the control device will generate an anthropomorphic image with expression changes and body movement changes.

[0067] If the image information and the sound information are collected at the same time, the control device can use the image information and the sound information at the same time to generate an anthropomorphic image, which completely matches the state of the user and accurately reflects the current state of the user using the electronic device.

[0068] In step S103, the anthropomorphic image is displayed on the screen of the electronic device, and there is no limitation on the display position of the anthropomorphic image. If the electronic device is in the screen-off state, the anthropomorphic image can be displayed in the center of the screen; if the screen of the electronic device is in the main menu interface or the negative one screen interface, the anthropomorphic image can be displayed in the blank area of ​​the screen. In addition, the user can also change the display position of the anthropomorphic image according to personal visual habits. For example, the default display position of the anthropomorphic image in the main menu interface is the lower right corner of the screen, and the user can move the anthropomorphic image to the upper right corner of the screen.

[0069] In the present disclosure, the electronic device generates an anthropomorphic image based on image information and / or sound information, and displays it on the screen to realize real-time monitoring of the user's status. Since the anthropomorphic image is generated based on the user's image information and sound information, the anthropomorphic image has the user's personal characteristics. Therefore, the anthropomorphic image is displayed in the screen-off state. On the basis of ensuring low power consumption and low memory usage, it can also enrich the display content of the screen, enhance the fun and vividness of the display, and improve the user's experience.

[0070] According to an exemplary embodiment, Figure 2 As shown, the display method in this embodiment includes:

[0071] S201, obtaining image information and / or voice information of a user, where the image information reflects facial information and / or body information of the user;

[0072] S202, determining the user's state based on the image information and / or the sound information, where the state includes at least one of the following: facial expression, emotion, and body movement;

[0073] S2021. Determine the quality of the image information based on the clarity and brightness of the image information;

[0074] S2022, determining whether the quality of the image information meets the conditions, if the quality of the image information meets the conditions, executing step S2023; if the quality of the image information does not meet the conditions, executing step S2024;

[0075] S2023, determining the user's expression based on the face recognition model and the image information, and / or determining the user's gesture based on the gesture recognition model and the image information;

[0076] S2024, determining the user's emotion based on the emotion recognition model and the image information and / or the sound information, and / or determining the user's gesture based on the image recognition model and the image information;

[0077] S203, generating an anthropomorphic image based on the image information and / or the sound information, as well as the status of the character model and the user;

[0078] S204: Display an anthropomorphic image.

[0079] Among them, steps S201 and S204 are the same as those in the above embodiment and will not be described again.

[0080] In step S202, since the image information can reflect the user's facial information and body information, and the sound information can reflect the volume and frequency of the user's speech, the control device can determine the user status based on the image information and the sound information, where the user status includes at least one of expression, emotion, and body movement. Expressions include raising the corners of the mouth, frowning, etc., emotions include happiness, sadness, anger, fear, surprise, etc., and body movements include movements of the head, hands, shoulders, etc., such as shaking the head, making gestures, turning the shoulders, etc.

[0081] In step S2021, the image information can reflect the user's facial information and body information. Since the recognition of user expressions based on image information is achieved by using a face recognition method, there are quality requirements for the images obtained by the camera module. If the image is too blurry or the brightness is too low, the control device cannot recognize the user's expression through the image. If the image is clear and the brightness is sufficient, the control device can recognize the user's expression and body movements through the image, where body movements include user gestures.

[0082] Therefore, the process of the control device determining the user's expression through the image information has quality requirements for the image information. Only when the quality of the image information meets the requirements can the control device accurately obtain the user's state from the image information. If the quality of the image information does not meet the requirements, the control device cannot use the image information to determine the user's expression and can only determine the user's gesture. The control device determines the quality of the image information based on the clarity and brightness of the image information.

[0083] When detecting the clarity of image information, the control device can use the Sobel algorithm or the Laplacian algorithm to detect the edge of the image information. The Sobel algorithm detects the edge in the image by calculating the difference between the grayscale value of each pixel in the image and the surrounding pixels. The specific process is: first convert the collected image into a grayscale image; perform Sobel operator calculation on the grayscale image to calculate the gradient values ​​in the horizontal and vertical directions; calculate the square sum of the gradient values ​​in the horizontal and vertical directions to obtain the gradient value of the image; count the gradient values ​​in the image, and evaluate the clarity of the image according to the size of the gradient value, wherein the average gradient value or the standard deviation of the gradient value can be selected as the clarity evaluation index to obtain the clarity evaluation result. The Laplacian algorithm detects the edge in the image by calculating the second-order derivative of each pixel in the image. The specific process is: first convert the collected image into a grayscale image; perform Laplacian operator calculation on the grayscale image to calculate the second-order derivative of the image; count the second-order derivative values ​​in the image to obtain the clarity evaluation result.

[0084] The control device can select the result obtained by either the Sobel algorithm or the Laplacian algorithm as the clarity result of the image information. In order to improve the accuracy, the control device can also evaluate the clarity of the image information using the results obtained by the two algorithms, that is, weighted average the results obtained by the two algorithms, and use the weighted average result as the final clarity evaluation result of the image information.

[0085] When detecting the brightness of image information, the control device can use a histogram algorithm or a Retinex algorithm to detect the brightness of image information. The specific process of the histogram algorithm to detect brightness is: first convert the collected image into a grayscale image, calculate the histogram of the grayscale image, and count the number of pixels at each grayscale level; calculate the cumulative distribution function of each grayscale level according to the histogram, and then transform the grayscale value of the image according to the cumulative distribution function; count the brightness values ​​of the pixels in the transformed grayscale image to obtain the brightness evaluation result. The brightness evaluation result output by the histogram algorithm usually uses the mean and standard deviation as evaluation indicators, where the larger the mean, the higher the image brightness; the smaller the standard deviation, the smaller the brightness difference in the image and the more uniform the image.

[0086] The specific process of Retinex algorithm to detect brightness is: first convert the collected image into LAB color space, separate the L channel as the brightness component; perform Gaussian filtering on the brightness component to remove noise in the image; calculate the reflection component and illumination component of the image based on the brightness component; enhance the reflection component to obtain the enhanced image; count the brightness values ​​of the pixels in the enhanced image to obtain the brightness evaluation result. The brightness evaluation result output by the Retinex algorithm usually uses the mean and standard deviation as evaluation indicators, where the larger the mean, the higher the image brightness; the smaller the standard deviation, the smaller the brightness difference in the image and the more uniform the image.

[0087] Only when the clarity and brightness of the image meet the standards, it is determined that the quality of the image information meets the conditions. Among them, the clarity of the image is related to the resolution, so the clarity judgment standard can be set according to the performance of the electronic device. It should be noted that since the ambient light around the user when using the electronic device will also affect the control device to determine the user's expression through the image information, the brightness judgment standard for the image is determined based on the ambient light intensity. When the ambient light intensity is large, the brightness judgment standard for the image is high, and when the ambient light intensity is small, the brightness judgment standard for the image is low. For example, the electronic device is currently in a room with curtains drawn on a cloudy day, and the ambient light intensity is 10lux. When the brightness of the image is higher than 10nit, it is considered that the brightness of the image meets the standard; the electronic device is in a room with lights on, and the ambient light intensity is 300lux. When the brightness of the image is higher than 50nit, it is considered that the brightness of the image meets the standard.

[0088] In step S2022, only when the clarity and brightness of the image information meet the respective judgment standards, the quality of the image information meets the conditions, and the control device can determine the user's expression and body movements based on the image information, so step S2023 is executed. When one of the clarity and brightness of the image information does not meet the respective judgment standards, the quality of the image information does not meet the conditions, and the control device cannot use the image information to determine the user's expression, so step S2024 is executed.

[0089] In step S2023, when the quality of the image information meets the conditions, if the image information includes the user's facial information and body information, the control device can determine the user's expression based on the face recognition model and the image information, and determine the user's gesture based on the gesture recognition model and the image information. If the image information only includes the user's facial information, the control device can determine the user's expression based on the face recognition model and the image information. If the image information includes the user's body information, the control device can determine the user's gesture based on the gesture recognition model and the image information.

[0090] In some embodiments, step S2023 determines the user's expression based on the face recognition model and the image information, including:

[0091] Based on the face recognition model and image information, determine the location of the user's facial key points and the recognition probability of the facial key points;

[0092] When the recognition probabilities of all facial key points are greater than or equal to a first threshold, the user's expression is determined based on the positions of the facial key points.

[0093] The network selection of the face recognition model is related to the location of image processing and the memory of the electronic device. Since the image information includes the user's facial information and has a high degree of privacy, privacy-protected image processing can be performed in the Trusted Execution Environment (TEE), which requires a very small network, such as BlazeFace, Mobilenet-Yolo, etc. If the image information only includes the user's body information, image processing can be performed in the hardware abstraction layer (HAL) or the application processor layer (AP), and a larger network can be used, such as HRNet, SANet, etc.

[0094] When the quality of the image information meets the conditions, the control device uses the face recognition model to detect facial key points of the image information. Among them, the number of facial key points can be selected from any value of 5-400, and the number of facial key points is positively correlated with the memory space of the electronic device. If the memory space of the electronic device is large, more key points can be selected to improve the recognition effect of the user's expression. For example, based on the distribution of the facial features of the human body, the facial key points can be selected from the left and right corners of the mouth, the center of the eyes, and the nose.

[0095] The image information is input into the face recognition model, and the face recognition model outputs the coordinates and recognition probability of each facial key point. When the recognition probability of all facial key points is greater than or equal to the first threshold, the control device matches the facial key position to the anthropomorphic image so that the anthropomorphic image can have an expression consistent with the user. If there is a facial key point whose recognition probability is less than the first threshold, the changes in the user's facial features cannot be determined, that is, the user's expression cannot be determined, then the image information can be input into the emotion recognition model, and the user's emotional changes can be determined by the emotion recognition model. For example, there are five facial key points, and the first threshold is 0.7. If the face recognition model determines that the recognition probabilities of the five facial key points of the image information are 0.8, 0.8, 0.75, 0.9, and 0.85, all of which are greater than the first threshold, the user's expression can be determined. If the face recognition model determines that the recognition probabilities of the five facial key points of the image information are 0.6, 0.8, 0.75, 0.65, and 0.85, and there are two facial key points whose recognition probabilities are less than the first threshold, the user's expression cannot be determined, so the emotion recognition model is selected to determine the user's emotion.

[0096] In some embodiments, step S2023 determines the user's gesture based on the gesture recognition model and the image information, including:

[0097] Based on the gesture recognition model and image information, determine the user's hand key point position and hand key point recognition probability;

[0098] When the recognition probability of all hand key points is greater than or equal to a second threshold, determining the user's gesture based on the hand key points;

[0099] When there is a hand key point whose recognition probability is less than the second threshold, the user's gesture cannot be determined.

[0100] The gesture recognition model can recognize dynamic gestures, including left swipe, right swipe, up swipe, down swipe and other gestures. The control device processes the image information based on the gesture recognition model to determine the position of the user's hand key points and the probability of hand key point recognition to determine the user's gesture. Among them, the network selection of the gesture recognition model is related to the location of the image processing and the memory of the electronic device. When privacy protection is required, image processing can be performed in TEE (Trusted Execution Environment) using a very small network, such as EfficientDet, CenterNet, etc. In addition, image processing can also be performed in the hardware abstraction layer (HAL, HardwareAbstraction Layer) or the Application Processor layer (AP, Application Processor), using a larger network, such as ST-GCN, CV-STAN, etc.

[0101] The control device inputs the image information into the gesture recognition model, and determines the position of the hand key points and the recognition probability of the hand key points through the data processing module, feature extraction module, hand key point detection module and posture estimation module. When the recognition probability of all hand key points is greater than or equal to the second threshold, the control device can determine the user's gesture based on the hand key points, and match the position of the hand key points to the anthropomorphic image so that the anthropomorphic image can have a gesture consistent with the user. If there is a hand key point whose recognition probability is less than the second threshold, the position of all hand key points cannot be determined, so the user's gesture cannot be determined, that is, the control device does not detect the gesture.

[0102] In step S2024, when the quality of the image information does not meet the condition, if the image information includes the user's body information, the control device can determine the user's gesture based on the image recognition model and the image information, and determine the user's emotion based on the emotion recognition model and the image information and / or the sound information. If the image information does not include the user's body information, the control device determines the user's emotion based on the emotion recognition model and the image information and / or the sound information.

[0103] The image recognition model can only recognize a fixed number of designated gestures, such as ok, one, two, three, four, etc., and cannot recognize dynamic gestures. The control device inputs low-quality image information that does not meet the conditions into the image recognition model. If the gesture in the image information is a fixed gesture that the image recognition model can recognize, the user's gesture can be determined, but if the gesture in the image information is not a fixed gesture that the image recognition model can recognize, the user's gesture cannot be determined. Among them, the network selection of the image recognition model is related to the location of image processing and the memory of the electronic device. When the electronic device leaves the factory, the corresponding network has been determined according to its own performance. When privacy protection is required, image processing can be performed in TEE (Trusted Execution Environment) using a very small network, such as MobileNet, EfficientNet, etc. In addition, image processing can also be performed in the hardware abstraction layer (HAL, Hardware Abstraction Layer) or the Application Processor layer (AP, Application Processor), using a larger network, such as ResNet, DenseNet, etc. It should be noted that in the screen-off state, the power consumption of the electronic device is expected to be as low as possible, so for image recognition models, etc., image recognition models composed of small networks are preferably used.

[0104] In some embodiments, the display method further includes:

[0105] When the gesture is different from the screen-lighting gesture, generating an anthropomorphic image based on the image information and / or the sound information;

[0106] When the gesture is the same as the screen-lighting gesture, the screen is turned on.

[0107] In order to reduce power consumption, when the user has not used the electronic device for a long time, the electronic device will be in the screen-off state. If the control device determines that the user's gesture is the same as the pre-stored screen-on gesture, the electronic device exits the screen-off state, lights up the screen, and displays the main menu interface, application interface, etc. For example, the pre-stored left swipe is a screen-on gesture. If the user's gesture is determined to be a left swipe based on a gesture recognition model or an image recognition model, the screen is turned on. If the control device determines that the user's gesture is different from the pre-stored screen-on gesture, and the electronic device has turned on the AOD function, the user's status is determined based on the image information and / or sound information, combined with other recognition models, and an anthropomorphic image consistent with the user's status is generated.

[0108] In some embodiments, the gesture is different from the screen-lighting gesture, including:

[0109] The gesture is different from the screen-lighting gesture and is the same as the gesture used to trigger the display of the anthropomorphic image.

[0110] The gesture used to trigger the display of an anthropomorphic image can be some specific gestures or any gesture except the screen-lighting gesture. When the control device can determine the user's gesture based on the collected image information, the anthropomorphic image can be generated and displayed based on the image information and / or sound information. For example, swiping to the left is set as the screen-lighting gesture, and any gesture except the screen-lighting gesture is a gesture that triggers the display of an anthropomorphic image. If the user's gesture is determined to be swiping to the right based on the gesture recognition model or the image recognition model, the control device generates an anthropomorphic image with a swiping to the right gesture, which is different from the screen-lighting gesture, and displays the anthropomorphic image on the screen.

[0111] When the image information does not meet the conditions, the control device can use the emotion recognition model to determine the user's state. The emotion recognition model consists of two parts, including a first emotion recognition model and a second emotion recognition model, wherein the first emotion recognition model recognizes emotions based on image information, and the second emotion recognition model recognizes emotions based on sound information. That is, if the control device collects image information and sound information at the same time, the image information and the sound information can be input into the first emotion recognition model and the second emotion recognition model in the emotion recognition model, and the image information and the sound information are used to jointly determine the user's emotions; if the control device only collects image information, the image information can be input into the first emotion recognition model, and the user's emotions are determined by the image information; if the control device only collects sound information, the sound information can be input into the second emotion recognition model, and the user's emotions are determined by the sound information.

[0112] The following describes the training process of the first emotion recognition model in the emotion recognition model.

[0113] First, a large network, such as the EmotiCon network, is constructed. High-quality image datasets, such as FER2013 and The Extended Cohn-Kanade Dataset, are used to train the large network, ensuring that the large network can accurately identify eight expressions, namely, anger, disgust, fear, happiness, calmness, sadness, surprise, and no emotion. It should be noted that no emotion includes images without facial information. After training a large network that can accurately identify expressions, the image information collected by the camera module is used as a real dataset to fine-tune the large network. That is, based on the existing large network, only the newly added real dataset is used for training to correct the parameters, so that the large network is more suitable for the image information collected by the camera module. The output result of the large network is a probability vector, which includes the predicted probability of each expression. For example, an image of a happy expression is input to the large network, and the large network outputs a probability vector of 60% happy, 30% angry, and 10% sad, where happy is the correct classification, while angry and sad are wrong classifications, and the wrong classification with the highest probability is angry. That is, the probability vector reflects which correct expression the image looks like and which incorrect answer it looks like. For example, some happy expressions have the highest probability of being misclassified as anger, while other expressions have the highest probability of being misclassified as fear.

[0114] After building a large network, a small lightweight neural network is built, such as a MobileNet series neural network. The image information collected by the camera module is used as a training data set, and the probability vector output by the large network is used as a soft target. The small network uses positive and negative samples for learning, learns the generalization ability of the large network, and makes the expression ability of the small network closer to that of the large network. The results of several frames of small networks in the training data set are multiplied by a preset probability distribution, such as a geometric distribution or a Poisson distribution. The geometric distribution represents that the most recent result in the past period of time can best represent the real result, so the result of the frame of image with the closest time in the multiple frames of images is given the highest weight value; the Poisson distribution represents that the result closest to the current time in the past period of time is the most important, so the result weight value of the frame of image closer to the current time is greater. That is, the results of several frames of images output by the small network are fused to determine the emotional changes. The small network that is finally trained is the second emotion recognition model. The control device inputs the collected several frames of image information into the second emotion recognition model, and the second emotion recognition model determines the emotions in the image information, fuses the multiple frames of results, and directly outputs them as the user's emotions.

[0115] In some embodiments, step S2024 determines the user's emotion based on the emotion recognition model and the sound information, including:

[0116] Preprocessing the sound information;

[0117] Based on the emotion recognition model and the preprocessed sound information, the user's emotions and emotional levels are determined.

[0118] The low-power microphone module can be in a normally open state and obtain the user's voice information at any time. Due to the low quality of the sound information, the sound information needs to be pre-processed for noise reduction. Since the recognition of emotions does not require the recognition of the voice content of the sound information, the pre-processing process only needs to remove noise such as noise. Statistical model algorithms or frequency domain filtering algorithms can be used to remove noise, and both algorithms can achieve good noise reduction effects. The statistical model algorithm models the noise by estimating the statistical distribution of the signal and noise, and removes the noise from the original signal. Common statistical model algorithms include Wiener filters and Kalman filters. The frequency domain filtering algorithm converts the audio signal to the frequency domain and uses a filter to filter out the noise to reduce noise. Common frequency domain filtering algorithms include FFT, DCT, Wavelet, etc.

[0119] The training process of the second emotion recognition model is the same as that of the first emotion recognition model. First, the denoised sound information is used as a training data set to train a large network with better results (such as a Transformer network). Then, a small network (such as a C-RNN network) is used to learn the generalization ability of the large network to obtain a small network with a recognition effect that is close to that of the large network. The small network after training is the second emotion recognition model. The preprocessed sound information is input into the second emotion recognition model, and the second emotion recognition model outputs the determined emotion result and the degree (probability) of the emotion. For example, the output result is 70% for happiness and 30% for surprise.

[0120] When the voice is high-pitched or sharp, anger and surprise are easily confused. In order to improve the accuracy of emotion recognition, when the sum of the probabilities of anger and surprise in the result output by the second emotion recognition model is greater than the third threshold, and the difference is less than the fourth threshold, the sound information is input into the third emotion recognition model to distinguish anger and surprise separately. Since the third emotion recognition model only outputs anger or surprise, the accuracy of the result is higher than the result determined by the second emotion recognition model. For example, the third threshold is 0.8, the fourth threshold is 0.2, and the result output by the second emotion recognition model is 40% anger, 50% surprise, and 10% happiness, where the sum of the probabilities of anger and surprise is 0.9 and the probability difference is 0.1, so the sound information needs to be input into the third emotion recognition model for re-recognition.

[0121] The third emotion recognition model first extracts features from the sound information, extracting 15 time domain features, 13 frequency domain features and 6 MFCC (Mel Frequency Cepstrum Coefficient) features. Among them, the 15 time domain features include: fluctuation, time domain energy, time domain variance, zero crossing rate, short-time average energy, short-time average zero crossing rate, short-time zero crossing rate cumulative value, short-time energy cumulative value, short-time zero crossing rate root mean square value, short-time energy root mean square value, fundamental frequency, peak amplitude, envelope, linear prediction coefficient, short-time autocorrelation function, time domain cross-correlation function. The 13 frequency domain features include: frequency spectral density, power spectral density, frequency band energy ratio, spectral entropy, spectral centroid, spectral average amplitude, spectral skewness, spectral kurtosis, frequency peak, spectral envelope, frequency offset, modulation frequency, and spectral dimension reduction. The six MFCC features include: Mel filter energy distribution on the frequency axis; energy distribution characteristics on the frequency axis; distribution characteristics of cepstral coefficients; correlation between cepstral coefficients; energy distribution characteristics on the time axis; and change characteristics on the time axis.

[0122] The above 34 features of the sound information are input into the three-layer MLP (Multilayer Perception) network in the third emotion recognition model for calculation, and anger or surprise is obtained as output.

[0123] In order to improve the accuracy of emotion recognition, when image information and sound information are collected at the same time, the results of the first emotion recognition model and the second emotion recognition model can be fused to determine the user's emotions. That is, after the convolution of the first emotion recognition model, it is input to the FC layer (fully connected layer) or the GAP layer (GlobalAvgPooling) to obtain n1 features. Similarly, the second emotion recognition model can obtain n2 features. The numerical values ​​corresponding to the n1 and n2 features are concatenated as input and input into a multi-layer MLP to calculate the result, and the user's emotional result can be obtained. If the sum of the probabilities of anger and surprise in the emotional result determined after fusion is greater than the third threshold, and the difference is less than the fourth threshold, the sound information is input into the third emotion recognition model for re-judgment.

[0124] In addition, in order to make the anthropomorphic image fully display the user's status and make the generated anthropomorphic image more vivid, when the quality of the image information meets the requirements, it can also be combined with sound information to use face recognition models, gesture recognition models, and emotion recognition models to jointly determine the user's status.

[0125] All the recognition models mentioned above can run in low power mode without waking up the main CPU, thus effectively reducing power consumption and memory usage.

[0126] In step S203, after determining the state of the user through image information and / or sound information, the control device will synchronously change the expression, body movements, etc. of the character model to generate an anthropomorphic image consistent with the user's state, and display the user's state through the anthropomorphic image. Among them, the character model can be a model similar to the user based on face recognition, or a model pre-stored in an electronic device, or a model determined by fine-tuning the face based on the information obtained from face recognition on the basis of a pre-stored model. In order to increase the fun, the character model can also be in the form of an animal, or a cartoon image that the user likes. For different display scenarios of anthropomorphic images, the character model includes three different images: head portrait, half body, and full body. For example, if an anthropomorphic image is displayed in the screen-off state, a full-body anthropomorphic image can be generated because the blank area of ​​the screen is large; if an anthropomorphic image is displayed in the main menu interface, a half-body anthropomorphic image can be generated because the blank area of ​​the screen is small.

[0127] Since the user's status may change at any time, in order to synchronously display the user's status, the control device may generate a dynamic anthropomorphic image, for example, the anthropomorphic image may be a gif image, an animation, etc. In addition, the control device may also generate a static anthropomorphic image, and the control device changes the anthropomorphic image based on the user's status at fixed intervals to display the user's status in stages.

[0128] In some embodiments, the anthropomorphic image is a dynamic image based on the image of the user, or a dynamic image based on the image of an object other than the user.

[0129] Since the character model can be based on a model that is consistent with the image of the user or a model that is inconsistent with the image of the user, the anthropomorphic image generated based on the image information and / or sound information, as well as the state of the character model and the user, can be a dynamic image that is consistent with the image of the user or a dynamic image that is inconsistent with the image of the user. For example, if the character model is made based on the image of the user, the generated anthropomorphic image is a dynamic image that not only has expressions, emotions, or body movements that are consistent with the state of the user, but also has the same distribution of facial features and the same body proportions as the user. For another example, if the character model is made based on the image of a cartoon cat, the anthropomorphic image is a dynamic image that displays expressions, emotions, or body movements that are consistent with the state of the user.

[0130] In some embodiments, the state of the anthropomorphic image is the same as or different from the state of the user.

[0131] The situation where the state of the anthropomorphic image is the same as the state of the user refers to the situation where the expression, emotion or body movement of the anthropomorphic image is the same as at least one of the expression, emotion or body movement of the user. For example, the control device can determine the three states of the user's expression, emotion and body movement based on the collected image information and / or sound information, and the control device can choose to generate an anthropomorphic image with only the same expression as the user's expression, or generate an anthropomorphic image with only the same body movement as the user's body movement, or generate an anthropomorphic image with only the same emotion as the user's emotion, or generate an anthropomorphic image with the same expression and body movement as the user's expression and body movement, or generate an anthropomorphic image with the same expression, emotion and body movement as the user's expression, emotion and body movement.

[0132] In addition, the control device can also generate an anthropomorphic image that is different from the user's state, for example, generate an anthropomorphic image that is opposite to the user's state, when the control device determines that the user's emotion is happy, generate an anthropomorphic image with a sad emotion; when the control device determines that the user has not changed his body movements, generate an anthropomorphic image of dancing. For another example, generate an anthropomorphic image that remains unchanged for a period of time, that is, the anthropomorphic image will not change according to the user's state and will not remain the same as the user's state at any time.

[0133] In the present disclosure, users can use voice or gesture commands to enable the function of generating anthropomorphic images, recognize the collected image information and sound information, and generate anthropomorphic images that show the user's status. Even low-quality images can detect the user's status in real time, thereby improving the utilization rate of image information. Anthropomorphic images can be displayed in the screen-off state or in other interfaces such as the negative one screen, enriching the display content of the screen, increasing the fun and liveliness, and giving the displayed content personal characteristics.

[0134] The exemplary embodiment of the present disclosure provides a display device, which is applied to electronic equipment. Figure 3 As shown, a block diagram of a display device shown in the present disclosure.

[0135] The block diagram includes: an acquisition module 31, a generation module 32, and a display module 33. The acquisition module 31 is used to acquire the user's image information and / or sound information, and the image information reflects the user's facial information and / or body information; the generation module 32 is used to generate an anthropomorphic image based on the image information and / or sound information; and the display module 33 is used to display the anthropomorphic image.

[0136] In an exemplary embodiment of the present disclosure, body movements are identified based on body movement information, and body movements include gestures. The generation module 32 is also used to: generate an anthropomorphic image based on image information and / or sound information when the gesture is different from the screen-lighting gesture; the display module 33 is also used to: light up the screen when the gesture is the same as the screen-lighting gesture.

[0137] In an exemplary embodiment of the present disclosure, the gesture is different from the screen-lighting gesture, including: the gesture is different from the screen-lighting gesture, and is the same as the gesture for triggering display of an anthropomorphic image.

[0138] In an exemplary embodiment of the present disclosure, the user's state is determined based on image information and / or sound information, and the state includes at least one of the following: facial expression, emotion, and body movement; the generation module 32 is also used to generate an anthropomorphic image based on the image information and / or sound information, as well as the character model and the user's state.

[0139] In an exemplary embodiment of the present disclosure, the anthropomorphic image is a dynamic image based on the image of a user, or a dynamic image based on the image of an object other than the user.

[0140] In an exemplary embodiment of the present disclosure, the state of the anthropomorphic image is the same as or different from the state of the user.

[0141] In an exemplary embodiment of the present disclosure, the generation module 32 is also used to: determine the quality of the image information based on the clarity and brightness of the image information; if the quality of the image information meets the requirements, determine the user's expression based on the face recognition model and the image information, and / or determine the user's gesture based on the gesture recognition model and the image information; if the quality of the image information does not meet the requirements, determine the user's emotion based on the emotion recognition model and the image information and / or sound information, and / or determine the user's gesture based on the image recognition model and the image information.

[0142] In an exemplary embodiment of the present disclosure, the generation module 32 is also used to: determine the user's facial key point positions and the recognition probability of the facial key points based on the face recognition model and image information; when the recognition probability of all facial key points is greater than or equal to a first threshold, determine the user's expression based on the facial key point positions.

[0143] In an exemplary embodiment of the present disclosure, the generating module 32 is further configured to: when there is a facial key point whose recognition probability is less than a first threshold, determine the user's emotion based on the emotion recognition model and the image information and / or the sound information.

[0144] In an exemplary embodiment of the present disclosure, the generation module 32 is further used to: pre-process the sound information; and determine the user's emotion and emotion level based on the emotion recognition model and the pre-processed sound information.

[0145] In an exemplary embodiment of the present disclosure, the generation module 32 is also used to: determine the user's hand key point position and hand key point recognition probability based on the gesture recognition model and image information; when all hand key point recognition probabilities are greater than or equal to a second threshold, determine the user's gesture based on the hand key points; when there is a hand key point recognition probability that is less than the second threshold, the user's gesture cannot be determined.

[0146] In an exemplary embodiment of the present disclosure, the display module 33 is further used to: when the screen is in a screen-off state, display an anthropomorphic image in the center of the screen.

[0147] Regarding the display device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0148] Figure 4 4 is a block diagram of an electronic device 400 according to an exemplary embodiment. For example, the electronic device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0149] Reference Figure 4 , the electronic device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0150] The processing component 402 generally controls the overall operation of the electronic device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0151] The memory 404 is configured to store various types of data to support operations on the electronic device 400. Examples of such data include instructions for any application or method operating on the electronic device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0152] The power supply component 406 provides power to the various components of the electronic device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 400.

[0153] The multimedia component 408 includes a screen that provides an output interface between the electronic device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the electronic device 400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0154] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the electronic device 400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0155] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0156] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the electronic device 400. For example, the sensor assembly 414 can detect the open / closed state of the electronic device 400, the relative positioning of components, such as the display and keypad of the electronic device 400, and the sensor assembly 414 can also detect the position change of the electronic device 400 or a component of the electronic device 400, the presence or absence of user contact with the electronic device 400, the orientation or acceleration / deceleration of the electronic device 400, and the temperature change of the electronic device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0157] The communication component 416 is configured to facilitate wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0158] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0159] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of an electronic device 400 to perform the above display method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0160] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables a processing device of the electronic device to execute a display method provided by an exemplary embodiment of the present disclosure.

[0161] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0162] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A display method, characterized in that: The display method comprises: Acquiring image information and / or voice information of a user, wherein the image information reflects facial information and / or body information of the user; generating an anthropomorphic image based on the image information and / or the sound information; The anthropomorphic image is displayed.

2. The display method according to claim 1, characterized in that: The method further includes: identifying a body movement according to the body information, wherein the body movement includes a gesture; The step of generating an anthropomorphic image based on the image information and / or the sound information includes: When the gesture is different from the screen-lighting gesture, generating an anthropomorphic image based on the image information and / or the sound information; When the gesture is the same as the screen-lighting gesture, the screen is turned on.

3. The display method according to claim 2, characterized in that: The gestures described above are different from the screen-lighting gestures and include: The gesture is different from the screen-lighting gesture and is the same as the gesture used to trigger the display of the anthropomorphic image.

4. The display method according to any one of claims 1 to 3, characterized in that: The method further comprises: Determine the state of the user based on the image information and / or the sound information, where the state includes at least one of the following: facial expression, emotion, and body movement; The step of generating an anthropomorphic image based on the image information and / or the sound information includes: The anthropomorphic image is generated based on the image information and / or the sound information, as well as the status of the character model and the user.

5. The display method according to claim 4, characterized in that: The anthropomorphic image is a dynamic image based on the image of the user, or a dynamic image based on the image of an object other than the user.

6. The display method according to claim 5, characterized in that: The state of the anthropomorphic image is the same as or different from the state of the user.

7. The display method according to claim 4, characterized in that: The determining the state of the user based on the image information and / or the sound information includes: Determining the quality of the image information based on the clarity and brightness of the image information; If the quality of the image information meets the condition, determining the user's expression based on a face recognition model and the image information, and / or determining the user's gesture based on a gesture recognition model and the image information; If the quality of the image information does not meet the conditions, the user's emotion is determined based on the emotion recognition model and the image information and / or the sound information, and / or the user's gesture is determined based on the image recognition model and the image information.

8. The display method according to claim 7, characterized in that: The determining the user's expression based on the face recognition model and the image information includes: Based on the face recognition model and the image information, determining the positions of key points on the user's face and the recognition probabilities of the key points on the face; When the recognition probabilities of all the facial key points are greater than or equal to a first threshold, the user's expression is determined based on the positions of the facial key points.

9. The display method according to claim 8, characterized in that: The determining the user's emotion based on the emotion recognition model and the image information and / or the sound information includes: When there is a facial key point whose recognition probability is less than the first threshold, the user's emotion is determined based on the emotion recognition model and the image information and / or the sound information.

10. The display method according to claim 9, characterized in that: The determining the user's emotion based on the emotion recognition model and the sound information includes: Preprocessing the sound information; Based on the emotion recognition model and the preprocessed sound information, the user's emotion and emotion level are determined.

11. The display method according to claim 7, characterized in that: The determining the user's gesture based on the gesture recognition model and the image information includes: Based on the gesture recognition model and the image information, determining the positions of key points of the user's hand and the recognition probability of the key points of the hand; When the recognition probabilities of all the hand key points are greater than or equal to a second threshold, determining the user's gesture based on the hand key points; When there is a hand key point whose recognition probability is less than the second threshold, the user's gesture cannot be determined.

12. The display method according to claim 1, characterized in that: The displaying of the anthropomorphic image comprises: The screen is in a screen-off state, and the anthropomorphic image is displayed at the center of the screen.

13. A display device, characterized in that: Applied to electronic equipment, the display device comprises: An acquisition module, used to acquire image information and / or sound information of a user, wherein the image information reflects facial information and / or body information of the user; A generating module, configured to generate an anthropomorphic image based on the image information and / or the sound information; The display module is used to display the anthropomorphic image.

14. An electronic device, characterized in that: The electronic device comprises: Screen; processor; a memory for storing processor-executable instructions; The processor is configured to execute executable instructions in the memory to implement the display method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium having executable instructions stored thereon, characterized in that: When the executable instruction is executed by a processor, the display method according to any one of claims 1 to 12 is implemented.

Images