Information processing method and electronic equipment

Abstract

The invention discloses an information processing method and electronic equipment, and the method comprises the steps: obtaining a collection image through an image collection device in response to a wake-up instruction of the image collection device if the electronic equipment is in a first mode; recognizing the collected image, obtaining a first recognition result of an identity represented by the biological characteristic information in the collected image, and obtaining a second recognition result of an emotion represented by the biological characteristic information in the collected image; if the first recognition result represents that the identity of the target user is met, the electronic equipment is controlled to be switched from a first mode to a second mode, and in the process that the electronic equipment is switched from the first mode to the second mode, a target object matched with the second recognition result is output; the power consumption of the electronic equipment in the first mode is lower than the power consumption of the electronic equipment in the second mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, and more particularly to an information processing method and an electronic device. BACKGROUND

[0002] With the wide application of electronic devices, in order to save the power consumption of the electronic devices, a low-power mode, such as a lock screen mode, of the electronic device is usually set. When a user unlocks the lock screen mode, a corresponding unlocking interface is entered, and the unlocking interface is generally a main display interface of the intelligent terminal, and the setting form is relatively single, which cannot meet the current state of the user and reduces the user experience effect. SUMMARY

[0003] Therefore, the present application provides the following technical solutions.

[0004] An information processing method comprises:

[0005] If the electronic device is in a first mode, a wake-up instruction of an image acquisition device is responded to, and an acquisition image is obtained through the image acquisition device;

[0006] The acquisition image is recognized to obtain a first recognition result of an identity represented by biological feature information in the acquisition image and a second recognition result of an emotion represented by the biological feature information in the acquisition image;

[0007] If the first recognition result represents an identity meeting a target user, the electronic device is controlled to switch from the first mode to a second mode, and in the process of the electronic device switching from the first mode to the second mode, a target object matching the second recognition result is outputted.

[0008] The power consumption of the electronic device in the first mode is lower than the power consumption of the electronic device in the second mode.

[0009] Optionally, the recognition of the acquisition image to obtain the first recognition result of the identity represented by the biological feature information in the acquisition image and the second recognition result of the emotion represented by the biological feature information in the acquisition image comprises:

[0010] The acquisition image is processed based on an image recognition model to obtain the first recognition result of the identity represented by the biological feature information in the acquisition image and the second recognition result of the emotion represented by the biological feature information in the acquisition image;

[0011] The image recognition model comprises an identity recognition branch and an emotion classification branch. The identity recognition branch is configured to recognize an identity represented by biological feature information in the collected image. The emotion classification branch is configured to recognize an emotion represented by the biological feature information in the collected image.

[0012] Optionally, the outputting the target object matched with the second recognition result comprises:

[0013] In response to configuring the target object as an interactive component, displaying the interactive component in a display interface of the electronic device after the electronic device switches to the second mode.

[0014] In response to configuring the target object as target output content, after outputting the target output content, displaying a display interface of the electronic device after the electronic device switches to the second mode; wherein the target output content represents output content independent of the display interface corresponding to the second mode.

[0015] Optionally, after outputting the target output content, displaying the display interface of the electronic device after the electronic device switches to the second mode comprises:

[0016] Based on the priority of the current notification information of the electronic device, determining the output duration of the target output content;

[0017] In response to the target output content outputting the target output duration, jumping to the display interface corresponding to the second mode of the electronic device.

[0018] Optionally, the control of the electronic device switching from the first mode to the second mode comprises:

[0019] Based on the second recognition result, determining the emotional state of the target user;

[0020] Controlling the electronic device to switch from the first mode to the second mode matched with the emotional state, so that the output of the electronic device in the second mode matches the output configuration of the emotional state.

[0021] Optionally, the electronic device comprises a foldable screen device, the first mode is a closed lock screen state of the foldable screen device, and the obtaining of the second recognition result of the emotion represented by the biological feature information in the collected image comprises:

[0022] In response to detecting that the user unfolds the foldable screen device, obtaining a force parameter of the user unfolding the foldable screen device;

[0023] Based on the force parameter, identifying the collected image to obtain the second recognition result of the emotion represented by the biological feature information in the collected image.

[0024] Optionally, the output of the target object matched with the second recognition result comprises:

[0025] determining a screen state corresponding to the folding screen device in a second mode; the second mode comprises a closed unlocked state or an unfolded unlocked state of the folding screen device;

[0026] determining an output mode of the target object matched with the second recognition result based on the screen state;

[0027] outputting the target object based on the target output mode.

[0028] Optionally, it further comprises:

[0029] obtaining scene feature information of the electronic device;

[0030] determining the target object matched with the second recognition result based on the scene feature information and the second recognition result.

[0031] Optionally, it further comprises:

[0032] recording each time the second recognition result is obtained and a corresponding time stamp;

[0033] generating emotion change trend data based on each time the second recognition result and the time stamp;

[0034] in response to a query request, displaying the emotion change trend data.

[0035] An electronic device comprises:

[0036] an image acquisition device, a biological feature recognition device and a control device;

[0037] the image acquisition device is configured to acquire an acquisition image in response to a wake-up instruction if the electronic device is in a first mode;

[0038] the biological feature recognition device is configured to recognize the acquisition image, obtain a first recognition result of an identity represented by biological feature information in the acquisition image, and obtain a second recognition result of an emotion represented by the biological feature information in the acquisition image;

[0039] the control device is configured to control the electronic device to switch from the first mode to a second mode if the first recognition result represents an identity meeting a target user, wherein a target object matched with the second recognition result is output during the process of the electronic device switching from the first mode to the second mode.

[0040] The power consumption of the electronic device in the first mode is lower than the power consumption of the electronic device in the second mode. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on the provided drawings.

[0042] Figure 1 A flowchart of an information processing method provided by the embodiments of the present application is shown in the figure.

[0043] Figure 2 A schematic diagram of processing collected images by different processing modules provided by the embodiments of the present application is shown in the figure.

[0044] Figure 3 A processing schematic diagram of an image recognition process provided by the embodiments of the present application is shown in the figure.

[0045] Figure 4 A schematic diagram of a home screen after unlocking provided by the embodiments of the present application is shown in the figure.

[0046] Figure 5 A schematic diagram of screen display changes when an electronic device is switched from a first mode to a second mode provided by the embodiments of the present application is shown in the figure.

[0047] Figure 6 A schematic diagram of a daily user mood change curve provided by the embodiments of the present application is shown in the figure.

[0048] Figure 7 A schematic diagram of a monthly user mood table provided by the embodiments of the present application is shown in the figure.

[0049] Figure 8 A structural schematic diagram of an electronic device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort belong to the scope of protection of the present application.

[0051] The terms "first" and "second" and the like in the present application are used to distinguish different objects, rather than to describe a particular order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can include steps or units not listed.

[0052] The electronic device involved in the embodiments of the present application can include various forms of devices, such as mobile terminal devices, vehicle-mounted devices, wearable devices, desktop devices, etc. Among them, the mobile terminal device can include a smart phone, a tablet computer or other mobile terminals, and the operating system of the mobile terminal device includes but is not limited to Android, IOS, Windows, Linux, Harmony, etc. The electronic device can have functions related to communication, image acquisition, processing and display. The function modules of the specific electronic device are related to the type and application scenario of the electronic device, and the embodiments of the present application do not limit this.

[0053] The embodiments of the present application provide an information processing method, applied to an electronic device, in the process of switching the electronic device from a first mode in low power consumption to a second mode, synchronously realizing identity verification and emotion recognition of a user of the electronic device, providing personalized output conforming to the current emotion for the user while realizing the mode switching verification of the electronic device, and improving the user experience effect.

[0054] Referring to Figure 1 A flowchart of an information processing method provided by the embodiments of the present application, the method can include the following steps:

[0055] S101, if the electronic device is in the first mode, obtaining the captured image through the image acquisition device in response to the wake-up instruction of the image acquisition device.

[0056] S102, identifying the captured image to obtain a first recognition result of the identity represented by the biological feature information in the captured image and a second recognition result of the emotion represented by the biological feature information in the captured image.

[0057] S103, if the first recognition result represents that the identity meets the target user, controlling the electronic device to switch from the first mode to the second mode. In the process of switching the electronic device from the first mode to the second mode, a target object matching the second recognition result is output.

[0058] The power consumption of the electronic device in the first mode is lower than the power consumption of the electronic device in the second mode. The first mode can be that part of the functional components of the electronic device are in a running state, and other functional components are in a non-running state or a low-power sleep state; the first mode can also be that the functional components such as CPU (Central Processing Unit) and GPU (Graphics Processing Unit) of the electronic device are in a low-power running state, and some necessary functional components corresponding to the current application scenario are in an active state. For example, the first mode can be a lock screen mode of the electronic device, or a sleep mode or a sleep mode of the electronic device. Correspondingly, the second mode of the electronic device can be a normal running mode of the electronic device, such as a mode in which the electronic device is in a working state.

[0059] The wake-up instruction of the image acquisition device can be a mode switching instruction of the electronic device by the user, such as a first mode being a lock screen mode of the electronic device, and the wake-up instruction of the image acquisition device can be an unlocking instruction of the electronic device, such as an instruction generated by the user triggering a face unlocking operation; or it can be a mode switching instruction of the electronic device recognized by the electronic device that the user can generate, such as when the electronic device is a folding screen device, the operation of the user from closing to unfolding the electronic device can also be recognized as a mode switching instruction, thereby generating a wake-up instruction for the image acquisition device. In the embodiments of the present application, the image acquisition device can be a device for image acquisition of a user's face region, such as a camera of the electronic device. If the electronic device is a mobile phone, the image acquisition device can be a front camera of the mobile phone. The image acquisition device can also be a fingerprint image acquisition device, and further, the image acquisition device can also include a fingerprint image acquisition device and a face image acquisition device.

[0060] In response to the wake-up instruction of the image acquisition device, the acquisition image can be obtained through the image acquisition device. Then, relevant information in the acquisition image is identified, on the one hand, the user identity corresponding to the biological feature information in the acquisition image is identified, and a first identification result is obtained; on the other hand, the user emotion reflected by the biological feature information is identified, and a second identification result is obtained. In the embodiments of the present application, the identity verification and emotion recognition are realized synchronously at the time of device mode switching, which ensures low power consumption while providing identification in line with the current emotion for the user, facilitating subsequent output of relevant information corresponding to the emotion and improving the user experience effect.

[0061] Taking a wake-up instruction as an unlocking instruction of an electronic device in a lock screen state as an example, the unlocking instruction can include user facial unlocking, and a corresponding image acquisition device can be a camera of the electronic device, etc. When the user unlocks by facial unlocking, the camera will acquire a facial image of the user, and the facial image can be used for identity verification and emotion recognition, which can be inputting the facial image into an identity verification module and an emotion recognition module respectively, so as to obtain a first recognition result and a second recognition result respectively. In order to be able to complete two recognition tasks based on the same acquired image synchronously, avoid repeated processing and calling of the image, improve processing efficiency, reduce device computing power consumption, meet the running requirements of the electronic device in a low-power mode (such as a lock screen state), the facial image can also be input into a unified biometric feature recognition module, and the module can realize identity verification and emotion recognition. If the first recognition result represents that the identity of the target user is met, that is, the identity recognition information of the current user matches the identity information of the target user who can unlock the electronic device and is pre-stored in the electronic device. The electronic device can be controlled to switch from the first mode to the second mode, such as controlling the electronic device to switch from the lock screen state to the unlocked state. Since the user's identity is verified at the same time, the second recognition result representing the user's emotion is also obtained, and in the process of switching the electronic device from the first mode to the second mode, a target object matched with the second recognition result is output. The target object represents output content matched with the user's emotion, which can affect the current emotional state of the user. For example, the target object can be at least one of audio, video, image and text matched with the current emotion of the user. For example, when the user's emotional state is identified as low, the corresponding target object can be positive and optimistic information, so as to guide the user to get rid of the low emotional state. In the embodiments of the present application, the target object can be output as a separate information, that is, output before formally entering the interface after unlocking. The target object can also be an interface element in the interface after unlocking and be presented in the interface after unlocking. In this way, personalized output conforming to the current emotion can be provided for the user, the user experience is improved, and the emotion recognition is combined with the mode switching of the electronic device, so that the integration degree and practicability of the function are improved.

[0062] Taking the wake-up instruction as an unlocking instruction of the electronic device in the lock screen state as an example, the unlocking instruction can include fingerprint unlocking, and the corresponding image acquisition device can be a fingerprint image acquisition device such as a camera of the electronic device. At this time, the fingerprint image acquisition device can be controlled to acquire a fingerprint image of the user, identity verification and recognition are performed through the fingerprint image to obtain a first recognition result, and emotion recognition is performed through the fingerprint image to obtain a second recognition result. When emotion recognition is performed through the fingerprint image, the emotion recognition can be performed using the fingerprint image, for example, a processing model corresponding to an emotion recognition module is trained in advance using fingerprint images labeled with emotion categories, so that the processing model can perform emotion classification based on the fingerprint image. The emotion classification is mainly performed by using the three-dimensional topographic characteristics of the fingerprint in the fingerprint image to determine the user's emotion. For example, the ridge line deformation feature (which can reflect the intensity of the user's pressing) and the sweat pore feature (such as whether the sweat pore is expanded) can be obtained from the fingerprint image. For example, when the user presses the fingerprint sensing area of the electronic device with great force, the fingerprint image shows the characteristics of ridge line deformation and sweat pore expansion. The emotion of the user can be recognized as anger according to these characteristics.

[0063] In some embodiments of the present application, when the emotion of the user is recognized through the fingerprint image, in some cases of large changes in emotion, the changes in emotion can be recognized through the fingerprint image. If further fine emotion changes are needed, the user's face image can be used for recognition. Correspondingly, the image acquisition device in the embodiments of the present application can include a fingerprint image acquisition device and a face image acquisition device. In response to the wake-up instruction of the image acquisition device, the acquisition image obtained by the image acquisition device includes: in response to the wake-up instruction of the fingerprint image acquisition device, the fingerprint image is acquired by the fingerprint image acquisition device, and the face image is acquired by the face image acquisition device. Wherein, the recognition of the acquisition image includes: obtaining the first recognition result of the identity represented by the fingerprint feature information in the fingerprint image, and obtaining the second recognition result of the emotion represented by the face feature information in the face image. In this way, fingerprint verification is performed at the same time, and the recognition of the user's emotion is also realized, so that the subsequent output after unlocking can better meet the needs of the user's current emotion, and affect the user's emotional state in subsequent use of the electronic device.

[0064] For example, if the electronic device is a mobile phone, the first mode is the lock screen state, and the power consumption of the mobile phone is low at this time. When the user presses the power button to prepare to unlock the mobile phone (triggers the wake-up instruction of the image acquisition device), the front camera starts and captures the user's face image (i.e., obtains the acquisition image). The mobile phone identifies the face image to determine that the user is the owner of the mobile phone (i.e., the first recognition result matches the target user), and identifies that the user's emotion at this moment is happy (obtains the second recognition result). During the process of switching the mobile phone from the lock screen state to the home screen after unlocking, the text information "Today's mood is really good, keep it up" is output.

[0065] The embodiment of the present application provides an information processing method, when an electronic device is in a first mode of low power consumption, an image collection device is used to obtain a collected image, an identity and an emotion recognition result are obtained by identifying the image, when the identity is consistent with a target user, the device is controlled to switch to a second mode of high power consumption, and a target object matched with the emotion is output during the switching process. The device mode switching opportunity is utilized, the identity verification and the emotion recognition are simultaneously implemented, the low power consumption is ensured, the personalized output consistent with the current emotion is provided for the user, the user experience is improved, and the emotion recognition and the device mode switching are combined, so that the integration degree and the practicability of the function are improved.

[0066] In some embodiments of the present application, referring to Figure 2 which shows a schematic diagram of processing a collected image by different processing modules provided by the embodiment of the present application. After the collected image is obtained by the image collection device 201, the collected image is input to the identity verification module 202 and the emotion recognition module 203 respectively, the collected image is processed by the identity verification module 202, a first recognition result is obtained, that is, an identity recognition result of whether the current user is a target user of the electronic device is obtained. The collected image is processed by the emotion recognition module 203, a second recognition result is obtained, that is, a current emotion state of the user, such as happy, negative, angry, sad, etc. is obtained. Then the first recognition result and the second recognition result are sent to the control module 204, a control instruction is generated by the control module 204, that is, it is determined whether the first mode of the electronic device is switched to the second mode, if the first recognition result represents that the identity of the target user is met, the electronic device is switched from the first mode to the second mode, and the control module 204 also controls the electronic device to output a target object matched with the second recognition result during the process of switching from the first mode to the second mode, such as outputting a dynamic component representing the current emotion of the user in the main interface corresponding to the second mode, such as the low emotion state of the user, the dynamic component can display an image that can make people happy, or the dynamic component can output a light and happy audio that can make people happy.

[0067] Correspondingly, in order to improve the recognition efficiency, reuse the collected image, reduce data processing redundancy, and enable the identity recognition and the emotion recognition to be quickly completed based on the same image, in an embodiment of the present application, the process of identifying the collected image, obtaining the first recognition result of the identity represented by the biological feature information in the collected image, and obtaining the second recognition result of the emotion represented by the biological feature information in the collected image includes:

[0068] The collected image is processed based on the image processing recognition model, the first recognition result of the identity represented by the biological feature information in the collected image is obtained, and the second recognition result of the emotion represented by the biological feature information in the collected image is obtained.

[0069] The image recognition model includes an identity recognition branch and an emotion classification branch. The identity recognition branch is configured to recognize an identity represented by biological feature information in the collected image. The emotion classification branch is configured to recognize an emotion represented by the biological feature information in the collected image.

[0070] Referring to Figure 3 Fig. 1 shows a processing schematic diagram of an image recognition process according to an embodiment of the present application. An image collection device 301 sends a collected image to an image recognition model 302. The image recognition model outputs a first recognition result and a second recognition result to a control module 303. The control module determines whether to switch a first mode of an electronic device to a second mode according to the first recognition result, and outputs a target object in a process of switching the electronic device from the first mode to the second mode according to the second recognition result.

[0071] The image recognition model can be a machine learning model, including two recognition branches, namely an identity recognition branch and an emotion classification branch. The identity recognition branch is specially configured to recognize a user identity represented by biological feature information in a collected image, and outputs a first recognition result by comparing extracted biological features (such as face contours, positions of facial features, etc.) with pre-stored target user features. The emotion classification branch is specially configured to recognize a user emotion represented by biological feature information in a collected image, and outputs a second recognition result by analyzing the morphology of biological features (such as the degree of mouth upturn, eyebrow position, eye contact, etc.). The identity recognition branch can be trained according to images including identity feature labels, so that it can predict the identity of a user based on a collected image. The emotion classification branch can be trained according to images including emotion type labels, so that it can classify the emotion of a user based on a collected image.

[0072] For example, the image recognition model is a deep learning model. When a user's face collection image is input, the identity recognition branch extracts key feature points (such as the coordinate positions of eye corners, nose tips, and mouth corners) of the face, compares them with the face feature points stored in the phone, determines whether it is the owner, and obtains a first recognition result. At the same time, the emotion classification branch analyzes the features of the face, such as finding that the mouth is upturned and the eye corners have laugh lines, and judges that the emotion is happy, and obtains a second recognition result.

[0073] In order to enhance the adaptability of the function based on emotion recognition and the richness of user experience, the output characteristics of the target object corresponding to the second recognition result can be determined according to the configuration needs of the user or the current scene characteristics, so as to output the target object.

[0074] In some embodiments, outputting a target object matching the second recognition result includes:

[0075] In response to the target object being configured as an interactive component, the interactive component is displayed in a display interface of the electronic device after the electronic device switches to the second mode.

[0076] In response to the target object being configured as target output content, after the target output content is output, a display interface of the electronic device after the electronic device switches to the second mode is displayed, where the target output content represents output content independent of a display interface corresponding to the second mode.

[0077] The target object being configured as an interactive component can be configured as a floating icon, a clickable widget, a widget, or the like. The interactive component is displayed in a display interface (such as a home screen or an application interface) of the electronic device after the electronic device switches to the second mode, and the user can interact with the component (such as clicking to view details). For example, refer to Figure 4 which shows a schematic diagram of a home screen after unlocking according to an embodiment of the present application. When the second recognition result is an anxiety emotion, if the target object is configured as an interactive component, a floating "relax" icon is displayed on the home screen of the mobile phone after unlocking. The user can click the icon to play soothing music.

[0078] If the target object is configured as target output content (such as a separate emotion prompt page or a piece of music), the target output content is first output. The content is independent of a display interface corresponding to the second mode, and after the output is completed, the display interface of the electronic device after the electronic device switches to the second mode is displayed. For example, refer to Figure 5 , in which Figure 5 The left side of the first diagram is a lock screen interface of the electronic device, and the second diagram is a display interface of the target output content. When it is detected that the emotion of the user is happy and positive, text such as "continue to maintain..." and positive and optimistic image content can be output in the interface. When the user clicks the display interface of the target output content (or when the display reaches a preset time length), the user can enter the home interface of the electronic device after unlocking, as shown in Figure 5 the third diagram from the left.

[0079] In this embodiment, two ways of outputting the target object matching the second recognition result are illustrated. One is to display the target object as an interactive component in the display interface of the second mode, and the other is to output the target object as independent target output content and then display the display interface of the second mode. Flexible and diverse target object output methods can be provided, which can be selected according to different scenarios and user needs, enhancing the adaptability of the function and the richness of the user experience.

[0080] When the target object is configured as the target output content, the output content is independent of the display interface corresponding to the second mode. The target output time length of the target output content can be pre-set, and the display interface corresponding to the second mode is automatically switched after the target output time length is reached. The display interface corresponding to the second mode can also be displayed based on a trigger operation of the user after the target output content is output, such as when the user slides the page corresponding to the target output content, the display interface corresponding to the second mode is automatically switched. The output time length of the target output content can also be determined according to the priority of the current notification information of the electronic device. Correspondingly, in an embodiment of the present application, the process of displaying the display interface of the electronic device after switching to the second mode after the target output content is output can include: determining the output time length of the target output content based on the priority of the current notification information of the electronic device; and jumping to the display interface corresponding to the second mode of the electronic device in response to the target output content being output for the target output time length.

[0081] In this embodiment, the current notification information (such as incoming call notification, short message notification, application push, etc.) of the electronic device is collected, and the priorities of the notification information (such as high priority for incoming calls and low priority for ordinary application pushes) are determined. The output time length of the target output content is determined according to the priority of the notification information, for example, the output time length corresponding to the high-priority notification is short, and the output time length corresponding to the low-priority notification can be longer. When the target output content is output according to the determined output time length, the display interface corresponding to the second mode of the electronic device is automatically jumped to. For example, when the mobile phone outputs the target output content (such as a piece of encouraging text) matched with the emotion, it is detected that there is an incoming call (high-priority notification), and the output time length of the target output content is determined to be 1 second. After 1 second, the incoming call interface (i.e., the display interface corresponding to the second mode) is immediately jumped to. If there is only one ordinary news push (low-priority notification) at present, the output time length is determined to be 5 seconds, and the main screen is jumped to after 5 seconds.

[0082] In the embodiments of the present application, the output time length of the target output content can be determined based on the priority of the current notification information, which ensures that important notifications are not delayed, and reasonably controls the display time of the target output content, balances the emotional guidance and the user's demand for normal use of the device, and improves the practicality of the function.

[0083] In the embodiments of the present application, the second recognition result representing the emotion can also be used to determine the corresponding emotional state application mode, which facilitates adjusting the use emotion of the user of the electronic device in subsequent use of the electronic device. In an embodiment, the process of controlling the electronic device to switch from the first mode to the second mode can include:

[0084] Based on the second recognition result, the emotional state of the target user is determined; the electronic device is controlled to switch from the first mode to a second mode matched with the emotional state, so that the output of the electronic device in the second mode matches the output configuration of the emotional state.

[0085] After verifying the user identity feature in the first recognition result, if it is determined that the user is the target user of the electronic device, the electronic device is controlled to switch from the first mode to the second mode, so as to match the emotional state of the current target user, and switch to the second mode matched with the emotional state, that is, according to the emotional state of the user, the electronic device enters the corresponding output mode, for example, if the emotional state is happy, the second mode is the unlock mode, so that the electronic device is unlocked and enters the output mode of "happy state"; if the emotional state is negative, the electronic device is unlocked and enters the output mode of "negative state".

[0086] According to the second recognition result (such as happy, sad, angry, etc.), the current emotional state of the target user is determined. The electronic device is controlled to switch from the first mode to the second mode matched with the determined emotional state, so that the output (such as interface theme, sound effect, recommended content, etc.) of the electronic device in the second mode conforms to the output configuration corresponding to the emotional state (such as happy mood corresponding to bright theme and cheerful sound effect, sad mood corresponding to soft theme and soothing sound effect). It should be noted that the user can actively exit the output mode matched with the emotional state. If the user does not exit the output mode matched with the emotional state, in addition to the output of the main interface of the electronic device presenting the output information matched with the emotional state, the output information matched with the emotional state can also be embodied in the interface of each application when the user starts each application. For example, when the user is excited, soft color tone can be used as the background color of the output of each application. In this embodiment, the target object matched with the second recognition result (i.e. the emotional recognition result) can be dynamically switched according to the current output information. After entering the second mode (such as the unlock state), the dynamic pendant matched with the current emotional state can be displayed in the main interface. When the user opens the corresponding application program, the content of the target object can be adjusted according to the current output information of the application program, such as displaying the prompt text matched with the emotional state in the frame of the application program. For example, if the second recognition result shows that the user's emotion is sad, it is determined that the current emotional state of the user is negative emotion. When the electronic device switches from the first mode of the lock screen to the second mode, the interface theme is switched to soft warm color tone, the system prompt sound is set to soothing music, and some light short video content is recommended. These outputs are all matched with the output configuration of the sad emotion.

[0087] Further, the current image of the user can also be collected according to the preset monitoring period to determine the current emotional state of the user, and the output content or display mode of the corresponding target object is adjusted according to the real-time emotional state. The output of the electronic device after entering the second mode is matched with the emotional state of the user in real time, realizing more personalized mode switching, so that the user can obtain an environment and content conforming to the emotional state of the user when using the device, and the use comfort of the user is improved.

[0088] In some use scenarios of electronic devices with specific forms, the emotional recognition result corresponding to the use scenario can also be optimized in real time in combination with the form or function of the electronic device, so as to improve the accuracy of user emotional recognition. In some embodiments of the present application, the electronic device can include a folding screen device, and the first mode is the closed lock screen state of the folding screen device. Correspondingly, the second recognition result of the emotion represented by the biological feature information in the collected image includes:

[0089] In response to detecting that the user unfolds the folding screen device, a force parameter of the user unfolding the folding screen device is obtained; and the collected image is identified based on the force parameter to obtain the second recognition result of the emotion represented by the biological feature information in the collected image.

[0090] The operation of the user unfolding the folding screen device is a wake-up instruction of the image collection device, and then the image collection device obtains the collected image and performs user identity verification and emotional recognition based on the collected image. In order to improve the accuracy of emotional recognition, the force parameter of the user unfolding the folding screen device and the image features in the collected image can be combined to determine the second recognition result representing the emotion. When the operation of the user unfolding the folding screen device is detected, the force parameter (such as the size of the unfolding pressure and the unfolding speed) of the user unfolding the device can be obtained through the sensors (such as pressure sensors and angular velocity sensors) built in the electronic device. The obtained force parameter and the collected image (such as the face image taken when the user unfolds the device) are analyzed and recognized together to comprehensively judge the emotion of the user and obtain a more accurate second recognition result.

[0091] For example, when the user unfolds the folding screen phone, the sensor of the phone detects that the unfolding speed is fast and the force is large (the force parameter), and the face image of the user is also taken, and the image shows that the user has a frown. In combination with the force parameter and the face image recognition, it is determined that the current emotion of the user is impatience, and the corresponding second recognition result is obtained, and the comprehensive judgment of multiple dimensions makes the result more accurate.

[0092] Further, in the embodiments of the present application, for the folding screen device, the output mode of the target object can also be determined and output according to the screen state in the second mode, so that the output of the target object is adapted to the state of the folding screen. Correspondingly, the processing process of the target object whose output matches the second recognition result in the embodiments of the present application can include:

[0093] determining the screen state of the folding screen device in the second mode; determining the target output mode of the target object whose output matches the second recognition result based on the screen state; and outputting the target object based on the target output mode.

[0094] The second mode includes a closed unlocked state or an unfolded unlocked state of the folding screen. For example, when the second mode is the closed unlocked state of the folding screen, the screen size is smaller than that in the unfolded unlocked state, and the target output mode of the target object can be determined as an independent interface output or an audio output. If in the unfolded unlocked state, the target output mode can be a mode of displaying the target object as a floating icon in the display interface. Specifically, the output mode (such as display size, position, layout, type of target object, and presentation mode) of the target object whose output matches the second recognition result can be selected according to the determined screen state, for example, a smaller icon is output in the closed unlocked state, and a complete page is output in the unfolded unlocked state. The target object is output according to the determined target output mode.

[0095] For example, the folding screen phone is in the unfolded unlocked state (the screen is fully unfolded) in the second mode, the second recognition result is a happy emotion, and the corresponding target object is a celebration animation. At this time, the output mode is determined as full-screen display of the animation. If the second mode is the closed unlocked state (the screen is only partially displayed), the output mode is to display a small celebration icon on the partial screen. In this embodiment, the screen characteristics of the folding screen device are fully considered, so that the target object can be presented in an appropriate way in different screen states, and the user experience on the folding screen device is improved.

[0096] In some embodiments of the present application, the electronic device can also include a device with voice intelligent wake-up. For this type of electronic device, the voice wake-up information can be a wake-up instruction of the image acquisition device. When the user produces the voice wake-up information, the image acquisition device of the electronic device can be triggered to obtain a captured image. The second recognition result representing the user's emotion is obtained by combining the voice information of the user and the captured image. In this way, the emotional characteristics (such as tone, speed, and tone) contained in the voice information and the biological characteristics (such as facial expression) in the image are complementary, avoiding the deviation of a single recognition dimension, especially suitable for emotion perception in a voice wake-up scene, and improving the reliability and applicability of emotion recognition.

[0097] For example, the voice intelligent wake-up device is a smart speaker in a first mode of low-power standby. A user says "Little A, turn on music" (a voice wake-up instruction), the microphone of the speaker collects the voice information of the user, and finds that the tone is low and the speed is slow; At the same time, the front camera is started, and the user's facial expression is photographed as the corners of the mouth droop and the eyes are dim. The device fuses and analyzes the voice information and the facial image, and determines that the user's emotion is low (the second recognition result). Since it is recognized that the user is the master of the device (the first recognition result matches the target user), the speaker switches from standby mode to working mode (the second mode), and outputs the voice response "You sound a little unhappy, play a soothing music for you" (the target object). The result combines voice and image information, which is more accurate than single recognition.

[0098] In order to enable the output target object to affect the current emotional state of the user and make it in a more optimal emotional state, in addition to determining the target object according to the second recognition result representing the emotion, the target object can also be determined in combination with the use scene, environment and user characteristics of the electronic device, so as to improve the accuracy of the application of the target object. Correspondingly, in the embodiment of the present application, the information processing method further comprises: obtaining scene characteristic information of the electronic device; determining the target object matched with the second recognition result based on the scene characteristic information and the second recognition result.

[0099] The scene characteristic information of the electronic device includes application environment information of the electronic device and user information of the electronic device, wherein the application environment information can include time, weather, light, place and the like. The user information can be some characteristic information of the target user of the current electronic device, such as preference information, age range information or commonly used software information. For example, when it is detected that the light of the current use scene of the electronic device is relatively dark, the display characteristics of the corresponding target object should be relatively soft, avoiding the discomfort of high-brightness display to the user's eyes and the like. For example, when it is detected that the current environment is relatively quiet, the target object can be output in the form of text or image, but not in the form of audio, so as to avoid affecting the user's use experience. For example, when it is detected that the current time is relatively late and the user's emotional state is recognized to be relatively irritable through image collection, it can be determined that the user's irritable mood is caused by insomnia, and the target object can be some sleep-aiding music. For example, the user likes to collect pictures or frequently switches wallpaper images, it can be determined that the user pays more attention to image content, and when determining the corresponding target object, the image content matched with the emotion corresponding to the second recognition result can be determined, so as to more easily attract the user's attention and improve the user experience effect.

[0100] For example, the second recognition result is that the user is in a tired mood, and the scene feature information of the electronic device shows that the current time is 10:00 PM (time information) and the geographic location is at home (geographic location information). Combining these information, the determined target object is a text prompt of "It's time to rest, recommend a sleep-aid music" and a sleep-aid music recommendation, which matches the tired mood and is consistent with the scene of being at home in the evening.

[0101] In this embodiment, the scene feature information of the electronic device and the second recognition result are combined when determining the target object, so that the target object is more consistent with the actual scene. The scene factor of using the device by the user is fully considered, and the situation that the target object does not match the scene is avoided, thereby improving the applicability and user acceptance of the target object.

[0102] In the embodiments of the present application, the emotional change trend data of the user can also be generated according to the emotional state of the user recognized each time, so as to facilitate the user to understand his own emotional state. Correspondingly, the information processing method further includes: recording each obtained second recognition result and a corresponding time stamp; generating emotional change trend data based on each second recognition result and the time stamp; and in response to a query request, displaying the emotional change trend data.

[0103] Each time the second recognition result is obtained, the time stamp (such as the specific year, month, day, hour, and minute) corresponding to the result is recorded. According to each recorded second recognition result and the corresponding time stamp, the emotional change trend data (such as an emotional curve, an emotional distribution table, etc.) is generated through statistical analysis (such as aggregation by hour, day, and week). The query request can be a message that is periodically pushed to the user to display emotional data, and the user feedback can be displayed information; or it can be a user-initiated query request, such as the user clicking the "My emotional record" function to display the generated emotional change trend data.

[0104] Correspondingly, in order to accurately reflect the trend of the user's emotional changes, the emotion state corresponding to each second recognition result can be quantified by quantifying the emotion classification, so as to obtain a quantified score representing the emotion state, and the score is statistically analyzed. For example, different scores corresponding to different emotion states can be set, such as the score corresponding to the “happy” state can be 1.0, and the score corresponding to the “sad” state can be 0.1. If the statistical unit is a day, the average value representing the user's emotion state in each hour can be calculated, and the emotion state corresponding to the average value is determined as the emotion state of the hour. For example, it can be determined that the emotion state closest to the average value is the current emotion state, for example, the average value of this hour is calculated to be 0.9, which is close to 1.0, so the emotion state of this hour is determined to be the happy state. In order to increase the readability, a small face icon representing happiness can be used to represent the emotion state. Based on this, the emotion trend data of each day or each week, each month (such as represented by an emotion state table) can be obtained, see Figure 6 and Figure 7 In Figure 6 , the mood curve of the user on September 4th is represented, and Figure 7 is a mood table of the user in August. In this way, the user's emotion state is expressed in a visual manner, which is more intuitive, and thus stimulates the user to adjust his attitude and avoid the influence of negative attitude.

[0105] For example, the phone records the emotion recognition result and time of the user when unlocking every day, and after a week, an emotion curve within a week is generated according to the data, the horizontal axis of the curve is the date and time, and the vertical axis is the emotion state (such as happy, calm, low, etc.). When the user clicks “view this week's emotion” (query request), the phone screen displays the emotion curve, and the user can intuitively understand the emotional changes of himself within a week. In this embodiment, the accumulation and analysis of emotion information are realized, the user is provided with a reference for emotion management, and the application value of the function is expanded from real-time emotion response to emotion tracking, which is convenient for providing better output information matched with the emotion of the user, and helps to adjust the user's emotion.

[0106] It should be noted that, in the embodiments of the present application, the processing mode of identifying and outputting the target object related to the matching of the user's emotional state based on the emotion can be started or stopped by the user based on the actual application requirements and experience. Moreover, the output of the target object related to the matching of the emotional state can be to set the corresponding output frequency or output time range, such as outputting the target object related to the emotional state only 3 times a day or outputting only in the time period of 8:00-9:00 in the morning, so that the output of the target object will not be too frequent and the user's aversion can be avoided. Correspondingly, the target object can also be output when the user has a significant emotional change. The emotional state can also be accumulated for multiple times, and the target object can be displayed at an appropriate time, or the target object can be displayed when the user actively inquires.

[0107] In the embodiments of the present application, the target object corresponding to the current recognized user emotional state can also be determined in a forward adjustment and reverse adjustment manner. For example, when it is recognized that the user is in a positive emotion (such as happy, joyful), the content matched with the emotion is outputted for the purpose of prolonging or amplifying the positive state, so as to realize the forward adjustment to determine the corresponding target object. For example, if the second recognition result is “happy”, the target object can be set as happy music, a dynamic icon with a celebration element, or a praise text of “Today's mood is really good, continue to maintain this vitality”. This adjustment strengthens the positive experience of the user through positive feedback, so that the good mood can be prolonged.

[0108] When it is recognized that the user is in a negative emotion (such as sad, anxious), the content with a dredging or relieving effect is outputted to neutralize or improve the negative state, so as to realize the reverse adjustment to determine the target object. For example, if the second recognition result is “anxious”, the target object can be set as a soothing natural sound effect, a graphic card containing deep breathing guidance, or an encouraging text of “Don't be anxious, come slowly, everything will be better”. This adjustment helps the user to relieve stress through targeted emotional dredging, and realizes the transition from negative emotion to stable or positive emotion, thereby providing effective support when the user is in a low mood.

[0109] In the embodiments of the present application, an electronic device is also provided, which is shown in Figure 8 The electronic device includes an image acquisition device 801, a biological feature recognition device 802, and a control device 803.

[0110] The image acquisition device 801 is configured to acquire an acquisition image in response to a wake-up instruction if the electronic device is in a first mode.

[0111] The biological feature recognition device 802 is configured to recognize the acquisition image, obtain a first recognition result of an identity represented by biological feature information in the acquisition image, and obtain a second recognition result of an emotion represented by the biological feature information in the acquisition image.

[0112] The control device 803 is configured to control the electronic device to switch from the first mode to the second mode if the first identification result satisfies the identity of the target user, wherein the electronic device outputs the target object matched with the second identification result in the process of switching from the first mode to the second mode; the power consumption of the electronic device in the first mode is lower than the power consumption of the electronic device in the second mode.

[0113] The biometric feature recognition module in the electronic device can be a module with face recognition and emotion recognition functions, or a module with fingerprint recognition and emotion recognition functions. The corresponding image acquisition device can be a face image acquisition device of the electronic device, or a fingerprint image acquisition device, or a device including a face image acquisition device and a fingerprint image acquisition device.

[0114] In some embodiments of the present application, the biometric feature recognition device can be configured with an image recognition model for processing the collected image, obtaining a first identification result of the identity represented by the biometric feature information in the collected image, and obtaining a second identification result of the emotion represented by the biometric feature information in the collected image; wherein the image recognition model includes an identity recognition branch and an emotion classification branch, the identity recognition branch is used to identify the identity represented by the biometric feature information in the collected image; the emotion classification branch is used to identify the emotion represented by the biometric feature information in the collected image.

[0115] In some embodiments, the control device can further include an output module, which is configured to:

[0116] In response to configuring the target object as an interactive component, displaying the interactive component in the display interface of the electronic device after switching to the second mode;

[0117] In response to configuring the target object as target output content, after outputting the target output content, display the display interface of the electronic device after switching to the second mode; wherein the target output content represents output content independent of the display interface corresponding to the second mode.

[0118] In some embodiments, after outputting the target output content, displaying the display interface of the electronic device after switching to the second mode, includes:

[0119] Based on the priority of the current notification information of the electronic device, determine the output duration of the target output content;

[0120] In response to the target output content outputting the target output duration, jump to the display interface corresponding to the second mode of the electronic device.

[0121] In some embodiments, the control device further comprises:

[0122] a first determination module configured to determine an emotional state of the target user based on the second recognition result;

[0123] a control module configured to control the electronic device to switch from a first mode to a second mode matching the emotional state, so that an output of the electronic device in the second mode matches an output configuration of the emotional state.

[0124] In some embodiments, the electronic device comprises a foldable screen device, the first mode is a closed lock screen state of the foldable screen device, and the control device further comprises a force sensing module configured to: in response to detecting that a user unfolds the foldable screen device, obtain a force parameter of the user unfolding the foldable screen device; and the biometric recognition device is further configured to identify the collected image based on the force parameter to obtain a second recognition result of an emotion represented by biometric information in the collected image.

[0125] Correspondingly, the control device is further configured to:

[0126] determine a screen state of the foldable screen device in the second mode; the second mode comprises a closed unlock state or an unfolded unlock state of the foldable screen device; determine a target object output mode matching the second recognition result based on the screen state; and output the target object based on the target output mode.

[0127] In some embodiments of the present application, the control device of the electronic device further comprises a target object determination module configured to: obtain scene feature information of the electronic device; and determine a target object matching the second recognition result based on the scene feature information and the second recognition result.

[0128] In some embodiments of the present application, the control device of the electronic device further comprises an analysis module configured to: record each obtained second recognition result and a corresponding time stamp; generate emotional change trend data based on each second recognition result and the time stamp; and in response to a query request, display the emotional change trend data.

[0129] It should be noted that the specific implementation of the function modules corresponding to each device and sub-unit device in the present embodiment can refer to the corresponding content in the foregoing, which will not be described in detail here.

[0130] In another embodiment of the present application, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the information processing method described above is implemented.

[0131] It should be noted that the specific implementation of the processor in this embodiment can refer to the corresponding content in the foregoing, which will not be described in detail here.

[0132] The various embodiments are described in a progressive manner in the specification, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0133] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0134] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0135] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An information processing method comprising: obtaining, by an image capturing device, a captured image in response to a wake-up instruction of the image capturing device if an electronic device is in a first mode; identifying the captured image to obtain a first identification result of an identity represented by biometric information in the captured image and a second identification result of an emotion represented by the biometric information in the captured image; controlling the electronic device to switch from the first mode to a second mode if the first identification result represents an identity satisfying a target user, wherein a target object matching the second identification result is output during the process of the electronic device switching from the first mode to the second mode; a power consumption of the electronic device in the first mode is lower than a power consumption of the electronic device in the second mode.

2. The method of claim 1, wherein the identifying the captured image to obtain the first identification result of the identity represented by the biometric information in the captured image and the second identification result of the emotion represented by the biometric information in the captured image comprises: processing the captured image based on an image recognition model to obtain the first identification result of the identity represented by the biometric information in the captured image and the second identification result of the emotion represented by the biometric information in the captured image; wherein the image recognition model comprises an identity recognition branch and an emotion classification branch, the identity recognition branch is configured to identify the identity represented by the biometric information in the captured image, and the emotion classification branch is configured to identify the emotion represented by the biometric information in the captured image.

3. The method of claim 1, wherein the outputting the target object matching the second identification result comprises: in response to configuring the target object as an interactive component, displaying the interactive component in a display interface of the electronic device after the electronic device switches to the second mode; in response to configuring the target object as target output content, displaying the display interface of the electronic device after the electronic device switches to the second mode after outputting the target output content, wherein the target output content represents output content independent of a display interface corresponding to the second mode.

4. The method of claim 3, wherein the displaying the display interface of the electronic device after the electronic device switches to the second mode after outputting the target output content comprises: determining an output duration of the target output content based on a priority of current notification information of the electronic device; in response to the target output content being output for the target output duration, jumping to the display interface corresponding to the second mode of the electronic device.

5. The method of claim 1, wherein the controlling the electronic device to switch from the first mode to the second mode comprises: determining an emotional state of the target user based on the second identification result; controlling the electronic device to switch from the first mode to a second mode matching the emotional state, so that an output of the electronic device in the second mode matches an output configuration of the emotional state.

6. The method of claim 1, wherein the electronic device comprises a foldable screen device, the first mode comprises a closed lock screen state of the foldable screen device, and the obtaining the second recognition result of the emotion represented by the biometric information in the collected image comprises: in response to detecting that a user unfolds the foldable screen device, obtaining a force parameter of the user unfolding the foldable screen device; and based on the force parameter, identifying the collected image to obtain the second recognition result of the emotion represented by the biometric information in the collected image.

7. The method of claim 6, wherein the outputting the target object matching the second recognition result comprises: determining a screen state corresponding to the second mode of the foldable screen device; the second mode comprises a closed unlock state or an unfolded unlock state of the foldable screen device; based on the screen state, determining an output mode of the target object matching the second recognition result; and outputting the target object based on the output mode.

8. The method of claim 1, further comprising: obtaining scene feature information of the electronic device; and based on the scene feature information and the second recognition result, determining the target object matching the second recognition result.

9. The method of claim 1, further comprising: recording each obtained second recognition result and a corresponding time stamp; based on each second recognition result and the time stamp, generating emotion change trend data; and in response to a query request, displaying the emotion change trend data.

10. An electronic device comprising: an image collection device, a biometric recognition device, and a control device; the image collection device is configured to, if the electronic device is in a first mode, collect a collected image in response to a wake-up instruction; the biometric recognition device is configured to identify the collected image to obtain a first recognition result of an identity represented by biometric information in the collected image and a second recognition result of an emotion represented by the biometric information in the collected image; and the control device is configured to, if the first recognition result represents an identity satisfying a target user, control the electronic device to switch from the first mode to a second mode, wherein during the process of the electronic device switching from the first mode to the second mode, a target object matching the second recognition result is outputted; and a power consumption of the electronic device in the first mode is lower than a power consumption of the electronic device in the second mode. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Images