Awakening method and electronic device for voice interaction function

Abstract

The present application provides a wake-up method and electronic device for a voice interaction function. When the microphone of the electronic device is not occupied, the electronic device detects the user status based on the user image captured by the camera. When it is detected that the user is looking at the electronic device at a close distance, the voice signals collected by multiple microphones are further used to detect whether the distribution difference of the voice signals on each microphone meets the preset conditions. If so, the voice interaction function of the electronic device is awakened.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to a wake-up method and electronic device for a voice interaction function. Background Art

[0002] At present, the voice interaction function in electronic devices is widely used, and how to wake up the voice interaction function of electronic devices is an urgent problem to be solved. Summary of the invention

[0003] The present application provides a wake-up method and electronic device for a voice interaction function. When the microphone of the electronic device is not occupied, the electronic device detects the user status based on the user image captured by the camera. When it is detected that the user is looking at the electronic device at a close distance, the voice signals collected by multiple microphones are further used to detect whether the distribution difference of the voice signals on each microphone meets the preset conditions. If so, the voice interaction function of the electronic device is awakened.

[0004] In a first aspect, the present application provides a method for waking up a voice interaction function, the method comprising: the electronic device captures a first image containing a user's face through a camera; determines the distance between the user's face and the electronic device, and the user's gaze direction based on the first image; the distance is within a preset range, and the user's gaze direction indicates that the user is looking at the electronic device; the electronic device captures a first voice signal through a first microphone, and captures a second voice signal through a second microphone, the first voice signal and the second voice signal having different intensities; the electronic device wakes up the voice interaction function.

[0005] After implementing the method provided in the first aspect of the present application, the electronic device can accurately identify the user's intention to wake up the voice interaction function based on the user status and user voice, and turn on the voice interaction function of the electronic device for the user.

[0006] In combination with the method provided in the first aspect, before the electronic device captures a first image containing a user's face through a camera, the method further includes: the electronic device detects that the electronic device is lifted or picked up.

[0007] In this way, considering that the user usually picks up or lifts the mobile phone when he wants to wake up the voice interaction function, it is judged in advance whether the electronic device is lifted or picked up, and subsequent image detection and voice detection are performed after the electronic device is lifted or picked up. This can avoid the electronic device from performing additional detection work and save power consumption of the electronic device.

[0008] In combination with the method provided in the first aspect, before the electronic device captures a first image containing a user's face through a camera, the method further includes: the electronic device detects that both the first microphone and the second microphone are not occupied.

[0009] In this way, scenarios that are not suitable for turning on the voice interaction function can be preliminarily excluded. Specifically, after the voice interaction function is turned on, the electronic device needs to occupy the microphone to collect the user's voice signal to respond to the user's voice input. Therefore, in order to avoid conflicts between the voice interaction function and other services that are using the microphone, before waking up the voice interaction function, it is pre-detected whether the microphone of the electronic device is occupied to exclude scenarios that are not suitable for turning on the voice interaction function, such as scenarios where the user uses the electronic device to answer calls, record audio, record videos, send voice messages or hold online meetings.

[0010] In combination with the method provided in the first aspect, after the electronic device captures a first image including a user's face through a camera, the method further includes: determining, based on the first image, whether the user's lip shape has changed.

[0011] In this way, after determining that the user's lip shape has changed, multiple microphones can be turned on to collect voice signals and detect whether there are differences in the voice signals on each microphone. This can avoid turning on the microphone when the user is not speaking, saving power consumption of the electronic device.

[0012] In combination with the method provided in the first aspect, the electronic device awakens the voice interaction function specifically including: the electronic device collects voice commands through a third microphone and executes tasks corresponding to the voice commands.

[0013] In this way, after waking up the voice interaction function, the corresponding tasks can be received consistently with the user's voice commands, thereby improving the user experience.

[0014] In combination with the method provided in the first aspect, after the electronic device wakes up the voice interaction function, the method also includes: the electronic device extracts voice instructions from the first voice signal and / or the second voice signal, and executes the task corresponding to the voice instruction.

[0015] In this way, missing the user's voice commands can be avoided. After waking up the voice interaction function, the electronic device can perform semantic analysis on the voice signal collected before waking up the voice interaction function to execute the corresponding task.

[0016] In combination with the method provided in the first aspect, after the electronic device wakes up the voice interaction function, the method also includes: the electronic device outputs a prompt message, and the prompt message is used to prompt that the voice interaction function is currently turned on; the prompt message includes: voice and / or interface elements displayed on the display screen.

[0017] In this way, the user can be reminded in time that the voice interaction function has been awakened, providing a better user experience.

[0018] In combination with the method provided in the first aspect, the first microphone is a microphone on the top of the electronic device, and the second microphone is a microphone on the bottom of the electronic device.

[0019] In combination with the method provided in the first aspect, the first microphone and the second microphone have different sensitivities; the intensities of the first voice signal and the second voice signal are different, specifically including: the intensities of the first voice signal or the second voice signal after compensation based on the sensitivity difference are different; or, the intensities of the first voice signal or the second voice signal are different, specifically including: the number of times the intensity difference of the corresponding sub-segment voice signals of the first voice signal and the second voice signal is greater than the first value is greater than the second value.

[0020] In this way, the electronic device can be assisted to further determine whether the user's speaking intention at this time is to wake up the voice interaction function.

[0021] In combination with the method provided in the first aspect, after the electronic device captures a first image containing a user's face through a camera, the method further includes: performing identity authentication based on the user's face in the first image.

[0022] In this way, the voice interaction function can be awakened through face recognition while ensuring user privacy.

[0023] In combination with the method provided in the first aspect, after the electronic device collects a first voice signal through a first microphone and collects a second voice signal through a second microphone, the method further includes: extracting voiceprint information from the first voice signal or the second voice signal, and performing identity authentication based on the voiceprint information.

[0024] In this way, voice interaction functions can be provided while ensuring user privacy through voiceprint authentication.

[0025] In combination with the method provided in the first aspect, the electronic device determines the distance between the user's face and the electronic device based on the first image, which is specifically achieved in the following manner: the electronic device analyzes the imaging pupil distance from the first image; the electronic device calculates the distance between the user's face and the electronic device based on the imaging pupil distance, the focal length of the camera and the actual pupil distance using the principle of similar triangles.

[0026] In this way, the distance between the user's face and the electronic device can be accurately known.

[0027] In combination with the method provided in the first aspect, the electronic device determines the gaze direction of the user based on the first image, which is specifically achieved in the following manner: the electronic device analyzes the position of the pupil in the eyeball from the first image to determine the gaze direction of the user.

[0028] In this way, it is possible to accurately know whether the user's gaze direction is the direction in which the user is looking at the electronic device.

[0029] In combination with the method provided in the first aspect, the aforementioned first voice signal is obtained by bandpass filtering the voice signal collected by the first microphone; the aforementioned second voice signal is obtained by bandpass filtering the voice signal collected by the second microphone.

[0030] This can filter out non-human voices such as environmental noise, making it easier for electronic devices to perform subsequent difference detection.

[0031] In a second aspect, the present application provides an electronic device, comprising: one or more processors, one or more memories, at least two microphones, a camera and a display screen; the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes a method as described in any one of the first aspects.

[0032] In a third aspect, the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a scenario provided for an embodiment of the present application;

[0034] Figure 2 A method flow chart provided for an embodiment of the present application;

[0035] Figures 3A-3D A schematic diagram of a set of operation interfaces for enabling the "close proximity wake-up" function provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the distance calculation principle between a face and an electronic device provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of an implementation process of determining a user's gaze direction provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of the distribution of the same speech signal detected by microphones at two distances provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the hardware architecture of an electronic device 100 provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the software architecture of the electronic device 100 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0042] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0043] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0044] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0045] The technology of using voice to achieve human-computer interaction has been widely used. However, in actual applications, if a user wants to interact with an electronic device through voice, the voice interaction function of the electronic device needs to be awakened first. In some achievable ways, the user can wake up the voice interaction function of the electronic device by inputting a wake-up word / breath sound, or by operating a physical button / virtual control.

[0046] The above-mentioned wake-up methods have many defects. For example, the method of inputting the wake-up word / breath sound by voice is not natural enough and has poor sociality. For another example, the method of operating the physical buttons / virtual controls is not only cumbersome and cannot free the user's hands, but also poses a privacy risk.

[0047] refer to Figure 1 , Figure 1 A schematic diagram of a scenario provided by the present application is exemplified.

[0048] like Figure 1 As shown, when a user holds an electronic device and places it in front of the user where he can see it, if the user speaks to the electronic device at a close distance, the electronic device can wake up the voice interaction function and complete various tasks such as response command input, information query or voice chat according to the voice input by the user.

[0049] Understandably, Figure 1 This is only an exemplary scenario and should not constitute a limitation on the implementation of the present application. For example, the present application is also applicable to the following scenarios: when the user does not pick up the electronic device but places the electronic device in front of the user where the user can see it, if the user speaks to the electronic device at a close distance at this time, the electronic device can also wake up the voice interaction function and complete various tasks such as response command input, information query or voice chat according to the voice input by the user.

[0050] Combination Figure 1 In the scenario shown, when the user intends to wake up the voice interaction function, he or she usually looks closely at the electronic device and speaks with his or her mouth close to one of the microphones of the electronic device, such as the bottom microphone.

[0051] In order to solve the above problems, the present application provides a method for waking up a voice interaction function and an electronic device. Specifically, the electronic device can detect the user status through the user image collected by the camera, and detect the distribution of the voice signals on each microphone through the voice signals collected by multiple microphones. After detecting that the user is in a state of looking at the electronic device at a close distance, and detecting that there is a difference in the distribution of the voice signals on each microphone, the voice interaction function is awakened.

[0052] In one achievable manner, the electronic device detects the user status through the user image captured by the camera, and detects the distribution of voice signals on each microphone through the voice signals collected by multiple microphones. The execution order is not limited, that is, the two can be executed simultaneously or one after another. For example, after first detecting that the user is in a state of closely watching the electronic device through the user image captured by the camera, the voice signals collected by multiple microphones are used to detect the distribution of voice signals on each microphone to determine whether there are differences between the voice signals. For another example, after first detecting that there are differences between the voice signals on each microphone through the voice signals collected by multiple microphones, the user status is detected through the user image captured by the camera to determine whether the user is in a state of closely watching the electronic device.

[0053] In another achievable method, before waking up the voice interaction function, the electronic device will also detect whether multiple microphones of the electronic device are occupied. If it is determined that they are not occupied, the electronic device will detect the user status through the user image captured by the camera, or detect the difference between the voice signals on each microphone through the voice signals collected by multiple microphones.

[0054] In another achievable manner, before the electronic device detects the user status, the electronic device may also first detect its own motion status. For example, the next step of detection, namely detecting the user status, is started only after detecting that the electronic device is in a lifted / picked up state.

[0055] In another achievable manner, when the electronic device detects the user status based on the user image, in addition to detecting the distance between the user and the electronic device and the user's gaze direction, the user's lip shape and the like may also be detected.

[0056] In the embodiment of the present application, the voice interaction function is equivalent to the function provided by the voice assistant in the electronic device. When the voice assistant is awakened, the user can interact with the electronic device by voice inputting voice to the voice assistant to control the electronic device to perform corresponding tasks, such as executing command input, viewing information or voice chat, etc.

[0057] After implementing the wake-up method for the voice interaction function provided in the present application, the electronic device can accurately identify the user's intention to wake up the voice interaction function, turn on the voice interaction function for the user, and improve the user experience.

[0058] Next, combine Figure 2 The method flow shown is used to introduce the wake-up method of the voice interaction function provided in this application.

[0059] like Figure 2 As shown, the method flow includes the following steps:

[0060] Optional S11 detects whether the microphone of the electronic device is occupied.

[0061] Specifically, the electronic device detects whether the microphone is occupied, specifically including detecting whether the first microphone and the second microphone are occupied. When it is detected that the first microphone and the second microphone are not occupied by other services, the subsequent S12 is executed. When it is detected that the microphone is occupied by other services, the subsequent steps are stopped and the detection of whether the microphone is occupied is continued until it is detected that the microphone is not occupied. The subsequent S12 will be continued. In the embodiment of the present application, the first microphone and the second microphone are microphones at two different positions in the electronic device, for example, the first microphone is a top microphone, and the second microphone is a bottom microphone.

[0062] It is understandable that S11 is an optional step provided in this application. If there are more than S11, the electronic device can directly execute S12 or S13 below. Preferably, executing S11 can preliminarily exclude scenarios that are not suitable for turning on the voice interaction function. Specifically, after the voice interaction function is turned on, the electronic device needs to occupy the microphone to collect the user's voice signal to respond to the user's voice input. Therefore, in order to avoid conflicts between the voice interaction function and other services that are using the microphone, before waking up the voice interaction function, it is necessary to detect whether the microphone of the electronic device is occupied in advance to exclude scenarios that are not suitable for turning on the voice interaction function, such as scenarios where users use electronic devices to answer calls, record audio, record videos, send voice messages or hold online meetings, etc.

[0063] In one practicable manner, the electronic device may detect whether the microphone is occupied by using an audio manager in the application framework layer of the electronic device.

[0064] In one practicable manner, the condition that triggers the electronic device to detect whether the microphone is occupied may be any of the following conditions: the electronic device is turned on, or the electronic device turns on a corresponding function such as turning on a "near distance wake-up" function.

[0065] refer to Figure 3A-3D , Figure 3A-3D The following is a schematic diagram of the operation interface for enabling the “near range wake-up” function.

[0066] like Figure 3A As shown, the user interface 310 displayed by the electronic device is a setting interface, and multiple setting options are displayed in the setting interface, including a smart assistant option 311. When the electronic device detects an operation on the smart assistant option 311, in response to the operation, the electronic device displays Figure 3B User interface 320 is shown.

[0067] like Figure 3BAs shown, the user interface 320 is the smart assistant details page, which displays a series of function options provided by the smart assistant, such as the voice assistant option 321, etc. When the electronic device detects an operation on the voice assistant option 321, in response to the operation, the electronic device displays Figure 3C User interface 330 is shown.

[0068] like Figure 3C As shown, the user interface 330 displays multiple options for waking up the voice assistant (also known as waking up the voice interaction function), and the controls corresponding to each option. For example, a switch control 331 corresponding to voice wake-up and a switch control 332 corresponding to proximity wake-up. At this time, the switch controls 331 and 332 of the electronic device remain in the off state. When the electronic device detects an operation on the switch control 332 corresponding to proximity wake-up, in response to the operation, the electronic device displays Figure 3D The user interface 330 is shown to prompt the user that the voice wake-up function has been turned on.

[0069] like Figure 3D As shown, the user interface 330 is Figure 3C The user interface 330 shown is similar, except that the state of the switch control 332 corresponding to the proximity wake-up is turned on.

[0070] Among them, voice wake-up means that after the user turns on the voice wake-up function and enters the wake-up word, the electronic device can detect whether the user says the wake-up word. If so, the voice interaction function of the electronic device is awakened to provide convenient services for the user.

[0071] Among them, proximity wake-up means that after the user turns on the proximity wake-up function, the electronic device can detect whether the user meets the corresponding conditions for proximity wake-up (for example, whether the microphone is occupied by other services, whether the user is looking at the electronic device closely when speaking, or whether there are differences in the distribution of voice signals on multiple microphones of the electronic device). If so, the voice interaction function of the electronic device can be awakened to provide convenient services to the user.

[0072] In addition, the above-mentioned voice assistant and close-range wake-up are merely optional names for corresponding functions, and the embodiments of this application do not limit this. The functions corresponding to the voice assistant and close-range wake-up are specifically recorded below, and a brief introduction is given here. In addition, the voice assistant can also be called intelligent voice, voice interaction, and close-range wake-up can also be called intelligent wake-up, etc.

[0073] It is understandable that the above merely introduces an exemplary operation interface for enabling the “near distance wake-up” function. In other embodiments, the electronic device may also enable the “near distance wake-up” function by default, and the embodiments of the present application do not limit this.

[0074] S12, collecting images through a camera.

[0075] Specifically, after detecting that the microphone is not occupied by other services, the electronic device can turn on the camera to collect images. Generally, the electronic device will continuously collect multiple images within a period of time, which are used by the electronic device to determine whether the user status meets the preset conditions in the subsequent S13.

[0076] In the embodiment of the present application, the camera is a front camera of the electronic device and is a low-power camera.

[0077] Optionally, in addition to being turned on only after detecting that the microphone is not occupied by other services, the front camera of the electronic device may also be turned on continuously, or be turned on after detecting that the electronic device is lifted or picked up, and the embodiments of the present application are not limited to this.

[0078] S13, detecting whether the user status meets a preset condition based on the image.

[0079] Specifically, the electronic device can detect whether there is a face in the current image through image analysis based on the image, and extract the user's eye information from the face information to determine the user's gaze direction and calculate the distance between the face and the electronic device, etc., to determine whether the user status meets the preset conditions. When it is detected that the user status meets the preset conditions, the subsequent S14 is executed. When it is detected that the user status does not meet the preset conditions, the subsequent steps are stopped and the user status is continued to be detected until it is detected that the user status meets the preset conditions before continuing to execute the subsequent S14. Among them, the image containing the face can also be called the first image.

[0080] Executing S13 can identify whether the user intends to wake up the voice interaction function according to the user status, and further eliminate the scenario where the user does not want to turn on the voice interaction function.

[0081] Optionally, the electronic device may also extract a face based on the captured image to perform face recognition, thereby determining whether the current user is a registered user of the electronic device.

[0082] In the embodiment of the present application, the preset conditions that the user status needs to meet include: the distance between the face and the electronic device is within a preset range (e.g., 15-25 cm), and the user is looking at the electronic device. Optionally, the preset conditions may also include: the user is speaking, and the face authentication is passed.

[0083] Next, the implementation method of detecting the distance between the face and the electronic device, the user's gaze direction, and whether the user is speaking based on the user image is described in detail.

[0084] (1) The distance between the face and the electronic device.

[0085] refer to Figure 4 The schematic diagram of the distance calculation principle between the face and the electronic device shown in the figure, the known conditions include: the actual pupil distance H, the focal length f and the imaging pupil distance h, and the unknown condition is the object distance D. Among them, the actual pupil distance H is obtained by counting the pupil distance information of a large number of users, the focal length f is the parameter of the front camera of the electronic device, the imaging pupil distance h refers to the pupil distance extracted from the captured user image, and the object distance D is approximately the distance between the face and the electronic device. Therefore, according to the similar triangle calculation principle, it can be known that D = (H*f) / h, and then D is calculated.

[0086] Optionally, H used in the present application is an average pupil distance obtained by counting pupil distance information of a large number of users, or may be an average pupil distance of a large number of users corresponding to the age of the user of the current electronic device, and the embodiments of the present application are not limited to this.

[0087] (2) The user’s gaze direction.

[0088] refer to Figure 5 As shown in the schematic diagram of the implementation process of determining the user's gaze direction, the electronic device can detect whether there is a face in the current user image based on the user image through image analysis. If there is a face, the image of both eyes in the face is extracted, and then the user's gaze direction is determined by analyzing the position of the pupil in the eyeball.

[0089] When the pupil is at the center of the eyeball and within a certain range from the center of the eyeball, it is considered that the user's gaze direction is the direction of the electronic device. Correspondingly, when the pupil is outside a certain range from the center of the eyeball, it is considered that the user is not looking at the electronic device.

[0090] In particular, when only the side face of the user is captured in the image, or the face outline is incomplete, it is considered as a non-face image. This is because when the front camera of the electronic device can only capture the side face of the user, it means that the user is very likely not looking at the electronic device at this time, and therefore it is very unlikely to wake up the voice interaction function. Therefore, in order to save power consumption, when only the side face or other incomplete face outlines are captured, the eyeball image will not be further extracted to determine the user's gaze direction.

[0091] (3) User’s lip shape.

[0092] Specifically, the electronic device can also extract the user's lip shape change information from multiple consecutive user images to detect whether the user is speaking, and even the content of the speech. When it is detected that the user's lip shape has changed, it is considered that the user is speaking, and it is determined that the user has the intention to wake up the voice interaction function. When it is detected that the user's lip shape has not changed, it is considered that the user is not speaking, and it is determined that the user has no intention to wake up the voice interaction function.

[0093] (4) Face recognition.

[0094] Specifically, the electronic device may also compare the collected user image with a pre-stored face image of the user. When the similarity meets the conditions, the face authentication is considered to have passed, otherwise, it is considered to have failed.

[0095] S14, collecting voice signals through multiple microphones.

[0096] Specifically, after detecting that the user status meets the preset conditions, the electronic device can turn on multiple microphones to collect voice signals. Usually, the electronic device continuously collects a segment of voice signals over a period of time, so that the electronic device can determine in subsequent S15 whether the distribution difference of the voice signals on each microphone meets the preset conditions.

[0097] In an embodiment of the present application, the plurality of microphones include at least two microphones disposed at different positions, for example, a microphone disposed at the top of the electronic device, and a microphone disposed at the bottom of the electronic device.

[0098] S15, detecting whether the distribution difference of the speech signals on the multiple microphones meets a preset condition.

[0099] Specifically, due to the different settings of the physical positions of the aforementioned microphones on the electronic device, if the acoustic parameters of the two microphones are consistent, when the user speaks to the electronic device at a close distance with a parallel / nearly parallel face, there is a difference in the distribution of the same voice signal collected by the two microphones for the same voice spoken by the user. The specific difference is that the energy distributed by the voice signal (also referred to as the second voice signal) on the microphone at the bottom of the electronic device is greater than the energy distributed by the voice signal (also referred to as the first voice signal) on the microphone at the bottom of the electronic device. This is because, when the user speaks to the electronic device at a close distance with a parallel / nearly parallel face, the distance between the microphone at the bottom of the electronic device and the user's mouth is closer than the distance between the microphone at the top of the electronic device and the user's mouth.

[0100] In summary, taking the top microphone and the bottom microphone as examples, whether the voice signal distribution difference on each microphone set in this application meets the preset conditions is as follows: for a fixed-duration speech, the fixed-duration speech is cut into a plurality of continuous fixed-duration sub-segments, and when the number of sub-segments where the signal strength on the bottom microphone is greater than the signal strength on the top microphone is greater than N, it is confirmed that the voice signal distribution difference on each microphone meets the preset conditions. N is a preset value, and this application does not limit this.

[0101] In a specific implementation, the electronic device may periodically compare the signal strength on the bottom microphone and the signal strength on the top microphone in the same sub-segment of time in a continuous speech segment based on the signal strength distribution in the time domain / frequency domain of the two microphones; whenever the signal strength on the bottom microphone is greater than the signal strength on the top microphone and the difference is greater than a preset strength difference (also called a first value), the count is increased by 1 until the continuous speech segment is compared and the count is greater than N (N may also be called a second value), then it is considered whether the difference in speech signal distribution on the two microphones meets a preset condition.

[0102] In another specific implementation, when the distance between the user's face and the electronic device is different, the signal strength detected by the microphone will also be different, so the aforementioned preset strength difference will also be different. When the distance is farther, the corresponding signal strength difference will be smaller. For details, please refer to Figure 6 , Figure 6 The diagram shows the distribution of the same voice signal detected by the microphones at two distances. It can be seen that when the distance between the user's face and the electronic device is far (for example, 30 cm), the signal strength / energy on the top microphone and the bottom microphone are respectively smaller than the signal strength / energy on the top microphone and the bottom microphone when the distance between the user's face and the electronic device is far (for example, 20 cm).

[0103] Optionally, since the frequency range of human voice is usually between 200Hz and 8000Hz, in order to avoid analyzing other sound signals collected by the microphone, before detecting whether the distribution difference of the voice signals on multiple microphones meets preset conditions, the voice signal can be band-pass filtered to filter out signals with frequencies below 200Hz and above 8000Hz.

[0104] It is understandable that when the acoustic parameters of multiple microphones on an electronic device are inconsistent, then in the process of detecting whether the difference in the distribution of voice signals on multiple microphones meets the preset conditions, it is necessary to compensate the signals on the microphones and compare the compensated signals to determine whether the preset conditions are met. Taking a specific example, when the acoustic parameters include sensitivity, if the sensitivity of the top microphone is lower than that of the bottom microphone, and the sensitivity difference is 16db, then in the process of detecting whether the difference in the distribution of voice signals on multiple microphones meets the preset conditions, it is necessary to first compensate the signal on the top microphone by 16db as a whole, or subtract 16db from the signal on the bottom microphone as a whole, and then compare the compensated signal on the top microphone with the signal on the bottom microphone.

[0105] In the embodiment of the present application, the above only gives an execution order of S12-S13 and S14-S15, that is, S12-S13 is executed first and then S14-S15. In another achievable method of the present application, the electronic device can execute S12-S13 and S14-S15 at the same time, or it can also execute S14-S15 first and then S12-S13, that is, the electronic device first detects whether the distribution difference of the voice signals on the multiple microphones meets the preset conditions through the voice signals collected by multiple microphones. If so, it will collect images through the camera and detect whether the user status meets the preset conditions based on the image.

[0106] S16, wake up the voice interaction function.

[0107] Specifically, after detecting that the distribution difference of voice signals on multiple microphones meets preset conditions, the electronic device wakes up the voice interaction function, which is equivalent to starting the voice interaction process.

[0108] Among them, starting the voice interaction function includes: collecting voice commands through the third microphone, and performing semantic analysis on the voice commands to perform corresponding tasks, such as command input, information query, voice chat, etc. The third microphone can be another microphone different from the first microphone and the second microphone, or the third microphone is the first microphone or the second microphone.

[0109] Optionally, in order to avoid missing the user's voice commands, the electronic device will also cache the voice signal collected by the microphone during the execution of S14-S15, and perform semantic analysis on the cached voice after waking up the voice interaction function in step S16 to execute the corresponding task.

[0110] Optionally, after the electronic device wakes up the voice interaction function, the electronic device may output prompt information to prompt the user that the voice interaction function has been woken up. The output form of the prompt information includes but is not limited to: prompting by voice, and prompting by displaying corresponding interface elements on the display screen.

[0111] Optionally, after waking up the voice interaction function, the electronic device can directly perform semantic analysis on the newly collected voice or the cached voice to execute the corresponding task, or first perform voiceprint verification based on the newly collected voice or the cached voice, and then analyze the semantics corresponding to the voice to execute the corresponding task after the voiceprint verification is successful. It is understandable that when the electronic device has previously performed face authentication on the collected user image and the face authentication is passed, the voiceprint authentication is no longer performed at this time.

[0112] Based on the wake-up method of the voice interaction function introduced above, the electronic device involved in this method is introduced below.

[0113] Electronic equipment can be equipped Or portable terminal devices with other operating systems, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, as well as cellular phones, personal digital assistants (PDA), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices and / or smart city devices, etc.

[0114] Figure 7 A schematic diagram of the hardware architecture of the electronic device 100 is shown.

[0115] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a camera 193, a display screen 194, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, an acceleration sensor 180C, etc.

[0116] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0117] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0118] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0119] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0120] In the embodiment of the present application, the processor 110 is used to control the corresponding software and hardware modules. Figure 2 In addition, after waking up the voice interaction function, the processor 110 can control the display screen or the audio module to output corresponding prompt information to prompt the user that the voice interaction function has been turned on. In addition, the processor can also perform voice analysis based on the collected voice signal to perform the corresponding task.

[0121] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0122] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0123] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0124] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0125] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0126] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0127] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0128] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0129] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0130] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0131] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0132] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0133] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microled, micro-oled, quantum dot light emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0134] In the embodiment of the present application, the electronic device can display the Figure 3A-3D The user interface shown in the figure, in addition, when the electronic device wakes up the voice interaction function and interacts with the user by voice, the electronic device will also display the voice interaction interface. The voice interaction interface can be information queried in response to the user's voice command, or the voice interaction interface will also output the corresponding text obtained by recognizing the user's voice, or it will also output the text corresponding to the voice assistant's reply to the user's voice, etc.

[0135] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0136] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0137] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0138] In the embodiment of the present application, the camera 193 includes a camera placed on the screen side of the electronic device, that is, a front camera, and the front camera can continue to operate in a low power consumption mode to collect user images for the electronic device to perform the aforementioned Figure 2 The S12-S13 described provide data support.

[0139] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0140] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0141] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0142] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0143] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.;

[0144] Non-volatile memory may include disk storage devices and flash memory.

[0145] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle; single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the storage unit potential level; universal flash storage (UFS), embedded multi media card (eMMC), etc. according to the storage specification.

[0146] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data, etc.

[0147] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0148] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0149] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0150] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0151] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0152] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0153] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0154] In the embodiment of the present application, the microphone 170C includes at least two microphones, such as a top microphone and a bottom microphone. The two microphones are used to collect voice signals to provide data support for the electronic device 100 to perform the aforementioned S14-S15. This can further prevent the user's random input of voice from erroneously waking up the voice interaction function of the electronic device.

[0155] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0156] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0157] The touch sensor 180B is also called a "touch panel". The touch sensor 180B can be arranged on the display screen 194, and the touch sensor 180B and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180B is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180B can also be arranged on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0158] The acceleration sensor 180C can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally X, Y, and Z axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0159] In the embodiment of the present application, the electronic device can also determine the motion posture of the electronic device, such as whether the user lifts or picks up the electronic device, by detecting the acceleration on the three axes of X, Y, and Z on the acceleration sensor 180C.

[0160] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0161] In an embodiment of the present application, the electronic device can wake up the voice interaction function of the electronic device by operating on the button 190.

[0162] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0163] Figure 8 Schematic diagram of the software architecture of the electronic device 100 according to an embodiment of the present application.

[0164] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0165] The application layer can include a series of application packages.

[0166] like Figure 8 As shown, the application package may include voice assistant, camera, gallery, calendar, call, map, navigation, Bluetooth, video, short message and other applications. Among them, the voice assistant may be a sub-function integrated in the setting application of the electronic device, or a separate application, which is not limited in the embodiment of the present application.

[0167] In the embodiment of the present application, when the voice assistant is awakened, the user can interact with the electronic device by voice inputting voice to the voice assistant to control the electronic device to perform corresponding tasks, such as executing command input, viewing information or voice chat, etc. Among them, waking up the voice assistant is equivalent to waking up the voice interaction function. The specific waking method can refer to the above description of Figure 2 The introduction of the method flow shown will not be repeated here.

[0168] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0169] like Figure 8 As shown, the application framework layer may include a media manager, a window manager, a content provider, a view system, a phone manager, a notification manager, and the like.

[0170] The media manager is used to manage media services, including services that occupy the microphone, such as microphone services occupied during calls, recordings, videos, and meetings. In one achievable method, when the electronic device turns on the "close-range wake-up" function provided in the voice assistant, the voice assistant can also control the front camera through the camera driver to capture user images to detect whether the user status meets the preset conditions, and can also control the top microphone and the bottom microphone through the microphone driver to collect voice to detect whether the distribution difference of the voice signals on the two microphones meets the preset conditions. When it is determined that the user status meets the preset conditions and the distribution difference of the voice signals on the two microphones meets the preset conditions, it confirms to wake up the voice assistant, that is, starts the subsequent voice interaction process. Optionally, the voice assistant will also detect from the media manager whether there are other services occupying the microphone. It will confirm to wake up the voice assistant only when it is determined that there are no other services occupying the microphone, and when it is determined that the user status meets the preset conditions and the distribution difference of the voice signals on the two microphones meets the preset conditions.

[0171] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0172] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0173] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0174] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0175] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0176] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0177] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0178] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0179] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0180] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0181] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0182] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0183] A 2D graphics engine is a drawing engine for 2D drawings.

[0184] The kernel layer is the layer between hardware and software. The kernel layer includes at least microphone driver, camera driver, display driver, and sensor driver.

[0185] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0186] When the touch sensor 180B receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input event (including touch coordinates, timestamp of the touch operation, and other information). The original input event is stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is a control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0187] It should be understood that each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0188] The present application also provides an electronic device, which may include: a memory and a processor, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0189] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0190] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0191] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0192] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0193] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0194] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0195] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0196] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0197] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0198] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0199] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0200] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for waking up a voice interaction function, characterized in that: The method is applied to an electronic device, the electronic device comprising a first microphone and a second microphone arranged at different positions, and the method comprises: The electronic device collects a first image including a user's face through a camera; Determine the distance between the user's face and the electronic device, and the user's gaze direction according to the first image; if the distance is within a preset range, the user's gaze direction indicates that the user is looking at the electronic device; The electronic device collects a first voice signal through the first microphone and collects a second voice signal through the second microphone; The intensity difference between the corresponding sub-segment voice signals in the first voice signal and the second voice signal is detected. If it is detected that the number of sub-segment voice signals whose intensity difference is greater than the first value is greater than the second value, the electronic device wakes up the voice interaction function.

2. The method according to claim 1, characterized in that Before the electronic device captures the first image including the user's face through a camera, the method further includes: The electronic device detects that the electronic device is lifted or picked up.

3. The method according to claim 1 or 2, characterized in that: Before the electronic device captures the first image including the user's face through a camera, the method further includes: The electronic device detects that both the first microphone and the second microphone are not occupied.

4. The method according to claim 1, characterized in that: After the electronic device captures a first image including a user's face through a camera, the method further includes: A change in the user's lip shape is determined according to the first image.

5. The method according to claim 1, characterized in that The electronic device wake-up voice interaction function specifically includes: The electronic device collects voice commands through the third microphone and executes tasks corresponding to the voice commands.

6. The method according to claim 1, characterized in that After the electronic device wakes up the voice interaction function, the method further includes: The electronic device extracts a voice instruction from the first voice signal and / or the second voice signal, and executes a task corresponding to the voice instruction.

7. The method according to claim 1, characterized in that After the electronic device wakes up the voice interaction function, the method further includes: The electronic device outputs prompt information, where the prompt information is used to prompt that the voice interaction function is currently turned on; The prompt information includes: voice and / or interface elements displayed on a display screen.

8. The method according to claim 1, characterized in that The first microphone is a microphone on the top of the electronic device, and the second microphone is a microphone on the bottom of the electronic device.

9. The method according to claim 1, characterized in that: The first microphone and the second microphone have different sensitivities; detecting the intensity difference between the first voice signal and the corresponding sub-segment voice signal in the second voice signal specifically includes: The first speech signal or the second speech signal is compensated based on the sensitivity difference; and the intensity difference between the corresponding sub-segment speech signals in the compensated first speech signal and the second speech signal is detected.

10. The method according to claim 1, characterized in that After the electronic device captures a first image containing a user's face through a camera, the method further includes: performing identity authentication based on the user's face in the first image.

11. The method according to claim 1, characterized in that After the electronic device collects the first voice signal through the first microphone and collects the second voice signal through the second microphone, the method further includes: Voiceprint information is extracted from the first voice signal or the second voice signal, and identity authentication is performed based on the voiceprint information.

12. The method according to claim 1, characterized in that The electronic device determines the distance between the user's face and the electronic device according to the first image, which is specifically achieved in the following manner: The electronic device analyzes the imaging pupil distance from the first image; The electronic device calculates the distance between the user's face and the electronic device based on the imaging pupil distance, the focal length of the camera and the actual pupil distance using the principle of similar triangles.

13. The method according to claim 1, characterized in that The electronic device determines the gaze direction of the user according to the first image, specifically in the following manner: The electronic device analyzes the position of the pupil in the eyeball from the first image to determine the gaze direction of the user.

14. An electronic device, characterized in that: include: One or more processors, one or more memories, at least two microphones, a camera, and a display; The one or more memories are coupled to one or more processors, and the one or more memories are used to store computer program codes, wherein the computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 13.

Images