Voice assistant interaction method and electronic equipment

Abstract

The invention provides a voice assistant interaction method and electronic equipment, the electronic equipment can determine whether to wake up a voice assistant according to own equipment posture and equipment action, and the equipment posture can be determined by combining measurement data of various sensors of the electronic equipment. For example, measurement data of an IMU, measurement data of an ultrasonic signal, touch information of a capacitive screen or measurement data of a proximity sensor and the like. The probability that the voice assistant is awakened by mistake is lower, and the energy utilization efficiency of the electronic equipment is higher. The pickup method and the feedback method executed by the electronic device can be determined according to the device posture of the electronic device, the voice interaction process of the user is more natural, and the usability of the voice assistant is better.

Description

Technical Field

[0001] The present application relates to the field of terminal device software, and more specifically, to a method and electronic device for voice assistant interaction. Background Art

[0002] Voice interaction is an important entry point for human-computer interaction between users and electronic devices. The power consumption, fluency, usability, response speed and intelligence of voice interaction are important factors that affect whether users use voice assistants and how often they use them. How to improve the usability of voice assistants while controlling the power consumption of electronic devices during voice interaction is an issue worth considering. Summary of the invention

[0003] The present application provides a method for interacting with a voice assistant, whereby an electronic device can determine whether to wake up a voice assistant based on its own device posture and device action, and use a sound pickup method and a feedback method that are compatible with the device posture of the electronic device. The probability of the voice assistant being woken up by mistake is lower, the power consumption of the electronic device is lower, the process of voice interaction by the user is more natural, and the voice assistant is more easy to use.

[0004] In a first aspect, a method for voice interaction is provided, which is applied to an electronic device, the method comprising: detecting a target action of the electronic device, determining that the electronic device is in a target posture, the target posture comprising a posture of a lower microphone of the electronic device close to a user's mouth, the target action comprising an action for converting the electronic device from a non-target posture to a target posture; in response to the target posture of the electronic device, displaying and / or playing first feedback information, the first feedback information being used to indicate that a voice assistant is successfully awakened; recording a voice instruction at least through the lower microphone of the electronic device; and in response to the voice instruction, displaying and / or playing second feedback information, the second feedback information being used to indicate an execution result of the voice instruction.

[0005] It should be understood that before displaying and / or playing the first feedback information, the electronic device can open or call a voice assistant application so that the application can interact with the user, such as obtaining the user's voice commands.

[0006] Before recording the sound of the user's voice interaction, the electronic device first feedbacks to the user that the voice assistant has been successfully awakened. The speaker of the electronic device does not need to be in a normally open state to record audio information. When the user uses the voice assistant, the power consumption of the electronic device is lower. In addition, when the user's mouth is close to the lower microphone of the electronic device, the electronic device can set the method of recording voice commands to a method that is compatible with the posture of the device, which is conducive to enabling the electronic device to obtain the user's voice commands more clearly and improve the efficiency of the electronic device calling the voice assistant. From another perspective, the implementation of this method is also conducive to reducing the power consumption of the electronic device during the recording process.

[0007] In combination with the first aspect, in certain implementations of the first aspect, the target posture includes a posture in which the lower microphone is close to the user's mouth and the ambient light sensor and / or the proximity sensor are not blocked, and in response to the target posture of the electronic device, a prompt screen is displayed and / or a first prompt audio is played through the lower speaker of the electronic device, and the first feedback information includes the prompt screen and / or the first prompt audio.

[0008] Here, the target posture may also be referred to as the first posture.

[0009] In some scenarios, the posture of the lower microphone close to the user's mouth and the ambient light sensor and / or proximity sensor being unblocked can also be understood as the posture of the electronic device when the lower microphone of the electronic device is close to the user's mouth and the user is not in the process of making or receiving a call. In this posture, the side of the electronic device where the front camera is set will be away from the user's head.

[0010] The technical solution specifically provides a method for an electronic device to send feedback information to a user when the electronic device is in a first posture. For an electronic device in the first posture, if the user is using a voice assistant, there is a high probability that the user is looking at the screen of the electronic device. The method of displaying the prompt screen through the display screen is more intuitive and easier to be noticed by the user, and the efficiency of the electronic device to send feedback information is higher. Similarly, when the user's mouth is close to the lower microphone, the lower speaker of the electronic device is closer to the user, and playing the prompt audio through the lower speaker is also conducive to the feedback information being noticed by the user, and the efficiency of the electronic device to send feedback information is higher. In short, the implementation of this technical solution is conducive to making the interaction between the user and the voice assistant more natural, improving the usability of the voice assistant, and increasing the frequency of users using the voice assistant.

[0011] In combination with the first aspect, in certain implementations of the first aspect, in response to the target posture of the electronic device, a first ultrasonic signal is sent; a second ultrasonic signal returned based on the first ultrasonic signal is received; and when the second ultrasonic signal indicates that the electronic device is in the target posture, a prompt screen is displayed and / or a first prompt audio is played through the lower speaker of the electronic device.

[0012] The electronic device can confirm whether the lower microphone of the electronic device is close to the user's mouth by sending an ultrasonic signal and recording the returned ultrasonic signal. When the confirmation method also indicates that the electronic device is in a posture where the lower microphone is close to the user's mouth, feedback information indicating that the voice assistant has been successfully awakened is sent to the user. Since the results determined by ultrasound are more reliable, the implementation of this technical solution is conducive to reducing the probability of the voice assistant of the electronic device being awakened by mistake, and is conducive to reducing the power consumption of the electronic device caused by the false awakening of the voice assistant.

[0013] In combination with the first aspect, in certain implementations of the first aspect, in response to a target posture of the electronic device, a first ultrasonic signal is sent through the lower speaker; in response to a second ultrasonic signal returned by the first ultrasonic signal contacting the user, a second ultrasonic signal is received through the lower microphone.

[0014] In this technical solution, the lower speaker near the user's mouth is used to send the first ultrasonic signal, and the lower microphone near the user's mouth is used to record the second ultrasonic signal. This technical solution is conducive to improving the accuracy and reliability of ultrasonic measurement results, reducing the probability of the voice assistant being woken up by mistake, and reducing the power consumption of electronic devices.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the target posture includes a posture in which the lower microphone of the electronic device is close to the user's mouth and the ambient light sensor and / or the proximity sensor are blocked, and in response to the target posture of the electronic device, a second prompt audio is played through the upper speaker of the electronic device, and the first feedback information includes the second prompt audio.

[0016] Here, the target posture may also be referred to as a second posture.

[0017] In some scenarios, the lower microphone is close to the user's mouth, and the ambient light sensor and / or proximity sensor are blocked. This can also be understood as the electronic device's lower microphone being close to the user's mouth and the user making a phone call. In this posture, the side of the electronic device where the front camera is set will be close to or in contact with the user's head.

[0018] This technical solution specifically provides a method for an electronic device to send feedback information to a user in a second posture. In the posture of making a call, the display screen of the electronic device is close to the user's ear, and the user cannot see the information on the display screen of the electronic device. The upper speaker (or receiver) of the electronic device is close to the user's ear, and the upper speaker is used to play the prompt audio, which is conducive to the feedback information being perceived by the user, and the efficiency of the electronic device in sending feedback information is higher. The implementation of this technical solution is conducive to making the interaction between the user and the voice assistant more natural, improving the usability of the voice assistant, and increasing the frequency of users using the voice assistant.

[0019] In combination with the first aspect, in certain implementations of the first aspect, touch information of the display screen of the electronic device is obtained in response to the target posture of the electronic device; when the touch information indicates that the display screen of the electronic device is touched, a second prompt audio is played through the upper speaker of the electronic device.

[0020] In combination with the first aspect, in some implementations of the first aspect, the touch information includes information of the outer ear contour.

[0021] In a possible implementation, the touch information in the present technical solution may include a capacitance value of a display screen of the electronic device.

[0022] The electronic device can use the touch information of the display screen to confirm whether the electronic device is in the posture of making or receiving a call. When it is confirmed that the electronic device is in the posture of making or receiving a call, feedback information prompting the user that the voice assistant has been successfully awakened is sent. The reliability of the result of the touch information of the display screen confirming the posture of the electronic device is higher. The implementation of this technical solution is conducive to reducing the probability of the voice assistant of the electronic device being awakened by mistake, and is conducive to reducing the power consumption of the electronic device caused by the false awakening of the voice assistant.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the lower microphone of the electronic device is used as a main recording unit, and the upper microphone of the electronic device is used as an auxiliary recording unit to record voice commands.

[0024] The voice command recording method provided by the present technical solution is adapted to the device posture, and using multiple microphones to record voice commands is beneficial for enabling the electronic device to obtain the user's voice commands more clearly, thereby improving the efficiency of the electronic device in calling the voice assistant.

[0025] In combination with the first aspect, in certain implementations of the first aspect, before recording voice instructions using the lower microphone of the electronic device as a main recording unit and the upper microphone of the electronic device as an auxiliary recording unit, the method also includes: recording first audio through the upper microphone and the lower microphone; and determining that the user is speaking close to the lower microphone based on the first audio.

[0026] Since the electronic device in the target posture does not directly correspond to the user currently using or intending to use the voice assistant, determining that the user is speaking close to the lower microphone of the electronic device through the upper and lower microphones of the electronic device is beneficial to improving the accuracy of judging the user's intention to use the voice assistant, reducing the probability of the voice assistant being mistakenly awakened, and reducing the power consumption of the electronic device.

[0027] In a second aspect, a method for voice assistant interaction is provided, which is applied to an electronic device, including: detecting a target action of the electronic device, determining that the electronic device is in a target posture, the target posture including a posture of an upper microphone of the electronic device close to a user's mouth, and the target action including an action for converting the electronic device from a non-target posture to a target posture; in response to the target posture of the electronic device, playing first feedback information, the first feedback information is used to indicate that the voice assistant is successfully awakened; recording a voice instruction at least through the upper microphone of the electronic device; in response to the voice instruction, playing second feedback information, the second feedback information is used to indicate the execution result of the voice instruction.

[0028] It should be understood that before playing the first feedback information, the electronic device can open or call the voice assistant application so that the application can interact with the user, such as obtaining the user's voice command.

[0029] Before recording the sound of the user's voice interaction, the electronic device first feedbacks to the user that the voice assistant has been successfully awakened. The speaker of the electronic device does not need to be in a normally open state to record audio information. When the user uses the voice assistant, the power consumption of the electronic device is lower. In addition, when the user's mouth is close to the upper microphone of the electronic device, the electronic device can set the recording method to a method that is compatible with the posture of the device, which is conducive to the electronic device being able to obtain the user's voice commands more clearly and improve the efficiency of the electronic device calling the voice assistant. From another perspective, the implementation of this method is also conducive to reducing the power consumption of the electronic device during the recording process.

[0030] In combination with the second aspect, in some implementations of the second aspect, in response to the target gesture of the electronic device, the first feedback information is played through an upper speaker of the electronic device.

[0031] Here, the target posture may also be referred to as the third posture.

[0032] The technical solution specifically provides a method for an electronic device to send feedback information to a user when the electronic device is in the third posture. For an electronic device in the third posture, if the user is using a voice assistant, the probability that the image on the display screen of the electronic device is perceived by the user is small, and the upper speaker of the electronic device is close to the user, and playing the prompt audio through the upper speaker is conducive to the feedback information being perceived by the user, and the efficiency of the electronic device in sending feedback information is higher. The implementation of the technical solution is conducive to making the interaction between the user and the voice assistant more natural, and is conducive to improving the usability of the voice assistant and increasing the frequency of users using the voice assistant.

[0033] In combination with the second aspect, in certain implementations of the second aspect, in response to a target posture of the electronic device, a first ultrasonic signal is sent; a second ultrasonic signal returned based on the first ultrasonic signal is received; and when the second ultrasonic signal indicates that the electronic device is in the target posture, first feedback information is played through an upper speaker of the electronic device.

[0034] The electronic device can confirm whether the upper microphone of the electronic device is close to the user's mouth by sending an ultrasonic signal and recording the returned ultrasonic signal. When the confirmation method also indicates that the electronic device is in a posture where the upper microphone is close to the user's mouth, feedback information indicating that the voice assistant has been successfully awakened is sent to the user. Since the reliability of the result determined by ultrasound is higher, the implementation of this technical solution is conducive to reducing the probability of the voice assistant of the electronic device being awakened by mistake, and is conducive to reducing the power consumption of the electronic device caused by the false awakening of the voice assistant.

[0035] In combination with the second aspect, in certain implementations of the second aspect, a first ultrasonic signal is sent through the upper speaker in response to a target gesture of the electronic device; and a second ultrasonic signal is received through the upper microphone in response to the first ultrasonic signal contacting the user and returning the second ultrasonic signal.

[0036] In this technical solution, the upper speaker near the user's mouth is used to send the first ultrasonic signal, and the upper microphone near the user's mouth is used to record the second ultrasonic signal. This technical solution is conducive to improving the accuracy and reliability of ultrasonic measurement results, reducing the probability of the voice assistant being woken up by mistake, and reducing the power consumption of electronic devices.

[0037] In combination with the second aspect, in certain implementations of the second aspect, the upper microphone of the electronic device is used as a main recording unit, and the lower microphone of the electronic device is used as an auxiliary recording unit to record voice instructions.

[0038] The voice command recording method provided by the present technical solution is adapted to the device posture, and using multiple microphones to record voice commands is beneficial for enabling the electronic device to obtain the user's voice commands more clearly, thereby improving the efficiency of the electronic device in calling the voice assistant.

[0039] In combination with the second aspect, in certain implementations of the second aspect, before recording a voice command using the upper microphone of the electronic device as a main recording unit and the lower microphone of the electronic device as an auxiliary recording unit, the method also includes: recording a first audio through the upper microphone and the lower microphone; and determining that the user is speaking close to the upper microphone based on the first audio.

[0040] Since the electronic device in the target posture does not directly correspond to the user currently using or intending to use the voice assistant, determining that the user is speaking close to the upper microphone of the electronic device through the upper and lower microphones of the electronic device is beneficial to improving the accuracy of judging the user's intention to use the voice assistant, reducing the probability of the voice assistant being mistakenly awakened, and reducing the power consumption of the electronic device.

[0041] According to a third aspect, a device for interacting with a voice assistant is provided, the device comprising an acquisition module and a processing module, the processing module being used to: detect a target action of the device, determine that the device is in a target posture, the target posture comprising a posture of a lower microphone of the device close to a user's mouth, the target action comprising an action for converting the device from a non-target posture to a target posture; in response to the target posture of the device, display and / or play first feedback information, the first feedback information being used to indicate that the voice assistant is successfully awakened; the acquisition module being used to: record voice instructions at least through the lower microphone of the device; the processing module being further used to: in response to the voice instruction, display and / or play second feedback information, the second feedback information being used to indicate the execution result of the voice instruction.

[0042] In combination with the third aspect, in certain implementations of the third aspect, the target posture includes a posture in which the lower microphone is close to the user's mouth and the ambient light sensor and / or the proximity sensor are not blocked, and the processing module is specifically used to: in response to the target posture of the device, display a prompt screen and / or play a first prompt audio through the lower speaker of the device, and the first feedback information includes the prompt screen and / or the first prompt audio.

[0043] In combination with the third aspect, in certain implementations of the third aspect, the processing module is specifically used to: send a first ultrasonic signal in response to the target posture of the device; the acquisition module is also used to: receive a second ultrasonic signal returned based on the first ultrasonic signal; when the second ultrasonic signal indicates that the device is in the target posture, display a prompt screen and / or play a first prompt audio through the lower speaker of the device.

[0044] In combination with the third aspect, in certain implementations of the third aspect, the processing module is also used to: send a first ultrasonic signal through the lower speaker in response to the target posture of the device; the acquisition module is also used to: receive a second ultrasonic signal through the lower microphone in response to a second ultrasonic signal returned when the first ultrasonic signal contacts the user.

[0045] In combination with the third aspect, in certain implementations of the third aspect, the target posture includes a posture in which the lower microphone of the device is close to the user's mouth and the ambient light sensor and / or the proximity sensor are blocked, and the processing module is specifically used to: in response to the target posture of the device, play a second prompt audio through the upper speaker of the device, and the first feedback information includes the second prompt audio.

[0046] In combination with the third aspect, in certain implementations of the third aspect, the processing module is specifically used to: obtain touch information of the display screen of the device in response to the target posture of the device; when the touch information indicates that the display screen of the device is touched, play a second prompt audio through the upper speaker of the device.

[0047] In combination with the third aspect, in certain implementations of the third aspect, the touch information includes information of the outer ear contour.

[0048] In combination with the third aspect, in certain implementations of the third aspect, the acquisition module is specifically used to: use the lower microphone of the device as a main recording unit and the upper microphone of the device as an auxiliary recording unit to record voice instructions.

[0049] In combination with the third aspect, in certain implementations of the third aspect, before the lower microphone of the device is used as the main recording unit and the upper microphone of the device is used as the auxiliary recording unit to record voice instructions, the acquisition module is also used to: record the first audio through the upper microphone and the lower microphone; the processing module is also used to: determine that the user is speaking close to the lower microphone based on the first audio.

[0050] In a fourth aspect, a device for interacting with a voice assistant is provided, the device comprising an acquisition module and a processing module, the processing module being used to: detect a target action of the device, determine that the device is in a target posture, the target posture comprising a posture of an upper microphone of the device close to a user's mouth, the target action comprising an action for converting the device from a non-target posture to a target posture; in response to the target posture of the device, play first feedback information, the first feedback information being used to indicate that the voice assistant has been successfully awakened; the acquisition module being used to: record voice instructions at least through the upper microphone of the device; the processing module being further used to: in response to the voice instruction, play second feedback information, the second feedback information being used to indicate the execution result of the voice instruction.

[0051] In combination with the fourth aspect, in certain implementations of the fourth aspect, the processing module is specifically used to: in response to the target posture of the device, play the first feedback information through the upper speaker of the device.

[0052] In combination with the fourth aspect, in certain implementations of the fourth aspect, the processing module is also used to: send a first ultrasonic signal in response to the target posture of the device; the acquisition module is also used to: receive a second ultrasonic signal returned based on the first ultrasonic signal; the processing module is also used to: play first feedback information through the upper speaker of the device when the second ultrasonic signal indicates that the device is in the target posture.

[0053] In combination with the fourth aspect, in certain implementations of the fourth aspect, the processing module is specifically used to: send a first ultrasonic signal through the upper speaker in response to the target posture of the device; the acquisition module is specifically used to: receive a second ultrasonic signal through the upper microphone in response to a second ultrasonic signal returned when the first ultrasonic signal contacts the user.

[0054] In combination with the fourth aspect, in certain implementations of the fourth aspect, the acquisition module is specifically used to: use the upper microphone of the device as a main recording unit and the lower microphone of the device as an auxiliary recording unit to record voice instructions.

[0055] In combination with the fourth aspect, in certain implementations of the fourth aspect, before the upper microphone of the device is used as the main recording unit and the lower microphone of the device is used as the auxiliary recording unit to record voice instructions, the acquisition module is also used to: record the first audio through the upper microphone and the lower microphone; the processing module is also used to: determine that the user is speaking close to the upper microphone based on the first audio.

[0056] In a fifth aspect, an electronic device is provided, which includes a processor and a memory, the memory is used to store program instructions, and the processor is used to: detect a target action of the electronic device, determine that the electronic device is in a target posture, the target posture includes the posture of the lower microphone of the electronic device close to the user's mouth, and the target action includes an action for converting the electronic device from a non-target posture to a target posture; in response to the target posture of the electronic device, display and / or play first feedback information, the first feedback information is used to indicate that the voice assistant is successfully awakened; record voice instructions at least through the lower microphone of the electronic device; in response to the voice instruction, display and / or play second feedback information, the second feedback information is used to indicate the execution result of the voice instruction.

[0057] In combination with the fifth aspect, in certain implementations of the fifth aspect, the target posture includes a posture in which the lower microphone is close to the user's mouth and the ambient light sensor and / or the proximity sensor are not blocked, and the processor is also used to: in response to the target posture of the electronic device, display a prompt screen and / or play a first prompt audio through the lower speaker of the electronic device, and the first feedback information includes the prompt screen and / or the first prompt audio.

[0058] In combination with the fifth aspect, in certain implementations of the fifth aspect, the processor is specifically used to: send a first ultrasonic signal in response to a target posture of the electronic device; receive a second ultrasonic signal returned based on the first ultrasonic signal; and display a prompt screen and / or play a first prompt audio through the lower speaker of the electronic device when the second ultrasonic signal indicates that the electronic device is in the target posture.

[0059] In combination with the fifth aspect, in certain implementations of the fifth aspect, the processor is specifically used to: send a first ultrasonic signal through a lower speaker in response to a target posture of the electronic device; and receive a second ultrasonic signal through a lower microphone in response to a second ultrasonic signal returned when the first ultrasonic signal contacts the user.

[0060] In combination with the fifth aspect, in certain implementations of the fifth aspect, the target posture includes a posture in which the lower microphone of the electronic device is close to the user's mouth and the ambient light sensor and / or the proximity sensor are blocked, and the processor is specifically used to: in response to the target posture of the electronic device, play a second prompt audio through the upper speaker of the electronic device, and the first feedback information includes the second prompt audio.

[0061] In combination with the fifth aspect, in certain implementations of the fifth aspect, the processor is specifically used to: obtain touch information of the display screen of the electronic device in response to the target posture of the electronic device; and play a second prompt audio through the upper speaker of the electronic device when the touch information indicates that the display screen of the electronic device is touched.

[0062] In combination with the fifth aspect, in certain implementations of the fifth aspect, the touch information includes information of the outer ear contour.

[0063] In combination with the fifth aspect, in certain implementations of the fifth aspect, the processor is further used to: record voice instructions using the lower microphone of the electronic device as a main recording unit and the upper microphone of the electronic device as an auxiliary recording unit.

[0064] In combination with the fifth aspect, in certain implementations of the fifth aspect, before recording voice commands using the lower microphone of the electronic device as the main recording unit and the upper microphone of the electronic device as the auxiliary recording unit, the processor is also used to: record first audio through the upper microphone and the lower microphone; and determine that the user is speaking close to the lower microphone based on the first audio.

[0065] In a sixth aspect, an electronic device is provided, which includes a processor and a memory, the memory is used to store program instructions, and the processor is used to: detect a target action of the electronic device, determine that the electronic device is in a target posture, the target posture includes a posture of an upper microphone of the electronic device close to a user's mouth, and the target action includes an action for converting the electronic device from a non-target posture to a target posture; in response to the target posture of the electronic device, play first feedback information, the first feedback information is used to indicate that the voice assistant is successfully awakened; record voice instructions at least through the upper microphone of the electronic device; in response to the voice instruction, play second feedback information, the second feedback information is used to indicate the execution result of the voice instruction.

[0066] In combination with the sixth aspect, in certain implementations of the sixth aspect, the processor is specifically used to: in response to a target gesture of the electronic device, play the first feedback information through an upper speaker of the electronic device.

[0067] In combination with the sixth aspect, in certain implementations of the sixth aspect, the processor is specifically used to: send a first ultrasonic signal in response to a target posture of the electronic device; receive a second ultrasonic signal returned based on the first ultrasonic signal; and play first feedback information through an upper speaker of the electronic device when the second ultrasonic signal indicates that the electronic device is in the target posture.

[0068] In combination with the sixth aspect, in certain implementations of the sixth aspect, the processor is specifically used to: send a first ultrasonic signal through an upper speaker in response to a target posture of the electronic device; and receive a second ultrasonic signal through an upper microphone in response to a second ultrasonic signal returned when the first ultrasonic signal contacts the user.

[0069] In combination with the sixth aspect, in certain implementations of the sixth aspect, the processor is specifically used to: use the upper microphone of the electronic device as a main recording unit and the lower microphone of the electronic device as an auxiliary recording unit to record voice instructions.

[0070] In combination with the sixth aspect, in certain implementations of the sixth aspect, before recording voice instructions using the upper microphone of the electronic device as the main recording unit and the lower microphone of the electronic device as the auxiliary recording unit, the processor is also used to: record first audio through the upper microphone and the lower microphone; and determine that the user is speaking close to the upper microphone based on the first audio.

[0071] In the seventh aspect, a computer program product is provided, which includes a computer program code. When the computer program code runs on a computer, the method in the first aspect and any possible implementation thereof or the method in the second aspect and any possible implementation thereof is executed.

[0072] In an eighth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the method in the first aspect and any possible implementation thereof or the method in the second aspect and any possible implementation thereof is executed.

[0073] In the ninth aspect, a chip is provided, comprising a processor for reading instructions stored in a memory, wherein when the processor executes the instructions, the chip implements the method in the first aspect and any possible implementation thereof or the method in the second aspect and any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a schematic diagram of the hardware architecture of an electronic device applicable to an embodiment of the present application.

[0075] Figure 2 It is a schematic diagram of the software architecture of an electronic device applicable to an embodiment of the present application.

[0076] Figure 3 This is a voice assistant interaction method provided in an embodiment of the present application.

[0077] Figure 4 It is a schematic diagram of the device postures of several electronic devices provided in the embodiments of the present application.

[0078] Figure 5 This is another voice assistant interaction method provided in an embodiment of the present application.

[0079] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0080] Figure 7 It is a schematic diagram of a method for an electronic device to determine whether a target action is detected provided by an embodiment of the present application.

[0081] Figure 8It is a schematic diagram of a method for an electronic device to detect the distance between a microphone and a user's mouth using ultrasound provided in an embodiment of the present application.

[0082] Fig. 9 It is a schematic diagram of a method for an electronic device to send feedback information provided in an embodiment of the present application.

[0083] Fig.10 It is a schematic diagram of a method for an electronic device to determine a position relationship between a user and the electronic device provided in an embodiment of the present application.

[0084] Fig.11 This is another voice assistant interaction method provided in an embodiment of the present application.

[0085] Fig.12 It is a schematic diagram of another method for an electronic device to determine whether a target action is detected provided by an embodiment of the present application.

[0086] Fig.13 This is another voice assistant interaction method provided in an embodiment of the present application.

[0087] Fig.14 It is a schematic diagram of another method for an electronic device to determine whether a target action is detected provided by an embodiment of the present application.

[0088] Fig.15 A voice assistant interaction device provided in an embodiment of the present application.

[0089] Fig.16 An electronic device is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0091] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one, two or more. The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0092] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0093] Figure 1 1 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0094] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0095] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0096] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0097] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0098] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0099] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0100] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0101] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0102] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0103] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0104] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0105] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0106] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0107] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0108] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0109] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor. The ISP is used to process the data fed back by the camera 193. The camera 193 is used to capture static images or videos. The digital signal processor is used to process digital signals, and can process other digital signals in addition to processing digital image signals. The video codec is used to compress or decompress digital videos.

[0110] NPU is a neural-network (NN) computing processor. It can quickly process input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn.

[0111] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement a data storage function.

[0112] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0113] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0114] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0115] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0116] In some examples, the electronic device 100 may include one or more speakers 170A. When there are multiple speakers 170A, the multiple speakers 170A may be distributed at different locations of the electronic device 100.

[0117] Exemplarily, at least one of the plurality of speakers 170A may be located on the same side of the display screen of the electronic device 100 , and at least one may be located on the other side opposite to the display screen of the electronic device 100 .

[0118] Exemplarily, the direction of the longer side of the electronic device 100 is the length direction, and the direction of the shorter side of the electronic device 100 is the width direction. The multiple speakers 170A may be distributed at two opposite upper and lower locations in the length direction or two opposite left and right locations in the width direction of the electronic device 100. For example, the position where the front camera of the electronic device 100 is set is the upper part, and the position on the same surface opposite to the upper part and far from the front camera is the lower part. The multiple speakers 170A may include an upper speaker and a lower speaker, the upper speaker is located at the upper part of the electronic device 100, or close to the front camera of the electronic device 100, and the lower speaker is located at the lower part of the electronic device 100, or far from the front camera of the electronic device 100.

[0119] It should be noted that the upper part and the lower part, up and down, and left and right are relative positional relationships. The above is only an exemplary description and should not be understood as a limitation to the present application.

[0120] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0121] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0122] When there are multiple microphones 170C, the multiple microphones 170C may be distributed at different locations of the electronic device 100 .

[0123] Exemplarily, at least one of the plurality of microphones 170C may be located on the same side of the display screen of the electronic device 100 , and at least one may be located on the other side opposite to the display screen of the electronic device 100 .

[0124] Exemplarily, the direction of the longer side of the electronic device 100 is the length direction, and the direction of the shorter side of the electronic device 100 is the width direction. The multiple microphones 170C may be distributed at two opposite upper and lower locations in the length direction or two opposite left and right locations in the width direction of the electronic device 100. For example, the position where the front camera of the electronic device 100 is set is the upper part, and the position on the same surface opposite to the upper part and far from the front camera is the lower part. The multiple microphones 170C may include an upper speaker and a lower speaker, the upper speaker is located at the upper part of the electronic device 100, or close to the front camera of the electronic device 100, and the lower speaker is located at the lower part of the electronic device 100, or far from the front camera of the electronic device 100.

[0125] It should be noted that the upper part and the lower part, up and down, and left and right are relative positional relationships. The above is only an exemplary description and should not be understood as a limitation to the present application.

[0126] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0127] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0128] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0129] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0130] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0131] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0132] Figure 2 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application (application, app) layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0133] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0134] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0135] like Figure 2 As shown, the application framework layer may include a window manager, an activity manager, a package manager, a resource manager, a view system, a phone manager, a notification manager, and the like.

[0136] Resource manager, also known as resource management service (RMS), provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.

[0137] The window manager, also known as the window management service (WMS), is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0138] The activity manager, also known as the activity manager service (AMS), manages all application processes in the system.

[0139] The package manager, also known as the package management service (PMS), is responsible for functions such as application installation and uninstallation, component query and matching, and permission management.

[0140] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0141] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0142] Android runtime includes core libraries and virtual machines, which are responsible for scheduling and management of the Android system.

[0143] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0144] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0145] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0146] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0147] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0148] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0149] A 2D graphics engine is a drawing engine for 2D drawings.

[0150] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0151] Before formally introducing the embodiments of the present application, some terms that may be used in the following content are first explained.

[0152] K-nearest neighbors algorithm (kNN), also known as the nearest neighbor method, is a non-parametric statistical method for classification and regression. The algorithm uses a vector space model for classification. The concept is that cases of the same category have high similarity to each other, and the possible classification of unknown category cases can be evaluated by calculating the similarity with known category cases.

[0153] Support vector machine (SVM), also known as support vector network, is a supervised learning model and related learning algorithm for analyzing data in classification and regression analysis. The SVM model represents instances as points in space, so that instances of separate categories are separated by as wide a clear interval as possible. Then, new instances are mapped to the same space and their categories are predicted based on which side of the interval they fall.

[0154] Random forest refers to a classifier that contains multiple decision trees, and its output category is determined by the majority of the categories output by individual trees.

[0155] Convolutional neural network (CNN) refers to a feedforward neural network whose artificial neurons can respond to surrounding units within a certain coverage area, and has excellent performance in large image processing. In at least one layer of a convolutional neural network, a mathematical operation called convolution is used instead of general matrix multiplication.

[0156] A recurrent neural network (RNN) is an artificial neural network that uses sequence data or time series data. It can be used for sequential or time problems, such as language translation, natural language processing, and speech recognition.

[0157] An autoencoder, also known as an automatic encoder, is an artificial neural network that is used to learn effective encoding of unlabeled data and belongs to unsupervised learning. An autoencoder consists of two main parts: the encoder is used to encode the input, and the decoder uses the encoding to reconstruct the input.

[0158] A chirp signal is a signal whose frequency changes (increases or decreases) over time.

[0159] Automatic speech recognition (ASR), also known as speech-to-text recognition or speech recognition technology, aims to automatically convert human speech into text responses using computers. Applications of speech recognition technology include voice dialing, voice navigation, indoor equipment control, and voice document retrieval.

[0160] A proximity sensor is a sensor that can detect the presence of nearby objects without contact. Proximity sensors usually emit an electromagnetic field or an electromagnetic radiation beam (such as infrared) and observe the changes in the electric field or return signal to achieve their functions.

[0161] Voice interaction is an important method for human-computer interaction between users and electronic devices. In the process of user interaction with a voice assistant, in order to improve the efficiency of voice interaction and reduce the power consumption of electronic devices, this application provides a method for voice assistant interaction, which is introduced as follows.

[0162] Figure 3 The flowchart of a voice assistant interaction method provided by an embodiment of the present application is shown. Before recording the sound of the user's voice interaction, the electronic device first feeds back to the user that the voice assistant has been successfully awakened, and the way in which the electronic device sends feedback information to the user is adapted to the device posture of the electronic device. The user's interaction process with the voice assistant is more natural, and the voice assistant is easier to use.

[0163] S110, in response to detecting a target posture and a target action, sending first feedback information according to a target feedback method.

[0164] Here, the target action may refer to an action of converting the electronic device from a non-target posture to a target posture, and the target action may be completed by a user operation.

[0165] In some examples, the target posture may include a first posture, a second posture, or a third posture, wherein the first posture is a posture in which the lower microphone of the electronic device is close to the user's mouth and the display screen of the electronic device is not close to the user's ear; the second posture is a posture in which the lower microphone of the electronic device is close to the user's mouth and the display screen of the electronic device is close to the user's ear; the third posture is a posture in which the upper microphone of the electronic device is close to the user's mouth.

[0166] Figure 4 The schematic diagrams of the aforementioned first posture, second posture and third posture are provided exemplarily, Figure 4 The left side of the diagram 401 is a schematic diagram of the first posture, the middle diagram 402 is a schematic diagram of the second posture, and the right side of the diagram 403 is a schematic diagram of the third posture. Figure 4 The exemplary description of the above-mentioned postures is only for the sake of understanding, and should not be regarded as limiting the embodiments of the present application. In other words, the above-mentioned first posture, second posture and third posture can also be other than Figure 4 The embodiments of the present application do not limit other postures than those shown in the schematic diagram.

[0167] In some scenarios, the first posture can also be understood as: the lower microphone of the electronic device is close to the user's mouth, and it is not a posture for making or receiving a call; or, the lower microphone of the electronic device is close to the user's mouth, and the ambient light sensor and / or the proximity sensor are not blocked, the posture of the electronic device. The second posture can also be understood as: the lower microphone of the electronic device is close to the user's mouth, and it is a posture for making or receiving a call. Or, the lower microphone of the electronic device is close to the user's mouth, and the ambient light sensor and / or the proximity sensor are blocked, the posture of the electronic device.

[0168] In some examples, the electronic device can obtain measurement data of one or more sensors of the electronic device, and the measurement data of these sensors can be used to indicate the device posture and device action of the electronic device. In other words, the electronic device can determine whether a target posture or target action is detected based on the measurement data of one or more sensors of the electronic device.

[0169] Exemplarily, the electronic device can determine whether the electronic device is in the first posture, the second posture, or the third posture according to the measurement data of the gravity sensor. For example, the electronic device can determine the components of gravity acceleration on the three axes of the electronic device according to the measurement data of the gravity sensor, and the angle between the three axes of the electronic device and the horizontal plane can be determined according to the components of gravity acceleration on the three axes, and the posture of the electronic device can be determined according to the angle.

[0170] In some examples, before determining the device posture of the electronic device, the electronic device may also determine the acceleration of the electronic device based on measurement data of an inertial measurement unit (IMU), and determine whether the electronic device is in a relatively stable state based on the acceleration. When the electronic device is in a relatively stable state, the electronic device determines its own device posture.

[0171] In some examples, when a target posture and target action are detected, the electronic device can use a target verification method to verify whether the electronic device is in the target posture, thereby reducing the probability of the voice assistant being mistakenly awakened or mistakenly touched, and improving the energy utilization efficiency of the electronic device.

[0172] Exemplarily, when the target posture includes the aforementioned first posture or third posture, the electronic device can use a speaker to send a first ultrasonic signal and use a microphone to receive a second ultrasonic signal returned based on the first ultrasonic signal, and verify whether the electronic device is in the target posture based on the recorded second ultrasonic signal and the sent first ultrasonic signal.

[0173] Here, the speaker may refer to the speaker 170A of the electronic device 100 mentioned above, such as the upper speaker or the lower speaker. The microphone may refer to the microphone 170C of the electronic device 100 mentioned above, such as the upper microphone and the lower microphone. For a detailed description of the speaker position or microphone position of the electronic device, please refer to the following description. Figure 6 The relevant description is not expanded here.

[0174] For example, when an electronic device detects that it is in a first posture according to the aforementioned method, the electronic device can use its lower speaker to send a first ultrasonic signal, use its lower microphone to record a second ultrasonic signal, and verify whether the electronic device is in the first posture based on the first ultrasonic signal and the second ultrasonic signal.

[0175] For another example, when the electronic device detects that it is in the third posture according to the aforementioned method, the electronic device can use its upper speaker to send a first ultrasonic signal, use its upper microphone to record a second ultrasonic signal, and verify whether the electronic device is in the second posture based on the first ultrasonic signal and the second ultrasonic signal.

[0176] The method of using the transmitted ultrasonic signal and the recorded ultrasonic signal to verify whether the electronic device is in the target posture will be specifically described in the following embodiments and will not be expanded here.

[0177] The power consumption of sending ultrasound and recording audio is higher than that of the aforementioned IMU and gravity sensor, but the measurement data used to determine the posture of the electronic device is more accurate. Using this method to verify the posture of the electronic device is beneficial to controlling the power consumption of the electronic device while improving the accuracy of target posture detection, and is beneficial to reducing the false touch rate during the use of the voice assistant.

[0178] Also exemplarily, when the target posture includes the aforementioned second posture, the electronic device can obtain touch information of the display screen, and verify whether the electronic device is in the second posture according to the touch information.

[0179] In some examples, the touch information may include information of the user's auricle. Exemplarily, the touch information may include a capacitance value of a display screen of an electronic device, and the capacitance value may be used to indicate information of the user's auricle.

[0180] When the electronic device is in the second posture, the distance between the user's ear and the electronic device display screen is relatively close, and the touch information of the electronic device display screen can be used to determine whether the user's ear is relatively close to the electronic device display screen, that is, to verify whether the electronic device is in the second posture. The method of using the touch information of the display screen to verify whether the electronic device is in the second posture will be specifically described in the following embodiments and will not be expanded here.

[0181] The power consumption of obtaining the touch information of the display screen is higher than that of the aforementioned IMU and gravity sensor, but the measurement data thereof is more accurate in determining the posture of the electronic device. Using this method to verify the posture of the electronic device is beneficial to controlling the power consumption of the electronic device while improving the accuracy of target posture detection, and is beneficial to reducing the false touch rate during the use of the voice assistant.

[0182] In some examples, when the target posture and target action are detected, the electronic device can also verify whether the electronic device is in the target posture by reading the measurement data of the proximity sensor and / or the measurement data of the ambient light sensor. When the electronic device is in the aforementioned first posture, second posture, and third posture, the corresponding measurement values ​​of the measurement data of the proximity sensor and the measurement data of the ambient light sensor are different. The method of how to use the measurement data of the proximity sensor and / or the measurement data of the ambient light sensor to verify whether the electronic device is in the target posture will be described in detail below and will not be expanded here.

[0183] In some examples, the electronic device may combine multiple methods of determining and verifying the target posture to determine whether the electronic device is in the target posture, for example, combining the measurement data of the IMU and the ultrasonic signal, combining the measurement data of the IMU, the measurement data of the proximity sensor, and the touch information of the display screen, or combining the measurement data of the gravity sensor, the ultrasonic signal, the measurement data of the proximity sensor, and the measurement data of the ambient light sensor.

[0184] Determining whether an electronic device is in a target posture through measurement data from multiple sensors of the electronic device using multiple methods is helpful in reducing the probability of accidental touches during the use of the voice assistant and is helpful in improving the energy utilization efficiency of the electronic device.

[0185] The first feedback information is used to indicate that the voice assistant is successfully awakened. For electronic devices in different postures, the method for the electronic device to send the first feedback information may be different, or in other words, the target feedback method for sending the first feedback information may be determined according to the device posture of the electronic device (i.e., the target posture).

[0186] In some examples, the target posture may include the aforementioned first posture, and the first feedback information may be a prompt screen and / or a first prompt audio. The electronic device may display a prompt screen through a display screen and / or play a first prompt audio through a speaker, and the prompt screen and / or the first prompt audio are used to indicate that the voice assistant wakes up successfully. Exemplarily, the electronic device may play the first prompt audio through a lower speaker.

[0187] In some examples, the target posture may include the aforementioned second posture, and the first feedback information may be a second prompt audio. The electronic device may play the second prompt audio through a speaker, and the second prompt audio may be used to indicate that the voice assistant wakes up successfully. Exemplarily, the electronic device may play the second prompt audio through an upper speaker.

[0188] In some examples, the target posture may include the aforementioned third posture, and the first feedback information may be a first prompt audio. The electronic device may play the first prompt audio through a speaker, and the first prompt audio may be used to indicate that the voice assistant wakes up successfully. Exemplarily, the electronic device may play the first prompt audio through an upper speaker.

[0189] The method of sending the first feedback information can adapt to different postures of the electronic device. In other words, the method of sending the first feedback information is more natural and more in line with the user's operating habits. This feedback method is conducive to making the user's human-computer interaction process through the voice assistant more natural.

[0190] It should be understood that the electronic device can also use more other ways to send the above-mentioned first feedback information, such as: one or more of vibration, display light effect, flash light effect, etc., and this application does not limit this.

[0191] S120, recording the voice command according to the target sound pickup method.

[0192] After sending the feedback information, the voice command begins to be recorded. The microphone of the electronic device does not need to be in a normally open state, which is conducive to reducing the power consumption of the electronic device, improving the energy utilization efficiency of the electronic device, and reducing the probability of the voice assistant being woken up by mistake, making the interaction process between the user and the voice assistant more natural.

[0193] The target sound pickup method refers to a method or strategy for an electronic device to record a voice command, and the target sound pickup method may be determined according to the target posture of the electronic device in S110.

[0194] In some examples, the target gesture includes the aforementioned first gesture or second gesture, and the target sound pickup method may be: recording the voice command at least through the lower microphone of the electronic device. Exemplarily, the lower microphone of the electronic device is used as the main recording unit and the upper microphone of the electronic device is used as the auxiliary recording unit to record the voice command.

[0195] In some examples, the target posture includes the aforementioned third posture, and the target sound pickup method may be: recording the voice command at least through the upper microphone of the electronic device. Exemplarily, the upper microphone of the electronic device is used as the main recording unit and the lower microphone of the electronic device is used as the auxiliary recording unit to record the voice command.

[0196] Exemplarily, here, the auxiliary recording unit can be used to perform noise reduction processing on the audio recorded by the main recording unit.

[0197] The target sound pickup method is determined according to the device posture of the electronic device. On the one hand, it is beneficial to improve the accuracy of the voice commands recorded by the electronic device, and to improve the efficiency of the voice assistant in executing the user's voice commands. To a certain extent, it is also beneficial to improve the energy utilization efficiency of the electronic device.

[0198] S130: In response to the voice command, send second feedback information according to the target feedback method.

[0199] Based on the recorded voice commands, the electronic device can determine the voice commands sent by the user, and can convert the voice commands into text messages through automatic voice recognition. Based on the text messages, the electronic device can call a voice assistant to complete the commands indicated by the text messages.

[0200] Here, the method of sending the second feedback information, ie, the target feedback method, may be determined according to the device posture of the electronic device.

[0201] In some examples, the target posture may include the aforementioned first posture, and the second feedback information may be a result prompt screen and / or a result prompt audio. The electronic device may display a result prompt screen through a display screen and / or play a result prompt audio through a speaker, and the result prompt screen and / or the result prompt audio are used to indicate the execution result of the voice command. Exemplarily, the electronic device may play the result prompt audio through a lower speaker.

[0202] In some examples, the target gesture may include the aforementioned second gesture, and the second feedback information may be a result prompt audio. The electronic device may play the result prompt audio through a speaker, and the result prompt audio may be used to indicate the execution result of the voice command. Exemplarily, the electronic device may play the result prompt audio through an upper speaker.

[0203] In some examples, the target gesture may include the aforementioned third gesture, and the second feedback information may be a result prompt audio. The electronic device may play the result prompt audio through a speaker, and the result prompt audio may be used to indicate the execution result of the voice command. Exemplarily, the electronic device may play the result prompt audio through a speaker.

[0204] It should be understood that the execution result here may include a successful execution result, an unsuccessful execution result, or a result that cannot be executed. In addition, the execution result may also be an execution result that indicates that the electronic device cannot detect the user's voice within a preset time period and stops or pauses the voice assistant interaction.

[0205] The method of sending the second feedback information can adapt to different postures of the electronic device. In other words, the method of sending the second feedback information is more natural and more in line with the user's operating habits. This feedback method is conducive to making the user's human-computer interaction process through the voice assistant more natural.

[0206] It should be understood that the electronic device can also use more other ways to send the above-mentioned second feedback information, such as: one or more of vibration, display light effect, flash light effect, indicator light effect, etc., and this application does not limit this.

[0207] For different device postures, electronic devices can adopt different wake-up schemes, sound pickup schemes and feedback schemes, such as Figure 5 As shown, for a situation where the lower microphone of the electronic device is close to the user's mouth and the user is not making or receiving a call (the posture of the electronic device 100 in FIG. 401 ), the electronic device can execute the following interaction scheme.

[0208] S201, determining that the electronic device is in a posture where the lower microphone is close to the user's mouth.

[0209] In some examples, while in a relatively stable state, the electronic device determines whether it is in a posture where the lower microphone is close to the user's mouth.

[0210] Whether the electronic device is in a relatively stable state can be determined based on one or more sensors on the electronic device.

[0211] Exemplarily, the electronic device can determine whether it is in a relatively stable state based on the measurement data of the IMU. For example, the electronic device can determine the current acceleration of the electronic device based on the measurement data of the IMU. When the acceleration of the electronic device is less than or equal to the acceleration threshold, the electronic device can determine that it is in a relatively stable state. On the contrary, when the acceleration of the electronic device is greater than the acceleration threshold, the electronic device can determine that it is in a moving state.

[0212] It should be noted that the relatively stable state here can also be understood as a relatively static state. Stability, static and movement are all relative descriptions.

[0213] In some examples, the posture of the electronic device can be determined based on one or more sensors on the electronic device.

[0214] Exemplarily, the electronic device can determine its own device posture according to the measurement data of the gravity sensor. For example, the electronic device can determine the components of the electronic device's gravity acceleration on the three axes of the electronic device according to the measurement data of the gravity sensor, and determine the angle between the three axes of the electronic device and the horizontal plane according to the components of the gravity acceleration on the aforementioned three axes, and the angle can be used to indicate the device posture of the electronic device. For ease of explanation, in the following content, "the electronic device is in a posture where the lower microphone is close to the user's mouth and is not in a state of making or receiving calls" is referred to as the first target posture.

[0215] by Figure 6 Taking the three-dimensional schematic diagram of the electronic device shown as an example, the three axes of the above electronic device can be respectively called x-axis, y-axis and z-axis, among which the x-axis direction can be regarded as the width direction of the electronic device, the y-axis direction can be regarded as the height direction of the electronic device, and the z-axis direction can be regarded as the thickness direction of the electronic device. Figure 6 The middle quadrilateral region 601 can be regarded as a horizontal plane, and the angles between the x-axis, y-axis and z-axis of the electronic device and the horizontal plane can be respectively referred to as a first angle (α1), a second angle (α2) and a third angle (α3).

[0216] The electronic device 100 may include an upper microphone 201A, a lower microphone 201B, an upper speaker 202A, a lower speaker 202B, a proximity sensor 203 , and an ambient light sensor 204 . Figure 6In the embodiment, the upper microphone 201A, the upper speaker 202A, the proximity sensor 203 and the ambient light sensor 204 can be located at the upper part of the electronic device 100, that is, the area of ​​the electronic device in the positive direction of the y-axis. The lower microphone 201B and the lower speaker 202B can be located at the lower part of the electronic device 100, that is, the area of ​​the electronic device in the negative direction of the y-axis.

[0217] It should be understood that Figure 6 The position distribution of multiple electronic components of the electronic device 100 is only an example, and the present application does not limit this.

[0218] In some examples, when the aforementioned first angle, second angle, and third angle simultaneously meet the preset conditions, the electronic device can determine that it is in the first target posture; on the contrary, when any of the aforementioned first angle, second angle, and third angle does not meet the preset conditions, the electronic device can determine that it is not in the first target posture.

[0219] S202: Determine whether the user has brought the lower microphone of the electronic device close to the mouth.

[0220] In order to improve the accuracy of electronic devices in determining the user's intention to wake up the voice assistant, reduce the probability of the electronic device's voice assistant being mistakenly awakened, and reduce the increase in power consumption caused by the use of the voice assistant, when it is determined that the electronic device is in the first target posture, the electronic device can determine whether it has undergone a change from a non-first target posture to a first target posture. In other words, the electronic device can determine whether the user has adjusted the electronic device from a non-first target posture to a first target posture. For ease of explanation, the action of "adjusting the electronic device from a non-first target posture to a first target posture" is referred to as the first target action below.

[0221] In some examples, the electronic device can save the sensor data for a period of time, and determine whether the user has performed the first target action based on the sensor data in the period of time. Exemplarily, the electronic device can continuously save the IMU measurement data for a certain period of time (e.g., 2 seconds or 1 second, etc.), and determine whether the user has performed the first target action based on the IMU measurement data in the period of time.

[0222] For example, based on the measurement data of the IMU within the time interval, the electronic device can use a machine learning-based classification algorithm and / or a rule-based judgment algorithm to determine whether the user has performed the first target action.

[0223] Classification algorithms based on machine learning may include: one or more of traditional machine learning methods such as nearest neighbor algorithms, support vector machines, random forests, and deep learning methods such as convolutional neural networks or autoencoders.

[0224] In some examples, the electronic device uses one or more of the above classification algorithms to obtain a probability value of "detecting the first target action". When the probability value is greater than or equal to a preset threshold, the electronic device determines that the user has performed the first target action and continues to perform S203 and subsequent operations. When the probability value is less than the preset threshold, the electronic device determines that the user has not performed the first target action and can restart S201.

[0225] For the rule-based judgment algorithm, in some examples, the electronic device can first divide the action into large movements and small movements according to the magnitude of acceleration and the movement time. For example, a large movement can be picking up the electronic device from a table or a trouser pocket to the mouth, and a small movement can be rotating the electronic device from a position close to the face to a certain angle so that the lower microphone of the electronic device is close to the user's mouth.

[0226] For example, the electronic device can determine its own displacement based on the acceleration and the motion time. If the displacement is greater than or equal to the displacement threshold, it is determined that a large motion has occurred; if the displacement is less than the displacement threshold, it is determined that a small motion has occurred. For different motion situations, the electronic device can use different methods to determine whether the first target action is detected. Figure 7 The figure shows a determination method provided by the present application.

[0227] For large movements, the distance of vertical upward movement can identify the action of "lifting up", such as lifting the electronic device from near the user's thigh to the mouth. The distance of horizontal movement can identify the action of "pulling the electronic device towards the user", or in other words, the action of moving the electronic device from a position away from the user to a position close to the user, such as moving the electronic device from a position away from the user to a position close to the user on a table in front of the user.

[0228] In some examples, in the event of a substantial movement, the electronic device may determine whether the electronic device has detected the first target action based on its own displacement component in the direction of gravity or in the horizontal direction.

[0229] For example, the electronic device can determine the component of the acceleration in the direction of gravity according to the angle between the direction of its own acceleration and the direction of gravity, and the moving distance of the electronic device in the direction of gravity can be obtained by performing a second integration of the acceleration in the direction of gravity during the movement time. When the moving distance in the direction of gravity is greater than or equal to the vertical displacement threshold, the electronic device determines that the first target action is detected; when the moving distance in the direction of gravity is less than the vertical displacement threshold, the electronic device determines that the first target action is not detected.

[0230] Similarly, by way of example, the electronic device can determine its first horizontal plane acceleration component and second horizontal plane acceleration component on the horizontal plane based on its own acceleration along the negative direction of the y-axis and the positive direction of the z-axis, and determine the composite acceleration of the electronic device on the horizontal plane based on the first horizontal plane acceleration component and the second horizontal plane acceleration component, and the moving distance of the electronic device on the horizontal plane can be obtained by performing a second integration of the composite acceleration within the movement time. When the moving distance on the horizontal plane is greater than or equal to the horizontal displacement threshold, the electronic device determines that the first target action is detected; when the moving distance on the horizontal plane is less than the horizontal displacement threshold, the electronic device determines that the first target action is not detected.

[0231] Here, since the negative direction of the y-axis and the positive direction of the z-axis of the electronic device usually point to the user, when calculating the distance moved in the horizontal direction, only the acceleration in the negative direction of the y-axis and the positive direction of the z-axis can be considered.

[0232] For small movements, the action can be identified by the change in the angle between the three axes of the electronic device and the horizontal plane. The amplitude of the angle change should be within a certain range. In some examples, since the process of rotating the electronic device close to the user's mouth generally does not include reciprocating motion, the change in the angle between the y-axis and the horizontal plane and the change in the angle between the x-axis and the horizontal plane are monotonically increasing or monotonically decreasing.

[0233] In some examples, when a small movement occurs, the electronic device can determine whether a target action is detected based on the angles (first angle α1, second angle α2, and third angle α3) between the three axes of the electronic device and the horizontal plane during the movement.

[0234] Exemplarily, the electronic device can determine the first angle change value (β1) corresponding to α1, the second angle change value (β2) corresponding to α2, and the third angle change value (β3) corresponding to α3 during its own motion. Based on the first angle change value, the second angle change value, and the third angle change value, the electronic device can determine whether the target action is detected. For ease of explanation, the first angle threshold (δ1), the second angle threshold (δ2), and the third angle threshold (δ3) are used here to represent the angle thresholds corresponding to β1, β2, and β3, respectively.

[0235] For example, in the case that β1≤δ1, β2≤δ3, and β3≤δ3, the electronic device may determine that the first target action is detected; otherwise, the electronic device may determine that the first target action is not detected.

[0236] Also exemplarily, the electronic device can continuously record the first angle and the second angle during its own movement, and then determine the first angle increment (γ1) of the first angle and the second angle increment (γ2) of the second angle at different times. Based on multiple first angle increments and multiple second angle increments at different times during the movement, the electronic device can determine whether the target action is detected.

[0237] For example, during the movement, if all γ1s are positive or negative, and all γ2s are positive or negative, the electronic device can determine that the first target action is detected; otherwise, the electronic device can determine that the first target action is not detected.

[0238] Still exemplarily, the electronic device may also combine the above two judgment methods to determine whether the target operation is detected. That is, whether the target action is detected is determined according to the first angle change value, the second angle change value, the third angle change value corresponding to the start and end time of the own movement, and multiple first angle increments and multiple second angle increments at different times during the movement.

[0239] For example, when β1≤δ1, β2≤δ3 and β3≤δ3, and all γ1s are positive or negative, and all γ2s are positive or negative, the electronic device can determine that the first target action is detected; otherwise, the electronic device can determine that the first target action is not detected.

[0240] S203, determining whether the measurement data of the proximity sensor and the measurement data of the ambient light sensor meet the requirements.

[0241] The lower microphone of the electronic device is close to the user's mouth and is not in the posture of making or receiving calls, and the upper part of the electronic device is not blocked. In this case, the distance determined based on the measurement data of the proximity sensor of the electronic device is greater than or equal to the distance threshold, that is, the measurement data of the proximity sensor will not indicate that the upper part of the electronic device is blocked, and the illuminance determined based on the measurement of the ambient light sensor will generally be greater than or equal to the illuminance threshold. In other words, when the reading of the proximity sensor and the measurement data of the ambient light sensor both meet the aforementioned preset conditions, the electronic device is more likely to be in the first target posture. On the contrary, if one or both of the detection data of the proximity sensor and the measurement data of the ambient light sensor do not meet the preset conditions, the electronic device is less likely to be in the first target posture. For example, in this case, the electronic device may be in a pocket or handbag.

[0242] If it is determined that the measurement data of the proximity sensor and the measurement data of the ambient light sensor meet the requirements, the electronic device can execute S204 and subsequent operations. If it is determined that the measurement data of the proximity sensor and / or the measurement data of the ambient light sensor do not meet the requirements, the electronic device can re-execute S201 operation.

[0243] S204, the lower speaker of the electronic device sends an ultrasonic signal, and the lower microphone detects the ultrasonic signal.

[0244] When it is determined that the electronic device is in the first target posture and the electronic device detects the first target action, the electronic device may execute S204, that is, control the lower speaker to send an ultrasonic signal and the lower microphone to detect the ultrasonic signal.

[0245] When it is determined that the electronic device is not in the first target posture, or when the electronic device does not detect the first target action, the electronic device may re-execute the operation of S201.

[0246] S205, determining the distance between the lower microphone and the user's mouth and acquiring feature information of the user's mouth.

[0247] According to the detection result of the ultrasonic signal by the lower microphone in S204, the distance between the lower microphone of the current electronic device and the user's mouth is determined, and the characteristic information of the user's mouth is obtained. Since different points of the user's mouth are at different distances from the microphone, different points have different effects on the reflection or absorption of ultrasonic signals. After the ultrasonic signal is reflected by the user's mouth, the audio clip of the microphone mobile phone contains this information. In other words, the characteristic information of the user's mouth can refer to the characteristic information of the user's mouth contained in the spectrum diagram of the ultrasonic signal detected by the microphone after the ultrasonic signal is blocked and reflected by the user's mouth.

[0248] Figure 8 The figure shows a processing method for determining the device posture of an electronic device using ultrasonic signals provided in an embodiment of the present application.

[0249] The electronic device emits a chirp signal of a preset frequency and a preset duration, and records audio of a certain duration, performs filtering on the recorded audio, and intercepts a spectrum of an audio segment in the recorded audio as an input of a classifier, and the output result of the classifier can be used to determine the distance between the lower microphone of the electronic device and the user's mouth. Here, the classifier can be RNN, CNN, or SVM, etc.

[0250] Specifically, in some examples, the electronic device can generate a chirp signal of 0.02s with a frequency range of 18kHz to 23kHz, and send it at a time interval of 0.02s for 0.2s. The electronic device can record 100ms of audio and apply a bandpass filter of 18kHz to 23kHz to the recorded audio signal. The original chirp signal and the recorded audio signal are used for matched filtering, and the maximum position in the result of the matched filtering can be regarded as the starting position of the chirp signal. In the recorded audio, a 0.02s audio segment is intercepted from the starting position, and the spectrum of the intercepted audio segment is calculated using a short-time Fourier transform or a fast Fourier transform as the input of the classifier. The classifier can determine the classification result of each complete signal based on the input, and the classification result can be used to determine the distance between the lower microphone and the user's mouth.

[0251] In the case where the ultrasonic signal is used to determine that the lower microphone of the electronic device is close to the user's mouth, the electronic device can execute S206, that is, after opening the voice assistant application, feedback that the voice assistant wake-up is successful. On the contrary, if the ultrasonic signal does not indicate that the lower microphone of the electronic device is close to the user's mouth, the electronic device can re-execute the operation of S201.

[0252] S206, feedback that the voice assistant is successfully awakened.

[0253] In some examples, such as Fig. 9 As shown, the electronic device can display a first prompt message 210 through a display screen. The first prompt message 210 can also be called a first prompt screen 210. The first prompt message 210 is used to prompt that the voice assistant is successfully awakened. The first prompt message 210 can include text information and / or light information (or light effect information), etc.

[0254] In some examples, such as Fig. 9 As shown, the electronic device can play the first prompt audio 220 through the lower speaker, and the first prompt audio 220 is used to prompt that the voice assistant is successfully woken up.

[0255] In some examples, the electronic device can simultaneously send a prompt message through the display screen and the lower speaker to indicate that the voice assistant has been successfully awakened.

[0256] The electronic device can also prompt the successful wake-up of the voice assistant through other methods, such as vibration information, etc. The electronic device can also change the feedback method according to the user's customized operations, and this application does not impose any restrictions on this.

[0257] When the electronic device is in the first target posture, the prompt information can be more easily perceived by the user by displaying the prompt information on the display screen and / or playing the audio prompt information using the lower speaker. This feedback method makes the user's interaction with the voice assistant more natural, which is conducive to improving the usability of the voice assistant.

[0258] S207, using the dual-channel audio feature, determining that the user is speaking close to the lower microphone.

[0259] In order to determine that the audio recorded by the microphone comes from the user of the electronic device, or in other words, to determine that the person speaking is the user of the electronic device, in some examples, while providing feedback that the voice assistant has been successfully woken up, the electronic device may record through the upper and lower microphones and utilize dual-channel audio features to determine that the user is speaking close to the lower microphone.

[0260] In some examples, the electronic device can combine audio information recorded by the upper microphone and the lower microphone to determine that the user is speaking close to the lower microphone. Fig.10 The figure shows a method provided by an embodiment of the present application for combining audio information recorded by an upper microphone and a lower microphone to determine that a user is speaking close to a lower microphone.

[0261] Exemplarily, the electronic device can save an audio signal with a length of 200ms and make a judgment every 100ms, and multiple judgment results can be saved as a historical cache. In some examples, the electronic device can separate the audio recorded by the upper microphone and the lower microphone, and use a low-pass filter with a cutoff frequency of 17kHz to remove the signal in the ultrasonic frequency band. The dual-channel audio signal is then used as the input of a CNN-based classifier, and the classification result output by the classifier is used to determine whether the user is speaking close to the lower microphone of the electronic device.

[0262] For example, the output results of the aforementioned CNN-based classifier can give the probabilities of the following three situations, and the situation where the probability is greater than or equal to the preset threshold is considered to be detected.

[0263] Category 1: The upper microphone of the electronic device is close to the user's mouth. For example, the distance between the upper microphone of the electronic device and the user's mouth is within 5 cm, and the user holds the electronic device and speaks.

[0264] Category 2: The lower microphone of the electronic device is close to the user's mouth. For example, the distance between the lower microphone of the electronic device and the user's mouth is within 5 cm, and the user holds the electronic device and speaks.

[0265] Category 3: The electronic device is not close to the user's mouth, or other situations, for example, the user uses the electronic device normally, such as talking while looking at the screen, talking in an environment with other people around the user, talking in a quiet environment around the user, or talking in a noisy environment around the user.

[0266] In some examples, the electronic device may also save the most recent n judgment results, vote on different types of results in the most recent n judgment results through a voting mechanism, and regard the result of the majority vote as the final output, where n is an integer greater than or equal to 1.

[0267] S208, selecting a microphone according to the sound pickup strategy and performing automatic speech recognition.

[0268] When the lower microphone of the electronic device is close to the user's mouth, the electronic device can record the user's voice command at least through the lower microphone, for example, executing a sound pickup strategy with the lower microphone as the main and the upper microphone as the auxiliary. For example, the electronic device can use the audio recorded by the upper microphone to perform auxiliary noise reduction processing on the audio recorded by the lower microphone.

[0269] S209, the voice assistant of the electronic device executes the command and generates feedback according to the corresponding feedback method.

[0270] After the electronic device calls the language assistant to execute the command, it can feedback the command execution result in a similar manner to the feedback of successful awakening of the language assistant in S206. For details, please refer to the content in S206, which will not be repeated here for the sake of brevity.

[0271] In some examples, the electronic device may use different feedback methods for different execution results, such as different types of feedback methods, or feedback methods of the same type but with different contents. This application does not limit this.

[0272] Fig.11 A method for a user to interact with a voice assistant is shown when the upper microphone of the electronic device is close to the user's mouth (the posture of the electronic device 100 in Figure 403).

[0273] S301, determining that the electronic device is in a posture where the upper microphone is close to the user's mouth.

[0274] In some examples, when in a relatively stable state, the electronic device determines whether its posture is a state in which the upper microphone is close to the user's mouth.

[0275] Whether the electronic device is in a relatively stable state can be determined based on one or more sensors on the electronic device. For example, the electronic device can be determined based on measurement data of an IMU.

[0276] In some examples, the posture of the electronic device can be determined based on one or more sensors on the electronic device. Exemplarily, the electronic device can determine the posture of the electronic device based on the measurement data of the gravity sensor. For example, the electronic device can determine the components of the gravity acceleration of the electronic device on the three axes of the electronic device based on the measurement data of the gravity sensor, and determine the angle between the three axes of the electronic device and the horizontal plane based on the components of the gravity acceleration on the aforementioned three axes, and the angle can be used to indicate the posture of the electronic device. For ease of explanation, the following content refers to "the electronic device is in a posture where the lower microphone is close to the user's mouth" as the third target posture.

[0277] by Figure 6 Taking the three-dimensional schematic diagram of the electronic device shown as an example, the angles between the x-axis, y-axis and z-axis of the electronic device and the horizontal plane can be respectively referred to as the first angle (α1), the second angle (α2) and the third angle (α3). In some examples, when the first angle, the second angle and the third angle meet the preset conditions at the same time, the electronic device can determine that it is in the third target posture; on the contrary, when any of the first angle, the second angle and the third angle does not meet the preset conditions, the electronic device can determine that it is not in the third target posture.

[0278] The specific execution method of S301 is similar to that of S201. For details, please refer to the relevant content in S201. For the sake of brevity, it will not be described here.

[0279] S302: Determine whether the user has brought the upper microphone of the electronic device close to the mouth.

[0280] In order to improve the accuracy of electronic devices in determining the user's intention to wake up the voice assistant, reduce the probability of the electronic device's voice assistant being mistakenly awakened, and reduce the increase in power consumption caused by the use of the voice assistant, when it is determined that the electronic device is in the third target posture, the electronic device can determine whether it has undergone a change from a non-third target posture to a third target posture. In other words, the electronic device can determine whether the user has adjusted the electronic device from a non-third target posture to a third target posture. For ease of explanation, the action of "adjusting the electronic device from a non-third target posture to a third target posture" is referred to as the third target action below.

[0281] In some examples, the electronic device may store sensor data for a period of time, and determine whether the user has performed the third target action based on the sensor data within the time period. For example, the electronic device may continuously store IMU measurement data for a certain period of time (e.g., 2 seconds or 1 second, etc.), and determine whether the user has performed the above action based on the IMU measurement data within the time interval.

[0282] For example, based on the measurement data of the IMU within the time interval, the electronic device can use a machine learning-based classification algorithm and / or a rule-based judgment algorithm to determine whether the user has performed the third target action.

[0283] Classification algorithms based on machine learning may include: one or more of traditional machine learning methods such as nearest neighbor algorithms, support vector machines, random forests, and deep learning methods such as convolutional neural networks or autoencoders.

[0284] In some examples, the electronic device uses one or more of the above classification algorithms to obtain a probability value of "detecting the third target action". When the probability value is greater than or equal to a preset threshold, the electronic device determines that the user has performed the third target action and continues to perform S203 and subsequent operations. When the probability value is less than the preset threshold, the electronic device determines that the user has not performed the third target action and can restart S301.

[0285] For the rule-based judgment algorithm, the electronic device may determine whether the third target action is detected according to its own moving distance in the vertical direction and / or the angle it rotates around the x-axis. Fig.12 A method for an electronic device to determine whether a third target action is detected is illustratively provided.

[0286] In some examples, the electronic device may determine that the third target action has been detected based on whether its vertical movement distance is greater than a vertical displacement threshold. If it is determined that the vertical movement distance is greater than or equal to the vertical displacement threshold, the electronic device may determine that the third target action has been detected; if it is determined that the vertical movement distance is less than the vertical displacement threshold, the electronic device may determine that the third target action has not been detected.

[0287] In some examples, the electronic device may determine that a third target action has been detected based on whether the angle through which the electronic device has rotated around the x-axis is greater than an angle threshold. If it is determined that the angle through which the electronic device has rotated around the x-axis is greater than or equal to the angle threshold, then the third target action has been detected. If it is determined that the angle through which the electronic device has rotated around the x-axis is less than the angle threshold, then the electronic device may determine that no third target action has been detected.

[0288] S303: Determine whether the measurement data of the proximity sensor and the measurement data of the ambient light sensor meet the requirements.

[0289] In some examples, when the upper microphone of the electronic device is close to the user's mouth, the proximity sensor is blocked and the reading of the ambient light sensor is reduced, or in other words, the distance detected by the proximity sensor is less than or equal to the distance threshold, and the illuminance detected by the ambient light sensor is less than or equal to the illuminance threshold. In other words, when both the reading of the proximity sensor and the measurement data of the ambient light sensor meet the preset conditions, it is more likely that the electronic device is in a posture where the upper microphone is close to the user's mouth. On the contrary, if one or both of the detection data of the proximity sensor and the measurement data of the ambient light sensor do not meet the preset conditions, it is less likely that the electronic device is in a posture where the upper microphone is close to the user's mouth. For example, in this case, the electronic device may be in a pocket or a handbag.

[0290] If it is determined that the measurement data of the proximity sensor and the measurement data of the ambient light sensor meet the requirements, the electronic device can execute S304 and subsequent operations. If it is determined that the measurement data of the proximity sensor and / or the measurement data of the ambient light sensor do not meet the requirements, the electronic device can re-execute S301 operation.

[0291] S304, the upper speaker of the electronic device sends an ultrasonic signal, and the upper microphone detects the ultrasonic signal.

[0292] When it is determined that the electronic device is in the third target posture and the electronic device detects the third target action, the electronic device may execute S304, that is, control the upper speaker to send an ultrasonic signal and the upper microphone to detect the ultrasonic signal.

[0293] When it is determined that the electronic device is not in the third target posture, or when the electronic device does not detect the third target action, the electronic device may re-execute the operation of S301.

[0294] S305, determining the distance between the upper microphone and the user's mouth and acquiring feature information of the user's mouth.

[0295] According to the detection result of the upper microphone for the ultrasonic signal in S304, the distance between the upper microphone of the current electronic device and the user's mouth is determined, and the characteristic information of the user's mouth is obtained. Since different points of the user's mouth are at different distances from the microphone, different points have different effects on the reflection or absorption of ultrasonic signals. After the ultrasonic signal is reflected by the user's mouth, the audio clip of the microphone mobile phone contains this information. In other words, the characteristic information of the user's mouth can refer to the characteristic information of the user's mouth contained in the spectrum diagram of the ultrasonic signal detected by the microphone after the ultrasonic signal is blocked and reflected by the user's mouth.

[0296] The electronic device emits a chirp signal of a preset frequency and a preset duration, and records audio of a certain duration, performs filtering on the recorded audio, and intercepts a spectrum of an audio segment in the recorded audio as an input of a classifier, and the output result of the classifier can be used to determine the distance between the lower microphone of the electronic device and the user's mouth. Here, the classifier can be RNN, CNN, or SVM, etc.

[0297] The specific execution method of S305 is similar to that of S205. For details, please refer to the relevant content in S205. For the sake of brevity, it will not be described here.

[0298] In the case where the upper microphone of the electronic device is determined to be close to the user's mouth by using the ultrasonic signal, the electronic device can execute S306, that is, after opening the voice assistant application, feedback that the voice assistant wake-up is successful. On the contrary, if the ultrasonic signal does not indicate that the upper microphone of the electronic device is close to the user's mouth, the electronic device can re-execute the operation of S301.

[0299] S306, feedback that the voice assistant wakes up successfully.

[0300] In some examples, the electronic device can display a second prompt message through a display screen. The second prompt message can also be called a second prompt screen. The second prompt message is used to prompt that the voice assistant has been successfully awakened. The second prompt message can be text information or light information (or light effect information), etc.

[0301] In some examples, the electronic device can play a third prompt audio through the upper speaker, and the third prompt audio is used to prompt that the voice assistant is successfully woken up.

[0302] In some examples, the electronic device can simultaneously send a prompt message through the display screen and the upper speaker to indicate that the voice assistant has been successfully awakened.

[0303] The electronic device can also prompt the successful wake-up of the voice assistant through other methods, such as vibration information, etc. The electronic device can also change the feedback method according to the user's customized operations, and this application does not impose any restrictions on this.

[0304] When the electronic device is in the third target posture, the prompt information can be more easily perceived by the user by playing audio prompt information through the upper speaker. This feedback method makes the interaction between the user and the voice assistant more natural, which is conducive to improving the usability of the voice assistant.

[0305] S307: Determine, using the dual-channel audio feature, that the user is speaking close to the lower microphone.

[0306] In order to determine that the audio recorded by the microphone comes from the user of the electronic device, or in other words, to determine that the person speaking is the user of the electronic device, in some examples, while providing feedback that the voice assistant has been successfully woken up, the electronic device may record audio simultaneously through the upper microphone and the lower microphone, and utilize dual-channel audio features to determine that the user is speaking close to the upper microphone.

[0307] In some examples, the electronic device can combine audio information recorded by the upper microphone and the lower microphone to determine that the user is speaking close to the lower microphone.

[0308] Exemplarily, the electronic device can save an audio signal with a length of 200ms and make a judgment every 100ms, and multiple judgment results can be saved as a historical cache. In some examples, the electronic device can separate the audio recorded by the upper microphone and the lower microphone, and use a low-pass filter with a cutoff frequency of 17kHz to remove the signal in the ultrasonic frequency band. The dual-channel audio signal is then used as the input of a CNN-based classifier, and the classification result output by the classifier is used to determine whether the user is speaking close to the lower microphone of the electronic device.

[0309] In some examples, the electronic device may also save the most recent n output results, vote on different types of results in the most recent n output results through a voting mechanism, and regard the result of the majority vote as the final output, where n is an integer greater than or equal to 1.

[0310] The specific execution method of S307 is similar to that of S207. For details, please refer to the relevant content in S207. For the sake of brevity, it will not be described here.

[0311] S308, selecting a microphone according to the sound pickup strategy and performing automatic speech recognition.

[0312] When the upper microphone of the electronic device is close to the user's mouth, the electronic device can record the user's voice command at least through the upper microphone, for example, executing a sound pickup strategy with the upper microphone as the main and the lower microphone as the auxiliary. For example, the electronic device can use the audio recorded by the lower microphone to perform auxiliary noise reduction processing on the audio recorded by the upper microphone.

[0313] S309, the voice assistant of the electronic device executes the command and generates feedback according to the corresponding feedback method.

[0314] After the electronic device calls the language assistant to execute the command, it can feedback that the command execution is completed in a similar manner to the feedback that the language assistant wakes up successfully in S306. For details, please refer to the content in S306, which will not be repeated here for brevity.

[0315] In some examples, the electronic device may use different feedback methods for different execution results, such as different types of feedback methods, or feedback methods of the same type but with different contents. This application does not limit this.

[0316] Fig.13 What is shown is another voice assistant interaction method provided by an embodiment of the present application. In this method, the electronic device can determine that the electronic device is in a posture for making or receiving a call (the posture of the electronic device 100 in Figure 402) based on its own sensors, and thus execute a corresponding wake-up scheme, sound pickup scheme, and feedback scheme based on the posture.

[0317] S401, determining that the electronic device is in a posture for making or receiving a call.

[0318] In some examples, when in a relatively stable state, the electronic device can determine whether its posture is that of making or receiving a phone call.

[0319] Whether the electronic device is in a relatively stable state can be determined based on one or more sensors on the electronic device. Exemplarily, the electronic device can be determined based on the measurement data of the IMU. For example, the electronic device can determine the current acceleration of the electronic device based on the measurement data of the IMU. When the acceleration of the electronic device is less than or equal to the acceleration threshold, the electronic device can determine that it is in a relatively stable state. On the contrary, when the acceleration of the electronic device is greater than the acceleration threshold, the electronic device can determine that it is in a moving state.

[0320] It should be noted that the relatively stable state here can also be understood as a relatively static state. Stability, static and movement are all relative descriptions.

[0321] In some examples, the posture of the electronic device can be determined based on one or more sensors on the electronic device. Exemplarily, the electronic device can determine the posture of the electronic device based on the measurement data of the gravity sensor. For example, the electronic device can determine the components of the gravity acceleration of the electronic device on the three axes of the electronic device based on the measurement data of the gravity sensor, and determine the angle between the three axes of the electronic device and the horizontal plane based on the components of the gravity acceleration on the aforementioned three axes, and the angle can be used to indicate the posture of the electronic device. For ease of explanation, the following content refers to "the posture of the electronic device in making and receiving calls" as the second target posture.

[0322] by Figure 6Taking the three-dimensional schematic diagram of the electronic device shown as an example, the angles between the x-axis, y-axis and z-axis of the electronic device and the horizontal plane 501 can be respectively referred to as the first angle (α1), the second angle (α2) and the third angle (α3). In some examples, when the first angle, the second angle and the third angle meet the preset conditions at the same time, the electronic device can determine that it is in the second target posture; on the contrary, when any of the first angle, the second angle and the third angle does not meet the preset conditions, the electronic device can determine that it is not in the second target posture.

[0323] S402: Determine whether the user has brought the electronic device close to the ear.

[0324] In order to improve the accuracy of electronic devices in determining the user's intention to wake up the voice assistant, reduce the probability of the electronic device's voice assistant being woken up by mistake, and reduce the increase in power consumption caused by the use of the voice assistant, when it is determined that the electronic device is in the posture of making or receiving a call, the electronic device can determine whether it has undergone a change from a non-second target posture to a second target posture. In other words, the electronic device can determine whether the user has performed an action to adjust the electronic device from a non-second target posture to a second target posture. For ease of explanation, the action of "adjusting the electronic device from a non-second target posture to a second target posture" is referred to as the second target action below.

[0325] In some examples, the electronic device can save the sensor data for a period of time, and determine whether the user has performed the second target action based on the sensor data in the time period. Exemplarily, the electronic device can continuously save the IMU measurement data for a certain period of time (e.g., 2 seconds or 1 second, etc.), and determine whether the user has performed the above action based on the IMU measurement data within the time interval.

[0326] For example, based on the measurement data of the IMU within the time interval, the electronic device can use a machine learning-based classification algorithm and / or a rule-based judgment algorithm to determine whether the user has performed the second target action.

[0327] Classification algorithms based on machine learning may include: one or more of traditional machine learning methods such as nearest neighbor algorithms, support vector machines, random forests, and deep learning methods such as convolutional neural networks or autoencoders.

[0328] In some examples, the electronic device uses one or more of the above classification algorithms to obtain a probability value of "detecting the second target action". When the probability value is greater than or equal to a preset threshold, the electronic device determines that the user has performed the second target action and continues to perform S403 and subsequent operations. When the probability value is less than the preset threshold, the electronic device determines that the user has not performed the second target action and can restart S401.

[0329] For the rule-based judgment algorithm, the electronic device can determine whether the target action is detected according to its own moving distance in the vertical direction and / or the angle of rotation around three axes (x-axis, y-axis and z-axis). Fig.14 A method for an electronic device to determine whether a second target action is detected is illustratively provided.

[0330] In some examples, the electronic device may determine that the second target action has been detected based on whether its own vertical movement distance is greater than a vertical displacement threshold. If it is determined that the vertical movement distance is greater than or equal to the vertical displacement threshold, the electronic device may determine that the second target action has been detected; if it is determined that the vertical movement distance is less than the vertical displacement threshold, the electronic device may determine that the second target action has not been detected.

[0331] In some examples, the electronic device may determine that the second target action has been detected based on whether the angle through which the electronic device rotates around the three axes is greater than an angle threshold. If it is determined that the angle through which the electronic device rotates around the three axes is greater than or equal to the angle threshold, then the second target action has been detected. If it is determined that the angle through which the electronic device rotates around the three axes is less than the angle threshold, then the electronic device may determine that the second target action has not been detected.

[0332] Exemplarily, the rotation angles of the electronic device around the x-axis, y-axis and z-axis are respectively the first rotation angle (θ1), the second rotation angle (θ2) and the third rotation angle (θ3). The rotation angle thresholds corresponding to the first rotation angle, the second rotation angle and the third rotation angle are respectively the first rotation angle threshold (σ1), the second rotation angle threshold (σ2) and the third rotation angle threshold (σ3). In the case where θ1≥σ1, θ2≥σ2, and θ3≥σ3, the electronic device determines that the second target action is detected; for the case where any of the three rotation angles does not meet the aforementioned conditions, the electronic device determines that the second target action is not detected.

[0333] For example, when θ1≥σ1, θ2≥σ2, and θ3≥σ3, the electronic device can determine that it is near the user's left ear, or in other words, the electronic device can determine that the user has performed an action of placing the electronic device near the left ear.

[0334] For another example, when θ1≥σ1, θ2≤-σ2, and θ3≤-σ3, the electronic device can determine that it is near the user's right ear, or in other words, the electronic device can determine that the user has performed the action of placing the electronic device near the right ear.

[0335] S403: Determine whether the measurement data of the proximity sensor and / or the measurement data of the ambient light sensor meets the requirements.

[0336] In most cases, the proximity sensor is triggered by the action of making or receiving a call. In a few cases, the proximity sensor is not triggered. In this case, you can determine whether the phone is close to the ear by whether the reading of the ambient light sensor is weakened. If the above conditions are not met, it is determined that the action of making or receiving a call is not detected.

[0337] In some examples, the electronic device can determine that the second target action is detected based on an event triggered by the proximity sensor. For example, during the movement of the electronic device, if the distance determined based on the measurement data of the proximity sensor is less than or equal to the distance threshold, the electronic device determines that the second target action is detected; if the distance determined based on the measurement data of the proximity sensor is greater than the distance threshold, the electronic device determines that the second target action is not detected.

[0338] In some examples, the electronic device can determine whether the electronic device is close to the ear, that is, whether the second target action is detected, based on whether the measured value of the ambient light sensor decreases during the electronic device's movement. Exemplarily, during the movement of the electronic device, if the measured value of the ambient light sensor decreases, it is determined that the second target action is detected. On the contrary, if the measured value of the ambient light sensor does not decrease, it is determined that the second target action is not detected.

[0339] In some examples, the electronic device may also determine whether the second target action is detected by combining the measurement data of the proximity sensor and the measurement data of the ambient light sensor. Exemplarily, when the measurement data of the proximity sensor and the measurement data of the ambient light sensor are both within a preset threshold range, the electronic device determines that the second target action is detected. Otherwise, the electronic device determines that the second target action is not detected.

[0340] S404: Determine whether the electronic device is close to the user's ear.

[0341] When the electronic device is placed close to the ear, the capacitive screen reading of the electronic device will change due to the contact between the external auricle and the electronic device screen. At this time, the shape of the auricle can be identified through the capacitive screen reading.

[0342] In some examples, the capacitive screen readings can be represented by a two-dimensional matrix, and generally, points with larger values ​​represent areas in contact with the skin. The capacitive screen readings can be used to determine whether the electronic device is close to the user's ear. Exemplarily, the electronic device can use machine learning-based methods such as kNN, SVM, and CNN to analyze the capacitive screen readings to determine whether the capacitive screen readings include target data, which is used to indicate the shape of the auricle, or in other words, the target data includes information about the user's auricle, thereby determining whether the electronic device is close to the user's ear.

[0343] In some scenarios, the above determination of whether the electronic device is close to the user's ear based on whether the capacitive screen reading includes the target data can also be understood as: determining whether the electronic device is close to the user's ear based on the touch information of the display screen of the electronic device. Exemplarily, the touch information may include the capacitance value of the display screen of the electronic device, and the capacitance value can be used to indicate the information of the user's auricle.

[0344] In the case where the touch information is used to determine that the electronic device is close to the user's ear, the electronic device can execute S405, that is, after opening the voice assistant application, feedback that the voice assistant wake-up is successful. On the contrary, if the touch information does not indicate that the electronic device is close to the user's ear, the electronic device can re-execute the operation of S401.

[0345] S405, feedback that the voice assistant is successfully awakened.

[0346] In some examples, the electronic device can play a second prompt audio through the lower speaker, and the second prompt audio is used to prompt that the voice assistant is successfully woken up.

[0347] The electronic device can also prompt the successful wake-up of the voice assistant through other methods, such as vibration information, etc. The electronic device can also change the feedback method according to the user's customized operations, and this application does not impose any restrictions on this.

[0348] When the electronic device is in the second target posture, the prompt information can be more easily perceived by the user by playing audio prompt information through the lower speaker. This feedback method makes the interaction between the user and the voice assistant more natural, which is conducive to improving the usability of the voice assistant.

[0349] S406: Determine, using the dual-channel audio feature, that the user is speaking close to the lower microphone.

[0350] In order to determine that the audio recorded by the microphone comes from the user of the electronic device, or in other words, to determine that the person speaking is the user of the electronic device, in some examples, while providing feedback that the voice assistant has been successfully woken up, the electronic device may record audio simultaneously through the upper microphone and the lower microphone, and utilize dual-channel audio features to determine that the user is speaking close to the lower microphone.

[0351] In some examples, the electronic device can combine the audio information recorded by the upper microphone and the lower microphone to determine that the user is speaking close to the lower microphone. Fig.10 The figure shows a method provided by an embodiment of the present application for combining audio information recorded by an upper microphone and a lower microphone to determine that a user is speaking close to a lower microphone.

[0352] Exemplarily, the electronic device can save an audio signal with a length of 200ms and make a judgment every 100ms, and multiple judgment results can be saved as a historical cache. In some examples, the electronic device can separate the audio recorded by the upper microphone and the lower microphone, and use a low-pass filter with a cutoff frequency of 17kHz to remove the signal in the ultrasonic frequency band. The dual-channel audio signal is then used as the input of a CNN-based classifier, and the classification result output by the classifier is used to determine whether the user is speaking close to the lower microphone of the electronic device.

[0353] In some examples, the electronic device may also save the most recent n output results, vote on different types of results in the most recent n output results through a voting mechanism, and regard the result of the majority vote as the final output, where n is an integer greater than or equal to 1.

[0354] The specific execution method of S406 is similar to that of S207. For details, please refer to the relevant content in S207. For the sake of brevity, it will not be described here.

[0355] S407, selecting a microphone according to the sound pickup strategy and automatically performing speech recognition.

[0356] When the electronic device is in a posture of making or receiving a call, the electronic device can record the user's voice command through at least the lower microphone, for example, executing a sound pickup strategy with the lower microphone as the main microphone and the upper microphone as the auxiliary microphone. For example, the electronic device can use the audio recorded by the upper microphone to perform auxiliary noise reduction processing on the audio recorded by the lower microphone.

[0357] S408: The voice assistant executes the command and generates feedback according to a corresponding feedback method.

[0358] After the electronic device calls the language assistant to execute the command, it can feedback that the command execution is completed in a similar manner to the feedback that the language assistant wakes up successfully in S405. For details, please refer to the content in S405, which will not be repeated here for the sake of brevity.

[0359] In some examples, the electronic device may use different feedback methods for different execution results, such as different types of feedback methods, or feedback methods of the same type but with different contents. This application does not limit this.

[0360] Combination of the above Figures 1 to 14 The method embodiment of the present application is described in detail. Fig.15 and Fig.16 The device embodiment of the present application is described. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0361] Fig.15 The present application provides a device 1500 for voice assistant interaction. The device 1500 may have the functions of the electronic device in the above method embodiment, and may be used to execute the steps performed by the functions of the electronic device in the above method embodiment. The function may be implemented by hardware, or by software or hardware executing corresponding software implementation. The hardware or software includes one or more modules corresponding to the above functions.

[0362] In a possible implementation, the voice assistant interaction device 1500 may include an acquisition module 1510 and a processing module 1520, and the acquisition module 1510 and the processing module 1520 are coupled to each other.

[0363] The acquisition module 1510 can be used to support the electronic device to obtain the user's input, such as executing Figure 3 etc. to record audio information.

[0364] The processing module 1520 is used to support the electronic device to execute the processing actions in the above method embodiments, such as sending feedback information, confirming device posture, device action, etc.

[0365] Optionally, the voice assistant interaction device 1500 may further include a storage unit 1530 for storing program codes and data of the voice assistant interaction device 1500 .

[0366] Fig.16 An electronic device 1600 provided in an embodiment of the present application, as shown in the figure, includes: at least one processor 1610 and a transceiver 1620. The processor 1610 is coupled to a memory and is used to execute instructions stored in the memory to control the transceiver 1620 to send and / or receive signals.

[0367] Optionally, the electronic device 1600 further includes a memory 1630 for storing instructions.

[0368] In some embodiments, the processor 1610 and the memory 1630 may be combined into a processing device, and the processor 1610 is used to execute the program code stored in the memory 1630 to implement the above functions. In specific implementation, the memory 1630 may also be integrated into the processor 1610, or independent of the processor 1610.

[0369] In some embodiments, the transceiver 1620 may include a receiver (or receiver) and a transmitter (or transmitter).

[0370] The transceiver 1620 may further include an antenna, and the number of antennas may be one or more. The transceiver 1620 may be a communication interface or an interface circuit.

[0371] When the electronic device 1600 is a chip, the chip includes a transceiver module and a processing module, wherein the transceiver module may be an input / output circuit or a communication interface; and the processing module may be a processor or a microprocessor or an integrated circuit integrated on the chip.

[0372] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method of preloading shaders in the above-mentioned embodiment.

[0373] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the method for preloading a shader in the above-mentioned embodiment.

[0374] In addition, an embodiment of the present application further provides a device, which may be a chip, a component or a module, and the device may include a connected processor and a memory. The memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory, so that the chip executes the method of preloading the shader in the above-mentioned method embodiments.

[0375] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0376] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0377] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0378] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0379] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0380] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0381] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0382] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for voice assistant interaction, applied to electronic equipment, characterized in that: include: Detecting a target action of the electronic device, determining that the electronic device is in a target posture, the target posture comprising a posture of a lower microphone of the electronic device close to a user's mouth, and the target action comprising an action for converting the electronic device from a non-target posture to the target posture; In response to the target posture of the electronic device, display and / or play first feedback information, where the first feedback information is used to indicate that the voice assistant is successfully awakened; recording voice commands via at least a lower microphone of the electronic device; In response to the voice instruction, second feedback information is displayed and / or played, where the second feedback information is used to indicate an execution result of the voice instruction.

2. The method according to claim 1, characterized in that The target posture includes a posture in which the lower microphone is close to the user's mouth and the ambient light sensor and / or the proximity sensor are not blocked, The displaying and / or playing first feedback information in response to the target posture of the electronic device includes: In response to the target posture of the electronic device, a prompt screen is displayed and / or a first prompt audio is played through a lower speaker of the electronic device, and the first feedback information includes the prompt screen and / or the first prompt audio.

3. The method according to claim 2, characterized in that The step of displaying a prompt screen and / or playing a first prompt audio through a lower speaker of the electronic device in response to the target gesture of the electronic device includes: In response to a target posture of the electronic device, sending a first ultrasonic signal; receiving a second ultrasonic signal returned based on the first ultrasonic signal; When the second ultrasonic signal indicates that the electronic device is in the target posture, the prompt screen is displayed and / or the first prompt audio is played through the lower speaker of the electronic device.

4. The method according to claim 3, characterized in that The sending of a first ultrasonic signal in response to a target posture of the electronic device comprises: In response to a target posture of the electronic device, sending the first ultrasonic signal through the lower speaker; The receiving a second ultrasonic signal returned based on the first ultrasonic signal comprises: The second ultrasonic signal is returned in response to the first ultrasonic signal contacting the user, and the second ultrasonic signal is received by the lower microphone.

5. The method according to claim 1, characterized in that The target posture includes a posture in which the lower microphone of the electronic device is close to the user's mouth and the ambient light sensor and / or the proximity sensor are blocked. The displaying and / or playing first feedback information in response to the target posture of the electronic device includes: In response to the target gesture of the electronic device, a second prompt audio is played through an upper speaker of the electronic device, and the first feedback information includes the second prompt audio.

6. The method according to claim 5, characterized in that The step of playing a second prompt audio through an upper speaker of the electronic device in response to the target gesture of the electronic device comprises: In response to the target posture of the electronic device, acquiring touch information of a display screen of the electronic device; When the touch information indicates that the display screen of the electronic device is touched, the second prompt audio is played through the upper speaker of the electronic device.

7. The method according to claim 6, characterized in that The touch information includes information of the outer ear contour.

8. The method according to any one of claims 1 to 7, characterized in that The recording of the voice command at least through the lower microphone of the electronic device includes: using the lower microphone of the electronic device as a main recording unit and the upper microphone of the electronic device as an auxiliary recording unit to record the voice command.

9. The method according to claim 8, characterized in that Before recording the first voice instruction using the lower microphone of the electronic device as a main recording unit and the upper microphone of the electronic device as an auxiliary recording unit, the method further includes: recording first audio through the upper microphone and the lower microphone; It is determined based on the first audio that the user is speaking close to the lower microphone.

10. A method for voice assistant interaction, applied to electronic equipment, characterized in that: include: Detecting a target action of the electronic device, determining that the electronic device is in a target posture, the target posture comprising a posture of an upper microphone of the electronic device close to a user's mouth, and the target action comprising an action for converting the electronic device from a non-target posture to the target posture; In response to the target posture of the electronic device, playing first feedback information, the first feedback information is used to indicate that the voice assistant is successfully awakened; recording a voice command via at least an upper microphone of the electronic device; In response to the voice instruction, second feedback information is played, where the second feedback information is used to indicate an execution result of the voice instruction.

11. The method according to claim 10, characterized in that The step of playing first feedback information in response to the target posture of the electronic device includes: In response to the target gesture of the electronic device, the first feedback information is played through an upper speaker of the electronic device.

12. The method according to claim 11, characterized in that The step of playing first feedback information through an upper speaker of the electronic device in response to the target posture of the electronic device comprises: In response to a target posture of the electronic device, sending a first ultrasonic signal; receiving a second ultrasonic signal returned based on the first ultrasonic signal; When the second ultrasonic signal indicates that the electronic device is in the target posture, the first feedback information is played through an upper speaker of the electronic device.

13. The method according to claim 12, characterized in that The sending of a first ultrasonic signal in response to a target posture of the electronic device comprises: In response to a target posture of the electronic device, sending the first ultrasonic signal through the upper speaker; The receiving a second ultrasonic signal returned based on the first ultrasonic signal comprises: The second ultrasonic signal is returned in response to the first ultrasonic signal contacting the user, and the second ultrasonic signal is received by the upper microphone.

14. The method according to any one of claims 10 to 13, characterized in that The recording of the voice command at least through the upper microphone of the electronic device includes: using the upper microphone of the electronic device as a main recording unit and the lower microphone of the electronic device as an auxiliary recording unit to record the voice command.

15. The method according to any one of claims 10 to 14, characterized in that Before recording the voice command using the upper microphone of the electronic device as a main recording unit and the lower microphone of the electronic device as an auxiliary recording unit, the method further includes: recording first audio through the upper microphone and the lower microphone; It is determined based on the first audio that the user is speaking close to the upper microphone.

16. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 1 to 9 or claims 10 to 15.

17. A memory management device, characterized in that: Comprising modules for implementing the method of any one of claims 1 to 9 or claims 10 to 15.

18. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the method according to any one of claims 1 to 9 or claims 10 to 15 is implemented.

19. A chip, characterized in that: The chip comprises a processor and a memory, wherein the processor is used to read instructions stored in the memory, and when the processor executes the instructions, the chip implements the method according to any one of claims 1 to 9 or claims 10 to 15.

Images