Voice interaction method and electronic device

Abstract

The present application relates to the field of information technology, and in particular to a voice interaction method and electronic device, which can simplify the process of voice interaction and help improve the user experience. The method is applied to a first electronic device, which includes a microphone, and the method includes: the first electronic device collects the user's voice data in a locked state, and the voice data is a first voice data including a preset voice feature, and the preset voice feature includes: when the user is close to the microphone of the first electronic device, the audio feature of the wind noise generated by speaking or breathing; after the voiceprint feature of the first voice data passes the voiceprint verification, the first electronic device executes the voice control event corresponding to the first voice data.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a voice interaction method and electronic device. Background Art

[0002] With the continuous development of voice interaction technology, more and more smart terminals can interact with users through voice. Smart terminals can recognize users' voice data and execute voice control events corresponding to the voice data.

[0003] However, when the smart terminal is in the locked screen state, the user needs to manually unlock it before the user's voice control event can be executed. For example, when the smart terminal is in the locked screen state, the user can first manually unlock the smart terminal and say the voice data of "call xx" to the smart terminal. In response to receiving the user's voice data, the smart terminal executes the voice control event of calling xx. For another example, when the smart terminal is in the locked screen state, the user can first say the voice data of "call xx" to the smart terminal. In response to receiving the user's voice data, the smart terminal voice prompts "Please unlock for me first", and the user needs to manually unlock the smart terminal. In response to the user's unlocking operation, the smart terminal executes the voice control event of calling xx.

[0004] It can be seen that when the smart terminal is in the locked screen state, the user needs to manually unlock it before the user's voice control event can be executed. The process of voice interaction between the user and the smart terminal is not simple and convenient, and the user experience is not good. Summary of the invention

[0005] In view of this, the present application provides a voice interaction method and an electronic device for simplifying the voice interaction process between a user and a smart terminal and improving the user experience.

[0006] In a first aspect, the present application provides a voice interaction method, which is applied to a first electronic device, and the first electronic device includes a microphone. In the method, the first electronic device collects a user's first voice data including a preset voice feature in a locked screen state. After the voiceprint feature of the first voice data passes the voiceprint verification, the first electronic device executes a voice control event corresponding to the first voice data. The preset voice feature includes: an audio feature of wind noise generated by speaking or breathing when the user approaches the microphone of the first electronic device.

[0007] The voice interaction method provided by the present application enables the first electronic device to analyze the collected voice data when the screen is locked, and only execute the voice control event corresponding to the first voice data issued by the target user (machine owner). The target user (machine owner) can execute the voice control event without the target user (machine owner) manually unlocking the device, which simplifies the voice interaction process and helps improve the user experience.

[0008] In a possible implementation of the first aspect, after the voiceprint feature of the first voice data passes the voiceprint verification, the first electronic device automatically unlocks the screen, and after unlocking the screen, the first electronic device executes the voice control event corresponding to the first voice data. In the voice interaction method provided by the present application, if the voiceprint feature of the first voice data passes the voiceprint verification (that is, the first voice data is sent by the target user), the first electronic device can automatically unlock the screen, and after unlocking the screen, execute the voice control event of the target user.

[0009] In a possible implementation of the first aspect, a first electronic device establishes a wireless communication connection with a second electronic device. In this implementation, after recognizing that the first voice data indicates that the second electronic device executes a voice control event, the first electronic device may recognize the voice control event corresponding to the first voice data, and then send a first control command to the second electronic device to instruct the second electronic device to execute the corresponding voice control event.

[0010] Specifically, the first electronic device recognizes that the first voice data indicates that the second electronic device executes a voice control event, and sends a first control command corresponding to the first voice data to the second electronic device. The first control command is used to instruct the second electronic device to execute a voice control event. In response to receiving the first control command, the second electronic device can execute the voice control event corresponding to the first voice data. In the voice interaction method provided in the present application, after the first electronic device establishes a wireless communication connection with the second electronic device, the first electronic device can control the second electronic device to execute the voice control event corresponding to the first voice data of the target user in a locked screen state.

[0011] In a possible implementation of the first aspect, the above-mentioned voice control event may be a voice-to-text event. The first electronic device recognizes that the voice control event indicated by the first voice data is a voice-to-text event, and may send a first control command to the second electronic device, where the first control command is specifically used to instruct the second electronic device to convert the second voice data from the first electronic device into text. The first electronic device may also send the second voice data collected by the first electronic device to the second electronic device. In response to receiving the second voice data, the second electronic device may convert the second voice data into text. In the voice interaction method provided in the present application, after the first electronic device establishes a wireless communication connection with the second electronic device, the first electronic device may, in a locked screen state, control the second electronic device to execute a voice-to-text event, and control the second electronic device to convert the second voice data of the target user into text in the second electronic device.

[0012] In a possible implementation of the first aspect, the second electronic device can identify the voice control event corresponding to the first voice data. Specifically, after the first electronic device recognizes that the first voice data indicates that the second electronic device executes the corresponding voice control event, the first voice data can be directly sent to the second electronic device to instruct the second electronic device to recognize the first voice data and execute the voice control event corresponding to the first voice data. This implementation method provides a possible implementation method for a first electronic device to control a second electronic device to execute a voice control event corresponding to the first voice data of a target user in a locked state.

[0013] In a possible implementation of the first aspect, the above-mentioned voice control event is a voice-to-text event. After the first electronic device sends the first voice data to the second electronic device, the second electronic device can recognize the first voice data. The second electronic device can recognize that the voice control event corresponding to the first voice data is a voice-to-text event. In this case, the second electronic device can send a second control command to the first electronic device. The second control command is used to instruct the first electronic device to send the voice data collected by the first electronic device to the second electronic device. The first electronic device can receive the second control command from the second electronic device, and then send the second voice data collected by the first electronic device to the second electronic device, and the second electronic device converts the second voice data into text. The implementation method provides a possible implementation method for a first electronic device to control a second electronic device to perform a voice-to-text event in a locked screen state.

[0014] In a possible implementation of the first aspect, a first electronic device establishes a wireless communication connection with multiple third electronic devices. After the voiceprint feature of the first voice data passes the voiceprint verification, the first electronic device sends a third control command corresponding to the first voice data to a target device selected / determined by the first electronic device among multiple third electronic devices, and the third control command is used to instruct the target device to execute the voice control event. In the voice interaction method provided in the present application, after the first electronic device establishes a wireless communication connection with multiple third electronic devices, the first electronic device can, in a locked screen state, send a third control command to the target device pointed to by the first electronic device to control the target device to execute the voice control event corresponding to the first voice data.

[0015] In a possible implementation of the first aspect, before the first electronic device sends a second control command corresponding to the first voice data to a target device selected / determined by the first electronic device among multiple third electronic devices, the first electronic device obtains the position information of each third electronic device among the multiple third electronic devices relative to the first electronic device, and the spatial posture information of the first electronic device; wherein the spatial posture information is used to indicate the direction of the first electronic device from the bottom to the top of the first electronic device; the first electronic device determines the target device selected / determined by the first electronic device from the multiple third electronic devices based on the position information and the spatial posture information. In the voice interaction method provided in the present application, before the first electronic device sends the third control command to the target device, it can determine the target device from the multiple third electronic devices based on the position information of the multiple third electronic devices and the spatial posture information of the first electronic device.

[0016] In a possible implementation of the first aspect, after the first electronic device collects the user's voice data in a locked state, the first electronic device extracts the voiceprint features of the first voice data based on the first voice data. The first electronic device compares the voiceprint features with the proprietary voiceprint features of the target user to identify whether the first voice data is sent by the target user. The proprietary voiceprint feature parameters of the target user are pre-stored in the first electronic device. If the first voice data is sent by the target user, the voiceprint features of the first voice data pass the voiceprint verification. In the voice interaction method provided in the present application, the first electronic device pre-stores the proprietary voiceprint features of the target user, and the first electronic device determines whether the first voice data passes the voiceprint verification by comparing the voiceprint features in the first voice data with the proprietary voiceprint features of the target user.

[0017] In a second aspect, the present application provides an electronic device, which is a first electronic device. The electronic device includes: a memory, a microphone, and one or more processors; the memory is coupled to the processor; wherein the microphone is used to collect voice data; the memory is used to store computer program codes, and the computer program codes include computer instructions; when the computer instructions are executed by the processor, the electronic device performs the following steps: in the locked screen state, collect the user's first voice data including preset voice features; wherein the preset voice features include: the audio features of wind noise generated by speaking or breathing when the user approaches the microphone of the above electronic device; after the voiceprint features of the first voice data pass the voiceprint verification, execute the voice control event corresponding to the first voice data.

[0018] In combination with the second aspect, in one possible design method, when the above-mentioned computer instructions are executed by the processor, the electronic device also performs the following steps: after the voiceprint feature of the first voice data passes the voiceprint verification, the screen is automatically unlocked, and after the screen is unlocked, the voice control event corresponding to the first voice data is executed.

[0019] In combination with the second aspect, in a possible design, the electronic device further includes a communication module, and the electronic device establishes a wireless communication connection with the second electronic device through the communication module. When the computer instructions are executed by the processor, the electronic device further performs the following steps: recognizing that the first voice data indicates that the second electronic device executes a voice control event, and sending a first control command corresponding to the first voice data to the second electronic device; wherein the first control command is used to instruct the second electronic device to execute the voice control event.

[0020] In conjunction with the second aspect, in a possible design, the voice control event is a voice-to-text event. When the computer instructions are executed by the processor, the electronic device further performs the following steps: recognizing that the voice control event indicated by the first voice data is voice-to-text, and sending the second voice data collected by the electronic device to the second electronic device.

[0021] In conjunction with the second aspect, in a possible design, the electronic device further includes a communication module, and the electronic device establishes a wireless communication connection with the second electronic device through the communication module. When the computer instructions are executed by the processor, the electronic device further performs the following steps: after recognizing that the first voice data indicates that the second electronic device executes the corresponding voice control event, the first voice data is sent to the second electronic device, indicating that the second electronic device executes the corresponding voice control event.

[0022] In conjunction with the second aspect, in a possible design, the voice control event is a voice-to-text event. When the computer instructions are executed by the processor, the electronic device further performs the following steps: after recognizing that the first voice data indicates that the second electronic device executes the corresponding voice control event, the first voice data is sent to the second electronic device; in response to a second control command, the second voice data is sent to the second electronic device, and the second control command is used to instruct the electronic device to send the voice data to the second electronic device.

[0023] In combination with the second aspect, in one possible design method, the above-mentioned electronic device also includes a communication module, and the electronic device establishes a wireless communication connection with multiple third electronic devices through the communication module. When the above-mentioned computer instructions are executed by the processor, the electronic device also performs the following steps: after the voiceprint feature of the first voice data passes the voiceprint verification, a third control command corresponding to the first voice data is sent to a target device selected / determined by the above-mentioned electronic device among multiple third electronic devices, and the third control command is used to instruct the target device to execute the voice control event.

[0024] In combination with the second aspect, in a possible design method, when the above-mentioned computer instructions are executed by the processor, the electronic device also performs the following steps: obtaining the position information of each third electronic device among multiple third electronic devices relative to the above-mentioned electronic device, and the spatial posture information of the above-mentioned electronic device; wherein the spatial posture information is used to indicate the direction in which the above-mentioned electronic device points from the bottom to the top of the above-mentioned electronic device; and determining the target device selected / determined by the above-mentioned electronic device from multiple third electronic devices based on the position information and the spatial posture information.

[0025] In conjunction with the second aspect, in a possible design, when the above-mentioned computer instructions are executed by the processor, the electronic device further performs the following steps: in the locked screen state, after collecting the user's voice data, extract the voiceprint features of the first voice data according to the first voice data; compare the voiceprint features with the target user's exclusive voiceprint features to identify whether the first voice data is sent by the target user. The target user's exclusive voiceprint feature parameters are pre-stored in the above-mentioned electronic device. If the first voice data is sent by the target user, the voiceprint features of the first voice data pass the voiceprint verification.

[0026] In a third aspect, the present application provides a chip system, which can be applied to an electronic device including a memory. The chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by a line. The interface circuit is used to receive a signal from the above-mentioned memory and send the signal to the processor, and the signal includes a computer instruction stored in the memory. When the processor executes the computer instruction, the electronic device executes the method as in the first aspect and any possible design thereof.

[0027] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect and any possible design thereof.

[0028] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of the first aspect and any possible design thereof.

[0029] It can be understood that the beneficial effects that can be achieved by the electronic device of any possible design of the second aspect, the chip system of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect can be referred to as the beneficial effects in the first aspect and any possible design of the electronic device, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the software architecture structure of an electronic device provided in an embodiment of the present application;

[0032] Figure 3 A flow chart of a voice interaction method provided in an embodiment of the present application;

[0033] Figure 4 A flow chart comparing a voice interaction method provided in an embodiment of the present application with a current voice interaction method;

[0034] Figure 5 A schematic diagram of a mobile phone controlling a PC to execute and convert speech to text events provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of a mobile phone controlling a large screen to execute a voice control event provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of a mobile phone controlling a smart home to execute a voice control event provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of a flow chart of a first electronic device executing a voice control event of an owner in a locked screen state provided in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of a process of controlling a second electronic device to execute a voice control event of an owner in a locked screen state provided by an embodiment of the present application;

[0039] Fig.10 A flowchart of a first electronic device controlling a third electronic device to execute a voice control event of an owner in a locked screen state is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] With the continuous development of voice interaction technology, more and more smart terminals can interact with users through voice. When users interact with smart terminals through voice, they can first wake up the smart terminal. For example, users can say a specified wake-up word to wake up the smart terminal. The wake-up words of smart terminals of different brands may be different. For example, users can say "Hello, YoYo" to wake up the smart terminal.

[0041] After waking up the smart terminal, the user can interact with the smart terminal by voice, and the smart terminal executes the user's voice control event. Specifically, the smart terminal can convert the collected user's voice data into computer-readable input information based on voice recognition technology, and execute the voice control event corresponding to the voice data. As an example, the user can say "Call colleague A" to the mobile phone, and the mobile phone can automatically call colleague A and display the dialing page.

[0042] Smart terminals can have a locked screen state and an unlocked state. In the locked screen state, the smart terminal can protect the user's privacy, prevent misoperation, and save power. In the locked screen state, the smart terminal can display the screen saver interface. In response to the user's operation, the smart terminal can execute some simple commands, such as answering a call, dialing an emergency contact number, turning on a flashlight, etc. The user can switch the smart terminal from the locked screen state to the unlocked state by various unlocking methods such as password unlocking, fingerprint unlocking, face unlocking, and button unlocking.

[0043] In the locked screen state, the smart terminal cannot execute the user's voice control command and needs to be manually unlocked by the user before execution. Take a mobile phone as an example. When the mobile phone is in the locked screen state, the user wants the mobile phone to send xx file to A. After the user manually unlocks the mobile phone, wake up the mobile phone and say the voice data "send xx file to A" to the mobile phone. In response to receiving the user's voice data, the mobile phone can parse the voice data and execute the voice control event of sending xx file to A.

[0044] As another example, when the phone is in lock screen state, the user says the wake-up word "Hello, YoYo" or long presses the power button of the phone to wake up the phone. After that, the user says the voice data of "send xx file to A" to the phone. In response to receiving the user's voice data, the phone can voice reply "OK, please help me unlock first". In response to the user performing the unlock operation, the phone executes the voice control event of sending xx file to A. If the user does not perform the unlock operation, the phone cannot execute the voice control event of sending xx file to A.

[0045] From the above two voice interaction processes, it can be seen that when the smart terminal is in the locked screen state, the user needs to manually unlock it before the user's voice control event can be executed. The voice interaction process between the user and the smart terminal is not simple and convenient, resulting in a poor user experience.

[0046] In order to simplify the process of voice interaction between users and smart terminals and improve the user experience, this application example provides a voice interaction method and electronic device. In the embodiment of the present application, the first electronic device can collect the user's first voice data in the lock screen state. After collecting the first voice data spoken by the user close to the microphone of the first electronic device, the first electronic device can perform a voiceprint verification on the voiceprint characteristics of the first voice data to determine whether the first voice data is sent by the target user (such as the owner). For the first voice data sent by the target user (such as the owner), the first electronic device can execute the voice control event corresponding to the first voice data.

[0047] It can be seen that the voice interaction method provided in the embodiment of the present application enables the electronic device to analyze the collected voice data when the screen is locked, and only execute the voice control event corresponding to the first voice data spoken by the target user (machine owner) close to the microphone of the first electronic device. The target user (machine owner) can execute the voice control event without the target user (machine owner) unlocking the electronic device, which simplifies the voice interaction process and helps to improve the user experience.

[0048] The method provided in the embodiments of the present application can be applied to electronic devices with data processing capabilities. The above-mentioned electronic devices may include servers, mobile phones, tablet computers, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), handheld computers, netbooks, smart home devices (e.g., smart TVs, smart screens, large screens, smart speakers, smart air conditioners, etc.), personal digital assistants (PDAs), wearable devices (e.g., smart watches, smart bracelets, etc.), vehicle-mounted devices, virtual reality devices, etc., and the embodiments of the present application do not impose any restrictions on this. In the embodiments of the present application, the above-mentioned electronic devices are electronic devices that can run operating systems and install applications. Optionally, the operating system running on the electronic device may be system, system, System, etc.

[0049] For example, please refer to Figure 1 , which shows a schematic diagram of the structure of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, an audio module 130, a speaker 130A, a microphone 130B, a display screen 140, a communication module 150, a power module 160, an input device 170, a sensor module 180, etc. Among them, the sensor module 180 may include a pressure sensor, a touch sensor, etc.

[0050] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0051] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110.

[0052] The controller is the nerve center and command center of the electronic device 100. It can generate operation control signals according to command operation codes and timing signals to complete the control of command acquisition and execution.

[0053] The application processor can run the operating system of the electronic device 100, which is used to manage the hardware and software resources of the electronic device 100. For example, it can manage and configure memory, determine the priority of system resource supply and demand, control input and output devices, operate the network, manage the file system, manage drivers, etc. The operating system can also be used to provide an operation page for users to interact with the system. Among them, various types of software can be installed in the operating system, such as drivers, applications (applications, Apps), etc.

[0054] NPU is a neural network (NN) computing processor, which can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of electronic device 100.

[0055] The processor 110 may also be provided with a memory for storing commands and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store commands or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the command or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0056] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc.

[0057] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0058] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0059] The internal memory 121 can be used to store one or more computer programs, which include commands. The processor 110 can enable the electronic device 100 to execute the application running method provided in some embodiments of the present application, as well as various applications and data management, etc. by running the above-mentioned commands stored in the internal memory 121. The internal memory 121 may include a code storage area and a data storage area. Among them, the data storage area can store data created during the use of the electronic device 100, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, a flash memory component, a universal flash storage (universal flash storage, UFS), etc. In some embodiments, the processor 110 can enable the electronic device 100 to execute the application running method provided in the embodiment of the present application, as well as other applications and data management by running commands stored in the internal memory 121, and / or commands stored in a memory provided in the processor 110.

[0060] The electronic device 100 can implement audio functions through the audio module 130, the speaker 130A, the microphone 130B, and the application processor, etc. For example, music playing, recording, etc. The audio module 130 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 130 can also be used to encode and decode audio signals. In some embodiments, the audio module 130 can be arranged in the processor 110, or some functional modules of the audio module 130 can be arranged in the processor 110.

[0061] The speaker 130A, also called a "horn", is used to convert audio electrical signals into sound signals.

[0062] Microphone 130B, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. The user can make a sound by approaching the microphone 130B with his mouth, and input the sound signal into the microphone 130B. For example, when the electronic device 100 in the embodiment of the present application processes the lock screen state, the microphone 130B can collect the user's voice data.

[0063] For example, in the embodiment of the present application, the electronic device 100 can convert the audio electrical signal into an ultrasonic signal through the speaker 130A and emit the ultrasonic signal outward. The electronic device 100 can also collect the ultrasonic signal returned by other electronic devices through the microphone 130B to measure the spatial position of other electronic devices relative to the electronic device 100, and the spatial posture information of the electronic device 100.

[0064] The communication function of the electronic device 100 can be realized through the antenna 1, the antenna 2 and the communication module 150.

[0065] The communication module 150 can provide wireless communication solutions including cellular, Wi-Fi, Bluetooth (BT), wireless data transmission modules (for example, 433MHz, 868MHz, 915MHz) applied to the electronic device 100. The communication module 150 can be one or more devices integrating at least one communication processing module. The communication module 150 receives electromagnetic waves via antenna 1 or antenna 2, filters and frequency modulates the electromagnetic wave signals, and sends the processed signals to the processor 110. The communication module 150 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through antenna 1 or antenna 2. For example, in an embodiment of the present application, the electronic device 100 can send a first control command or a third control command to a second electronic device or a third electronic device through wireless communication technology. The electronic device 100 can receive a second control command from a second electronic device through wireless communication technology.

[0066] The electronic device 100 implements the display function through a GPU, a display screen 140, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 140 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program commands to generate or change display information.

[0067] The display screen 140 is used to display images, videos, etc. The display screen 140 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 140, where N is a positive integer greater than 1. In the embodiment of the present application, the display screen 140 can be used to display a UI and receive user operations on the UI.

[0068] In some embodiments, the display screen 140 is provided with a pressure sensor, a touch sensor, and the like. The pressure sensor is used to sense pressure signals and can convert pressure signals into electrical signals. When a touch operation is applied to the display screen 140, the electronic device 100 detects the strength of the touch operation based on the pressure sensor. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor. The touch sensor, also known as a "touch panel", can form a touch screen, also known as a "touch screen", with the display screen 140. The touch sensor is used to detect touch operations applied thereto or thereabout. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can also be provided through the display screen 140.

[0069] The power module 160 can be used to supply power to various components included in the electronic device 100. In some embodiments, the power module 160 can be a battery, such as a rechargeable battery.

[0070] The input device 170 may include a keyboard, a mouse, etc. The keyboard is used to input English letters, numbers, punctuation marks, etc. into the electronic device 100, thereby issuing commands to the electronic device 100, inputting data, etc. The mouse is an indicator for the vertical and horizontal coordinates of the display system of the electronic device 100, and is used to input commands, etc. to the electronic device 100. Among them, the input device 170 can be connected to the electronic device 100 through a wired connection, for example, the input device 170 is connected to the electronic device 100 through a GPIO interface, a USB interface, etc. The input device 170 can also be connected to the electronic device 100 wirelessly, for example, the input device 170 is connected to the electronic device 100 through Bluetooth, infrared, etc.

[0071] Take the above electronic device 100 as a mobile phone as an example. The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device 100.

[0072] Figure 2 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0073] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0074] The application layer can include a series of application packages.

[0075] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0076] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0077] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a call manager, a resource manager, a notification manager, and the like.

[0078] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the display screen, capture the display screen, etc.

[0079] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0080] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display page can be composed of one or more views. For example, a display page including a text notification icon can include a view for displaying text and a view for displaying images.

[0081] The call manager is used to provide communication functions of the electronic device 100, such as management of call status (including connection, disconnection, etc.).

[0082] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0083] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the display in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0084] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system.

[0085] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0086] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0087] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0088] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0089] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0090] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0091] A 2D graphics engine is a drawing engine for 2D drawings.

[0092] The kernel layer is the layer between hardware and software. The kernel layer can contain display drivers, audio drivers, sensor drivers, etc.

[0093] For ease of understanding, the first electronic device is a mobile phone combined Figure 3 , introduces the voice interaction method provided in an embodiment of the present application. Figure 3 An example flow chart of a voice interaction method provided in an embodiment of the present application.

[0094] In S301, the mobile phone collects voice data of the user in a locked screen state, where the voice data is first voice data including preset voice features.

[0095] Among them, the mobile phone may include a microphone. The mobile phone can collect the user's voice data through the microphone. The above-mentioned lock screen state may include a screen-off lock screen state and a screen-on lock screen state. In other words, the mobile phone can execute a voice interaction method provided in an embodiment of the present application in the screen-off lock screen state or the screen-on lock screen state.

[0096] In the embodiment of the present application, the first voice data is voice data including preset voice features. The preset voice features include: the audio features of wind noise generated by speaking or breathing when the user is close to the microphone of the mobile phone. It should be understood that if the user speaks close to the microphone, the voice data collected by the mobile phone through the microphone may include the above-mentioned preset voice features. At this time, the voice data is the first voice data. If the user does not speak close to the microphone, the microphone of the mobile phone will not collect voice data including the preset voice features.

[0097] The preset voice feature may be configured in the mobile phone before it leaves the factory; or, the preset voice feature may be a voice feature of voice data collected by the mobile phone when the mobile phone reminds the user to approach the microphone.

[0098] In some embodiments, when the mobile phone is in the lock screen state, multiple voice data of the user can be collected in real time to determine whether the multiple voice data include preset voice features. After collecting the first voice data including the preset voice features, the mobile phone can perform voiceprint verification on the voiceprint features of the first voice data.

[0099] The preset voice features can be extracted from a plurality of audio sample data, where the plurality of audio sample data are voice data spoken by the user close to the microphone of the mobile phone. The mobile phone can extract the audio features of wind noise generated by the user's speaking or breathing based on the plurality of audio sample data.

[0100] The present application does not limit the specific type of audio features of wind noise. The audio features of wind noise may be, for example, decibel value features of wind noise, sound wave frequency features of wind noise, time domain features of audio signals of wind noise, frequency domain features of audio signals of wind noise, etc.

[0101] Taking the decibel value feature as an example, wind noise is the noise generated by the airflow generated when the user speaks or breathes hitting the microphone. The closer the user is to the microphone, the greater the force of the airflow generated when the user speaks or breathes hitting the microphone, the greater the noise generated, and the higher the decibel value. The farther the user is from the microphone, the smaller the force of the airflow generated when the user speaks or breathes hitting the microphone, the smaller the noise generated, and the lower the decibel value. Therefore, the preset voice feature can be, for example, that the decibel value of wind noise is greater than Xdb. If the decibel value of wind noise in the voice data collected by the mobile phone is greater than Xdb, the voice data is the first voice data including the preset voice feature. If the decibel value of wind noise in the voice data collected by the mobile phone is less than Xdb, the voice data is not the first voice data, and the mobile phone may not perform the voiceprint verification operation.

[0102] The mobile phone performs voiceprint verification on the voiceprint feature of the first voice data to identify whether the first voice data is sent by a target user, and the target user may be the owner of the mobile phone. The mobile phone can perform voiceprint verification on the voiceprint feature of the first voice data through voiceprint recognition technology.

[0103] Exemplarily, voiceprint features may include acoustic features, lexical features, prosodic features, and voice and accent features, etc., which are not limited in the present application. Voiceprint features of voice data sent by different users are different, so the voiceprint features of voice data sent by a user can be used to identify the user.

[0104] Before the mobile phone performs voiceprint verification on the voiceprint features of the first voice data, it can collect the voice of the target user and extract the proprietary voiceprint features that can characterize the target user from the voice. Proprietary voiceprint features have strong stability, are not easy to imitate, and are noise-resistant. The proprietary voiceprint features of the target user can be stored in a voiceprint model database. The mobile phone can extract the voiceprint features of the first voice data. The extracted voiceprint features are matched with the proprietary voiceprint features of the target user in the database for similarity, and the similarity between the voiceprint features of the first voice data and the proprietary voiceprint features of the target user is obtained. Based on the similarity, it is verified whether the first voice data is sent by the target user. Specifically, if the similarity between the two is higher than a preset value, the first voice data is sent by the target user, and the voiceprint feature verification of the first voice data passes; if the similarity between the two is lower than a preset value, the first voice data is not sent by the target user, and the voiceprint feature verification of the first voice data fails.

[0105] In S302, after the voiceprint feature of the first voice data passes the voiceprint verification, the mobile phone executes the voice control event corresponding to the first voice data.

[0106] If the voiceprint feature of the first voice data passes the voiceprint verification, it means that the first voice data is sent by the target user, and the mobile phone can execute the voice control event corresponding to the first voice data.

[0107] In some embodiments, after the mobile phone collects the first voice data, it can analyze the first voice data and identify the voice control event corresponding to the first voice data. Then, the mobile phone can execute the voice control event.

[0108] Exemplarily, the mobile phone may convert the first voice data into text based on voice recognition technology, and the mobile phone may understand the converted text based on natural language processing technology to identify the voice control event corresponding to the first voice data.

[0109] The mobile phone may have a built-in speech recognition engine and a natural language processing engine to parse the first speech data and identify the speech control event corresponding to the first speech data. In some embodiments, the speech recognition engine and the natural language processing engine may be deployed on a cloud server, and the mobile phone may transmit the first speech data to the cloud. After being processed by the cloud server, the mobile phone receives the speech control event that can be transmitted to the cloud.

[0110] As an example, the first voice data is "I want to listen to music", and the mobile phone recognizes that the voice control event corresponding to the first voice data is "play music". The mobile phone can execute the voice control event, for example, open an audio player and play music.

[0111] As another example, the first voice data is "how to get to xx place", and the mobile phone recognizes that the voice control event corresponding to the first voice data is "navigate to xx place". The mobile phone can execute the voice control event, for example, the mobile phone opens the map software and navigates to xx place.

[0112] In other embodiments, for example, the voice control event corresponding to the first voice data may be a voice-to-text event. After the mobile phone analyzes the first voice data and recognizes that the voice control event corresponding to the first voice data is a voice-to-text event, the voice data collected by the mobile phone may be converted into text.

[0113] For example, when the mobile phone is in the lock screen state, in response to receiving the first voice data "converted to text", the mobile phone can automatically unlock and open the voice-to-text interface. The microphone collects the voice data spoken by the user next, and the mobile phone voice data is converted to text on the mobile phone and displayed on the voice-to-text interface.

[0114] Combine the following Figure 4 Taking file transmission as an example, the differences between the voice interaction method provided in the embodiment of the present application and the current voice interaction method are introduced.

[0115] like Figure 4 As shown, in the current technology, when the mobile phone is in the locked screen state, the owner (target user) controls the mobile phone by voice. The owner can say "Hello, YoYo" or long press the power button to wake up the mobile phone. After that, the owner can say "Send xx file to A". In response to the first voice data of the owner, the mobile phone can voice prompt "Please help me unlock first". The owner can unlock the mobile phone by fingerprint, face, password, etc. In response to the user's unlocking operation, the mobile phone executes the voice control event of "Send xx file to A".

[0116] In an embodiment of the present application, when the mobile phone is in the locked screen state, the owner (target user) controls the mobile phone by voice, and the owner can speak the voice data "Send xx file to A" close to the microphone of the mobile phone. After the mobile phone collects the first voice data including the preset voice feature, it can perform a voiceprint verification on the voiceprint feature of the first voice data to determine whether the first voice data is sent by the owner. If the voiceprint feature of the first voice data passes the voiceprint verification, that is, the first voice data is sent by the owner, the mobile phone automatically executes the voice control event of "Send xx file to A". If the voiceprint feature of the first voice data does not pass the voiceprint verification, the mobile phone may not respond.

[0117] It can be seen that in the voice interaction method provided in the embodiment of the present application, the mobile phone analyzes the collected voice data when the screen is locked, and only executes the voice control event corresponding to the first voice data sent by the target user (machine owner). The target user (machine owner) can execute the voice control event without the target user (machine owner) unlocking the electronic device, which simplifies the voice interaction process and helps to improve the user experience.

[0118] After the voiceprint feature of the first voice data passes the voiceprint verification, the mobile phone can automatically unlock the screen, and execute the voice control event corresponding to the first voice data after the screen is unlocked. In some cases, the mobile phone can also execute the voice control event corresponding to the first voice data in the locked screen state without unlocking the screen.

[0119] The mobile phone can choose to automatically unlock the screen and then execute the voice control event or directly execute the voice control event without unlocking the screen according to the type of the voice control event corresponding to the first voice data.

[0120] As an example, for voice control events that require the display of the execution process or results, the mobile phone can automatically unlock the screen before executing. For voice control events that do not require the display of the execution process or results, the mobile phone can execute without unlocking the screen. For example, for the first voice data "I want to listen to music", the mobile phone can execute it without unlocking the screen. Specifically, the mobile phone displays the screen saver interface in the lock screen state. In response to the user's first voice data, the mobile phone still displays the screen saver interface, and opens the player in the background and plays music. For another example, for the first voice data "How to get to xx place", the mobile phone can automatically unlock the screen, and after unlocking the screen, execute the voice control event of opening the software and navigating. Specifically, the mobile phone displays the screen saver interface in the lock screen state. In response to the user's first voice data, the mobile phone unlocks the screen and displays the main interface. After that, the mobile phone automatically opens the map software and displays the application interface for navigating to xx place.

[0121] In other embodiments, the mobile phone can be connected to a second electronic device. When the mobile phone is in lock screen state, it can collect the user's first voice data. If the voiceprint feature of the first voice data passes the voiceprint verification, the mobile phone can control the second electronic device to execute the voice control event corresponding to the first voice data.

[0122] The second electronic device may include a server, a mobile phone, a tablet computer, a laptop computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a smart home device (for example, a smart TV, a smart screen, a large screen, a smart speaker, a smart air conditioner, a smart fan, a smart desk lamp, etc.), a personal digital assistant (PDA), a wearable device (for example, a smart watch, a smart bracelet, etc.), a vehicle-mounted device, a virtual reality device, etc., and the embodiments of the present application do not impose any restrictions on this.

[0123] The mobile phone can establish a wireless communication connection with the second electronic device. For example, the mobile phone can establish a short-range wireless communication connection with the second electronic device. Specifically, the mobile phone can establish a wireless LAN 802.11 (wireless fidelity, Wi-Fi) connection, Bluetooth (Bluetooth, BT) connection, short-range communication (near field communication, NFC), etc. with the second electronic device. In some embodiments, the mobile phone can also establish an account connection with the second electronic device.

[0124] The first voice data may include the name and code of the second electronic device, and is used to indicate voice data of a voice control event executed by the second electronic device. The first voice data may be, for example, "PC shutdown", "PC opens voice-to-text software", "large screen search for xxx TV series", "desk lamp dims the light", "speaker plays the next song", etc.

[0125] In some embodiments, when the mobile phone recognizes that the first voice data indicates execution by the second electronic device, the mobile phone can recognize the voice control event corresponding to the first voice data and send a first control command to the second electronic device. In response to receiving the first control command, the second electronic device can execute the voice control event corresponding to the first voice data. The first control command can include the voice control event recognized by the mobile phone. In response to receiving the first control command, the second electronic device can execute the voice control event in the first control command.

[0126] For example, the first voice data is "the speaker plays the next song". The mobile phone recognizes that the first voice data indicates that it is executed by the speaker. Based on the first voice data, the mobile phone can identify the voice control event "play the next song" corresponding to the first voice data according to the voice recognition technology and the natural language understanding technology. The mobile phone sends the first control command to the speaker. The first control command is used to instruct the speaker to execute the voice control event of "play the next song". The first control command may include the voice control event of "play the next song". The speaker can receive the first control command and execute the voice control event of playing the next song.

[0127] For another example, the first voice data is "big screen search xx TV series", and the mobile phone recognizes that the first voice data indicates that it is executed by the big screen. Based on the first voice data, the mobile phone can recognize that the voice control event corresponding to the first voice data is "search xx TV series". The mobile phone sends a first control command including "search xx TV series" to the big screen. The big screen receives the first control command and executes the voice control event of searching for xx TV series.

[0128] In other embodiments, after the mobile phone recognizes that the first voice data indicates that a corresponding voice control event is to be executed by a second electronic device, the mobile phone can directly send the first voice data to the second electronic device, and the second electronic device recognizes the voice control event corresponding to the first voice data and executes the voice control event corresponding to the first voice data.

[0129] The second electronic device may have the ability to identify the voice control event corresponding to the first voice data. As an example, the second electronic device may obtain an instruction executable by the second electronic device based on the first voice data. The second electronic device executes the instruction to execute the voice control event corresponding to the first voice data.

[0130] Take the first voice data in the previous text as an example. For example, the mobile phone recognizes the first voice data "the speaker plays the next song" and instructs the speaker (i.e., the second electronic device) to execute. The mobile phone can send the first voice data to the speaker, instructing the speaker to execute the voice control event corresponding to the first voice data. The speaker recognizes the voice control event of "play the next song" based on the first voice data. The speaker can execute the voice control event of playing the next song.

[0131] For another example, the mobile phone recognizes the first voice data "big screen search xx TV series" and instructs the big screen (i.e., the second electronic device) to execute. The mobile phone can send the first voice data to the big screen, instructing the big screen to execute the voice control event corresponding to the first voice data. The big screen recognizes the voice control event of "search xx TV series" based on the first voice data. The big screen can execute the voice control event of searching for xx TV series.

[0132] In other embodiments, as mentioned above, the voice control event corresponding to the first voice data may be a voice-to-text event. The mobile phone recognizes that the voice control event corresponding to the first voice data is a voice-to-text event, and the mobile phone recognizes that the first voice data indicates that the voice-to-text event is executed by the second electronic device. The mobile phone may send a first control command to the second electronic device. The first control instruction is used to instruct the second electronic device to convert the second voice data sent by the first electronic device into text. The mobile phone sends the second voice data collected by the mobile phone to the second electronic device. In response to receiving the second voice data, the second electronic device converts the second voice data into text.

[0133] As an example, Figure 5 As shown. The mobile phone establishes a wireless communication connection with the PC, and the mobile phone responds to the user's first voice data "PC to text" in the lock screen state. The mobile phone recognizes that the first voice data indicates that the PC executes a voice-to-text event. The mobile phone sends a first control command to the PC, which is used to instruct the PC to execute a voice-to-text event. The PC receives the first control command and opens the voice-to-text interface. The mobile phone microphone collects the second voice data "Record today's meeting minutes..." spoken by the user next, and sends the second voice data to the PC, which converts the second voice data into text. The text on the PC side displays: Record today's meeting minutes.

[0134] In other embodiments, after the mobile phone sends the first voice data to the second electronic device, the second electronic device may identify the voice control event corresponding to the first voice data. When the second electronic device identifies that the voice control event corresponding to the first voice data is a voice-to-text event, the second electronic device may send a second control command to the first electronic device. The second control command is used to instruct the mobile phone to send the voice data collected by the mobile phone to itself. In response to receiving the second control command, the mobile phone sends the second voice data collected by the mobile phone to the second electronic device. In response to receiving the second voice data, the second electronic device converts the second voice data into text.

[0135] Specifically, continue with Figure 5 For example. The mobile phone directly sends the first voice data "PC to text" to the PC. The PC recognizes that the voice control event corresponding to the voice data is a voice-to-text event. The PC opens the voice-to-text interface and sends a second control command to the mobile phone, which is used to instruct the mobile phone to send the collected voice data that needs to be converted into text to itself. The mobile phone microphone collects the user's next voice data "Record today's meeting minutes..." and sends the voice data to the PC. The PC converts the voice data into text and displays it on the voice-to-text interface on the PC side.

[0136] In some cases, the second electronic device can display a text box for editing information, and when the cursor is in the text box, the second electronic device can automatically execute a voice-to-text event and insert the converted text into the text box. For example, the PC displays a voice-to-text interface, and the cursor is in the text box. When the mobile phone is in the locked screen state, it collects the owner's first voice data "Tomorrow's work plan..." and sends it to the PC. The PC receives the first voice data "Tomorrow's work plan..." sent by the mobile phone, automatically executes a voice-to-text event, converts the voice into text, and displays on the voice-to-text interface on the PC side: Tomorrow's work plan...

[0137] For example, Figure 6 As shown, the large screen displays a search box, and the cursor is in the search box. When the mobile phone is in the locked screen state, the first voice data "XX movie" of the owner is collected. The mobile phone can convert the first voice data into text, and send the text to the large screen, and insert it into the search box of the large screen as text input.

[0138] The mobile phone can establish wireless communication connections with multiple third electronic devices, and the third electronic devices may include servers, mobile phones, tablet computers, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), handheld computers, netbooks, smart home devices (for example, smart TVs, smart screens, large screens, smart speakers, smart air conditioners, smart fans, smart desk lamps, etc.), personal digital assistants (PDAs), wearable devices (for example, smart watches, smart bracelets, etc.), vehicle-mounted devices, virtual reality devices, etc. The embodiments of the present application do not impose any restrictions on this.

[0139] When the mobile phone is in lock screen state, the first voice data is collected. If the voiceprint features of the first voice data pass the voiceprint verification, the mobile phone can send a third control command corresponding to the first voice data to a target device selected / determined by the mobile phone among multiple third devices, to instruct the target device to execute the voice control event corresponding to the first voice data.

[0140] As an example, a mobile phone can establish a wireless communication connection with a bedside lamp, a fan, a smart speaker, and a smart TV (television, TV). The mobile phone can select / determine a target device from the above four third electronic devices, for example, the target device can be a bedside lamp. When the mobile phone is in the locked screen state, the user's first voice data is collected. If the voiceprint feature of the first voice data passes the voiceprint verification, the mobile phone can send a third control command to the bedside lamp, and the bedside lamp can receive the third control command and execute the voice control event.

[0141] Before the mobile phone sends the third control command to the target device, the mobile phone can select / determine the target device from multiple third electronic devices based on the relative position information between the multiple third electronic devices and the mobile phone and the spatial posture information of the mobile phone.

[0142] As an example, the mobile phone can obtain the position information of the bedside lamp, fan, smart speaker, and TV relative to the mobile phone, for example, the distance of the third electronic device relative to the mobile phone can be obtained. The mobile phone can also obtain the spatial posture information of the mobile phone, which is used to indicate the direction of the mobile phone from the bottom to the top. In some embodiments, the mobile phone can detect the position information of the third electronic device relative to the mobile phone through ultrasonic sensing technology, and the mobile phone can also sense the spatial posture information of the mobile phone through ultrasonic sensing technology.

[0143] like Figure 7 As shown, the angle between the direction from the bottom to the top of the phone and the direction of the short side of the phone is 90°. The phone can determine that the fan located in the direction from the bottom to the top of the phone is the target device. The angle between the direction from the bottom to the top of the phone and the smart TV is 40°, and the angle between the short side of the phone and the smart TV is 50°. When the phone is rotated 40° to the right, the angle between the short side of the phone and the smart TV is 90°. At this time, the phone can determine that the smart TV located in the direction from the bottom to the top of the phone is the target device. Similarly, when the phone is rotated 30° to the left, the phone can determine that the bedside lamp located in the direction from the bottom to the top of the phone is the target device.

[0144] Specifically, the mobile phone may store a plurality of preset control distances of the third electronic device relative to the mobile phone. Figure 7 As shown, the preset control distance between the bedside lamp and the mobile phone is 100cm, the preset control distance between the fan and the mobile phone is 100cm, the preset control distance between the TV and the mobile phone is 400cm, and the preset control distance between the smart speaker and the mobile phone is 80cm. Each of the above-mentioned third electronic devices may include a microphone and a speaker for receiving ultrasonic waves sent by the mobile phone and sending ultrasonic waves with an identity mark to the mobile phone. The mobile phone can determine the distance of each third electronic device relative to the mobile phone based on the received ultrasonic waves with an identity mark. For example, the mobile phone can receive an ultrasonic wave with an identity mark sent by a fan, and determine that the relative distance between the fan and the mobile phone is 80cm, which is less than the preset control distance (that is, the fan is within the controllable range of the mobile phone).

[0145] The mobile phone can obtain its own spatial posture information, such as Figure 7As shown, the mobile phone obtains spatial posture information indicating that the third electronic device in the direction from the bottom of the mobile phone to the top is a fan, and the fan is within the controllable range of the mobile phone. The mobile phone can select / determine the fan as the target device. The mobile phone can collect the first voice data of user A, user B (the owner), and user C, but only controls the fan to execute the voice control event corresponding to the first voice data of user B.

[0146] In some cases, the relative distance between the third device and the mobile phone is greater than the preset control distance. Even if the spatial posture information of the mobile phone indicates that the mobile phone is pointing at the third device, the mobile phone cannot control the third device to perform a voice control event.

[0147] For example, the mobile phone determines that the distance between the bedside lamp and the mobile phone is 120 cm, which is greater than the preset control distance (i.e., the bedside lamp is not within the controllable range of the mobile phone). If the mobile phone's spatial posture information indicates that the mobile phone is pointing at the bedside lamp, since the bedside lamp is not within the controllable range of the mobile phone, the mobile phone cannot send the third control command corresponding to the first voice message to the bedside lamp (i.e., it cannot control the bedside lamp to execute the voice control event).

[0148] In some cases, when the spatial posture information of the mobile phone indicates a direction from the bottom of the mobile phone to the top, and there are multiple third electronic devices, the mobile phone can select a third electronic device that is closer to the mobile phone as the target device.

[0149] Combine the following Figure 8 , Fig. 9 and Fig.10 , the voice interaction method provided in the embodiment of the present application is introduced. Figure 8 The figure shows a flow chart of device A executing the voice control event of the owner when the device A is in the locked screen state.

[0150] like Figure 8 As shown, device A, such as a mobile phone, detects voice data collected by the microphone when it is not unlocked (i.e., in the lock screen state), detects whether there is airflow and voice input in the voice data, and determines whether the user is close to the microphone to speak based on the airflow and voice in the voice data. Specifically, device A determines whether the voice data includes the audio characteristics of wind noise generated by speaking or breathing when the user is close to the microphone of device A based on the airflow and voice in the voice data.

[0151] If the voice data includes the above-mentioned audio features, the voiceprint features of the voice data are verified to identify whether the language data is sent by the owner (target user). Only the owner can wake up device A for voice command interaction.

[0152] If the voiceprint feature of the voice data passes the voiceprint verification, that is, the voice data is sent by the device owner, the voice data is parsed to obtain the voice control event corresponding to the voice data. Device A does not need to be unlocked to execute the voice control event corresponding to the voice data.

[0153] Fig. 9 The figure shows a flow chart of device A controlling device B to execute the voice control event of the owner when the device A is in the locked screen state. Fig. 9 As shown, after device A and device B are connected through an account or short distance, when device A is not unlocked (i.e., in the lock screen state), in response to the user speaking close to device A, first voice data is collected, and the first voice data mentions a voice control event for device B. On the device A side, the first voice data is converted into text, and based on the instructions obtained based on the text, device A forwards the instructions to device B. The instructions are received on the device B side, and the instructions are executed to execute the voice control event.

[0154] Fig.10 The figure shows that device A is in the locked screen state, and device A controls device B to execute the owner's voice control event. Fig.10 As shown, after device A and device B are connected through an account or short-distance connection, when device A is not unlocked (i.e., in the lock screen state), device A can detect the relative position between the two devices through position detection technology such as ultrasonic sensing technology. In response to the user speaking close to device A, device A determines that device B is the target device, and device A collects the first voice data. Device A can convert the first voice data into instructions or text and send it to device B. Device B can receive the instruction, execute the voice control event corresponding to the instruction, or use the text sent by device A as input.

[0155] It can be seen that the voice interaction method provided in the embodiment of the present application integrates the self-learning owner's voiceprint recognition technology, so that the first electronic device only executes the owner's voice control events; it integrates the device connection and location awareness technology, and can perform voice wake-up of the owner's microphone across devices to control the second electronic device or the third electronic device to execute the owner's voice control events.

[0156] The present application embodiment provides an electronic device, which includes: a memory, a microphone, and one or more processors. The microphone is used to collect voice data. The memory is coupled to the processor, and the memory is used to store computer program code. The computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute the various functions or steps executed by the mobile phone in the above method embodiment. The structure of the electronic device can refer to Figure 1 The structure of the electronic device 100 is shown.

[0157] The present application also provides a computer storage medium, which includes computer instructions. When the computer instructions are in the above electronic device (such as Figure 1 When the method is executed on the electronic device 100 shown in the figure, the electronic device executes each function or step in the above method embodiment.

[0158] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step in the above method embodiment.

[0159] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected by lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device, a microphone of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0160] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0161] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0162] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0163] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0165] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A voice interaction method, characterized in that: Applied to a first electronic device, the first electronic device includes a microphone, and the method includes: The first electronic device performs the following operations in the locked screen state: Collecting voice data of the user, where the voice data is first voice data including preset voice features, where the preset voice features include: audio features of wind noise generated by the user speaking or breathing when the user approaches the microphone of the first electronic device; The first electronic device establishes a wireless communication connection with a plurality of third electronic devices. After the voiceprint feature of the first voice data passes the voiceprint verification, the first electronic device obtains the position information of each third electronic device in the plurality of third electronic devices relative to the first electronic device, and the spatial posture information of the first electronic device; wherein the spatial posture information is used to indicate the direction in which the first electronic device points from the bottom to the top of the first electronic device; the first electronic device determines the target device selected / determined by the first electronic device from the plurality of third electronic devices according to the position information and the spatial posture information; The first electronic device instructs the target device to execute a voice control event corresponding to the first voice data.

2. The method according to claim 1, characterized in that The method further comprises: The first electronic device establishes a wireless communication connection with the second electronic device. After the voiceprint feature of the first voice data passes the voiceprint verification, the first electronic device recognizes that the first voice data indicates that the second electronic device executes the voice control event, and sends a first control command corresponding to the first voice data to the second electronic device. The first control command is used to instruct the second electronic device to execute the voice control event.

3. The method according to claim 2, characterized in that The voice control event is a voice-to-text event, and the first control command is specifically used to instruct the second electronic device to convert the second voice data from the first electronic device into text; the method further includes: The first electronic device recognizes that the voice control event indicated by the first voice data is a voice-to-text event, and sends the second voice data collected by the first electronic device to the second electronic device.

4. The method according to claim 1, characterized in that: The method further comprises: The first electronic device establishes a wireless communication connection with the second electronic device. After the voiceprint feature of the first voice data passes the voiceprint verification, the first electronic device recognizes that the first voice data indicates that the second electronic device executes the voice control event, and sends the first voice data to the second electronic device to instruct the second electronic device to recognize the first voice data and execute the voice control event corresponding to the first voice data.

5. The method according to claim 4, characterized in that The voice control event is a voice-to-text event; after sending the first voice data to the second electronic device, the method further includes: The first electronic device receives a second control command from the second electronic device, where the second control command is used to instruct the first electronic device to send voice data collected by the first electronic device to the second electronic device, and the second control command is sent after the second electronic device recognizes that the voice control event indicated by the first voice data is a voice-to-text event; In response to the second control command, the first electronic device sends the second voice data collected by the first electronic device to the second electronic device.

6. The method according to any one of claims 1 to 5, characterized in that: The first electronic device instructing the target device to execute a voice control event corresponding to the first voice data includes: After the voiceprint feature of the first voice data passes the voiceprint verification, the first electronic device sends a third control command corresponding to the first voice data to a target device selected / determined by the first electronic device among the multiple third electronic devices; wherein the third control command is used to instruct the target device to execute the voice control event.

7. The method according to claim 1, characterized in that After the first electronic device collects the user's voice data in a locked state, the method further includes: The first electronic device extracts a voiceprint feature of the first voice data according to the first voice data; The first electronic device compares the voiceprint feature with a target user's unique voiceprint feature to identify whether the first voice data is sent by the target user, wherein the target user's unique voiceprint feature is pre-stored in the first electronic device; Wherein, if the first voice data is sent by the target user, the voiceprint feature of the first voice data passes the voiceprint verification.

8. An electronic device, characterized in that: The electronic device comprises: a memory, a microphone and one or more processors; the memory is coupled to the processor; wherein the microphone is used to collect voice data; the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 7.

Images