Voice interaction method, electronic device, and storage medium

Abstract

The application provides a voice interaction method, an electronic device and a computer readable storage medium. The voice interaction method comprises: performing voice interaction with a first user, and collecting a first audio signal in an angle range where the first user is located and a second audio signal in an angle range where a second user is located in a voice collection period of the voice interaction, the second user being a user who performs voice interaction with the electronic device in a set history period; judging whether a start time of a first voice signal in the first audio signal is located within a first period, and determining a target voice signal from the first voice signal and a second voice signal according to a judgment result, the second voice signal being a voice signal contained in the second audio signal, and the first period being a period after a start time of the voice collection period by a first time length; and responding to the target voice signal. The application can accurately determine a target interaction person in a multi-person interaction scene.

Description

Technical Field

[0001] This application relates to the field of software technology, and more particularly to a voice interaction method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the development of audio processing technology and artificial intelligence (AI), more and more electronic devices (such as smart speakers and smart robots) are equipped with voice interaction capabilities. During voice interaction, electronic devices need to capture the user's voice. To improve the accuracy of voice capture, electronic devices will capture sound from the target direction (i.e., the direction in which the person interacting is currently located) and suppress sound from other directions to reduce the interference of environmental noise on the user's voice signal.

[0003] In some situations, electronic devices may be in multi-user interaction scenarios. That is, besides the current user interacting with the electronic device, there are other users around the device who may interact with it via voice (referred to as "potential users"). Because the electronic device only captures voice from the direction of the current user, the voices of potential users are suppressed. When a potential user speaks, the electronic device cannot perceive the content of their voice and therefore cannot respond to them. Summary of the Invention

[0004] Some embodiments of this application provide a voice interaction method, an electronic device, and a computer-readable storage medium. The following describes this application from multiple aspects, and the embodiments and beneficial effects of the following aspects can be referred to each other.

[0005] In a first aspect, embodiments of this application provide a voice interaction method for an electronic device. The method includes: engaging in voice interaction with a first user, and during a voice acquisition period of the voice interaction, acquiring a first audio signal within an angular range of the first user and a second audio signal within an angular range of the second user, wherein the second user is a user engaging in voice interaction with the electronic device during a set historical period; determining whether the start time of the first voice signal in the first audio signal falls within a first time period, and determining a target voice signal from the first and second voice signals based on the determination result, wherein the second voice signal is a voice signal contained in the second audio signal, and the first time period is a period of time after the start time of the voice acquisition period for a first duration; and responding to the target voice signal.

[0006] According to the embodiments of this application, the target interactor can be determined among the first user and the second user based on the first duration, which can reasonably take into account the voice interaction needs of the first user and the second user, accurately determine the target interactor, and thus improve the user experience in multi-user interaction scenarios.

[0007] In some embodiments, at least one of the start time of the first speech signal and the start time of the second speech signal is located within a first time period; determining the target speech signal from the first speech signal and the second speech signal based on the determination result includes: if the start time of the first speech signal is located within the first time period, then the first speech signal is determined as the target speech signal; otherwise, the target speech signal is determined based on the temporal overlap between the second speech signal and the first speech signal.

[0008] According to the implementation method of this application, as long as the first user speaks within the first time period, the electronic device will identify the first user as the target interaction person (that is, keep the first user as the current interaction person) so as to give priority to meeting the voice interaction needs of the first user.

[0009] If the first user does not speak within the first time period, it is considered that the first user has a low willingness to interact, and the second user may be identified as the target interaction person in order to take into account the second user's voice interaction needs.

[0010] In some embodiments, determining the target speech signal based on the temporal overlap between the second speech signal and the first speech signal includes: if the second speech signal and the first speech signal overlap in time, then the first speech signal is determined as the target speech signal; if the second speech signal and the first speech signal do not overlap in time, then the second speech signal is determined as the target speech signal.

[0011] In some implementations, the first duration is determined based on the interaction willingness value P of the first user and / or the number of interactions M between the first user and the electronic device within a set time period, wherein the interaction willingness value P is used to characterize the first user's willingness to engage in voice interaction with the electronic device.

[0012] In some implementations, the interaction willingness value P is determined based on the first user's facial angle and / or the distance between the first user and the electronic device.

[0013] In some implementations, the first duration is k1×P+k2×min{M,n}, where k1 and k2 are preset constants, and n is an integer between 3 and 6.

[0014] In some implementations, the second voice signal does not include the wake word of the electronic device.

[0015] In some implementations, the historical time period is defined as the second duration preceding the start time of the voice acquisition period.

[0016] Secondly, embodiments of this application provide an electronic device, including: a memory for storing instructions executable by one or more processors of the electronic device; and a processor, which, when executing the instructions in the memory, causes the electronic device to perform the voice interaction method provided in any embodiment of the first aspect of this application. The beneficial effects achievable through this second aspect can be referred to in the context of the beneficial effects of any embodiment of the first aspect of this application, and will not be repeated here.

[0017] Thirdly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the voice interaction method provided in any embodiment of the first aspect of this application. The beneficial effects achievable through this third aspect can be referenced to the beneficial effects of any embodiment of the first aspect of this application, and will not be repeated here. Attached Figure Description

[0018] Figure 1 This is an exemplary application scenario of the embodiments of this application;

[0019] Figure 2 An exemplary structural diagram of the electronic device provided in the embodiments of this application;

[0020] Figure 3 An exemplary flowchart of the voice interaction method provided in the embodiments of this application;

[0021] Figure 4 A timing diagram illustrating the voice interaction between the electronic device provided in this application embodiment and the current user;

[0022] Figure 5 An exemplary flowchart of the user voice acquisition process provided in this application embodiment;

[0023] Figure 6 A schematic diagram illustrating the angle range of the user as provided in an embodiment of this application;

[0024] Figure 7 An exemplary flowchart of a target speech signal determination method provided in an embodiment of this application;

[0025] Figure 8A A schematic diagram of the target speech signal determination rules provided in the embodiments of this application. Figure 1 ;

[0026] Figure 8B A schematic diagram of the target speech signal determination rules provided in the embodiments of this application. Figure 2 ;

[0027] Figure 9 A schematic diagram of the target speech signal determination rules provided in the embodiments of this application. Figure 3 ;

[0028] Figure 10A A schematic diagram of the target speech signal determination rules provided in the embodiments of this application. Figure 4 ;

[0029] Figure 10B A schematic diagram of the target speech signal determination rules provided in the embodiments of this application. Figure 5 ;

[0030] Figure 11 This is another exemplary application scenario of the embodiments of this application;

[0031] Figure 12 These are schematic diagrams of voice interaction methods in some embodiments;

[0032] Figure 13 This is a schematic diagram of a voice interaction method in some other embodiments;

[0033] Figure 14 A block diagram of an electronic device provided according to an embodiment of this application is shown;

[0034] Figure 15 A schematic diagram of the structure of a System on Chip (SOC) provided in an embodiment of this application is shown. Detailed Implementation

[0035] The specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0036] To facilitate understanding, the audio processing technologies that may be involved in this application will be introduced first.

[0037] (1) Beamforming: Beamforming technology can determine the direction of a sound source. Beamforming technology relies on a microphone array. When a sound source emits sound, the sound signals received by each microphone (i.e., each sound acquisition channel) in the microphone array are delayed. Beamforming technology can locate the sound source by using the delay information of each channel (e.g., determine the direction angle, elevation angle, and distance of the sound source).

[0038] Beamforming technology can also capture sound within a target angle. Beamforming technology can perform phase shifting, weighting, and other processing on the sound signals of each channel in a microphone array, thereby enhancing the sound signal within the target angle and suppressing sound signals from other directions, so as to achieve sound capture within the target angle (e.g., within ±30° directly in front of the electronic device).

[0039] (2) Voice Activity Detection (VAD), also known as "voice boundary detection" or "endpoint detection," is a technique that distinguishes between speech and non-speech signals in an audio signal. It identifies the start and end points of the speech signal, thus separating it from the audio signal. This allows subsequent speech recognition to focus solely on the speech signal, improving its accuracy.

[0040] This application provides a voice interaction method for determining a suitable target person in a multi-person interaction scenario to meet the user's voice interaction needs.

[0041] In this application, the electronic device can be any form of device, such as a smart speaker, in-vehicle system, large-screen device, mobile phone, tablet, wearable device, camera, etc., as long as it has voice interaction function. In the following text, a smart robot (e.g., Xiaoyi Elf) is used as an example of an electronic device.

[0042] Figure 1 Exemplary application scenarios of embodiments of this application are illustrated. Figure 1 In the scenario, electronic device 100 (specifically, an intelligent robot) is engaging in voice interaction (referred to as "interaction") with user A. That is, user A is the current person interacting with electronic device 100. For ease of interaction, user A is positioned directly in front of electronic device 100. The content of the interaction between electronic device 100 and user A is, for example:

[0043] User A: "Xiao Yi Xiao Yi";

[0044] Electronic devices: "I am here";

[0045] User A: "Do you have a favorite animal?"

[0046] Electronic devices: "I like furry animals, they look so warm."

[0047] User A: "Then you must like this panda toy."

[0048] Electronic device: "Hmm, I like this more than I like you";

[0049] User A: "xxxxx..."

[0050] The above example illustrates a user-initiated voice interaction. That is, after user A actively says the wake-up word "Hey Celia," electronic device 100 is activated and begins interacting with user A. In other examples, voice interaction can also be initiated by electronic device 100. For instance, when electronic device 100 observes user A staring at it for an extended period (e.g., 10 seconds), it can play a preset voice message (e.g., "Do you have any questions for me?") to proactively initiate a voice interaction with user A.

[0051] Continue to refer to Figure 1 User B is also present in the vicinity of electronic device 100. User B is the user who just ended a voice interaction with electronic device 100 (for example, half a minute ago). This application does not limit the reason why user B ended the voice interaction with electronic device 100. For example, user B does not respond to the voice played by electronic device 100, thus actively ending the voice interaction with electronic device 100; or, during the voice interaction with user B, electronic device 100 hears a wake word from user A (for example, "Hey Celia"), thereby ending the voice interaction with user B and starting the voice interaction with user A.

[0052] Since User B has just interacted with Electronic Device 100 via voice, User B may continue to interact with Electronic Device 100 via voice. That is, User B is a potential user of Electronic Device 100. However, in some embodiments, when Electronic Device 100 interacts with User A via voice, it suppresses sounds from directions other than User A's location to improve the accuracy of voice acquisition. For example, the electronic device only acquires sounds within ±30° directly in front, while suppressing sounds from other directions. Thus, when User B is speaking, Electronic Device 100 cannot acquire User B's voice and therefore cannot perceive User B's interaction needs.

[0053] To address this, this application provides a voice interaction method for identifying a target user in a multi-user interaction scenario, thereby improving the user experience. Specifically, during a voice interaction with user A, the electronic device 100 collects audio signals from both the direction of user A (as the "current user" or "first user") and the direction of user B (as a "potential user" or "second user"). Based on the collected audio signals, the electronic device 100 determines whether the time user A begins speaking falls within a set priority waiting period (also known as the "first period"), and determines the target user (the user to whom the electronic device 100 will respond) between user A and user B based on the determination result. For example, if user A begins speaking within the set priority waiting period, user A is identified as the target user; otherwise, user B may be identified as the target user.

[0054] In this application, the target interactor can be determined from user A and user B based on the priority waiting period. This can reasonably take into account the voice interaction needs of the current interactor (e.g., user A) and the potential interactor (e.g., user B), accurately determine the target interactor, and thus improve the user experience in multi-person interaction scenarios.

[0055] The following describes specific embodiments of this application. In the following embodiments, an intelligent robot is used as an example of electronic device 100. However, it is understood that this application is not limited thereto.

[0056] Figure 2 An exemplary structural diagram of the electronic device 100 provided in this embodiment is shown. (Refer to...) Figure 2 The electronic device 100 includes a processor 110, a camera 120, a microphone 130, a speaker 140, a communication module 150, a memory 160, and a sensor 170.

[0057] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0058] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors. The processor can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0059] Camera 120 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 120, where N is a positive integer greater than 1.

[0060] Microphone 130, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. Electronic device 100 can be equipped with multiple (e.g., three, four, or more) microphones 130 to form a microphone array. As a voice front-end device, the microphone array, in addition to acquiring sound signals, can also perform functions such as sound signal acquisition, noise reduction, sound source identification, and directional recording.

[0061] Speaker 140, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music or make hands-free calls through speaker 140.

[0062] The communication module 150 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. Through the communication module 150, the electronic device can communicate with other devices (e.g., a cloud server).

[0063] The memory 160 can be used to store computer executable program code, which includes instructions. The memory 160 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.). Furthermore, the memory 160 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the memory 160 and / or instructions stored in memory disposed in the processor. The instructions stored in the memory 160 may include instructions that, when executed by at least one of the processors 110, cause the electronic device 100 to implement the voice interaction method provided in the embodiments of this application.

[0064] Sensor 170 may include a distance sensor and a proximity light sensor. The distance sensor is used to measure distance. For example, the distance sensor can measure distance using infrared or laser. The proximity light sensor may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. Electronic device 100 emits infrared light outward through the LED. Electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near electronic device 100. When insufficient reflected light is detected, electronic device 100 can determine that there is no object near electronic device 100. Electronic device 100 may use proximity light sensor 180G to detect whether there is a user nearby.

[0065] In addition, the electronic device may include a rotating mechanism to enable the device to turn like a servo motor. For example, the rotating mechanism allows the electronic device to rotate from an angle facing user A to an angle facing user B.

[0066] In addition, electronic devices may include several functional units. For example, ASR algorithm units, sound source localization algorithm units (e.g., beamforming algorithm units), sound source suppression algorithm units, and other speech recognition units, as well as face recognition algorithm units and other visual recognition units.

[0067] The following combination Figure 1 The illustrated scenario describes the specific process of the voice interaction method provided in this embodiment. (See reference...) Figure 3 The voice interaction method provided in this embodiment includes the following steps:

[0068] S110: The electronic device engages in voice interaction with User A (also known as the "current interactor" or "first user"). During the voice acquisition phase of this voice interaction, the electronic device acquires the audio signal Autio_A within the angular range of User A's location and the audio signal Autio_B within the angular range of User B (also known as the "potential interactor" or "second user").

[0069] Figure 4 The diagram shows a timing sequence of voice interaction between an electronic device and User A. The white boxes represent User A's speech, and the gray boxes represent content played by the electronic device. (Reference) Figure 4 The voice interaction process between the electronic device and user A includes alternating voice playback and voice acquisition phases. During the voice playback phase, the electronic device plays its own audio message; after playing the audio message, the electronic device enters the voice acquisition phase to monitor the user's voice. When the electronic device determines that the user has finished speaking, it ends the voice acquisition phase to proceed to the next voice playback phase. In other embodiments, the voice acquisition phase may end when the electronic device does not detect user speech within a set time (e.g., 8 seconds). Typically, during the voice playback phase, the electronic device may mute its microphone to avoid capturing external sounds.

[0070] In this embodiment, during each voice acquisition stage, the electronic device determines whether there is a potential human to interact with. If a potential human to interact with is present, the electronic device acquires audio signals from both the current human's directional range and the potential human's directional range; if no potential human to interact with is present, the electronic device acquires only audio signals from the current human's directional range.

[0071] The following is based on the current stage of voice acquisition (i.e. Figure 4 Taking voice acquisition stage 3 as an example, the process of electronic devices acquiring user voice is introduced. The start time of the current voice acquisition stage is T. S . refer to Figure 5 The process of electronic devices collecting user voice data includes the following steps:

[0072] S111: The electronic device has been identified as having a potential human interaction.

[0073] Potential interactors are users who have engaged in voice interaction with the electronic device within a defined historical time period P1. Because they have previously interacted with the electronic device via voice, potential interactors may still be able to engage in voice interaction with it. In this embodiment, the defined historical time period P1 is T. S The second time interval T2 before the current time. That is, the starting time of historical time interval P1 is set to T. S -T2, terminal time is T SIn other words, the potential interactor is a user who has interacted with the electronic device within the most recent second duration period. At this time, the potential interactor has a higher probability of engaging in voice interaction with the electronic device. This embodiment does not limit the specific value of the second duration. In some examples, the second duration is 0.5 to 2 minutes, for example, 0.5 minutes, 1 minute, and 1.3 minutes. It should be noted that the numerical range in this application includes end values. For example, the data range of 0.5 to 2 minutes includes 0.5 minutes and 2 minutes.

[0074] The following example, using User B, illustrates how electronic devices determine the presence of potential users. During a voice interaction with User A, the electronic device determines that User B is nearby. For instance, the device can determine this through image recognition or by using sensors (e.g., proximity sensors). In some examples, the electronic device identifies users within a set distance (e.g., within 3 meters) as being in its vicinity.

[0075] Furthermore, the electronic device identifies user B's identity, for example, through facial recognition, voiceprint recognition, or other methods. Then, the electronic device can determine whether user B is a potential user by querying its stored voice interaction records. For example, the voice interaction records store identifiers of users who have recently interacted with the electronic device via voice, as well as the start and end times of each interaction.

[0076] In this embodiment, the historical time period P1 is set to [T S -60s, T S According to historical interaction records, the most recent time period for user B's interaction with the electronic device is [T]. S -80s, T S [-40s] Therefore, User B has engaged in voice interaction with the electronic device within the set historical time period P1, thereby the electronic device determines that User B is a potential interactor.

[0077] S112: The electronic device determines the angular range (denoted as angle_A) where user A is located and the angular range (denoted as angle_B) where user B is located.

[0078] Figure 6A schematic diagram of angle ranges angle_A and angle_B is shown. For example, the front of the electronic device is 0°, and the direction of clockwise rotation around the electronic device is the positive direction. The electronic device can locate user A and user B using visual positioning, distance sensor ranging, sound source localization, etc. After locating user A and user B, the electronic device can determine user A's angle as α (user A is directly in front of the electronic device, therefore α = 0°) and user B's angle as β (e.g., 75°). In other embodiments, user A may not be directly facing the electronic device; in this embodiment, α can be other values, for example, α = -12°.

[0079] Considering factors such as positioning errors and sound diffusion, to more reliably collect the voice signals of users A and B, the electronic device not only collects the audio signals from the locations of users A and B, but also the audio signals from their surroundings. That is, the electronic device collects the audio signals within the angular range (angle_A) where user A is located and the angular range (angle_B) where user B is located. In this embodiment, angle_A = α ± c1, angle_B = β ± c2. This embodiment does not limit the values ​​of c1 and c2. For example, c1 = c2 = 20° to 40°, such as c1 = c2 = 30°.

[0080] S113: The electronic device acquires the audio signal (denoted as Audio_A, as the first audio signal) in the angle range angle_A and the audio signal (denoted as Audio_B, as the second audio signal) in the angle range angle_B.

[0081] Specifically, at time T S The electronic device activates its microphone and acquires audio signals through a microphone array. Then, it processes the audio signals acquired by the microphone array using beamforming algorithm A to obtain audio signal Audio_A; and processes the audio signals acquired by the microphone array using beamforming algorithm B to obtain audio signal Audio_B.

[0082] In this context, the target angle of beamforming algorithm A is set to angle_A. Therefore, the audio signal Audio_A is the audio signal after suppressing the sound signal outside angle_A. That is, angle_A is the audio signal within the angle range angle_A.

[0083] The target angle of beamforming algorithm B is set to angle_B. Therefore, the audio signal Audio_B is the audio signal after suppressing the sound signal outside angle_B. That is, angle_B is the audio signal within the angle range angle_B.

[0084] The above describes an exemplary method for electronic devices to collect user voice. The following returns to... Figure 3 The following steps of the voice interaction method provided in this embodiment will be introduced.

[0085] S120: The electronic device determines whether the start time of the voice signal (denoted as Voice_A, as the first voice signal) in the audio signal Audio_A is within the priority waiting period (also known as the "first period"), and determines the target voice signal (the voice signal that the electronic device is about to reply to) from the voice signal Voice_A and the voice signal Voice_B (as the second voice signal) according to the determination result, wherein the voice signal Voice_B is the voice signal contained in the audio signal Audio_B.

[0086] If user A speaks during the current voice acquisition period, the audio signal Audio_A will include the voice signal Voice_A. Similarly, if user B speaks during the current voice acquisition period, the audio signal B will include the voice signal Voice_B. The electronic device can detect the voice signals Voice_A and Voice_B in the audio signals Audio_A and Audio_B using the Voice Activity Detection (VAD) algorithm, and determine the endpoints (start and end points) of the voice signals Voice_A and Voice_B.

[0087] In this embodiment, the electronic device determines whether the starting point of the voice signal Voice_A (also known as the start time of Voice_A) is within the priority waiting period, and determines one of the voice signals Voice_A and Voice_B as the target voice signal based on the determination result. The priority waiting period is time T. S Then, a first time interval of duration T is elapsed. That is, the starting point of the priority waiting period is time T. S The endpoint is time T. E (T E =T S +T).

[0088] In some embodiments, the first duration T is the time when user A is most likely to speak (e.g., 3 seconds), meaning that if user A will continue to interact with the electronic device, user A is highly likely to speak within the priority waiting period. This embodiment determines the target voice signal based on the priority waiting period, which can better align with the user's interaction intentions and improve the user experience. For the sake of narrative coherence, the specific method for determining the first duration T will be described later.

[0089] refer to Figure 7 The process by which the electronic device provided in this embodiment determines the target speech signal includes the following steps:

[0090] S121: The electronic device determines the start time T of the voice signal Voice_A. AS Is it within the priority waiting period?

[0091] Electronic devices can detect the start time T of the voice signal Voice_A using the VAD algorithm. AS It's understandable that Voice_A's start time T... AS That is, the moment when User A begins to speak.

[0092] Electronic devices at a given time T AS Then, it is compared with the end time T of the priority waiting period. E Compare. If time T AS Earlier than or equal to time T E The electronic device determines the start time T of the voice signal Voice_A. AS If the device is within the priority waiting period, step S122 is executed; otherwise, step S123 is executed.

[0093] S122: The electronic device identifies the voice signal Voice_A as the target voice signal.

[0094] In this embodiment, as long as the start time T of the voice signal Voice_A is... AS During the priority waiting period, Voice_A is identified as the target voice signal (the voice signal that the electronic device will respond to). In other words, as long as User A speaks during the priority waiting period, the electronic device identifies User A as the target interactor (i.e., keeps User A as the current interactor) and prioritizes meeting User A's voice interaction needs.

[0095] Figure 8A An example of identifying the speech signal Voice_A as the target speech signal is given. (Reference) Figure 8A The start time T of Voice_A AS The start time T of Voice_B BS All are within the priority waiting period, and time T AS Earlier than time T BS (That is, User A speaks earlier than User B). In this example, due to time T... AS Since it is within the priority waiting period, the electronic device identifies the voice signal Voice_A as the target voice signal.

[0096] Optionally, after the electronic device detects the start of the voice signal Voice_A (i.e., at time T) AS After that, you can stop acquiring the audio signal Audio_B (or the voice signal Voice_B).

[0097] Figure 8B Another example of identifying the speech signal Voice_A as the target speech signal is given. (Reference) Figure 8B The start time T of Voice_A AS The start time T of Voice_B BS All are within the priority waiting period, but time T BS Earlier than time T AS (That is, User B speaks earlier than User A). In this example, although User B speaks first, due to time T... AS Since it is within the priority waiting period, the electronic device will still identify the voice signal Voice_A as the target voice signal. Optionally, after the electronic device detects the start of the voice signal Voice_A (i.e., at time T), AS After that, you can stop acquiring the audio signal Audio_B (or the voice signal Voice_B) and discard time T. AS The previously collected voice signal, Voice_B.

[0098] If User A does not speak during the priority waiting period, it is assumed that User A has a low willingness to interact, and User B may be identified as the target interactor to accommodate User B's voice interaction needs. The following combines... Figure 7 The subsequent steps will be introduced.

[0099] S123: The electronic device determines whether the voice signal Voice_A and the voice signal Voice_B overlap in time.

[0100] In this embodiment, the start time T of the voice signal Voice_A AS The start time T of the voice signal Voice_B BS At least one of them is located within the priority waiting period. When the voice signal Voice_A begins at time T... AS When outside the priority waiting period, the electronic device determines the target voice signal based on the temporal overlap between voice signals Voice_A and Voice_B.

[0101] Wherein, if the start time T of the voice signal Voice_A AS The time T before or equal to the end of the voice signal Voice_B BE (Also known as "Voice_B's endpoint T") BEWhen the electronic device determines that voice signals Voice_A and Voice_B overlap in time, it executes step S124; otherwise, it determines that voice signals Voice_A and Voice_B do not overlap in time, and executes step S125. As described above, the endpoints of each voice signal, for example, the start time T of voice signal Voice_A... AS The end time T of the voice signal Voice_B BE These can all be determined using the Voice Activity Detection (VAD) algorithm.

[0102] S124: The electronic device identifies the voice signal Voice_A as the target voice signal.

[0103] When voice signals Voice_A and Voice_B overlap in time, the electronic device identifies Voice_A as the target voice signal. In other words, if User A begins speaking before User B has finished speaking, User A is still considered the target interactor (keeping User A as the current interactor unchanged).

[0104] Figure 9 Another example of identifying the speech signal Voice_A as the target speech signal is given. (Reference) Figure 9 The start time T of Voice_B BS Located within the priority waiting period, ending time T BE Located outside the priority waiting period. Voice_A's start time T AS Located outside the priority waiting period, but at time T AS Earlier than the end time T of Voice_B BE (That is, the time when user A starts speaking is earlier than the time when user B finishes speaking). In this example, because the voice signals Voice_A and Voice_B overlap in time, the electronic device identifies the voice signal Voice_A as the target voice signal.

[0105] Optionally, in some embodiments, after the electronic device detects the start point of the voice signal Voice_A (i.e., time T) AS After that, you can stop acquiring the audio signal Audio_B (or the voice signal Voice_B) and discard time T. AS The previously acquired voice signal, Voice_B. In this embodiment, after detecting the start point of voice signal Voice_A, if it is determined that voice signal Voice_B has not yet ended (i.e., the end point T of voice signal Voice_B has not yet been detected), BE Then, the start time T of Voice_A can be determined. ASEarlier than the end time T of Voice_B BE .

[0106] S125: The electronic device identifies the voice signal Voice_B as the target voice signal.

[0107] When voice signals Voice_A and Voice_B do not overlap in time, the electronic device identifies Voice_B as the target voice signal. In other words, when User B has finished speaking and User A has not yet started speaking, the electronic device identifies User B as the target interactor (i.e., the electronic device switches the current interactor from User A to User B).

[0108] Figure 10A An example of identifying the speech signal Voice_B as the target speech signal is given. (Reference) Figure 10A The start time T of Voice_B BS All are within the priority waiting period, ending at time T. BE Located outside the priority waiting time. The start time T of Voice_A. AS Later than the end time T of Voice_B BE (That is, when User B finishes speaking, User A has not yet started speaking). In this example, since the voice signals Voice_A and Voice_B do not overlap in time, the electronic device identifies Voice_B as the target voice signal. Optionally, after the electronic device detects the end of Voice_B (i.e., at time T)... BE After that, the acquisition of audio signals can be stopped (e.g., the microphone can be turned off and audio signals Audio_A can not be acquired), and the voice signal Voice_B can be responded to (i.e., step S130 is executed).

[0109] Figure 10B An example of identifying the speech signal Voice_B as the target speech signal is given. (Reference) Figure 10B The start time T of Voice_B BS and the end time T BE All are within the priority waiting period. The start time T of Voice_A AS Located outside the priority waiting time. In this example, the electronic device detects the endpoint T of Voice_B. BE After that, continue waiting until time T. E When the electronic device determines at time T E If user A has not spoken by the time the signal is received, the electronic device identifies the voice signal Voice_B as the target voice signal. Optionally, the electronic device waits until time T. EThen, the acquisition of audio signals can be stopped (e.g., the microphone can be turned off and audio signal Audio_A can not be acquired), and the voice signal Voice_B can be responded to (i.e., step S130 is executed).

[0110] S130: The electronic device responds to the target voice signal.

[0111] After identifying the target speech signal, the electronic device responds to it (i.e., replies to the target user). For example, the electronic device uploads the target speech signal to a cloud server. The cloud server uses Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) algorithms to perform semantic recognition on the target speech signal to determine the content of the response text. Here, ASR is a technology for converting speech to text, and NLP is a technology for enabling electronic devices to "understand" human language.

[0112] After determining the response text, the cloud server sends it to the electronic device. Upon receiving the response text, the electronic device converts it into a speech stream using a Text-to-Speech (TTS) algorithm and outputs (e.g., plays) the speech stream to respond to the target speech signal (or the target interacting person). During the playback of the speech stream, the electronic device can mute its microphone and not collect audio signals. In other embodiments, the electronic device can also determine the response text content using local ASR or NLP algorithms.

[0113] In addition to voice responses, electronic devices can also respond using facial expressions and gestures. For example, after identifying the target user as User B, the electronic device can turn to face User B, thereby enhancing its intelligence and anthropomorphism.

[0114] In summary, this embodiment provides a voice interaction method that determines the target user based on a priority waiting period. This method can reasonably balance the voice interaction needs of the current user (e.g., user A) and potential users (e.g., user B), thereby improving the user experience in multi-user interaction scenarios.

[0115] For example, if user A (the current interactor) speaks during the priority waiting period, the electronic device identifies user A as the target interactor and prioritizes meeting user A's voice interaction needs. If user A does not speak during the priority waiting period, it is assumed that user A's willingness to interact is low, and user B (the potential interactor) may be identified as the target interactor to accommodate user B's voice interaction needs as well.

[0116] This embodiment is an exemplary description of the technical solution of this application, and those skilled in the art can make other modifications.

[0117] For example, in this embodiment, if user A does not speak during the priority waiting period, the target interactor is determined based on the temporal overlap of voice signals Voice_A and Voice_B. In other embodiments, if user A does not speak during the priority waiting period, but user B does speak during the priority waiting period, user B is identified as the target interactor. This embodiment allows for relatively faster identification of the target interactor. Furthermore, this embodiment can also increase the priority of potential interactors.

[0118] For example, in this embodiment, the start time T of the voice signal Voice_A AS The start time T of the voice signal Voice_B BS At least one of them is located within a priority waiting period. In other embodiments, when the start time T of the voice signal Voice_A... AS The start time T of the voice signal Voice_B BS When all are outside the priority waiting period, the electronic device can continue to monitor the voice signal Voice_A, and if it is outside the set time (e.g., T... S If the start point of the voice signal Voice_A is detected 8 seconds before the target voice signal is detected, then the voice signal Voice_A will be used as the target voice signal; otherwise, the electronic device will end the current voice interaction.

[0119] For example, in some embodiments, considering that the wake word may directly lead to the switching of the target user, the wake word of the electronic device is not included in the voice of user B (i.e., the voice signal Voice_B).

[0120] For example, in this embodiment, there is only one potential interacting person around the electronic device, specifically user B. In other scenarios, there may be multiple potential interacting persons around the electronic device. (See reference...) Figure 11 There are two potential interactors around the electronic device, specifically user B and user C. In this embodiment, the electronic device can select one of user B and user C as a provisional interactor, and determine the target interactor from the provisional interactor and user A according to the method described in step S120.

[0121] This application does not limit the method of selecting the provisional interactor. For example, the user who has most recently interacted with the electronic device via voice between user B and user C may be selected as the provisional interactor; or, the user who speaks first during the current voice acquisition phase may be selected as the provisional interactor.

[0122] The following describes the method for determining the first duration T provided in this embodiment.

[0123] In this embodiment, the first duration T is determined based on the time when user A is most likely to speak. That is, assuming user A has the intention to continue interacting, user A will at the latest within the first duration (i.e., time T). E (Before) he started speaking.

[0124] In this embodiment, the first duration is determined based on user A's interaction willingness value P and the number of interaction rounds M between user A and the electronic device within a set time period. The interaction willingness value P represents user A's willingness to interact with the electronic device. The larger the interaction willingness value P, the greater the likelihood that user A will engage in voice interaction with the electronic device.

[0125] Specifically, the first duration T = k1 × P + k2 × min{M, n}, where k1 and k2 are preset constants, and n is an integer between 3 and 6. The details are explained below.

[0126] (1) Interaction willingness value P. In this embodiment, the interaction willingness value is a value between 0 and 1. However, this application is not limited to this. In other embodiments, the interaction willingness value P can also be other values, such as a value between 1 and 5.

[0127] In this embodiment, the interaction willingness value P is based on user A's facial angle. And / or the distance D between User A and the electronic device is determined. User A's facial angle. This indicates the degree to which User A's face is directly facing the electronic device. When User A's face is directly facing the front of the electronic device, User A's facial angle is 0°; the greater the angle at which User A is turned to the electronic device, the greater the facial angle. The larger the value, the greater the user's willingness to interact (P). Generally, the more directly user A faces the electronic device, the greater their willingness to interact (P). User A's facial angle... It can be determined through image recognition.

[0128] The distance D between user A and the electronic device can be determined using methods such as image recognition, distance sensor ranging, and sound source localization. For example, when the distance D between user A and the electronic device is within a set range (e.g., 0.5 to 1 times the height of the electronic device), user A is considered to have a high willingness to interact value P; the greater the deviation of distance D from the set range, the smaller the willingness to interact value P of user A is considered.

[0129] In some embodiments, the interaction willingness value P is the facial angle. The weighted sum of distance D. In this embodiment, the facial angle The weights of the distance D can be constants determined empirically.

[0130] In other embodiments, the interaction willingness value P can be determined using an AI algorithm. A pre-trained AI model can be stored in the electronic device to represent facial angles. The mapping relationship between distance D and interaction willingness value P. Electronic devices measure facial angles... After determining the distance D, the interaction willingness value P can be calculated using this AI model.

[0131] (2) Interaction Round M. Interaction round M is the number of interactions between user A and the electronic device within a set time period P2. In this application, one question-and-answer session between the electronic device and the user is counted as one interaction round. For example, if user A asks, "Do you have a favorite animal?" and the electronic device answers, "I like furry animals, they look so warm," this is counted as one interaction round.

[0132] The interaction round M reflects the frequency of interaction between user A and the electronic device. The higher the interaction frequency (i.e., the larger the M value), the greater the likelihood of user A interacting. In this embodiment, the time period P2 is set as time T. S The third time period prior. In other words, M represents the number of interactions between user A and the electronic device within the most recent third time period. Thus, M can more accurately characterize the likelihood of user A continuing to interact with the electronic device. For example, the third time period is 0.5 to 2 minutes, such as 1 minute.

[0133] In some embodiments, M represents the number of consecutive interactions between user A and the electronic device within a set time period P2. Continuous interaction means that no other user intervenes during the interaction between user A and the electronic device. If another user intervenes, the number of interaction rounds M is recalculated starting from 0.

[0134] (3) k1, k2. k1 and k2 are the weights of the interaction willingness value P and the interaction round M, respectively, used to adjust the weights of the interaction willingness value P and the interaction round M in the first duration T. In this embodiment, since the interaction willingness value P is a value between 0 and 1, and M is a value greater than 1, k1 is greater than k2 in order to balance the weights of the interaction willingness value P and the interaction round M in the first duration T. For example, k1 is 3 to 5 times k2. For example, k1 = 2, k2 = 0.5.

[0135] (4) n. n is used to limit the upper limit of the first duration T. Considering that in some scenarios, user A will interact with the electronic device many times, for example, M=15, the first duration T may have an excessively large value. In order to avoid the first duration T from increasing indefinitely, the interaction rounds term is set to min{M, n} to limit the upper limit of the first duration T.

[0136] The above describes the method for determining the first duration T. For example, in one example, the interaction willingness value P = 0.9, the number of interaction rounds M = 3, k1 = 2, k2 = 0.5, and n = 5, so T = 3.3s can be obtained; in other examples, the interaction willingness value P = 0.3, the number of interaction rounds M = 1, k1 = 2, k2 = 0.5, and n = 5, so T = 1.1s can be obtained.

[0137] In this embodiment, the first duration is determined based on the interaction willingness value P and the interaction round M, which can accurately predict the likelihood of user A's interaction. The greater the likelihood of user A's interaction, the longer the electronic device will wait for user A, thus reasonably meeting user A's voice interaction needs.

[0138] The method for determining the first duration T has been described above. However, this application is not limited to this.

[0139] For example, in other embodiments, the first duration T can be determined based on either the interaction willingness value P or the interaction round M. For example, T = 3 × P.

[0140] For example, in this embodiment, the first duration T is dynamically adjusted based on the interaction willingness value P and the number of interaction rounds M. In other embodiments, the first duration T can be a fixed value determined empirically, for example, the first duration is 3 seconds. This embodiment can simplify the process of determining the first duration T and reduce computational overhead.

[0141] Compared with the voice interaction methods provided in other embodiments, the voice interaction method provided in this application embodiment can reasonably select the target interaction person in a multi-person interaction scenario. The following is a comparison with the voice interaction methods provided in other embodiments.

[0142] Figure 12 One implementation method is shown. Specifically, Figure 12 A sound pickup method and apparatus are provided. Initially, the sound pickup beam is pointed at the target sound source. When a change in the orientation of the recording device is detected, the direction of the sound pickup beam is dynamically adjusted to ensure that the target sound source and the sound pickup beam are pointing in the same direction, thereby attenuating or blocking the sound signals of other noise sources.

[0143] Figure 12 The implementation shown dynamically adjusts the pickup beam direction based on the orientation of the recording device after identifying the target sound source. However, it does not address how the robot can pick up sound in multi-user interaction scenarios, or how to select the target user among multiple users.

[0144] Figure 13 Another implementation is shown. Specifically, Figure 13A sound pickup method is provided, including the following steps: S1001, when the angle of the received sound source is within a preset angle, the face of the current camera is acquired; S1002, the face with the closest angle to the sound source is selected as the speaker, that is, the face with the closest angle is tracked to achieve the purpose of tracking the current speaker; finally, in step S1003, the robot angle is adjusted so that the center of the speaker's face falls in front of the center of the robot, so as to facilitate the response to the speaker's voice signal.

[0145] Figure 13 The implementation shown only selects the face closest to the current sound source angle among all the faces in front of it for face tracking, in order to track the current speaker. However, it does not solve the problems of how the robot determines the sound pickup direction and how to select the target interactive person among multiple users in a multi-user interaction scenario.

[0146] Compared to Figure 12 and Figure 13 As shown, the voice interaction method in this embodiment can reasonably take into account the voice interaction needs of the current interactor and potential interactors in a multi-person interaction scenario, accurately determine the target interactor, and thus improve the user experience in a multi-person interaction scenario.

[0147] Now for reference Figure 14 The diagram shows a block diagram of an electronic device 400 according to one embodiment of the present application. The electronic device 400 may include one or more processors 401 coupled to a controller hub 403. In at least one embodiment, the controller hub 403 communicates with the processor 401 via a multi-branch bus such as a Front Side Bus (FSB), a point-to-point interface such as a QuickPath Interconnect (QPI), or a similar connection 406. The processor 401 executes instructions controlling general types of data processing operations. In one embodiment, the controller hub 403 includes, but is not limited to, a Graphics & Memory Controller Hub (GMCH) (not shown) and an Input / Output Hub (IOH) (which may be on a separate chip) (not shown), wherein the GMCH includes memory and a graphics controller and is coupled to the IOH.

[0148] Electronic device 400 may also include a coprocessor 402 and a memory 404 coupled to a controller hub 403. Alternatively, one or both of the memory and GMCH may be integrated within the processor (as described in this application), with memory 404 and coprocessor 402 directly coupled to processor 401 and controller hub 403, which is located on a single chip with IOH.

[0149] Memory 404 may be, for example, Dynamic Random Access Memory (DRAM), Phase Change Memory (PCM), or a combination of both. Memory 404 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable storage medium stores instructions, specifically, temporary and permanent copies of those instructions. The instructions may include, when executed by at least one of the processors, causing the electronic device 400 to perform, as... Figure 3 , Figure 5 , Figure 7 The instructions for the method are shown. When the instructions are executed on a computer, the computer performs the method disclosed in the above embodiments.

[0150] In one embodiment, the coprocessor 402 is a dedicated processor, such as, for example, a high-throughput many-integrated core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, general-purpose computing on graphics processing units (GPGPU), or an embedded processor, etc. Optional properties of the coprocessor 402 are indicated by dashed lines. Figure 14 middle.

[0151] In one embodiment, electronic device 400 may further include a Network Interface Controller (NIC) 406. The network interface 406 may include a transceiver for providing a radio interface for electronic device 400 to communicate with any other suitable device (such as a front-end module, antenna, etc.). In various embodiments, the network interface 406 may be integrated with other components of electronic device 400. The network interface 406 can implement the functions of the communication unit in the above embodiments.

[0152] Electronic device 400 may further include input / output (I / O) devices 405. I / O 405 may include: a user interface designed to enable a user to interact with electronic device 400; a peripheral component interface designed to enable peripheral components to also interact with electronic device 400; and / or sensors designed to determine environmental conditions and / or location information related to electronic device 400.

[0153] It is worth noting that, Figure 14 This is merely an example. That is, although... Figure 14 The electronic device 400 shown includes multiple devices such as a processor 401, a controller hub 403, and a memory 404. However, in practical applications, devices using the methods of this application may include only a portion of the devices in the electronic device 400. For example, it may include only the processor 401 and the network interface 406. Figure 14 The properties of the optional devices are shown by dashed lines.

[0154] Now for reference Figure 15 The diagram shown is a block diagram of a System-on-Chip (SoC) 500 according to an embodiment of this application. Figure 15 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 15 In this SoC 500, the following components are included: an interconnect unit 550 coupled to the processor 510; a system proxy unit 580; a bus controller unit 590; an integrated memory controller unit 540; a group or one or more coprocessors 520, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 530; and a direct memory access (DMA) unit 560. In one embodiment, the coprocessor 520 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, general-purpose computing on graphics processing units (GPGPU), a high-throughput MIC processor, or an embedded processor.

[0155] Static Random Access Memory (SRAM) cell 530 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable storage medium stores instructions, specifically, temporary and permanent copies of those instructions. These instructions may include, when executed by at least one of the processors, causing the SoC implementation to... Figure 3 , Figure 5 , Figure 7 The instructions for the method are shown. When the instructions are executed on a computer, the computer performs the method disclosed in the above embodiments.

[0156] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0157] All methods and implementations of this application can be implemented in the form of software, magnetic files, firmware, etc.

[0158] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0159] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this paper are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0160] One or more aspects of at least one embodiment can be implemented by representational instructions stored on a computer-readable storage medium, the instructions representing various logics in a processor, which, when read by a machine, cause the machine to create logic for performing the techniques described herein. These representations, referred to as “Intellectual Property (IP) cores,” can be stored on a tangible computer-readable storage medium and provided to multiple customers or production facilities for loading into manufacturing machines that actually manufacture the logic or processor.

[0161] In some cases, an instruction translator can be used to translate instructions from a source instruction set to a target instruction set. For example, an instruction translator can transform (e.g., using static binary transformation, including dynamically compiled dynamic binary transformation), morph, emulate, or otherwise translate instructions into one or more other instructions that will be processed by the core. Instruction translators can be implemented in software, hardware, firmware, or a combination thereof. Instruction translators can be on the processor, off the processor, or partially on and partially off the processor.

Claims

1. A voice interaction method for electronic devices, characterized in that, The method includes: The system engages in voice interaction with a first user, and during the voice acquisition period of the voice interaction, it acquires a first audio signal within the angular range of the first user and a second audio signal within the angular range of the second user, where the second user is the user who engages in voice interaction with the electronic device during a set historical period. Determine whether the start time of the first speech signal in the first audio signal is within the first time period, and determine the target speech signal from the first speech signal and the second speech signal based on the determination result. The second speech signal is the speech signal contained in the second audio signal, and the first time period is the time period after the start time of the speech acquisition time period for a first duration. Respond to the target speech signal.

2. The method according to claim 1, characterized in that, At least one of the start times of the first speech signal and the second speech signal falls within the first time period; The step of determining the target speech signal from the first speech signal and the second speech signal based on the judgment result includes: If the start time of the first voice signal is within the first time period, then the first voice signal is determined as the target voice signal; otherwise, the target voice signal is determined according to the temporal overlap between the second voice signal and the first voice signal.

3. The method according to claim 2, characterized in that, Determining the target speech signal based on the temporal overlap between the second speech signal and the first speech signal includes: If the second speech signal and the first speech signal overlap in time, then the first speech signal is determined as the target speech signal; If the second speech signal and the first speech signal do not overlap in time, then the second speech signal is determined as the target speech signal.

4. The method according to claim 1, characterized in that, The first duration is determined based on the interaction willingness value P of the first user and / or the number of interactions M between the first user and the electronic device within a set time period, wherein the interaction willingness value P is used to characterize the voice interaction willingness of the first user and the electronic device.

5. The method according to claim 4, characterized in that, The interaction willingness value P is determined based on the first user's facial angle and / or the distance between the first user and the electronic device.

6. The method according to claim 4, characterized in that, The first duration is k1×P+k2×min{M,n}, where k1 and k2 are preset constants, and n is an integer between 3 and 6.

7. The method according to claim 1, characterized in that, The second voice signal does not include the wake word of the electronic device.

8. The method according to claim 1, characterized in that, The set historical time period is the second duration preceding the start time of the voice acquisition period.

9. The method according to claim 8, characterized in that, The second duration is 0.5 to 2 minutes.

10. An electronic device, characterized in that, include: A memory for storing instructions executed by one or more processors of the electronic device; A processor, when executing the instructions in the memory, can cause the electronic device to perform the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method described in any one of claims 1 to 9.

Images