Voice interaction method

Abstract

Embodiments of the present application provide a voice interaction method, the method comprising: collecting user voice, and inputting the user voice into a first voice model of an audio signal processor to obtain a preliminary screening confidence; when the preliminary screening confidence is greater than or equal to a preliminary screening confidence threshold, inputting the user voice into a second voice model of an application processor to obtain a target confidence; calculating, by the application processor, a sound source angle corresponding to the user voice, and determining a target confidence threshold corresponding to the sound source angle; when the target confidence is greater than or equal to the target confidence threshold, determining that the user voice is a voice instruction, and performing voice interaction in response to the voice instruction. Embodiments of the present application preliminarily screen the user voice through the first voice model of the audio signal processor, thereby reducing system power consumption, and the second voice model of the application processor can accurately determine whether the user voice is a voice instruction.

Description

Technical Field

[0001] This application relates to the field of terminal technology, specifically to voice processing technology, and more particularly to a voice interaction method. Background Technology

[0002] With the development of terminal technology, electronic devices such as smartphones, personal computers, and smart speakers have voice assistant functions, enabling users to interact with these devices via voice to control them to perform specified operations, such as playing music, making phone calls, setting alarms, or setting to-do lists. When a user interacts with an electronic device via voice, the device captures the user's voice and uses a voice recognition model to identify whether the voice is a voice command. If it is a voice command, the device responds and performs the corresponding operation. The voice recognition model is typically located in the audio digital signal processor (ADSP) of the electronic device. Due to the limited memory and processing power of the ADSP, the network structure of the voice recognition model is relatively simple, and its iterative ability is weak. This results in low voice recognition accuracy, leading to the inability to recognize voice commands or misidentifying user voice as a voice command, thus preventing the electronic device from correctly responding to user commands and performing voice interactions. Summary of the Invention

[0003] In view of the above, it is necessary to provide a voice interaction method and related equipment to solve the problem that the voice recognition accuracy of the voice recognition model set in the audio signal processor is low, which causes electronic devices to be unable to correctly respond to user commands for voice interaction.

[0004] In a first aspect, this application provides a voice interaction method applied to an electronic device. The method includes: acquiring audio data and inputting the audio data into a first speech model of an audio signal processor of the electronic device to obtain an initial screening confidence level, wherein the initial screening confidence level is used to characterize the accuracy of the first speech model in determining that the audio data is near-field speech; when the initial screening confidence level is greater than or equal to an initial screening confidence level threshold, inputting the audio data into a second speech model of an application processor of the electronic device to obtain a target confidence level, wherein the target confidence level is used to characterize the accuracy of the second speech model in determining that the audio data is a voice command; calculating the sound source angle corresponding to the audio data through the application processor and determining the target confidence level threshold corresponding to the sound source angle; when the target confidence level is greater than or equal to the target confidence level threshold, determining that the audio data is the voice command and responding to the voice command to perform voice interaction.

[0005] Through the above technical solution, when audio data is collected, the first speech model performs a preliminary judgment, filtering out audio data that is clearly not close-range speech. This eliminates the need for the second speech model to recognize audio data that is clearly not close-range speech. Furthermore, because the first speech model is relatively simple and runs on the audio signal processor side, the system power consumption during speech recognition is low. After the first speech model initially determines that the audio data is close-range speech, the audio data is sent to the second speech model on the application processor side. Because the application processor has larger memory and stronger computing power, the second speech model is more complex than the first speech model, and the judgment result is more accurate. Therefore, it can accurately determine whether the audio data is a voice command.

[0006] In one possible implementation, the first speech model is trained using audio features corresponding to multiple speech commands and audio features corresponding to multiple non-speech commands as training data. The multiple speech commands are speech output by sound sources at a distance less than or equal to a first preset distance from the electronic device, and are used as positive samples for training the first speech model. The multiple non-speech commands are speech output by sound sources at a distance greater than or equal to a second preset distance from the electronic device, and are used as negative samples for training the first speech model.

[0007] By using the above technical solution, the speech features corresponding to multiple speech commands are used as positive samples, and the speech features corresponding to multiple non-speech commands are used as negative samples to train the first speech model, which can effectively improve the accuracy of the first speech model in speech recognition.

[0008] In one possible implementation, the first speech model is a convolutional neural network model. The step of inputting the audio data into the first speech model of the audio signal processor of the electronic device to obtain an initial screening confidence level includes: preprocessing the audio data and extracting audio features from the preprocessed audio data; inputting the audio features into the first speech model and extracting convolutional features of the audio features through the convolutional layers of the first speech model; performing nonlinear transformation processing on the convolutional features of the audio features through the activation layers of the first speech model; performing pooling processing on the convolutional features of the audio features through the pooling layers of the first speech model; classifying the convolutional features of the audio features through the fully connected layers of the first speech model, determining the judgment result of the audio features, and calculating the initial screening confidence level.

[0009] Through the above technical solution, the first speech model uses a convolutional neural network model for speech recognition. It performs speech recognition through audio feature extraction, dimensionality reduction, classification and other operations, which effectively improves the accuracy of speech recognition.

[0010] In one possible implementation, the second speech model includes multiple sub-speech models. The multiple sub-speech models are generated by training the audio features corresponding to multiple speech commands and the audio features corresponding to multiple non-speech commands as training data. The multiple speech commands are speech output by sound sources at a distance less than or equal to the first preset distance from the electronic device, and are used as positive samples for training the second speech model. The multiple non-speech commands are speech output by sound sources at a distance greater than or equal to the second preset distance from the electronic device, and are used as negative samples for training the second speech model.

[0011] By using the above technical solution, the speech features corresponding to multiple speech commands are used as positive samples, and the speech features corresponding to multiple non-speech commands are used as negative samples to train the second speech model, which can effectively improve the accuracy of the first speech model in speech recognition.

[0012] In one possible implementation, the second speech model includes multiple sub-models, such as a convolutional neural network model and a long short-term memory network. The step of inputting the audio data into the second speech model of the application processor of the electronic device to obtain the target confidence score includes: calculating the corresponding confidence score using the convolutional neural network model in the second speech model; calculating the corresponding confidence score using the long short-term memory network in the second speech model; and calculating the target confidence score based on the confidence scores obtained from the convolutional neural network model and the long short-term memory network.

[0013] Through the above technical solution, the second speech model can integrate the judgment results of convolutional neural network model and long short-term memory network to determine whether audio data is a speech command, effectively improving the accuracy of recognizing audio data as a speech command. In addition, long short-term memory network can solve the vanishing / exploding gradient problem of the model, is suitable for processing sequential data such as speech, and has a flexible network structure that can adapt to various feature data, thereby further improving the efficiency and accuracy of speech recognition.

[0014] In one possible implementation, the target confidence level is the average, weighted average, or median of the confidence levels calculated by the plurality of sub-speech models.

[0015] The above technical solution can integrate the judgment results of multiple sub-speech models to determine whether audio data is a voice command, effectively improving the accuracy of recognizing audio data as a voice command.

[0016] In one possible implementation, the number of model parameters of the second speech model is greater than the number of model parameters of the first speech model, and / or the number of feature extraction layers of the second speech model is greater than the number of feature extraction layers of the first speech model, and / or the number of feature samples used to train the second speech model is greater than the number of feature samples used to train the first speech model.

[0017] By employing the above technical solutions, when the application processor has more memory and stronger computing power than the audio signal processor, the accuracy of speech recognition by the second speech model can be effectively improved by setting the number of feature samples used to train the second speech model to be greater than the number of feature samples used to train the first speech model, or by setting the number of model parameters of the second speech model to be greater than the number of model parameters of the first speech model, or by setting the number of feature extraction layers of the second speech model to be greater than the number of feature extraction layers of the first speech model.

[0018] In one possible implementation, the step of calculating the sound source angle corresponding to the audio data through the application processor includes: receiving dual-channel audio data through the application processor and extracting the audio features of the dual-channel audio data; performing cross-correlation calculation on the dual-channel audio data using a generalized cross-correlation-phase transform algorithm to obtain the cross-correlation function between the dual-channel audio data; normalizing the cross-correlation function between the dual-channel audio data; performing phase transform weighting on the normalized cross-correlation function to convert the temporal correlation of the audio signals of the dual-channel audio data into phase information in the frequency domain; and determining the angle of the sound source of the dual-channel audio data relative to the microphone based on the phase information in the frequency domain.

[0019] Using the above technical solution, the application processor can accurately calculate the sound source angle corresponding to the audio data by employing the generalized cross-correlation-phase transformation algorithm.

[0020] In one possible implementation, determining the target confidence threshold corresponding to the sound source angle includes: determining a preset sound source angle range into which the sound source angle corresponding to the audio data falls; and determining the target confidence threshold corresponding to the preset sound source angle range into which the sound source angle corresponding to the audio data falls, based on the preset sound source angle range into which the sound source angle corresponding to the audio data falls and the correspondence between multiple preset sound source angle ranges and multiple target confidence thresholds.

[0021] By using the above technical solution, the correspondence between multiple preset sound source angle ranges and multiple target confidence thresholds is pre-set, so that audio data in different preset sound source angle ranges can be recognized as voice commands. This relaxes the restrictions on sound source angles for voice commands. When issuing voice commands, users do not need to set the angle between the electronic device and their mouth to a specific angle, reducing user operations and conforming to user habits.

[0022] In one possible implementation, the method further includes: when the target confidence level is greater than or equal to the target confidence level threshold, inputting audio data into a third speech model of the application processor; when the third speech model determines that the audio data is the voice of a specified user, determining that the audio data is the voice command, and responding to the voice command to perform voice interaction.

[0023] The above technical solution uses a third voice model to determine whether the audio data belongs to a specified user, ensuring that only the specified user's voice commands can interact with the electronic device, thereby protecting the privacy and security of the electronic device and ensuring that voice commands are responded to correctly.

[0024] In one possible implementation, the method further includes: acquiring multiple speech segments captured by the microphone of the electronic device, and extracting the voiceprint features of each speech segment; training a Gaussian mixture model corresponding to the speech of the specified user based on the voiceprint features of each speech segment; and determining the trained Gaussian mixture model as the text-independent voiceprint model corresponding to the speech of the specified user.

[0025] The above technical solution uses a Gaussian mixture model to cluster multiple segments of a specified user's speech, generating a text-independent voiceprint model for voiceprint recognition of collected audio data, effectively improving the accuracy of voiceprint recognition.

[0026] In one possible implementation, inputting the audio data into the third speech model of the application processor includes: extracting the voiceprint features of the audio data; inputting the voiceprint features of the audio data into the text-independent voiceprint model corresponding to the speech of the specified user, to obtain a determination result of whether the audio data is the speech of the specified user and the confidence level of the determination result.

[0027] By employing the above technical solution and using a text-independent voiceprint model to perform voiceprint recognition on the collected audio data, the accuracy of voiceprint recognition is effectively improved.

[0028] In one possible implementation, the method further includes: performing a first noise reduction process on the audio data by the audio signal processor.

[0029] The above technical solution reduces noise in the audio data before the first speech model performs speech recognition, thus avoiding noise affecting the speech recognition of the first speech model and effectively improving the accuracy of the speech recognition results of the first speech model.

[0030] In one possible implementation, the first noise reduction processing of the audio data by the audio signal processor includes: acquiring the audio data through two microphones of the electronic device respectively, and the audio signal processor performing noise reduction processing on the dual-channel audio data acquired by the two microphones through an acoustic echo cancellation algorithm.

[0031] The above technical solution employs an acoustic echo cancellation algorithm to perform a first noise reduction process on audio data, which filters out the self-noise of electronic devices and prevents the self-noise from affecting the accuracy of the first speech model in speech recognition.

[0032] In one possible implementation, the audio signal processor performs noise reduction processing on the dual-channel audio data acquired by the two microphones using an acoustic echo cancellation algorithm, including: preprocessing the dual-channel audio data by converting it into a frequency domain signal; adaptively reducing noise in the frequency domain signal corresponding to the dual-channel audio data using a regularized minimum mean square error (MME) filtering algorithm; adjusting the filter weights based on the difference between the frequency domain signal and the desired output signal; and completing the noise reduction of the dual-channel audio data when the difference between the frequency domain signal and the desired output signal is less than or equal to a preset difference value; and smoothing the dual-channel audio data using a Kalman filter algorithm. The process involves establishing a state model and a measurement model for the dual-channel audio data; predicting the state of the dual-channel audio data at the current moment based on the state transition equation, the state model of the dual-channel audio data acquired at the previous moment, and the measurement model; correcting the predicted state of the dual-channel audio data at the current moment based on the actual observed state of the dual-channel audio data at the current moment; and predicting the state of the dual-channel audio data at the next moment based on the state transition equation, the predicted state model of the dual-channel audio data acquired at the current moment, and the measurement model, until the difference between two adjacent state estimates is less than or equal to a preset state estimate, and the covariance matrix is ​​less than or equal to a preset covariance matrix, thus completing the noise reduction of the dual-channel audio data.

[0033] By employing the above technical solutions, regularized minimum mean square error filtering algorithm and Kalman filtering algorithm are used to reduce noise in audio data, which effectively improves the noise reduction effect and thus enhances the accuracy of the first speech model in speech recognition.

[0034] In one possible implementation, the method further includes performing a second noise reduction process on the audio data via the application processor.

[0035] The above technical solution reduces noise in the audio data before the second speech model performs speech recognition, thus avoiding noise affecting the speech recognition of the second speech model and effectively improving the accuracy of the speech recognition results of the second speech model.

[0036] In one possible implementation, the second noise reduction processing of the audio data by the application processor includes: preprocessing the audio data by the application processor to obtain multiple sub-audio data segments; extracting audio features of each sub-audio data segment; identifying whether each sub-audio data segment is clean audio data by the application processor; if a sub-audio data segment is determined to be clean speech data, outputting the sub-audio data segment; or if a sub-audio data segment is determined not to be clean speech data, subtracting the clean speech data segment from the clean speech data segment to obtain the difference between the sub-audio data segment and the clean speech data segment; filtering out noise in the sub-audio data segment based on the difference between the sub-audio data segment and the clean speech data segment, thus completing the second noise reduction processing of the audio data segment.

[0037] Through the above technical solution, the application processor uses a speech denoising model to perform a second denoising process on the user's speech to filter out external noise from the electronic device, thus preventing external noise from affecting the accuracy of the second speech model in speech recognition.

[0038] In one possible implementation, the method further includes: determining, through the audio signal processor, whether the audio parameters of the audio data are greater than or equal to a preset audio parameter threshold; if the audio parameters of the audio data are greater than or equal to the preset audio parameter threshold, inputting the audio data into the first speech model to obtain the initial screening confidence level.

[0039] The above technical solution determines whether the audio parameters of the audio data are greater than or equal to a preset audio parameter threshold before the first stage of user voice recognition is performed. The threshold comparison method is used to determine whether the audio data obviously does not meet the conditions of the voice command, and the audio data that obviously does not belong to the issued voice command is filtered out. The first and second voice models do not need to recognize the audio data that obviously does not belong to the issued voice command, which effectively improves the efficiency of voice discrimination and reduces system power consumption.

[0040] In one possible implementation, the audio parameters are at least one of the energy, amplitude, and intensity of the user's voice.

[0041] The above technical solution uses at least one of the following audio parameters—energy, amplitude, and intensity—to characterize audio data, making it easier to filter audio data through threshold comparison.

[0042] In one possible implementation, the method further includes: determining, by the audio signal processor, whether the audio parameters of the audio data are greater than or equal to the preset audio parameter threshold, and determining whether the first pose parameter of the electronic device is greater than or equal to the first preset pose parameter threshold; if the audio parameters of the audio data are greater than or equal to the preset audio parameter threshold, and the first pose parameter of the electronic device is greater than or equal to the first preset pose parameter threshold, inputting the audio data into the first speech model of the audio signal processor to obtain the initial screening confidence level.

[0043] With the above technical solution, before performing the first stage of recognition on the collected audio data, in addition to using threshold comparison to determine whether the audio data obviously does not meet the conditions for issuing voice commands, a threshold comparison is also used to determine whether the pose of the electronic device during the audio data collection process obviously does not meet the conditions for issuing voice commands. This further filters out audio data that obviously does not belong to the issuance of voice commands, eliminating the need for the first and second speech models to recognize audio data that obviously does not belong to the issuance of voice commands, effectively improving the efficiency of speech discrimination and reducing system power consumption.

[0044] In one possible implementation, the first pose parameter is the variance of the three-axis acceleration of the electronic device, and the first preset pose parameter threshold includes a preset three-axis acceleration threshold.

[0045] The above technical solution uses the variance of the triaxial acceleration of the electronic device to characterize the first pose parameter of the electronic device, which facilitates the filtering of the first pose parameter of the electronic device by means of threshold comparison.

[0046] In one possible implementation, the method further includes: determining, by the application processor, whether the second pose parameter of the electronic device is greater than or equal to a second preset pose parameter threshold; if the second pose parameter of the electronic device is greater than or equal to the second preset pose parameter threshold, inputting the audio data into the second speech model of the application processor to obtain the target confidence level.

[0047] Through the above technical solution, on the application processor side, before performing the second stage of recognition on the collected audio data, model recognition is used to accurately determine whether the posture of the electronic device during the audio data collection process obviously does not meet the conditions for issuing voice commands. Since the application processor has strong computing power, the model recognition method can more accurately filter out audio data that obviously does not belong to the issuance of voice commands, eliminating the need for a second voice model to recognize audio data that obviously does not belong to the issuance of voice commands, effectively improving the efficiency of voice discrimination and reducing system power consumption.

[0048] In one possible implementation, the second pose parameter is the pose confidence of the three-axis acceleration of the electronic device in motion, and the second preset pose parameter threshold is the confidence threshold corresponding to the second pose parameter.

[0049] The above technical solution uses the pose confidence of the three-axis acceleration of the electronic device to characterize the second pose parameter of the electronic device, which facilitates the filtering of the second pose parameter of the electronic device through model recognition.

[0050] Secondly, this application provides an electronic device, the electronic device including a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device performs the above-described voice interaction method.

[0051] Thirdly, this application provides a chip coupled to a memory in an electronic device, the chip being used to control the processor of the electronic device to execute the aforementioned voice interaction method.

[0052] Fourthly, this application provides a computer storage medium storing program instructions that, when executed on an electronic device, cause the processor of the electronic device to perform the aforementioned voice interaction method.

[0053] Furthermore, the technical effects brought about by the second to fourth aspects can be found in the descriptions of the methods in the above-mentioned method section, and will not be repeated here. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the existing voice interaction interface.

[0055] Figure 2 This is a schematic diagram of another interface for voice interaction using existing technologies.

[0056] Figure 3 This is a schematic diagram of another interface for voice interaction using existing technologies.

[0057] Figure 4 This is a schematic diagram of another interface for voice interaction using existing technologies.

[0058] Figure 5 This is a schematic diagram of the existing voice interaction process.

[0059] Figure 6 This is a software architecture diagram of an electronic device provided in an embodiment of this application.

[0060] Figure 7 This is a flowchart of a voice interaction method provided in an embodiment of this application.

[0061] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0062] Figure 9 This is a schematic diagram of the sound source angle provided in one embodiment of this application.

[0063] Figure 10 This is a flowchart of calculating the initial screening confidence level provided in one embodiment of this application.

[0064] Figure 11 This is a framework diagram of a second speech model for speech discrimination provided in an embodiment of this application.

[0065] Figure 12 This is a flowchart of calculating the target confidence level provided in one embodiment of this application.

[0066] Figure 13 This is a flowchart of a voice interaction method provided in another embodiment of this application.

[0067] Figure 14 This is a framework diagram of a third speech model for speech discrimination provided in an embodiment of this application.

[0068] Figure 15 This is a flowchart of training a third speech model provided in one embodiment of this application.

[0069] Figure 16 This is a flowchart provided in one embodiment of the present application for determining whether audio data is the voice of a specified user.

[0070] Figure 17 This is a framework diagram of the second noise reduction process provided in an embodiment of this application.

[0071] Figure 18 This is a flowchart of a voice interaction method provided in another embodiment of this application.

[0072] Figure 19 This is a coordinate schematic diagram of an electronic device provided in an embodiment of this application.

[0073] Figure 20 This is a flowchart of a voice interaction method provided in another embodiment of this application.

[0074] Figure 21 This is a flowchart of a voice interaction method provided in another embodiment of this application.

[0075] Figure 22 This is a hardware architecture diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0076] In one embodiment of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in one embodiment of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to limit the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. Where there is no conflict, the following embodiments and features described herein can be combined with each other.

[0078] With the development of terminal technology, many electronic devices such as smartphones, personal computers, and smart speakers have voice assistant functions, enabling users to interact with these devices via voice to control them to perform specific operations, such as controlling home appliances, playing music, making phone calls, setting alarms, or setting to-do lists. However, when users want to control electronic devices via voice, they usually need to first activate the voice assistant using a wake word (i.e., a preset voice command) or a button command. The wake word is the name of the voice assistant set by the electronic device manufacturer, such as YOYO, Xiaoyi, Xiaodu, or Xiao Ai. Button commands can be generated by clicking or long-pressing a button on the electronic device, such as long-pressing the power button. Only after the voice assistant is activated can it receive and recognize the voice commands issued by the user to achieve voice interaction or control the electronic device to perform the specified operations. In particular, during the interaction process, if the user needs to engage in multiple voice interactions with the voice assistant before the assistant understands the user's command, the user needs to call out the voice assistant's name before each voice command. For example, assuming the voice assistant's name is AA, the user's voice message is "AA, please play music from J." The voice assistant's response is "We have found several music videos of J. Which one would you like to play?" The user's response still needs to include the voice assistant's name, such as "AA, please play the first video." This voice interaction process is cumbersome and does not conform to user habits, thus affecting the user experience.

[0079] In some application scenarios where a wake word isn't required to activate the voice assistant, users can wake it up without adding a specific wake word to the voice command. For example, they can simply bring the voice assistant close to the electronic device and speak the command, allowing the device to respond with voice interaction or perform a specified operation. This makes voice interaction with the electronic device more natural. Furthermore, not using a specific wake word during voice interaction aligns better with user habits. However, if a wake word isn't needed to activate the voice assistant and interact with the electronic device, the user's frequent speaking might lead to accidental activation, resulting in unnecessary voice interactions or unnecessarily controlled operations. This increases system power consumption and could even cause misoperations, negatively impacting the user experience.

[0080] See Figure 1The diagram illustrates a conventional voice interaction interface. Electronic devices use a specific wake-up word to activate the voice assistant. When the screen is off, the user speaks the wake-up word near the device or presses and holds the power button to issue a voice command to activate the assistant. The wake-up word can be the name of the voice assistant or any custom voice content. The voice assistant's name can be the assistant itself, a name set by the device manufacturer, or a user-defined name. For example, the wake-up word could be "AA". When the electronic device recognizes that the user's speech contains the wake-up word, or receives the user's press and hold of the power button, or clicks the preset voice wake-up button / icon on the device, it activates the voice assistant and outputs a prompt to guide the user to the next step. This prompt can be a sound, information displayed after the screen lights up, or a combination of both. The user then continues to issue voice commands to interact with the electronic device through the voice assistant. Clearly, this voice interaction is cumbersome, inefficient, and does not conform to user habits.

[0081] Electronic devices can also activate voice assistants without using a specific wake word, instead allowing the voice assistant to directly respond to the user's voice commands for voice interaction. For example, see [link to relevant documentation]. Figure 2 As shown, the electronic device's settings application or drop-down menu includes a control to enable the voice assistant function. After the user activates the voice assistant function by triggering the control, the electronic device displays the following: Figure 3 The interface shown prompts the user to use breath-activated voice assistant and guides them on how to control the electronic device to perform specific operations. For example, the prompt might read, "Raise your phone, bring the bottom of the phone close to your mouth (within 7 cm), and aim it at the bottom microphone to begin your conversation." With the voice assistant enabled, users can send voice commands directly to control the electronic device to perform specific operations, whether the screen is on or off. Figure 1 When the screen is off, as shown, the user can bring their phone close to the electronic device and say, "I want to listen to J's songs," and then issue a voice command to play J's songs, such as... Figure 4 As shown, the voice assistant responds to a voice command, opens the music player, and plays song J. However, since users typically carry or keep electronic devices with them, this voice interaction method could lead to accidental activation of the voice assistant due to casual conversations, resulting in unintended actions. Adding restrictions to voice commands to prevent these unintended actions would require users to perform a series of steps to ensure the commands meet these restrictions, increasing the user's workload and causing inconvenience.

[0082] Regarding the aforementioned wake-up-free solution, an improved solution has been proposed in the prior art, see reference. Figure 5 The diagram illustrates a prior art voice interaction process. For example, in voice interaction, the process of acquiring user voice and identifying whether it constitutes a voice command includes two stages. In the first stage, the amplitude and pose parameters (e.g., the variance of the electronic device's acceleration) of the acquired user voice are determined. If the amplitude of the user voice is greater than or equal to a preset amplitude threshold, and the pose parameters of the electronic device are greater than or equal to a preset pose parameter threshold, the user voice is then recognized in the second stage. In the second stage, the user voice and the pose parameters of the electronic device within 2 seconds are input, and the user voice features and the pose parameter features of the electronic device are extracted. These features are then input into a voice and pose parameter fusion model. The fusion model determines whether the user voice is a voice command and outputs the result.

[0083] The threshold comparison of user voice amplitude and pose parameters, as well as the recognition of user voice and pose parameters by the fusion model, are performed on the audio signal processor side of the electronic device. The audio signal processor is usually a chip in the electronic device that is mainly used to process audio signals. It has small memory and limited processing power. The network structure of the fusion model, i.e. the speech recognition model, set in it is also relatively simple and has weak iteration ability. As a result, the speech recognition accuracy of the speech recognition model is low. It may not be able to recognize voice commands, or may misrecognize user voice as voice commands, causing the electronic device to be unable to respond to the voice commands issued by the user, or to misrespond to user voice that is not a voice command, thus resulting in poor voice interaction effect.

[0084] To address the aforementioned issues, this application proposes a voice interaction method. A first voice model, configured on the audio signal processor side, performs preliminary screening of audio data to determine whether it is near-field speech. A second voice model, configured on the application processor side, determines whether the audio data is a voice command and outputs the determination result. Compared to the audio signal processor, the application processor of the electronic device has stronger processing power. The network results of the second voice model configured on the application processor can be more complex and have stronger iterative update capabilities. Therefore, the second voice model has higher speech recognition accuracy, ensuring accurate determination of whether audio data is a voice command and effectively optimizing the voice interaction effect.

[0085] See Figure 6 The diagram shown is a software architecture diagram of an electronic device provided in an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. For example, the Android system, from top to bottom, consists of the application layer 101, framework layer 102, Android runtime and system libraries 103, hardware abstraction layer 104, kernel layer 105, and hardware layer 106.

[0086] Application layer 101 may include a series of application packages. For example, application packages may include applications such as camera, gallery, calendar, calling, map, navigation, WLAN, Bluetooth, music, video, SMS, device control services, etc.

[0087] The framework layer 102 provides an Application Programming Interface (API) and programming framework for applications in the application layer. The application framework layer includes predefined functions. For example, it may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0088] The window manager manages window programs. It can obtain screen size, determine the presence of a status bar, lock the screen, and capture screenshots. The content provider stores and retrieves data, making it accessible to applications. This data can include videos, images, audio, made and received calls, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon can include views for displaying text and views for displaying images. The phone manager provides communication functionality for electronic devices, such as managing call status (including connection and disconnection). The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, and video files. The notification manager allows applications to display notifications in the status bar, conveying informational messages that disappear automatically after a short pause without user interaction. For example, the notification manager is used to notify of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the system's top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, causing electronic devices to vibrate, and flashing indicator lights.

[0089] The Android Runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core Android libraries.

[0090] Application layer 101 and framework layer 102 run in a virtual machine. The virtual machine executes the Java files of the application layer and framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0091] System library 103 may include multiple functional modules. For example, a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0092] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The Media Library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D Graphics Processing Library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D Graphics Engine is the drawing engine for 2D graphics.

[0093] Hardware Abstraction Layer 104 runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to the upper layers.

[0094] Kernel layer 105 is the layer between hardware and software. Kernel layer 105 contains at least the display driver, camera driver, audio driver, and sensor driver.

[0095] Kernel layer 105 is the core of the operating system for electronic devices. It is the first layer of software extension based on the hardware, providing the most basic functions of the operating system. It is the foundation for the operation of the operating system, responsible for managing system processes, memory, device drivers, files, and network systems, and determining the system's performance and stability. For example, the kernel can determine the timing of an application's operation on a certain part of the hardware.

[0096] Kernel layer 105 includes hardware-dependent programs such as interrupt handlers and device drivers, as well as basic, common, and frequently running modules such as clock management and process scheduling modules, and critical data structures. The kernel layer can be located within the processor or embedded in internal memory.

[0097] Hardware layer 106 includes the hardware of electronic devices, such as displays, buttons, cameras, etc.

[0098] For a detailed description of the implementation process of the voice interaction method, please refer to the descriptions in the various embodiments below.

[0099] See Figure 7 The diagram shown is a flowchart of a voice interaction method provided in an embodiment of this application. The method is applied in an electronic device, and the voice interaction method includes: S101: Collect audio data and input the audio data into the first speech model to obtain the initial screening confidence level.

[0100] See Figure 8 The diagram shown is a structural schematic of an electronic device according to an embodiment of this application. In one embodiment of this application, the electronic device 100 includes a microphone 170C disposed at the bottom (e.g., near the charging port), which can collect audio data in real time. Figure 6 As shown, the electronic device 100 also includes an audio signal processor 1061. After the microphone 170C at the bottom collects audio data, it sends the audio data to the audio signal processor 1061. A first speech model is set in the audio signal processor 1061. By inputting single-channel audio data into the first speech model, an initial screening confidence level is obtained. The initial screening confidence level characterizes the accuracy of the first speech model's determination that the audio data is near-field speech. Near-field speech refers to speech output by a sound source at a distance less than or equal to a first preset distance from the electronic device.

[0101] In another embodiment of this application, the electronic device 100 includes two microphones 170C disposed at the bottom (e.g., near the charging port) and the top (e.g., near the front-facing camera), which can respectively acquire audio data in real time. After acquiring audio data, the two microphones 170C respectively send the audio data to the audio signal processor 1061, thereby sending dual-channel audio data to the audio signal processor 1061. A first speech model is set in the audio signal processor 1061, and an initial screening confidence level is obtained by inputting the dual-channel audio data into the first speech model. Dual-channel speech can provide more acoustic information; for example, the energy difference between the two channels of speech can be used as a feature of the audio data, thereby improving the accuracy of speech discrimination.

[0102] In one embodiment of this application, the first speech model can be a standalone convolutional neural network model. The first speech model is trained using audio features corresponding to multiple speech commands as training data. For example, audio features are extracted from multiple speech commands as training data. The speech command can be speech output from a sound source at a distance less than or equal to a first preset distance from the electronic device (specifically, the distance to the microphone of the electronic device). The audio features can be time-domain features, frequency-domain features, or a combination of time-domain and frequency-domain features of the audio data. The first preset distance is less than 10 centimeters. For example, the first preset distance can be 6 centimeters, 7 centimeters, 8 centimeters, or other distances; this embodiment does not limit this. By inputting single-channel or dual-channel audio data into the first speech model, the first speech model can identify whether the received audio data is near-field speech, thereby initially determining whether the audio data is a speech command. It can also output the confidence level of the determination result, i.e., the initial screening confidence level. The initial screening confidence level can be the softmax probability output of the convolutional neural network model. By identifying whether the collected audio data is voice output from a sound source at a distance less than or equal to a first preset distance from the electronic device using the first speech model, it can be determined whether the audio data is near-field speech, that is, whether the user intends to issue a voice command. This avoids speech that is far from the electronic device being recognized as a voice command for voice interaction, thereby preventing the voice assistant from being accidentally woken up.

[0103] In another embodiment of this application, the first speech model may also be a deep neural network (DNN), a long short-term memory (LSTM) network, a Transformer model, a convolutional recurrent neural network (CRNN) model, or other models.

[0104] In another embodiment of this application, a first speech model can be generated by training audio features corresponding to multiple speech commands and audio features corresponding to multiple non-speech commands. Speech commands are speech output from a sound source at a distance less than or equal to a first preset distance from the electronic device, and are used as positive samples for training the first speech model. Non-speech commands are speech output from a sound source at a distance greater than or equal to a second preset distance from the electronic device, and are used as negative samples for training the first speech model. The second preset distance can be the same as the first preset distance, or less than or equal to 10 centimeters; for example, the second preset distance is 7 centimeters or 8 centimeters. The second preset distance can also be different from the first preset distance, for example, 9 centimeters, 10 centimeters, 11 centimeters, or other distances. By training the first speech model using audio features corresponding to speech commands as positive samples and audio features corresponding to non-speech commands as negative samples, the accuracy of the judgment results of the first speech model can be improved.

[0105] S102: If the initial screening confidence level is greater than or equal to the initial screening confidence level threshold, proceed to S103; if the initial screening confidence level is less than the initial screening confidence level threshold, return to S101. In one embodiment of this application, the initial screening confidence level can be 0.7, 0.8, 0.9, or other values, which are only examples and are not limited to these in actual applications.

[0106] In one embodiment of this application, if the initial screening confidence level is greater than or equal to the initial screening confidence threshold, it indicates that the audio data has passed the first stage of judgment on the audio signal processor side and has been preliminarily determined to be near-field speech.

[0107] S103, input the audio data into the second speech model to obtain the target confidence level.

[0108] like Figure 6 As shown, the electronic device 100 also includes an application processor (AP) 1062, and an audio signal processor 1061 communicates with the application processor 1062 through the kernel layer 105. To further accurately determine whether the audio data is a voice command, the audio signal processor 1061 sends the audio data (single-channel audio data or single-channel audio data) to the application processor 1062.

[0109] In one embodiment of this application, a second speech model is disposed in an application processor 1062. The second speech model includes multiple sub-speech models, which include, but are not limited to, any combination of two or more models selected from convolutional neural network models, deep neural network models, long short-term memory networks, Transformer models, and convolutional recurrent neural network models. Multiple sub-speech models are generated by training multiple sub-speech models using audio features corresponding to multiple speech commands as training data. Speech commands are speech output from a sound source at a distance less than or equal to a first preset distance from the electronic device. Audio features can be time-domain features, frequency-domain features, or a combination of time-domain and frequency-domain features of the audio data. By inputting single-channel or dual-channel audio data into the first speech model, the second speech model can identify whether the audio data is speech output from a sound source at a distance less than or equal to the first preset distance from the electronic device, thereby determining whether the audio data is a speech command. It can also output the target confidence score of the determination result. The target confidence score is used to characterize the accuracy of the second speech model in determining whether the audio data is a speech command. It is also used to determine the confidence score of whether the audio data is a speech command. The target confidence score can be the average, weighted average, or median of the confidence scores output by multiple sub-speech models of the second speech model.

[0110] In another embodiment of this application, the audio features corresponding to multiple voice commands and the audio features corresponding to multiple non-voice commands can be used as training data to train and generate a second voice model. Voice commands are speech output from a sound source at a distance less than or equal to a first preset distance from the electronic device, and are used as positive samples for training the second voice model. Non-voice commands are speech output from a sound source at a distance greater than or equal to the second preset distance from the electronic device, and are used as negative samples for training the second voice model. By using the audio features corresponding to multiple voice commands as positive samples and the audio features corresponding to multiple non-voice commands as negative samples to train the second voice model, the accuracy of the second voice model's discrimination results can be improved.

[0111] Compared to the first speech model, the second speech model includes multiple sub-speech models. It can combine the discrimination results of multiple sub-speech models to determine whether the audio data is a speech command, and the discrimination result is more accurate.

[0112] In another embodiment of this application, the model type of the second speech model can be the same as that of the first speech model, and the number of models in the second speech model can also be the same as the number of models in the first speech model. However, the number of feature samples used to train the second speech model is greater than the number of feature samples used to train the first speech model, making the judgment result of the second speech model more accurate. For example, the number of feature samples used to train the first speech model is 100, and the number of feature samples used to train the second speech model is 1000.

[0113] In another embodiment of this application, the number of model parameters in the second speech model is greater than the number of model parameters in the first speech model. For example, the first speech model has one of each type of parameter, while the second speech model has three of each type of parameter. Parameter types may include the number of iterations, learning rate, weights, bias values, number of convolutional kernels, and kernel size, etc.

[0114] In another embodiment of this application, the model type of the second speech model can be the same as that of the first speech model, but the network result of the second speech model is more complex. For example, the number of feature extraction layers in the second speech model is greater than that in the first speech model. By extracting more features, more audio data features can be extracted, thereby improving the accuracy of speech discrimination. For example, the first speech model has 3 feature extraction layers, while the second speech model has 6 feature extraction layers.

[0115] S104, calculate the sound source angle corresponding to the audio data, and determine the target confidence threshold corresponding to the sound source angle.

[0116] In one embodiment of this application, the application processor can also run a sound source angle detection algorithm, which is a Generalized Cross Correlation-Phase Transform (GCC-PHAT) algorithm. After the application processor receives the audio data and extracts the audio features of the audio data, it performs cross-correlation calculation on the dual-channel audio data using the GCC-PHAT algorithm to obtain the cross-correlation function of the dual-channel audio data. The audio signals x1 and x2 received by the two microphones (e.g., the bottom microphone and the top microphone of the electronic device) are respectively: (1); (2); In the calculation formulas (1) and (2), the audio signals x1 and x2 are the audio signals obtained after performing a fast Fourier transform on the audio data collected by the microphone, s(t) is the sound source signal, n1(t) and n2(t) are the ambient noise, and τ1 and τ2 are the propagation time of the sound source signal from the sound source to the two microphones.

[0117] The cross-correlation function between the audio signals x1 and x2 received by the two microphones is phase-transformed and weighted. The formula for calculating the phase-transformation weighting is (3): (3).

[0118] The formula (4) for calculating the cross-correlation function between the phase-transformed weighted audio signals x1 and x2 is as follows: (4).

[0119] In the calculation formula (4), x1 and x2 are the audio signals collected by the two microphones, α1 and α2 are the phases of the audio signals collected by the two microphones, and τ is the current time. 12 Let τ be the time delay between the audio signals acquired by the two microphones. The time delay τ between the audio signals acquired by the two microphones is obtained by estimating the peak value of the cross-correlation function. 12 Based on the time delay τ between the audio signals collected by the two microphones 12 The angle of the sound source is calculated from the distance d between the two microphones.

[0120] See Figure 9 The diagram shown is a schematic representation of the sound source angle according to an embodiment of this application. The distance between the two microphones of the electronic device is d, the sound source angle is θ, and Δr is the distance difference between the two microphones and the sound source, where Δr = τ. 12 ×c, where c is the propagation speed of the audio signal. The angle of the sound source is θ = arcsin(τ). 12 ×c / d).

[0121] In one embodiment of this application, the application processor pre-sets a correspondence between multiple preset sound source angle ranges and multiple target confidence thresholds. After determining the sound source angle of the audio data, it determines the preset sound source angle range into which the sound source angle of the audio data falls. Based on the preset sound source angle range into which the sound source angle of the audio data falls and the correspondence between the multiple preset sound source angle ranges and the multiple target confidence thresholds, it determines the target confidence threshold corresponding to the preset sound source angle range into which the sound source angle of the audio data falls. The preset sound source angle range is between -60° and 60°. For example, the correspondence between multiple preset sound source angle ranges and multiple target confidence thresholds includes: the target confidence threshold corresponding to the preset sound source angle range of 0-30 degrees is 0.7, the target confidence threshold corresponding to the preset sound source angle range of 30-60 degrees is 0.8, and the target confidence threshold corresponding to the preset sound source angle range of -30-0 degrees is 0.9. If the calculated sound source angle of the audio data is 45 degrees, then the sound source angle of the audio data falls within the preset sound source angle range of 30-60 degrees, and the corresponding target confidence threshold is 0.8.

[0122] In one embodiment of this application, by setting corresponding target confidence thresholds according to different sound source angle ranges, the restrictions on sound source angles for voice commands can be relaxed. When issuing voice commands, users do not need to set the angle between the electronic device and their mouth to a specified angle, reducing user operations and conforming to user habits.

[0123] S105: If the target confidence level is greater than or equal to the target confidence level threshold, proceed to S106; if the target confidence level is less than the target confidence level threshold, return to S101. In one embodiment of this application, the target confidence level threshold and the initial screening confidence level threshold may be different. For example, the target confidence level may be greater than the initial screening confidence level, specifically 0.75, 0.85, 0.95, or other values, and is not limited to these in practical applications. In other embodiments of this application, the target confidence level threshold may also be the same as the initial screening confidence level threshold.

[0124] S106, determine that the audio data is a voice command, and respond to the voice command to perform voice interaction.

[0125] In one embodiment of this application, when the target confidence level is greater than or equal to the target confidence threshold, it indicates that the audio data has been determined as a voice command by the application processor through the second stage of voice discrimination, thereby responding to the voice command to perform voice interaction.

[0126] Through the above embodiments, when audio data is collected, the first speech model first performs a preliminary judgment on the audio data, filtering out non-near-field speech, that is, filtering out audio data that is obviously not a voice command. This eliminates the need for the second speech model to recognize audio data that is obviously not a voice command. In addition, since the first speech model is relatively simple and runs on the audio signal processor side, the system power consumption of the speech recognition process is low. After the first speech model initially determines that the audio data is a voice command, the audio data is sent to the second speech model on the application processor side. Because the application processor has a larger memory and stronger computing power, the second speech model is more complex than the first speech model, and the judgment result is more accurate. Therefore, it can accurately determine whether the audio data is a voice command.

[0127] See Figure 10 The diagram shown is a flowchart illustrating the calculation of initial screening confidence using a first speech model according to an embodiment of this application. The process of calculating the initial screening confidence using a convolutional neural network model as an example is explained below.

[0128] S1011, preprocess the audio data and extract the audio features of the preprocessed audio data.

[0129] In one embodiment of this application, preprocessing the audio data includes: removing silent segments from the audio data and standardizing the audio intensity. Then, the preprocessed audio data is segmented into frames, dividing the preprocessed audio signal into short time segments, for example, using a window function with a length of 20-30 milliseconds. The segmented audio signal is then windowed, with each frame of the audio signal windowed using a window function (e.g., a Hamming window) to reduce spectral leakage. Then, a Fast Fourier Transform (FFT) is performed on each frame of the audio signal to obtain its frequency domain characteristics. Then, the square of the amplitude of each frame of the audio signal is calculated to obtain the power spectral density of each frame. The power spectral density of each frame of the audio signal is filtered through a filter bank to obtain the filtered energy. The logarithm of the filtered energy value is taken, and an inverse Fourier transform is performed on the logarithmic energy spectrum to obtain the cepstral coefficients. The cepstral coefficients ranked first by a predetermined number (e.g., 12 or 13) are selected as the final Mel Frequency Cepstrum Coefficients (MFCCs). In other embodiments of this application, the audio features of the extracted audio data may also be linear prediction cepstrum coefficients (LPCC).

[0130] S1012, input the audio features into the first speech model, and extract the convolutional features of the audio features through the convolutional layer of the first speech model.

[0131] In one embodiment of this application, the first speech model includes a convolutional layer, an activation layer, a pooling layer, and a fully connected layer. The convolutional layer receives the input audio features, performs convolution processing on the audio features, and extracts the convolutional features corresponding to the audio features.

[0132] S1013 performs nonlinear transformation processing on the convolutional features of audio features through the activation layer of the first speech model.

[0133] In one embodiment of this application, after the convolutional layer extracts the convolutional features corresponding to the audio features, the convolutional features are passed to the activation layer. In the activation layer, the convolutional features of the audio features are subjected to nonlinear transformation processing by an activation function (such as the ReLU function or Sigmoid).

[0134] S1014, the convolutional features of the audio features are pooled through the pooling layer of the first speech model.

[0135] In one embodiment of this application, the activation layer performs a nonlinear transformation on the convolutional features corresponding to the audio features and then passes the convolutional features to the pooling layer. In the pooling layer, the convolutional features are pooled using max pooling or average pooling, thereby reducing the dimensionality of the convolutional features.

[0136] S1015: The convolutional features of the audio features are classified through the fully connected layer of the first speech model to determine the judgment result of the audio features and calculate the initial screening confidence.

[0137] In one embodiment of this application, the pooling layer performs pooling processing on the convolutional features corresponding to the audio features, and then passes the convolutional features to the fully connected layer. In the fully connected layer, the convolutional features are classified using the softmax function to determine the judgment result of the audio features, that is, to determine whether the convolutional features belong to near-field speech or not, and to determine the probability distribution of the audio features belonging to near-field speech as the initial screening confidence level of the judgment result.

[0138] See Figure 11 The diagram shown illustrates the framework for speech discrimination using a second speech model provided in an embodiment of this application. After receiving dual-channel audio data, features of the dual-channel audio data are extracted and input into the second speech model. The second speech model then outputs a determination result indicating whether the audio data is a speech command and a target confidence level.

[0139] See Figure 12 The diagram shown is a flowchart illustrating the calculation of target confidence using a second speech model according to an embodiment of this application. The process of calculating target confidence using the second speech model is explained using multiple sub-models of the second speech model, including a convolutional neural network model and a long short-term memory network, as an example.

[0140] S1031, calculate the corresponding confidence level using the convolutional neural network model in the second speech model.

[0141] In one embodiment of this application, the method for calculating the confidence score of a convolutional neural network model is similar to... Figure 10 The methods in the illustrated embodiments are the same and will not be described in detail here.

[0142] S1032, calculate the corresponding confidence level through the long short-term memory network in the second speech model.

[0143] In one embodiment of this application, the Long Short-Term Memory (LSTM) network includes an input gate, a forget gate, an output gate, state units, and an LSTM result. The calculation formula for the input gate is as follows: (5); The formula for calculating the forgetting gate is: (6); The formula for calculating the output gate is: (7); The formula for calculating the state unit is: (8); The formula for calculating the LSTM result is: (9).

[0144] In the above calculation formulas (1) to (5), x t Let W be the audio feature input at time t, W be the weight matrix, b be the bias vector, and σ be the sigmoid function. The formula for calculating the sigmoid function is: (10) g and h are the input activation function and the output activation function, respectively. The input activation function and the output activation function are tanh functions, and the formula for calculating the tanh function is: (11).

[0145] In one embodiment of this application, the audio features extracted in S1011 are input into a Long Short-Term Memory (LSTM) network. The LSTM network outputs a determination result of whether the audio features are voice commands, and the calculated LSTM result is used as the confidence level of the LSTM network. The confidence level of the LSTM network can also be calculated using the softmax function.

[0146] S1033, calculate the target confidence level based on the confidence level obtained from the convolutional neural network model and the long short-term memory network.

[0147] In one embodiment of this application, the target confidence score is obtained by calculating the average value of the convolutional neural network model and the long short-term memory network. The sub-models of the second speech model in this embodiment are not limited to this; other speech models can be added as sub-models as needed. The target confidence score can also be increased by calculating the weighted average or median of multiple sub-models as the number of sub-models increases.

[0148] In one embodiment of this application, Long Short-Term Memory (LSTM) networks can solve the vanishing / exploding gradient problem of models, are suitable for processing sequential data such as speech, and have a flexible network structure that can adapt to various feature data, thereby improving the efficiency and accuracy of speech recognition.

[0149] See Figure 13 The diagram shown is a flowchart of a voice interaction method provided in another embodiment of this application. The method is applied in an electronic device, and the voice interaction method includes: S201, Collect audio data and input the audio data into the first speech model to obtain the initial screening confidence level.

[0150] S202: If the initial screening confidence level is greater than or equal to the initial screening confidence level threshold, execute S203; if the initial screening confidence level is less than the initial screening confidence level threshold, return to execute S201.

[0151] S203, input the audio data into the second speech model to obtain the target confidence level.

[0152] S204, calculate the sound source angle corresponding to the audio data, and determine the target confidence threshold corresponding to the sound source angle.

[0153] S205, if the target confidence level is greater than or equal to the target confidence level threshold, execute S206; if the target confidence level is less than the target confidence level threshold, return to execute S201.

[0154] S206: Input the audio data into the third speech model. If the audio data is determined to be the speech of the specified user, execute S207; if the audio data is determined not to be the speech of the specified user, return to execute S201.

[0155] In one embodiment of this application, a third speech model is configured in the application processor 1062. The third speech model can be a text-independent voiceprint model, which can be trained and generated using the voiceprint features of a specified user as training data. By inputting audio data (single-channel audio data or dual-channel audio data) into the third speech model, the third speech model can determine whether the audio data belongs to the specified user through voiceprint feature recognition. In one embodiment of this application, the specified user includes the owner of the electronic device and / or multiple users set by the owner of the electronic device (e.g., the owner's relatives or friends).

[0156] S207, determine that the audio data is a voice command, and respond to the voice command to perform voice interaction.

[0157] In one embodiment of this application, when the target confidence level is greater than or equal to the target confidence level threshold, it indicates that the audio data has passed the second stage of voice discrimination on the application processor side. When it is determined that the audio data is the voice of the specified user, the audio data is finally determined as a voice command, thereby responding to the voice command to perform voice interaction.

[0158] The specific implementation methods of S201-S205 are the same as those of S101-S105, and will not be described in detail here.

[0159] Through the above embodiments, the third voice model is used to determine whether the audio data is the voice of a specified user, so as to ensure that only the voice commands of the specified user can interact with the electronic device, thereby ensuring the privacy and security of the electronic device and ensuring that the voice commands are responded to correctly.

[0160] See Figure 14The diagram shows a framework diagram for speech discrimination using a third speech model provided in an embodiment of this application. S201: Obtain audio features identified by the first speech model. S202: Extract the speaker's representation corresponding to the audio data, such as voiceprint features. S203: Determine whether the electronic device has completed the extraction of the designated user's voiceprint features. If the electronic device has completed the extraction of the designated user's voiceprint features, proceed to S204; if the electronic device has not completed the extraction of the designated user's voiceprint features, proceed to S208. S204: Determine whether the speaker's voiceprint features are those of the designated user using a voiceprint feature decision device. If the speaker's voiceprint features are those of the designated user, proceed to S205; if the speaker's voiceprint features are not those of the designated user, proceed to S207. S205: Determine that the speaker is the designated user. S206: Update the designated user's voiceprint feature database based on the speaker's speech. S207: Determine that the speaker is not the designated user. S208: Obtain multiple speech segments collected by the electronic device and cluster the multiple speech segments.

[0161] See Figure 15 The diagram shown is a flowchart of training a third speech model according to an embodiment of this application. The process of training the third speech model is explained using a text-independent speaker model as an example.

[0162] S301: Acquire multiple speech segments captured by the microphone and extract the voiceprint features of each speech segment.

[0163] In one embodiment of this application, the voiceprint features of each speech segment can also be Mel-Cepstral Coefficients. The method for extracting Mel-Cepstral Coefficients is the same as the method for extracting Mel-Cepstral Coefficients in S1011, and will not be described in detail here.

[0164] S302, train a Gaussian mixture model corresponding to the voice of a specified user based on the voiceprint features of each speech segment.

[0165] In one embodiment of this application, the calculation formula for the Gaussian mixture model is as follows: (12) in, It is a D-dimensional random vector. It is an M-group Gaussian probability density function with D dimensions. The formula for calculating the D-dimensional Gaussian probability density function is: (13).

[0166] P i It is the mixture number of M groups of Gaussian vectors, where, . This is the parameter set of the Gaussian mixture model. A unique Gaussian mixture model can be determined based on the parameter set.

[0167] In one embodiment of this application, the training process of the Gaussian mixture model includes: setting an initial parameter set. The expected value maximization algorithm (EM algorithm) is used for iterative calculation. until .in, , Determined according to the K-means algorithm Let D be the covariance matrix, i=1,…,M, and D be the dimensions of the MFCC parameter vector, for example, 20, and assume it to be a diagonal matrix. , for A set of vectors.

[0168] In one embodiment of this application, the EM algorithm includes: selecting T training vectors. ; Calculate the posterior probability Where λ is the GMM parameter set determined in the previous iteration; calculate ;calculate ;calculate Determine if the condition is met. If satisfied The training of the GMM model corresponding to the specified user's voice is complete. If the following conditions are met... Return to the step of calculating the posterior probability and proceed to the next iteration.

[0169] S303, the trained Gaussian mixture model is determined as the text-independent voiceprint model corresponding to the speech of the specified user.

[0170] In another embodiment of this application, the K-means clustering algorithm can also be used to cluster the audio features of a specified user to generate a text-independent voiceprint model.

[0171] See Figure 16 The diagram shown is a flowchart illustrating how to determine whether audio data belongs to a specified user, according to an embodiment of this application. The process of determining whether audio data belongs to a specified user is explained using a text-independent voiceprint model as an example.

[0172] S2061, Extract the voiceprint features from the audio data.

[0173] In one embodiment of this application, the voiceprint features of the audio data can also be Mel-Cepstral Coefficients. The method for extracting Mel-Cepstral Coefficients is the same as the method for extracting Mel-Cepstral Coefficients in S1011, and will not be described in detail here.

[0174] S2062, input the voiceprint features of the audio data into the text-independent voiceprint model to obtain the determination result of whether the audio data is the voice of the specified user and the corresponding voiceprint confidence.

[0175] In one embodiment of this application, the voiceprint features of the audio data are input into a text-independent voiceprint model, and the posterior probability corresponding to the voiceprint features of the audio data is calculated by a Gaussian mixture model. It is then determined whether the posterior probability is greater than or equal to a preset probability. If the posterior probability is greater than or equal to the preset probability, the audio data is determined to be the voice of a specified user, and the posterior probability is used as the confidence level of the determination result that the audio data is the voice of the specified user.

[0176] In another embodiment of this application, if the text-independent voiceprint model is a K-means clustering model, the Euclidean distance between the voiceprint features of the audio data and the cluster centers of the voiceprint features corresponding to the voice of the specified user can be calculated. If the Euclidean distance between the voiceprint features of the audio data and the cluster centers of the voiceprint features corresponding to the voice of the specified user is less than or equal to a preset distance threshold, the audio data is determined to be the voice of the specified user; if the Euclidean distance between the voiceprint features of the audio data and the cluster centers of the voiceprint features corresponding to the voice of the specified user is greater than the preset distance threshold, the audio data is determined not to be the voice of the specified user.

[0177] In one embodiment of this application, the self-learning and verification process of the text-independent voiceprint model includes: collecting user speech through a microphone; determining whether the text-independent voiceprint model has completed learning; if the text-independent voiceprint model has not completed learning, acquiring multiple speech segments collected from the electronic device as wake-up-free data; extracting voiceprint features from the wake-up-free data; comparing the voiceprint features of the wake-up-free data with voiceprint features in the voiceprint database to determine whether the voiceprint features of the wake-up-free data match the voiceprint features in the voiceprint database; if the voiceprint features of the wake-up-free data are not human voice features, determining that the voiceprint features of the wake-up-free data do not match the voiceprint features in the voiceprint database, and continuing to collect user speech through the microphone; if the voiceprint features of the wake-up-free data and the features in the voiceprint database are both human voice features, determining that the voiceprint features of the wake-up-free data match the voiceprint features in the voiceprint database, adding the voiceprint features of the wake-up-free data to the voiceprint database, and updating the voiceprint database; and determining whether the number of voiceprint features in the voiceprint database is greater than or equal to a preset number. If the number of voiceprint features in the voiceprint database is greater than or equal to a preset number, the multiple voiceprint features in the voiceprint database are clustered and averaged. If the number of voiceprint features in the voiceprint database is less than the preset number, the user's voice is collected again via microphone. For example, the preset number could be 20, 30, 40, or other values. A text-independent voiceprint model is built based on the clustered and averaged voiceprint features, and the self-learning of the text-independent voiceprint model is completed. If the text-independent voiceprint model has not completed learning, it is determined whether the collected audio data passes the first-stage breath-awakening speech discrimination. If the collected audio data passes the first-stage breath-awakening speech discrimination, the voiceprint features of the collected audio data are extracted. If the collected audio data does not pass the first-stage breath-awakening speech discrimination, audio data is collected again via microphone. The voiceprint features of the audio data are verified in the backend. The voiceprint features of the audio data are input into the text-independent voiceprint model, and the text-independent voiceprint model is used to identify the voiceprint features of the audio data to determine whether the audio data is the voice of the specified user.

[0178] In one embodiment of this application, before inputting the audio data into the first speech model, the voice interaction method further includes: performing a first noise reduction process on the audio data using an audio signal processor.

[0179] In one embodiment of this application, the audio signal processor further includes a first noise reduction module, which can perform noise reduction processing on the audio using an acoustic echo cancellation (AEC) algorithm. The system determines whether the echo scene flag of the electronic device is a first flag or a second flag. If the echo scene flag is the first flag, audio data is collected through two microphones at the bottom and top of the electronic device, respectively. The first noise reduction module then performs noise reduction processing on the dual-channel audio data collected by the two microphones using the acoustic echo cancellation algorithm.

[0180] In one embodiment of this application, the echo scene flag of the electronic device includes a first flag and a second flag. For example, if the first flag is 1, it indicates that the electronic device is in a noisy scene and the speech needs to be denoised. If the second flag is 0, it indicates that the electronic device is not in a noisy scene and the speech does not need to be denoised.

[0181] In one embodiment of this application, the echo scene flag can be set according to the echo scene option. If the user enables the echo scene option, the electronic device sets the echo scene flag to a first flag; if the user disables the echo scene option, the electronic device sets the echo scene flag to a second flag. In another embodiment of this application, the echo scene flag is automatically set by the electronic device. The microphone can collect the sound around the electronic device in real time, and when the sound is collected, it determines whether the collected sound includes speech. If the collected sound does not include speech, it indicates that there is noise in the environment around the electronic device, and the echo scene flag is automatically set to the first flag. If the collected sound does not include any sound other than speech, it indicates that there is no noise in the environment around the electronic device, and the echo scene flag is automatically set to the second flag.

[0182] In one embodiment of this application, the acoustic echo cancellation algorithm includes a Normalized Least Mean Square (NLMS) filtering algorithm and a Kalman filtering algorithm. When the microphone acquires dual-channel audio data, it sends the dual-channel audio data to the audio signal processor. The first noise reduction module preprocesses the received dual-channel audio data, performs a short-time Fourier transform on the dual-channel audio data, converts the dual-channel audio data into a frequency domain signal, and uses the Normalized Least Mean Square (NLMS) filtering algorithm to adaptively reduce noise in the frequency domain signal corresponding to the dual-channel audio data. The filter weights are adjusted according to the difference between the frequency domain signal and the desired output signal. When the difference between the frequency domain signal and the desired output signal is 0 or less than or equal to a preset difference value, the Normalized Least Mean Square (NLMS) filtering algorithm converges, completing the noise reduction of the dual-channel audio data.

[0183] In one embodiment of this application, after the regularized minimum mean square error filtering algorithm completes the noise reduction of the dual-channel audio data, the Kalman filtering algorithm is used to further smooth the dual-channel audio data, establishing a state model and a measurement model for the dual-channel audio data. The state model is represented by a state vector, which includes multiple parameters of the speech signal, such as amplitude, phase, and frequency. The measurement model includes the frequency domain features corresponding to each parameter of the speech signal. Based on the state transition equation, the state model of the dual-channel audio data acquired at the previous time step, and the measurement model, the state of the dual-channel audio data at the current time step (including multiple parameters and frequency domain features) is predicted. The predicted state of the dual-channel audio data at the current time step is corrected based on the actual observed state of the dual-channel audio data at the current time step. Then, based on the state transition equation, the predicted state model of the dual-channel audio data acquired at the current time step, and the measurement model, the state of the dual-channel audio data at the next time step is predicted. This process is iterated until the Kalman filtering algorithm converges, completing the noise reduction of the dual-channel speech signal. After each iteration, the state estimate and covariance matrix of the Kalman filter algorithm are calculated. If the difference between two adjacent state estimates is 0 or less than or equal to the preset state estimate, and the covariance matrix is ​​less than or equal to the preset covariance matrix, the Kalman filter algorithm is considered to have converged.

[0184] In one embodiment of this application, the audio signal processor uses an acoustic echo cancellation algorithm to perform a first noise reduction process on the audio data to filter out the self-noise of the electronic device (i.e., the sound emitted by the electronic device itself, such as music being played), so as to avoid the self-noise affecting the accuracy of speech recognition.

[0185] In one embodiment of this application, after the audio data after the first noise reduction processing is recognized by the first speech model and before it is input into the second speech model, the voice interaction method further includes: performing a second noise reduction processing on the audio data through an application processor.

[0186] See Figure 17 The diagram shown is a framework diagram of the second noise reduction process provided in one embodiment of this application. In one embodiment of this application, the application processor of the electronic device further includes a second noise reduction module. Audio features of the acquired dual-channel audio data are extracted, and these features are input into the second noise reduction module. The second noise reduction module is a speech noise reduction model. The speech noise reduction model is used to perform a second noise reduction process on the dual-channel audio data, and the second noise-reduced audio data is output.

[0187] In one embodiment of this application, the second noise reduction module can be a convolutional neural network model, which is trained and generated from the audio features of clean user speech (i.e., user speech without noise). If the initial screening confidence level is greater than or equal to the initial screening confidence level threshold, the audio signal processor initiates the speech processing process of the application processor, sends the audio data to the application processor, preprocesses the audio data to obtain multiple sub-audio data, extracts the audio features of each sub-audio data, and uses the second noise reduction module of the application processor to identify whether each sub-audio data is clean user speech. If a sub-audio data is determined to be clean user speech, it is directly output. If a sub-audio data is determined not to be clean user speech, the clean user speech is subtracted from the audio data, that is, the audio signal of the audio data is subtracted from the speech signal of the clean user speech to obtain the difference between the audio data and the clean user speech. The noise in the audio data is filtered out based on the difference between the audio data and the clean user speech, thereby completing the second noise reduction processing of the audio data.

[0188] In another embodiment of this application, a speech activity detection algorithm can be used to identify whether the collected sound includes speech. If speech is detected, the collected sound is input into a second noise reduction module, and the noise-reduced speech is output through the second noise reduction module. The second noise reduction module can also be a convolutional recurrent network (CRN), a recurrent neural network noise reduction model RNNoise, a U-shaped network UNET, etc.

[0189] In one embodiment of this application, the application processor uses a speech denoising model to perform a second denoising process on the audio data to filter out external noise of the electronic device (i.e., environmental noise of the electronic device) and avoid external noise affecting the accuracy of speech recognition.

[0190] See Figure 18 The diagram shown is a flowchart of a voice interaction method provided in another embodiment of this application. The method is applied in an electronic device, and the voice interaction method includes: S501: Acquire audio data and determine whether the audio parameters of the audio data are greater than or equal to a preset audio parameter threshold. If the audio parameters of the audio data are determined to be greater than or equal to the preset audio parameter threshold, proceed to S502; if the audio parameters of the audio data are determined to be less than the preset audio parameter threshold, continue to proceed to S501.

[0191] In one embodiment of this application, the audio parameter is the energy of the audio data. Wherein, the energy E x The calculation formula is: (14).

[0192] In the calculation formula (14), N is the frame length for dividing the audio data signal into frames, and x n The audio signal is part of the audio data signal sequence.

[0193] In other embodiments of this application, the audio parameters may also be other parameters such as the amplitude and intensity of the audio data.

[0194] S502, input the audio data into the first speech model to obtain the initial confidence level.

[0195] S503: If the initial screening confidence level is greater than or equal to the initial screening confidence level threshold, execute S504; if the initial screening confidence level is less than the initial screening confidence level threshold, return to execute S501.

[0196] S504: Input the audio data into the second speech model to obtain the target confidence level.

[0197] S505, calculate the sound source angle corresponding to the audio data, and determine the target confidence threshold corresponding to the sound source angle.

[0198] S506, determine whether the target confidence level is greater than or equal to the target confidence level threshold. If the target confidence level is greater than or equal to the target confidence level threshold, proceed to S507; if the target confidence level is less than the target confidence level threshold, return to proceed to S501.

[0199] S507: Input the audio data into the third speech model. If the audio data is determined to be the speech of the specified user, execute S508; if the audio data is determined not to be the speech of the specified user, return to execute S501.

[0200] S508 determines that the audio data is a voice command from a specified user and responds to the audio data to perform voice interaction.

[0201] The specific implementation methods of S502-S508 are the same as those of S201-S207, and will not be described in detail here.

[0202] Through the above embodiments, before performing the first stage of recognition on the collected audio data, it is determined whether the audio parameters of the audio data are greater than or equal to a preset audio parameter threshold. The threshold comparison method is used to determine whether the audio data obviously does not meet the conditions of the voice command, and the audio data that obviously does not belong to the issued voice command is filtered out. There is no need for the first voice model and the second voice model to recognize the audio data that obviously does not belong to the issued voice command, which effectively improves the efficiency of voice discrimination and reduces system power consumption.

[0203] In another embodiment of this application, S501 can be replaced by: determining whether the audio parameters of the audio data are greater than or equal to a preset audio parameter threshold, and determining whether the first pose parameter of the electronic device is greater than or equal to a first preset pose parameter threshold. If it is determined that the audio parameters of the audio data are greater than or equal to the preset audio parameter threshold, and the first pose parameter of the electronic device is greater than or equal to the first preset pose parameter threshold, S502 is executed; if it is determined that the audio parameters of the audio data are less than the preset audio parameter threshold, or the first pose parameter of the electronic device is less than the first preset pose parameter threshold, S501 is continued.

[0204] In another embodiment of this application, the method for calculating the audio parameters of the audio data is the same as that in S501 described above, and will not be repeated here.

[0205] In another embodiment of this application, the first pose parameter of the electronic device is the variance of the three-axis acceleration of the electronic device, and the first preset pose parameter threshold includes a preset three-axis acceleration threshold. See also... Figure 19 The diagram shown is a coordinate schematic of an electronic device according to an embodiment of this application. A three-dimensional Cartesian coordinate system is established with the center point of the electronic device as the origin, the longer side as the X-axis, the shorter side as the Y-axis, and the direction perpendicular to the display screen as the Z-axis. The electronic device also includes an Inertial Measurement Unit (IMU), which includes, but is not limited to, an accelerometer, a gyroscope, an angular velocity sensor, a magnetometer, and a gravity sensor. The IMU can detect the three-axis angular velocity and acceleration of an object. The IMU detects the three-axis acceleration of the electronic device in real time and calculates the variances of the detected X-axis accelerations, Y-axis accelerations, and Z-axis accelerations respectively. The variances of the X-axis accelerations are compared with preset X-axis acceleration variance thresholds, the variances of the Y-axis accelerations are compared with preset Y-axis acceleration variance thresholds, and the variances of the Z-axis accelerations are compared with preset Z-axis acceleration variance thresholds.

[0206] If the variance of multiple X-axis accelerations is greater than or equal to a preset X-axis acceleration variance threshold, and the variance of multiple Y-axis accelerations is greater than or equal to a preset Y-axis acceleration variance threshold, and the variance of multiple Z-axis accelerations is greater than or equal to a preset Z-axis acceleration variance threshold, then the first pose parameter of the electronic device is determined to be greater than or equal to a first preset pose parameter threshold, thereby determining that the electronic device is in motion.

[0207] If the variance of multiple X-axis accelerations is less than a preset X-axis acceleration variance threshold, or the variance of multiple Y-axis accelerations is less than a preset Y-axis acceleration variance threshold, or the variance of multiple Z-axis accelerations is less than a preset Z-axis acceleration variance threshold, the first pose parameter of the electronic device is determined to be less than a first preset pose parameter threshold, and thus the electronic device is determined not to be in motion.

[0208] Through the above embodiments, before performing the first stage of recognition on the collected audio data, in addition to using threshold comparison to determine whether the audio data obviously does not meet the conditions for issuing voice commands, a threshold comparison is also used to determine whether the pose of the electronic device during the audio data collection process obviously does not meet the conditions for issuing voice commands. This further filters out audio data that obviously does not belong to the issuance of voice commands, eliminating the need for the first and second speech models to recognize audio data that obviously does not belong to the issuance of voice commands, effectively improving the efficiency of speech discrimination and reducing system power consumption.

[0209] See Figure 20 The diagram shown is a flowchart of a voice interaction method provided in another embodiment of this application. The method is applied in an electronic device, and the voice interaction method includes: S701: Acquire audio data, determine whether the audio parameters of the audio data are greater than or equal to a preset audio parameter threshold, and determine whether the first pose parameter of the electronic device is greater than or equal to a first preset pose parameter threshold. If it is determined that the audio parameters of the audio data are greater than or equal to the preset audio parameter threshold, and the first pose parameter of the electronic device is greater than or equal to the first preset pose parameter threshold, execute S702; if it is determined that the audio parameters of the audio data are less than the preset audio parameter threshold, or the first pose parameter of the electronic device is less than the first preset pose parameter threshold, return to execute S701.

[0210] S702 inputs the audio data into the first speech model to obtain the initial confidence level.

[0211] S703: If the initial screening confidence level is greater than or equal to the initial screening confidence level threshold, execute S704; if the initial screening confidence level is less than the initial screening confidence level threshold, return to execute S701.

[0212] S704, determine whether the second pose parameter of the electronic device is greater than or equal to the second preset pose parameter threshold. If it is determined that the second pose parameter of the electronic device is greater than or equal to the second preset pose parameter threshold, execute S705; if it is determined that the second pose parameter of the electronic device is less than the second preset pose parameter threshold, return to execute S701.

[0213] In one embodiment of this application, the second pose parameter of the electronic device is the pose confidence of the three-axis acceleration of the electronic device in motion, and the second preset pose parameter threshold is the confidence threshold corresponding to the second pose parameter.

[0214] In one embodiment of this application, the application processor of the electronic device further includes a pose detection model, which may be a convolutional neural network model. The pose detection model is trained and generated using multiple triaxial accelerations of the electronic device in motion as a feature set. If the initial screening confidence level is greater than or equal to the initial screening confidence level threshold, the audio signal processor initiates the speech processing process of the application processor, sends the multiple triaxial accelerations of the electronic device detected by the inertial measurement unit to the application processor, and determines whether the multiple triaxial accelerations of the electronic device are triaxial accelerations in motion through the pose detection model of the application processor, and calculates the pose confidence level of the determination result. The pose confidence level is used to characterize the probability that the triaxial accelerations of the electronic device are triaxial accelerations in motion.

[0215] S705 inputs the audio data into the second speech model to obtain the target confidence level.

[0216] S706, calculate the sound source angle corresponding to the audio data, and determine the target confidence threshold corresponding to the sound source angle.

[0217] S707, determine whether the target confidence level is greater than or equal to the target confidence level threshold. If the target confidence level is greater than or equal to the target confidence level threshold, proceed to S708; if the target confidence level is less than the target confidence level threshold, return to proceed to S701.

[0218] S708: Input the audio data into the third speech model and determine whether the audio data is the voice of the specified user. If the audio data is determined to be the voice of the specified user, proceed to S709; if the audio data is determined not to be the voice of the specified user, return to and proceed to S701.

[0219] S709 determines that the audio data is a voice command from a specified user and responds to the voice command to perform voice interaction.

[0220] Through the above embodiments, on the application processor side, before performing the second stage of recognition on the collected audio data, model recognition is used to accurately determine whether the pose of the electronic device during the audio data collection process obviously does not meet the conditions for issuing voice commands. Since the application processor has strong computing power, the model recognition method can more accurately filter out audio data that obviously does not belong to the issuance of voice commands, eliminating the need for a second voice model to recognize audio data that obviously does not belong to the issuance of voice commands, effectively improving the efficiency of voice discrimination and reducing system power consumption.

[0221] See Figure 21The diagram shown is a flowchart of a voice interaction method provided in another embodiment of this application. The method is applied in an electronic device. In this other embodiment, the speech discrimination of the second speech model, the speech discrimination of the voiceprint model, and the discrimination of the second pose model can be processed in parallel. The voice interaction method includes: S901: Acquire audio data, determine whether the audio parameters of the audio data are greater than or equal to a preset audio parameter threshold, and determine whether the first pose parameter of the electronic device is greater than or equal to a first preset pose parameter threshold. If it is determined that the audio parameters of the audio data are greater than or equal to the preset audio parameter threshold, and the first pose parameter of the electronic device is greater than or equal to the first preset pose parameter threshold, execute S902; if it is determined that the audio parameters of the audio data are less than the preset audio parameter threshold, or the first pose parameter of the electronic device is less than the first preset pose parameter threshold, continue executing S901.

[0222] S902, input the audio data into the first speech model to obtain the initial screening confidence level.

[0223] S903, determine whether the initial screening confidence level is greater than or equal to the initial screening confidence level threshold. If the initial screening confidence level is greater than or equal to the initial screening confidence level threshold, execute S904; if the initial screening confidence level is less than the initial screening confidence level threshold, return to execute S901.

[0224] S904, determine whether the second pose parameter of the electronic device is greater than or equal to the second preset pose parameter threshold.

[0225] S905 inputs the audio data into the second speech model to obtain the target confidence level.

[0226] S906, calculate the sound source angle corresponding to the audio data, and determine the corresponding target confidence threshold based on the sound source angle.

[0227] S907, determine whether the target confidence level is greater than or equal to the target confidence level threshold.

[0228] S908 inputs audio data into the third speech model to determine whether the audio data is the voice of the specified user.

[0229] S909: Obtain three judgment results: whether the second pose parameter is greater than or equal to the second preset pose parameter threshold, whether the target confidence score is greater than or equal to the target confidence threshold, and whether the audio data is the voice of the specified user. Determine whether the three judgment results meet preset conditions. If the three judgment results meet the preset conditions, execute S910; if the three judgment results do not meet the preset conditions, return to execute S901.

[0230] In one embodiment of this application, when all three determination results are yes, it is determined that the three determination results meet the preset conditions; when any one of the three determination results is no, it is determined that the three determination results do not meet the preset conditions. In other embodiments of this application, when the determination results of whether the target confidence level is greater than or equal to the target confidence level threshold and whether the audio data is the voice of the specified user are yes, it is determined that the three determination results meet the preset conditions; when the determination results of whether the target confidence level is greater than or equal to the target confidence level threshold or whether the audio data is the voice of the specified user are no, it is determined that the three determination results do not meet the preset conditions.

[0231] S910 determines that the audio data is a voice command from a specified user and responds to the voice command to perform voice interaction.

[0232] This application also provides an electronic device 100, see reference. Figure 22 As shown, the electronic device 100 may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. The embodiments of this application do not impose special restrictions on the specific type of electronic device 100.

[0233] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, Universal Serial Bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0234] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0235] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0236] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0237] The processor 110 may also include a memory for storing instructions and data. In one embodiment of this application, the memory in the processor 110 is a cache memory. The memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instructions or data again, it can directly retrieve them from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0238] In one embodiment of this application, the processor 110 may include one or more interfaces. These interfaces may include an Inter-integrated Circuit (I2C) interface, an Inter-integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bus (USB) interface, etc.

[0239] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In one embodiment of this application, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0240] The I2S interface can be used for audio communication. In one embodiment of this application, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to realize communication between the processor 110 and the audio module 170. In one embodiment of this application, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to realize the function of answering phone calls through a Bluetooth headset.

[0241] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In one embodiment of this application, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In another embodiment of this application, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0242] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In one embodiment of this application, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In one embodiment of this application, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback via Bluetooth headphones.

[0243] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a Camera Serial Interface (CSI) and a Display Serial Interface (DSI). In one embodiment of this application, the processor 110 and the camera 193 communicate via the CSI interface to realize the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to realize the display function of the electronic device 100.

[0244] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In one embodiment of this application, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0245] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. Furthermore, the interface can be used to connect other electronic devices 100, such as AR devices.

[0246] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0247] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0248] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0249] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0250] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0251] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In one embodiment of this application, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In another embodiment of this application, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0252] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In one embodiment of this application, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and housed within the same device as the mobile communication module 150 or other functional modules.

[0253] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including Wireless Local Area Networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0254] In one embodiment of this application, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the Beidou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0255] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for voice interaction, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0256] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light-Emitting Diode (AMOLED), a Flexible Light-Emitting Diode (FLED), a Minied, Microled, Micro-OLED, or a Quantum Dot Light-Emitting Diode (QLED), etc. In one embodiment of this application, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0257] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0258] The ISP is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In one embodiment of this application, the ISP can be set in the camera 193.

[0259] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In one embodiment of this application, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0260] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0261] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0262] NPU stands for Neural Network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech discrimination, and text understanding.

[0263] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0264] Random access memory can include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), and double data rate synchronous dynamic random-access memory (DDR SDRAM, such as fifth-generation DDR SDRAM, which is generally called DDR5 SDRAM). Non-volatile memory can include disk storage devices and flash memory.

[0265] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multi-media card (eMMC), etc.

[0266] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0267] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0268] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0269] Internal memory 121 or external memory interface 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by processor 110. The one or more computer programs include multiple instructions, which, when executed by processor 110, can implement the screen display detection method executed on electronic device 100 in the above embodiments, so as to realize the screen display detection function of electronic device 100.

[0270] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0271] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In one embodiment of this application, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0272] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0273] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0274] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0275] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0276] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0277] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0278] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0279] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In one embodiment of this application, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100. This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on the electronic device 100, the electronic device 100 executes the aforementioned related methods to implement the voice interaction method in the above embodiments.

[0280] This application also provides a computer program product that, when run on a computer, causes the computer to execute the aforementioned related actions to implement the voice interaction method described in the above embodiments.

[0281] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein, the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the voice interaction methods in the above-described method embodiments.

[0282] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0283] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0284] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0285] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0286] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0287] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts or all or part of the technical solutions that contribute to the prior art, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0288] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A voice interaction method applied to electronic devices, characterized in that, The electronic device includes a first microphone and a second microphone, the distance between the first microphone and the second microphone being d, and the method includes: When a sound source approaches the electronic device and speaks into the first microphone or the second microphone, audio data is collected through the first microphone and the second microphone. Based on the time delay between the audio data collected by the first microphone and the second microphone, the sound source angle θ formed by the sound source and the plane where the first microphone and the second microphone are located is calculated; When the sound source angle θ is within the first preset angle range, and the distance between the sound source and the electronic device is less than or equal to the first preset distance, the audio data is determined to be a voice command.

2. The voice interaction method according to claim 1, characterized in that, The first preset distance is less than or equal to 10 centimeters.

3. The voice interaction method according to claim 1, characterized in that, The first preset angle range is between -60° and 60°.

4. The voice interaction method according to claim 1, characterized in that, The first microphone is located at the bottom of the electronic device, and the second microphone is located at the top of the electronic device.

5. The voice interaction method according to claim 1, characterized in that, The sound source angle θ satisfies the following relationship: θ = arcsin(Δr / d), Wherein, Δr is the difference between the distance from the sound source to the first microphone and the distance from the sound source to the second microphone.

6. The voice interaction method according to claim 1, characterized in that, The method further includes: When the sound source angle θ is within the second preset angle range, the threshold of the first preset distance is set to the first distance value; When the sound source angle θ is within the third preset angle range, the threshold of the first preset distance is set to the second distance value; Wherein, the degree of angular deviation in the second preset angle range is less than that in the third preset angle range, and the first distance value is greater than the second distance value.

7. The voice interaction method according to claim 1, characterized in that, The method further includes: performing a voice interaction operation in response to the voice command, including: A voice interaction interface is displayed on the screen of the electronic device; Based on the content of the voice command, control the electronic device to perform corresponding functional operations.

8. The voice interaction method according to claim 1, characterized in that, The method further includes: Before determining that the audio data is a voice command, it is determined whether the pose parameters of the electronic device meet preset conditions; When the pose parameters meet the preset conditions, the step of determining that the audio data is a voice command is executed.

9. The voice interaction method according to claim 8, characterized in that, The pose parameters include the lift angle of the electronic device or the triaxial acceleration of the electronic device.

10. The voice interaction method according to claim 1, characterized in that, The method further includes: When the sound source angle θ is not within the first preset angle range, or when the distance between the sound source and the electronic device is greater than the first preset distance, the audio data is not responded to.

11. The voice interaction method according to claim 1, characterized in that, The electronic device includes an audio signal processor and an application processor, and the method further includes: The audio data is input into the first speech model of the audio signal processor, and the first speech model receives the audio data as an audio data event. When the first speech model determines that the audio data meets the preset conditions, the first speech model outputs a successful trigger event.

12. The voice interaction method according to claim 11, characterized in that, The method further includes: The trigger success event is received through a sound triggering framework.

13. The voice interaction method according to claim 12, characterized in that, The method further includes: The sound triggering framework transmits the successful trigger event to the application of the application processor.

14. The voice interaction method according to claim 11, characterized in that, The first speech model that inputs the audio data into the audio signal processor includes: The audio data is received from the digital signal processor via the audio signal processor; The audio data is input into the first speech model as an audio data event. When the first speech model determines that the audio data is near-field speech, the first speech model outputs the successful trigger event.

15. The voice interaction method according to claim 11, characterized in that, The method further includes: The audio data is input into the first speech model of the audio signal processor to obtain the initial confidence level, including: The audio data is preprocessed, and the audio features of the preprocessed audio data are extracted. The audio features are input into the first speech model, and the convolutional features of the audio features are extracted through the convolutional layer of the first speech model. The convolutional features of the audio features are nonlinearly transformed through the activation layer of the first speech model; The convolutional features of the audio features are pooled through the pooling layer of the first speech model. The convolutional features of the audio features are classified by the fully connected layer of the first speech model to determine the judgment result of the audio features, and the initial screening confidence is calculated.

Images