Multimodal intelligent terminal ai voice wake-up method and device based on voice control

Abstract

The application relates to the technical field of voice analysis, in particular to a multi-modal intelligent terminal AI voice wake-up method and device based on voice control, which comprises the following steps: extracting the features of each voice sample by using a voice model, and extracting the features of each facial action sample by using a video model; voice samples containing a wake-up voice are recorded as positive samples, and voice samples not containing a wake-up voice are recorded as negative samples; the features of the facial action samples corresponding to the positive and negative samples are used to update the parameters of the voice model and update the facial action samples corresponding to the positive and negative samples; then the parameters of the video model are updated by using the updated facial action samples; and the above parameter updating process is repeated. The application realizes efficient and accurate voice wake-up functions in a complex and changeable voice environment without significantly increasing the model parameters.

Description

Technical Field

[0001] This invention relates to the field of voice analysis technology, and more specifically to a method and apparatus for AI voice wake-up of multimodal intelligent terminals based on voice control. Background Technology

[0002] Voice wake-up, as the "first point of entry" for user interaction with devices, is of paramount importance. It refers to the device continuously monitoring its environment while in standby mode, only being activated and entering a full voice interaction mode when a preset keyword is detected. Traditional voice wake-up typically relies on manually designed acoustic features such as Mel-frequency cepstral coefficients (MFCC) and Fourier transform techniques, or on large-scale speech recognition models to identify wake-up words and initiate the wake-up process.

[0003] However, achieving efficient and accurate wake-up in complex and ever-changing real-world environments still faces significant challenges. These challenges mainly include: environmental noise interference, multiple conversations and reverberation, and significant wake-up delays. To improve accuracy in complex environments, deeper models with larger parameters and more complex structures are usually required, but this leads to high computational costs and significant wake-up delays. In addition, relying solely on audio information and lacking the utilization of other complementary modalities results in a lack of effective auxiliary judgment criteria during voice wake-up, making the voice wake-up results unreliable.

[0004] In summary, existing technologies struggle to achieve efficient and accurate voice wake-up functionality in complex and ever-changing speech environments without significantly increasing model complexity and computational overhead. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method and apparatus for AI voice wake-up of a multimodal intelligent terminal based on voice control.

[0006] The AI ​​voice wake-up method and device for multimodal intelligent terminals based on voice control of the present invention adopts the following technical solution:

[0007] One embodiment of the present invention provides an AI voice wake-up method for a multimodal smart terminal based on voice control, the method comprising the following steps:

[0008] D1: Each speech sample corresponds to a facial action sample. The speech model is used to extract the features of each speech sample, and the video model is used to extract the features of each facial action sample. The features of the pre-recorded wake-up speech and facial action samples are denoted as F1 and F2, respectively.

[0009] D2: Speech samples containing wake-up speech are denoted as positive samples, and speech samples without wake-up speech are denoted as negative samples. The features of the facial action samples corresponding to the positive and negative samples are denoted as the first feature and the second feature. The average of the features of the positive samples and the first feature is then compared with the F1 score to obtain the first difference; the average of the features of the negative samples and the second feature is then compared with the F1 score to obtain the second difference; the parameters of the speech model and the facial action samples corresponding to the positive and negative samples are updated to make the first difference less than the second difference.

[0010] D3: The difference between the features of the updated facial motion sample corresponding to the negative sample and F2 is recorded as the third difference, and the difference between the features of the updated facial motion sample corresponding to the positive sample and F2 is recorded as the fourth difference. The parameters of the video model are updated using the updated facial motion sample so that the third difference is greater than the fourth difference.

[0011] D4: Repeat D2~D3 until the distributions of the first and second differences have the smallest difference with the distributions of the third and fourth differences, and then perform wake-up recognition.

[0012] Preferably, each voice sample corresponds to a facial action sample, and the specific steps include the following:

[0013] Facial movement samples collected during the recording of wake-up voice are recorded as baseline movement samples;

[0014] Initialize an action generation vector for any speech sample, and add noise to the baseline action sample using the action generation vector as the standard deviation to obtain a facial action sample for any speech sample.

[0015] Preferably, whenever D2 is executed, the action generation vectors corresponding to all speech samples are normalized to obtain the normalized vector of each speech sample; when the Euclidean distance between the normalized vectors corresponding to positive and negative samples is less than a first preset threshold, the Euclidean distance between the normalized vectors corresponding to positive and negative samples is recorded as the recognition distance.

[0016] The parameters of the updated speech model and the facial action samples corresponding to the updated positive and negative samples are such that the first difference is smaller than the second difference, while the difference between the first difference and the second difference is negatively correlated with the recognition distance.

[0017] Preferably, the steps for updating the parameters of the speech model and updating the facial action samples corresponding to the positive and negative samples, such that the first difference is less than the second difference, and that the difference between the first and second differences is negatively correlated with the recognition distance, include the following:

[0018] For any pair of positive and negative samples, the action generation vectors corresponding to the positive and negative samples are treated as parameters. The parameters of the video model are frozen, and the speech model and video model are treated as a whole model. A loss function S1 is set for the whole model:

[0019]

[0020] in, Indicates the difference in objectives. Indicates the first difference, Y0 represents the loss function of the autoencoder corresponding to the speech model, representing the second difference.

[0021] , where d0 represents the recognition distance; exp() represents an exponential function with the natural constant as the base;

[0022] The parameters of the speech model and the action generation vector are iteratively updated using the loss function S1. Then, the facial action samples corresponding to the positive and negative samples are regenerated using the updated action generation vector. The iterative update process stops when the first difference is less than the second difference.

[0023] Preferably, the specific steps for updating the parameters of the video model using the updated facial motion samples, such that the third difference is greater than the fourth difference, are as follows:

[0024] For any pair of positive and negative samples, the third and fourth differences are obtained. A loss function S2 is set based on the third and fourth differences. The loss function S2 and the difference between the fourth and third differences are positively correlated.

[0025] The parameters of the video model are iteratively updated using the loss function S2, and the iterative update process continues when the third difference is greater than the fourth difference.

[0026] Preferably, the step of stopping when the distributions of the first difference and the second difference have the minimum difference with the distributions of the third difference and the fourth difference includes the following specific steps:

[0027] Repeating the process from D2 to D3 once is called performing one alternating iteration process;

[0028] The first and second differences obtained in each alternation iteration constitute set C1, and the third and fourth differences obtained in each alternation iteration constitute set C2;

[0029] After each alternating iteration, the parameters of the video model and the speech model are updated once. The loss functions used for the video model and the speech model during the parameter update are S3 and S4, respectively. S3 and S4 are positively correlated with the distribution differences of sets C1 and C2, respectively.

[0030] Preferably, the specific steps for obtaining the distribution differences between set C1 and set C2 are as follows:

[0031] Perform linear normalization on all elements in sets C1 and C2 together; plot the normalized elements in set C1 as a histogram curve, denoted as the first curve, and plot the normalized elements in set C2 as a histogram curve, denoted as the second curve; use the DTW distance between the first curve and the second curve as the distribution difference.

[0032] Preferably, the wake-up recognition includes the following specific steps:

[0033] At each moment, a voice sample Q1 and a corresponding facial action sample Q2 are collected in real time using a microphone. The features of the voice sample Q1 are extracted using a voice model, and the features of the facial action sample Q2 are extracted using a video model. The mean F of the features of the voice sample Q1 and the facial action sample Q2 is obtained. The mean of F1 and F2 is denoted as F3. When the Euclidean distance between F and F3 is less than a first preset wake-up threshold, wake-up is initiated. When the Euclidean distance between F and F3 is greater than or equal to the first preset wake-up threshold, wake-up is not initiated.

[0034] Preferably, the specific steps for obtaining the facial motion samples are as follows:

[0035] When the wake-up voice is recorded or the voice sample Q1 is collected, the camera is used to capture the user's video. Facial key point detection technology is used to obtain all key points of the user's face in each frame of the video. The positions of the same key point in all frames of the face image constitute a sequence. The sequence corresponding to all key points in all frames of the face image constitutes a facial action sample.

[0036] Another embodiment of the present invention provides a voice-controlled multimodal intelligent terminal AI voice wake-up device, which includes a microphone, a camera, a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it reads the voice and video collected by the microphone and the camera, and executes all the steps of the above-described voice-controlled multimodal intelligent terminal AI voice wake-up method.

[0037] The beneficial effects of the technical solution of the present invention are:

[0038] This invention utilizes speech and video models to perform voice wake-up from different modalities, solving the problem that relying solely on audio information and lacking utilization of other complementary modalities leads to a lack of effective auxiliary judgment criteria during voice wake-up.

[0039] Furthermore, this invention updates the parameters of the speech model and the facial action samples corresponding to the positive and negative samples, making the first difference smaller than the second difference. In this process, the features of the facial action samples corresponding to the speech samples are used to assist the speech model in further learning and training (i.e., further fine-tuning the parameters of the speech model). Simultaneously, considering that the facial action samples assigned to all speech samples may have insufficient auxiliary capabilities when assisting the speech model in further learning and training, resulting in the speech model's feature extraction ability not being maximized, the facial action samples corresponding to the speech samples are also reallocated (i.e., updating the facial action samples corresponding to the positive and negative samples). Thus, without increasing the number of model parameters and complexity, the feature extraction ability of the speech model is initially improved.

[0040] Furthermore, this invention denotes the difference between the features of the updated facial action samples corresponding to negative samples and F2 as the third difference, and the difference between the features of the updated facial action samples corresponding to positive samples and F2 as the fourth difference. The parameters of the video model are then updated using the updated facial action samples, ensuring that the third difference is greater than the fourth difference. This process re-updates the video model's parameters using the updated facial action samples. By re-updating the video model's parameters, the feature extraction capability of the video model can be adapted to the distribution of the updated facial action samples, preventing insufficient feature extraction capability from affecting the aforementioned process of using facial action samples to assist the speech model in further learning and training.

[0041] Finally, this invention alternately executes the update processes of the speech model parameters and facial action samples, as well as the update process of the video model parameters, to minimize the differences between the distributions of the first and second differences and the distributions of the third and fourth differences. This avoids the contradiction between the speech model's output features tending towards voice wake-up and the video model's output features tending towards non-wake-up, or between the speech model's output features tending towards non-wake-up and the video model's output features tending towards voice wake-up, further improving the accuracy of voice wake-up when combining the speech and video models.

[0042] In summary, this invention combines speech and video models and updates the parameters appropriately, achieving efficient and accurate voice wake-up in complex and variable speech environments without significantly increasing model complexity and computational overhead. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 The flowchart illustrates the steps of a voice-controlled multimodal smart terminal AI voice wake-up method according to an embodiment of the present invention. Detailed Implementation

[0045] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the voice-controlled multimodal intelligent terminal AI voice wake-up method and apparatus proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0047] It should be noted that there are many commonly used speech recognition methods, such as Whisper and FunAudio-ASR large-model technologies; these methods can all accurately recognize target speech signals. However, the purpose of this invention is voice wake-up, not speech recognition. Voice wake-up refers to the process of continuously monitoring the audio stream in the environment, detecting whether it contains preset specific keywords or key speech segments, and activating the device for full voice interaction after detecting the keywords or key speech segments. Unlike traditional continuous speech recognition, voice wake-up only needs to detect the target keywords or key speech segments from the continuous speech stream, without needing to accurately recognize non-keywords or key speech segments. Directly using existing large-model speech recognition for voice wake-up results in significant issues such as reduced device power consumption, high cost, and noticeable wake-up time delay.

[0048] Conventional wake-up methods involve pre-recording a wake-up voice and extracting its features. During wake-up, the system collects audio streams from the environment in real time and compares the features of the audio streams with those of the wake-up voice to determine whether device wake-up is necessary (e.g., waking up vehicles, in-vehicle devices, smart home devices, etc.). Commonly used and efficient voice feature extraction methods include Fourier transform algorithms and Mel-frequency cepstral coefficients. However, these methods are prone to false wake-ups in complex and variable voice environments (e.g., strong ambient noise or numerous sound sources). Therefore, this invention provides a specific solution—a voice-controlled multimodal intelligent terminal AI voice wake-up method and device—to achieve voice wake-up in complex and variable voice environments.

[0049] Comparative Example:

[0050] Record a wake-up voice message from the user, and capture a video of the user's face using a camera while recording the wake-up voice message.

[0051] A speech model is trained; this model is an autoencoder, which includes an encoder and a decoder. Acquired speech samples are input into the encoder, which extracts features from the speech samples and outputs a high-dimensional vector. This high-dimensional vector is then input into the decoder, which reconstructs the high-dimensional vector and outputs a result identical to the speech sample. The high-dimensional vector represents the features of the speech sample.

[0052] In this embodiment, the voices of different people communicating are collected as speech samples. Each speech sample in this embodiment refers to a speech signal with a duration of 3 seconds. In some embodiments, a large number of speech samples can be generated from a large amount of text using text-to-speech methods (such as FastSpeech technology). At the same time, different speech samples (including user-recorded wake-up voice) and collected environmental noise are mixed or superimposed to perform large-scale amplification of real speech samples. All speech samples constitute a dataset, which contains speech data under complex and variable environments.

[0053] The dataset is used to train a speech model. The trained speech model is then used to extract a high-dimensional vector for each speech sample, which is denoted as the feature of each speech sample. In this embodiment, the dimension of the high-dimensional vector is set to 128.

[0054] In this embodiment, both the encoder and decoder use LSTM networks. Other embodiments may use TCN networks; this embodiment does not impose a specific limitation. The high-dimensional vector described in this embodiment uses the softmax activation function, therefore the value of each dimension in the high-dimensional vector is normalized. Autoencoder technology is a well-known technique, and its specific principles and training process will not be elaborated upon in this embodiment.

[0055] Furthermore, when recording wake-up voice commands, a video of the user's face can be captured using a camera.

[0056] This method uses facial recognition technology to extract the user's face image from each frame of a video. Then, it employs facial landmark detection technology (such as a DCNN network) to obtain all key points on the user's face within each frame, including facial contour key points and lip key points. The positions of the same key point across all frames constitute a sequence. Multiple sequences are represented by all key points across all frames, and these sequences are recorded as facial motion samples in the video, describing the facial movements when the user speaks. In this embodiment, considering that lip key points are most relevant to the speech signal, only the sequences corresponding to lip key points in the facial motion samples are retained.

[0057] This embodiment utilizes a video model to extract features from facial motion samples.

[0058] Features are extracted from the recorded wake-up speech using a speech model, denoted as the baseline speech features, and represented as F1. Features are extracted from facial motion samples captured in the video during the recording of the wake-up speech using a video model, denoted as the baseline motion features, and represented as F2.

[0059] In this embodiment, every second is considered a moment. The ambient audio stream is collected in real time through the microphone, and video data is collected through the camera. At each moment, the audio stream collected in the most recent 3 seconds is used as a speech sample. The speech model is used to extract the features of the speech sample and determine whether the features are similar to the baseline speech features F1.

[0060] Meanwhile, for the video captured within the most recent 3 seconds at each moment, the features of the facial action samples in the video are obtained, and the features of the facial action samples are extracted using the video model to determine whether the features are similar to the baseline action features F2.

[0061] If the feature is similar to both the baseline speech feature F1 and the baseline action feature F2, the device is woken up; otherwise, the device is not woken up.

[0062] In this embodiment, voice data and video data are combined for voice wake-up, which improves the accuracy of voice wake-up in complex and ever-changing voice environments to a certain extent.

[0063] Example 1:

[0064] The above comparative embodiments still have the following problems:

[0065] (1) Regarding feature extraction from speech samples, this embodiment uses autoencoder technology to replace Fourier transform algorithms, Mel-Cepstral coefficients, and other techniques, making the feature extraction results of speech samples more accurate. This greater accuracy is based on the assumption that the autoencoder has a sufficiently large number of parameters. However, when the number of autoencoder parameters is large, there will be a significant wake-up delay problem. Conversely, when the number of autoencoder parameters is small, the wake-up delay problem is not significant, but it is limited by the autoencoder's limited feature extraction capability and the complex and variable nature of the speech environment, leading to potentially inaccurate feature extraction results for speech samples. For example, it may be unable to distinguish between speech samples containing wake-up speech and environmental noise information and those without wake-up speech. In summary, the speech model in the comparative embodiment cannot efficiently and accurately extract features from speech samples.

[0066] (2) In the comparative embodiments, due to the inability to obtain a large number of videos corresponding to each speech sample in the dataset, it is impossible to train an autoencoder to extract features from each facial action sample using a large number of facial action samples. Therefore, the comparative embodiments can only use Fourier transform algorithms, Mel-Cepstral coefficients, and other techniques as video models to extract features from facial action samples. However, the feature extraction capability of such video models is limited, and the accuracy of the feature extraction results for facial action samples cannot be guaranteed. In summary, the comparative embodiments cannot obtain a video model to efficiently and accurately extract features from facial action samples.

[0067] To resolve this issue, please refer to Figure 1 The diagram illustrates the steps of the AI ​​voice wake-up method for a multimodal smart terminal based on voice control provided in this embodiment. The method includes the following steps:

[0068] Step S101: Randomly assign a facial action sample to each voice sample, extract the features of each user's voice using the voice model, and extract the features of each facial action sample using the video model.

[0069] The dataset in the comparative example contains a large number of speech samples.

[0070] For video of the user's face captured by the camera when the wake-up voice is recorded, the facial motion samples in the video are used as baseline motion samples.

[0071] In this embodiment, a large number of speech samples lack corresponding videos. This is because acquiring videos is more difficult than acquiring speech samples, and many speech samples are generated by expanding upon other speech samples, making it impossible to obtain corresponding videos. Therefore, a large number of speech samples in the dataset lack corresponding facial motion samples.

[0072] In this embodiment, a facial motion sample is assigned to each speech sample using the following method:

[0073] Considering that the purpose of this embodiment is to determine whether voice wake-up is needed by judging whether the facial action sample conforms to the baseline action feature F2, without needing to identify the specific voice content based on the facial action sample, it is not necessary to assign a facial action sample that completely corresponds to the voice content to each voice sample.

[0074] Based on this, the number of key points contained in the baseline action sample is denoted as N0, meaning the baseline action sample consists of a sequence of N0 key points, with each sequence representing the temporal change of the position of each key point (see the comparative embodiment for details). For any speech sample, N0 values ​​are randomly initialized, each value corresponding to a sequence of key points. The vector formed by these N0 values ​​is denoted as the action generation vector. The ordinate of each position in the sequence of each key point is summed with a Gaussian noise to obtain the noise sequence of each key point. The Gaussian noise is generated with each value as the standard deviation.

[0075] Thus, an action generation vector is initialized for any speech sample, and N0 noise sequences are generated based on this action generation vector. These N0 noise sequences are denoted as the facial action samples of the speech sample.

[0076] In this embodiment, the video model is also an autoencoder (independent of the autoencoder corresponding to the speech model). This autoencoder includes an encoder and a decoder. Facial motion samples are input into the encoder, which extracts features from the facial motion samples and outputs a high-dimensional vector. This high-dimensional vector is then input into the decoder, which reconstructs the high-dimensional vector and outputs a result identical to the facial motion sample. The high-dimensional vector represents the features of the facial motion sample.

[0077] The autoencoder is trained using facial motion samples corresponding to all speech samples.

[0078] In the comparative embodiment, the baseline action feature F2 refers to the high-dimensional vector obtained after inputting the baseline action sample into the autoencoder. In this embodiment, the features (i.e., high-dimensional vectors) output by the speech model and the features (i.e., high-dimensional vectors) output by the video model have the same dimension.

[0079] It should be noted that when a small amount of noise interference is added to the baseline action sample, it simulates the facial (or mouth) movements when a speech sample containing a wake-up voice occurs. When a large amount of noise interference is added to the baseline action sample, the resulting facial action sample is severely distorted compared to the baseline action sample. In this case, it is used to simulate the facial (or mouth) movements when a speech sample without a wake-up voice occurs. Since this embodiment does not need to identify the specific speech content of the speech sample without a wake-up voice, the speech content contained in the severely distorted facial action sample is not considered.

[0080] Thus, this embodiment has obtained a speech model and a video model, as well as facial motion samples generated based on the motion generation vector of each speech sample.

[0081] Step S102: Update the parameters of the speech model and update the facial action samples corresponding to the positive and negative samples.

[0082] (1) Voice samples containing wake-up voice are designated as positive samples. For example, wake-up voice and voice samples with other voices and environmental noise superimposed on the wake-up voice are both positive samples. User voice samples that do not contain wake-up voice are designated as negative samples. Positive samples can be used to wake up devices, while negative samples cannot be used to wake up devices.

[0083] This embodiment considers the limited feature extraction capability of the speech model. This limitation or insufficiency mainly manifests in: the inability to distinguish between positive and negative samples, incorrectly using negative samples to wake up the device, or failing to wake up the device when using positive samples. The reason for this limited feature extraction capability is that, given a small number of speech model parameters, the dataset composed of all speech samples suffers from significant interference (interference from complex and variable speech environments). This prevents the speech model from accurately learning the speech information present in the speech samples. In other words, the trained speech model gets stuck in a local optimum state without accurately learning the speech information present in the speech samples, and cannot escape this local optimum state to further learn the features of the speech samples. Therefore, this embodiment utilizes the features of facial action samples corresponding to the speech samples to assist the speech model in further learning and training (i.e., further fine-tuning the parameters of the speech model). Simultaneously, this embodiment considers that the facial action samples allocated to all speech samples may have insufficient auxiliary capabilities when assisting the speech model in further learning and training, resulting in the speech model's feature extraction capability not being maximized. Therefore, it is also necessary to reallocate the facial action samples corresponding to the speech samples (i.e., updating the facial action samples corresponding to positive and negative samples).

[0084] (2) Randomly select a positive sample and a negative sample as a sample pair. Use the action generation vector of the positive sample to generate the facial action sample corresponding to the positive sample (see step S101 for details). Input the facial action sample into the video model to obtain the features of the facial action sample, which is denoted as the first feature. Similarly, use the action generation vector of the negative sample to generate the facial action sample corresponding to the negative sample (see step S101 for details). Input the facial action sample into the video model to obtain the features of the facial action sample, which is denoted as the second feature.

[0085] Positive samples are input into the speech model to obtain the features of the positive samples. The average of the features of the positive samples and the first feature is calculated, and then the Euclidean distance is calculated with F1 to obtain the first difference. Negative samples are input into the speech model to obtain the features of the negative samples. The average of the features of the negative samples and the second feature is calculated, and then the Euclidean distance is calculated with F1 to obtain the second difference.

[0086] The purpose of averaging with the first feature is to apply the feature extraction results from the video model to the speech model, thereby combining the video and speech models to achieve the goal of using facial action samples to assist the speech model in further learning and training. The purpose of calculating the Euclidean distance with F1 is to describe the difference between the features contained in the speech samples (i.e., positive or negative samples) and the features F1 corresponding to the wake-up speech when using facial action samples to assist speech model training. This difference (i.e., the first difference or the second difference) can describe whether the features extracted by the speech model with the assistance of the facial action sample features extracted by the video model are the same as or different from the baseline speech features F1.

[0087] Furthermore, the parameters of the speech model are updated using the stochastic gradient descent algorithm, which also updates the motion generation vectors of the facial motion samples corresponding to the positive and negative samples.

[0088] (3) Repeat the process described in (2) until the first difference for all sample pairs is less than the second difference, or stop after repeating (2) a first preset number of times.

[0089] In the above process, the first difference is made smaller than the second difference. The purpose is to ensure that the features corresponding to user voice samples containing wake-up voice are closer to (or similar to) the baseline voice feature F1 than the features of user voice samples without wake-up voice.

[0090] Step S103: Update the parameters of the video model using the updated facial motion samples.

[0091] In the above process, facial action samples are used to assist the speech model in further learning and training, overcoming the limitation that the speech model may be unable to distinguish between positive and negative samples. For example, when the features that the speech model judges as negative samples are more obviously close to the baseline speech features F1 than the positive samples, facial action samples corresponding to positive and negative samples are introduced to break the local optimal solution state of the speech model parameters (that is, to further fine-tune the trained speech model), thereby avoiding the occurrence of this situation.

[0092] Meanwhile, the above process updates the action generation vector corresponding to each speech sample, thereby realizing the redistribution or update of facial action samples. This allows the facial action samples redistributed for each speech sample (including positive and negative samples) to fully utilize or enhance the ability of the auxiliary speech model to be further trained (i.e., the facial action samples try to crack the local optimal state of the speech model's parameters).

[0093] Furthermore, this embodiment uses the updated facial motion samples to update the parameters of the video model. This process, by updating the video model's parameters, ensures that the video model's feature extraction capability is adapted to the distribution of the updated facial motion samples, thus preventing insufficient feature extraction capability from affecting the aforementioned process of using facial motion samples to assist the speech model in further learning and training.

[0094] The parameters of the video model are updated using the updated facial motion samples, specifically including:

[0095] (1) Randomly select a positive sample and a negative sample as a sample pair. Regenerate the facial action sample corresponding to the negative sample using the updated action generation vector corresponding to the negative sample (i.e., update the facial action sample). Input the updated facial action sample into the video model. The Euclidean distance between the obtained features and F2 is denoted as the third difference. Regenerate the facial action sample corresponding to the positive sample using the updated action generation vector corresponding to the positive sample (i.e., update the facial action sample). Input the updated facial action sample into the video model. The Euclidean distance between the obtained features and F2 is denoted as the fourth difference.

[0096] The purpose of calculating the Euclidean distance with F2 is to describe the difference between the features contained in the updated facial motion sample and the baseline motion features F2. This difference (i.e., the third or fourth difference) can describe whether the facial motion sample features extracted from the video model are the same as or different from the baseline motion features F2.

[0097] Furthermore, update the parameters of the video model.

[0098] (2) Repeat the process of (1) until the third difference for all sample pairs is greater than the fourth difference, or stop after repeating (1) a second preset number of times.

[0099] The third difference is greater than the fourth difference. The purpose is to make the facial action sample features containing wake-up speech more obviously close to the baseline action feature F2 compared to the facial action sample features without wake-up speech, so as to ensure that the video model can distinguish whether there is a speech action with wake-up speech based on the facial action sample.

[0100] Step S104: Repeat the alternating process of updating the parameters of the speech model and the facial action samples, as well as the update process of the parameters of the video model.

[0101] Repeat steps S102 to S103 above. In this process, by alternately executing the update process of the speech model and facial action samples and the update process of the video model, the feature extraction process of the video model for facial action samples is used to assist the feature extraction process of the speech model. This avoids the problem of insufficient feature extraction ability caused by the speech model getting stuck in local optima. It also makes up for the inaccuracy of the feature extraction results of the speech model by combining it with the feature extraction process of the video model.

[0102] It should be noted that the subsequent method for voice wake-up in this embodiment is to combine the features output by the speech model and the video model (e.g., take the average) to determine whether voice wake-up is needed. To avoid the contradiction where the features output by the speech model favor voice wake-up while the features output by the video model favor non-wake-up, or vice versa, this embodiment needs to further guide and constrain the feature extraction results of the speech model and the video model.

[0103] The specific method is as follows: each repetition of steps S102 to S103 is recorded as one alternating iteration process. In each alternating iteration, step S102 obtains several first differences and second differences, which are denoted as set C1, representing the feature distribution of different speech samples (including positive and negative samples) relative to the baseline speech feature F1; in each alternating iteration, step S103 obtains several third differences and fourth differences, which are denoted as set C2, representing the feature distribution of facial action samples corresponding to different speech samples (including positive and negative samples) relative to the baseline action feature F2.

[0104] In this embodiment, after each alternating iteration, the parameters of the video model and the speech model are updated once to minimize the distribution difference between set C1 and set C2.

[0105] Step S105: Use the speech model and video model for voice wake-up.

[0106] Voice wake-up is performed using the speech model and video model obtained after step S104, including:

[0107] At each moment, the system collects real-time voice data of the environment via microphone and video data via camera. At each moment, the voice data collected within the last 3 seconds is taken as voice sample Q1, and facial action samples Q2 are extracted from the video collected within the last 3 seconds. The voice sample Q1 is input into the voice model to obtain the features of voice sample Q1, and the facial action sample Q2 is input into the video model to obtain the features of facial action sample Q2. The mean F of the features of voice sample Q1 and facial action sample Q2 is obtained. The mean F of the baseline voice feature F1 and the baseline action feature F2 is obtained and denoted as F3. When the Euclidean distance between F and F3 is less than a first preset wake-up threshold, the device is woken up; when the Euclidean distance between F and F3 is greater than or equal to the first preset wake-up threshold, the device is not woken up.

[0108] Specifically, if no user face is detected in the video (or the user face is present for less than 3 seconds), or facial key points are missing (e.g., when occluded), then the device will be woken up if the feature of the speech sample and the F3 Euclidean distance are less than the second preset wake-up threshold; otherwise, the device will not be woken up. The second preset wake-up threshold being less than the first preset wake-up threshold indicates that in special cases such as the absence of a complete facial movement, only the speech model is used for wake-up, and more stringent wake-up conditions are applied.

[0109] In this embodiment, the first preset wake-up threshold is set to 0.35, and its preferred value is [0.2, 0.5]. In this embodiment, the second preset wake-up threshold is set to half of the first preset wake-up threshold.

[0110] This concludes the example.

[0111] Example 2:

[0112] In this embodiment, it is considered that facial features in video data are also affected by the environment, such as changes in lighting, when collecting video data. This interference can also affect the wake-up process.

[0113] In this embodiment, the following method is used to solve the problem.

[0114] Specifically, after each step S102 is executed, the action generation vectors corresponding to all speech samples are normalized (using a linear normalization algorithm to normalize the same dimension of all action generation vectors) to obtain the normalized vector of each speech sample.

[0115] Obtain the Euclidean distance between the normalized vectors corresponding to any two speech samples. When the Euclidean distance is less than a first preset threshold, mark the two speech samples as environmental interference samples, and record the Euclidean distance between the normalized vectors as the recognition distance between the environmental interference samples. In this embodiment, the first preset threshold is set to 0.2, and its preferred value range is [0, 0.5].

[0116] The difference in action generation vectors between environmental interference samples is small, meaning that the difference in facial action samples between the corresponding two speech samples is small. This makes them susceptible to environmental interference such as changes in ambient lighting. In other words, when environmental interference is introduced into these two facial action samples, the small difference in facial action samples may cause the environmental interference to overwhelm the differences in facial action samples, making it impossible for the video model to distinguish between the two facial action samples. In order to avoid interference with the wake-up recognition result, this embodiment needs to distinguish these two speech samples more clearly from the speech information in the speech model.

[0117] Therefore, whenever step S102 is executed again, if the positive and negative samples in the sample pair are environmental interference samples, then the difference between the first and second differences corresponding to the positive and negative samples is negatively correlated with the recognition distance of the environmental interference samples. This means that the smaller the recognition distance between the positive and negative samples (indicating that the video model cannot distinguish the corresponding facial action samples well under environmental interference), the more necessary it is to distinguish the positive and negative samples in the speech model.

[0118] This embodiment further improves the accuracy of wake-up.

[0119] Example 3:

[0120] In the comparative embodiment, after acquiring all key points within each frame, the positions of all key points need to be normalized. This is to avoid interference from lip movements and video frame size, retaining only the relative changes in the positions of the key points. As an example, the specific process is as follows:

[0121] The key point at the left corner of the mouth among all key points is taken as the reference point. The difference between the position of each key point and the position of the reference point is calculated to obtain the relative position of each key point. The x-coordinate of the relative position of each key point is linearly normalized, and the y-coordinate of the relative position of each key point is also linearly normalized to obtain the normalized position of each key point. The normalized positions of the same key point on the face in all frames constitute a sequence. All key points in all frames correspond to multiple sequences, and these sequences are recorded as facial motion samples in the video.

[0122] Step S101 in Embodiment 1 includes: randomly initializing N0 values. As an example, the specific method is as follows:

[0123] In the real number interval [0, Nmax], N0 real numbers are randomly selected as initial values, where Nmax represents a preset value used to represent the maximum allowable movement distance of the key point. In this embodiment, Nmax is obtained as follows: for each key point in the baseline action sample, the distance between the two farthest positions in the sequence is recorded as the farthest distance, and the maximum value of the farthest distance of the sequences corresponding to all key points is taken as Nmax.

[0124] Step S101 in Example 1 includes: Gaussian noise is generated with each value as the standard deviation. As an example, it specifically includes:

[0125] A Gaussian distribution function with a mean of 0 is constructed using each value as the standard deviation. Gaussian noise is obtained by random sampling according to this Gaussian distribution function.

[0126] Step S102 of Example 1 includes: stopping after all the first differences corresponding to all sample pairs are less than the second differences, or after repeating (2) a first preset number of times. As an example, it specifically includes:

[0127] The motion generation vectors of the facial motion samples corresponding to the positive and negative samples are also regarded as parameters that can be updated (note that the parameters of the video model are frozen at this time), and the speech model and video model are regarded as a whole model.

[0128] Each time step (2) of S102 is executed, a loss function S1 is set for the overall model:

[0129]

[0130] Where Y0 represents the loss function of the autoencoder corresponding to the speech model, a1 represents the first difference, a2 represents the second difference, and exp represents the exponential function with the natural constant as the base.

[0131] Then, based on the loss function S1, the stochastic gradient descent algorithm is used to update the parameters of the speech model and the action generation vector.

[0132] When (2) is executed repeatedly, the loss function S1 will be updated repeatedly, causing the loss function S1 to optimize towards the minimum value. When the loss function S1 optimizes towards the minimum value, It will continuously decrease (that is, a1 gradually decreases and a2 gradually increases). When the first difference a1 corresponding to all sample pairs is less than the second difference a2, the repeated execution (2) stops. In order to avoid the repeated execution (2) from falling into an infinite loop, this embodiment stops after the first preset number of repeated executions (2).

[0133] As an optional example, the first preset number of times is set to 10 times the number of speech samples in the dataset.

[0134] As a preferred example, considering that step S102 will be repeated in step S104 of embodiment one, the first preset number of times is set to 0.1 times (rounded up) the number of speech samples in the dataset to reduce the number of training times for the speech model.

[0135] In Example 1, step S103 includes: the third difference corresponding to all sample pairs is greater than the fourth difference, or the second preset number of times (1) is repeated. As an example, this process specifically includes:

[0136] Each time step (1) of S103 is repeated, a loss function S2 is set for the video model:

[0137]

[0138] Where Y1 represents the loss function of the autoencoder corresponding to the video model, a4 represents the fourth difference, and a3 represents the third difference.

[0139] Then, the stochastic gradient descent algorithm is used to update the parameters of the video model using the loss function S2.

[0140] When step (1) of S103 is executed repeatedly, the loss function S2 will be updated repeatedly, causing the loss function S2 to optimize towards the minimum value. When the loss function S2 optimizes towards the minimum value, It will continuously decrease (that is, a4 gradually decreases and a3 gradually increases). When the third difference a3 corresponding to all sample pairs is greater than the fourth difference a4, the repeated execution (1) is stopped. In order to avoid the repeated execution (1) process falling into an infinite loop, this embodiment stops after the second preset number of repeated executions (1).

[0141] As an example, the second preset number of times is set to 0.1 times the number of speech samples in the dataset (rounded up).

[0142] Step S104 of Example 1 includes: After each alternating iteration, the parameters of the video model and the speech model are updated once to minimize the distribution difference between set C1 and set C2. As an example, the specific process is as follows:

[0143] After each alternating iteration, the loss functions for the speech model and video model are set to S3 and S4, respectively.

[0144]

[0145]

[0146] in, This represents the difference in distribution between set C1 and set C2.

[0147] Then, the stochastic gradient descent algorithm is used to update the parameters of the speech model and the video model using loss functions S3 and S4, respectively.

[0148] As the alternating iteration process continues, the parameters of the speech model and video model are continuously updated using loss functions S3 and S4, respectively, so that S3 and S4 are optimized towards the minimum value, thereby minimizing the distribution difference between set C1 and set C2.

[0149] In this embodiment, the alternating iteration process (i.e., the process from steps S102 to S103) is executed 1000 times and then stopped.

[0150] As an example, the method for obtaining the distribution difference between set C1 and set C2 is as follows:

[0151] Perform linear normalization on all elements in sets C1 and C2 together.

[0152] Plot the normalized elements in set C1 as a histogram curve, denoted as the first curve, and plot the normalized elements in set C2 as a histogram curve, denoted as the second curve. Use the DTW distance between the first curve and the second curve as the distribution difference.

[0153] The DTW distance is obtained by the DTW algorithm, a well-known technique used to calculate the difference between sequences or curves.

[0154] In the histogram, the horizontal axis represents the element value, and the vertical axis represents the probability.

[0155] As an example, plotting the normalized elements of set C1 or C2 as a histogram curve includes the following methods:

[0156] Several intervals are set. In this embodiment, the intervals are: [0, 0.05), [0.05, 0.1), [0.1, 0.15), ..., [0.95, 1].

[0157] For any set C1 or C2, the ratio of the number of elements in each interval to the total number of elements in the set is taken as the probability of each interval. The median of each interval and the probability of each interval are used as the x and y axes, respectively, to form a coordinate point. The coordinate points of all intervals form a histogram curve.

[0158] Example 4:

[0159] Example 2 includes: when step S102 is executed again, if the positive and negative samples in the sample pair are environmental interference samples, then the difference between the first and second differences corresponding to the positive and negative samples is negatively correlated with the recognition distance of the environmental interference samples. As an example, this process specifically includes:

[0160] When it is determined that the positive and negative samples contained in the sample pair are environmental interference samples, the loss function S1 in Example 3 is replaced with the following formula:

[0161]

[0162] in Indicates the difference in objectives. Where d0 represents the recognition distance. The smaller the recognition distance, the larger the target difference m0.

[0163] Then, the stochastic gradient descent algorithm is used to update the parameters of the speech model and the action generation vector using the loss function S1. When the loss function S1 optimizes towards its minimum, it will... (That is, the difference between the first difference and the second difference) approaches m0, thereby achieving the goal that the difference between the first difference and the second difference is negatively correlated with the recognition distance of environmental interference samples.

[0164] Example 5:

[0165] As an example, the methods for obtaining the loss function Y0 of the autoencoder corresponding to the speech model include:

[0166] Let h1 be the speech sample input to the autoencoder, where h1 is a speech signal sequence. Let h2 be the output of the decoder in the autoencoder, where h2 is a speech signal sequence of the same length as h1. Treat h1 and h2 as two vectors, and denote the Euclidean distance between them as the loss function Y0.

[0167] As an example, the methods for obtaining the loss function Y1 of the autoencoder corresponding to the video model include:

[0168] The sequence corresponding to the nth keypoint in the facial motion sample input to the autoencoder is represented as follows: The decoder output in an autoencoder is also a sequence of N0 keypoints, where the sequence corresponding to the nth keypoint is represented as... .

[0169]

[0170] in, express and European distance, The number of key points in the facial motion sample.

[0171] Example 6:

[0172] This embodiment provides a voice-controlled multimodal intelligent terminal AI voice wake-up device. The device includes a microphone, a camera, a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it uses the microphone and camera to collect wake-up voice and voice sample Q1, uses the camera to collect video, and executes all the steps of all the above embodiments.

[0173] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A voice control-based multi-modal intelligent terminal AI voice wake-up method, characterized in that, The method includes the following steps: D1: Each speech sample corresponds to a facial action sample. The speech model is used to extract the features of each speech sample, and the video model is used to extract the features of each facial action sample. The features of the pre-recorded wake-up speech and facial action samples are denoted as F1 and F2, respectively. D2: Speech samples containing wake-up speech are denoted as positive samples, and speech samples without wake-up speech are denoted as negative samples. The features of the facial action samples corresponding to the positive and negative samples are denoted as the first feature and the second feature. The average of the features of the positive samples and the first feature is then calculated with the Euclidean distance of F1 to obtain the first difference; the average of the features of the negative samples and the second feature is then calculated with the Euclidean distance of F1 to obtain the second difference; the parameters of the speech model and the facial action samples corresponding to the positive and negative samples are updated to make the first difference less than the second difference. D3: The Euclidean distance between the updated facial motion sample corresponding to the negative sample and F2 is recorded as the third difference, and the Euclidean distance between the updated facial motion sample corresponding to the positive sample and F2 is recorded as the fourth difference. The parameters of the video model are updated using the updated facial motion samples so that the third difference is greater than the fourth difference. D4: Repeat D2~D3 until the distributions of the first and second differences have the smallest difference with the distributions of the third and fourth differences, and then perform wake-up recognition. Each voice sample corresponds to a facial action sample, and the specific steps involved are as follows: Facial movement samples collected during the recording of wake-up voice are recorded as baseline movement samples; Initialize an action generation vector for any speech sample, and add noise to the baseline action sample with the action generation vector as the standard deviation to obtain a facial action sample for any speech sample. Whenever D2 is executed, the action generation vectors corresponding to all speech samples are normalized to obtain the normalized vector of each speech sample; when the Euclidean distance between the normalized vectors corresponding to positive and negative samples is less than the first preset threshold, the Euclidean distance between the normalized vectors corresponding to positive and negative samples is recorded as the recognition distance. The parameters of the updated speech model and the facial action samples corresponding to the updated positive and negative samples are such that the first difference is smaller than the second difference, while the difference between the first difference and the second difference is negatively correlated with the recognition distance. 2.The AI voice wake-up method based on a voice-controlled multi-modal intelligent terminal according to claim 1, wherein, The steps involved in updating the parameters of the speech model and updating the facial action samples corresponding to the positive and negative samples, such that the first difference is less than the second difference, and that the difference between the first and second differences is negatively correlated with the recognition distance, are as follows: For any pair of positive and negative samples, the action generation vectors corresponding to the positive and negative samples are treated as parameters. The parameters of the video model are frozen, and the speech model and video model are treated as a whole model. A loss function S1 is set for the whole model: wherein, represents a target difference, represents a first difference, represents a second difference, Y0represents a loss function of the autoencoder corresponding to the speech model; where do represents the recognition distance; exp() represents the exponential function with base of the natural constant; The parameters of the speech model and the action generation vector are iteratively updated using the loss function S1. Then, the facial action samples corresponding to the positive and negative samples are regenerated using the updated action generation vector. The iterative update process stops when the first difference is less than the second difference. 3.The AI voice wake-up method based on a multi-modal intelligent terminal controlled by voice according to claim 1, wherein, The specific steps involved in updating the parameters of the video model using the updated facial motion samples, so that the third difference is greater than the fourth difference, are as follows: For any pair of positive and negative samples, the third and fourth differences are obtained. A loss function S2 is set based on the third and fourth differences. The loss function S2 and the difference between the fourth and third differences are positively correlated. The parameters of the video model are iteratively updated using the loss function S2, and the iterative update process continues when the third difference is greater than the fourth difference. 4.The AI voice wake-up method based on a voice-controlled multi-modal intelligent terminal according to claim 1, wherein, The process of stopping when the distributions of the first and second differences have the minimum difference with the distributions of the third and fourth differences includes the following specific steps: Repeating the process from D2 to D3 once is called performing one alternating iteration process; The first and second differences obtained in each alternation iteration constitute set C1, and the third and fourth differences obtained in each alternation iteration constitute set C2; After each alternating iteration, the parameters of the video model and the speech model are updated once. The loss functions used for the video model and the speech model during the parameter update are S3 and S4, respectively. S3 and S4 are positively correlated with the distribution differences of sets C1 and C2, respectively. 5.The AI voice wake-up method based on a voice-controlled multi-modal intelligent terminal according to claim 4, wherein, The specific steps for obtaining the distribution differences between set C1 and set C2 are as follows: Perform linear normalization on all elements in sets C1 and C2 together; plot the normalized elements in set C1 as a histogram curve, denoted as the first curve, and plot the normalized elements in set C2 as a histogram curve, denoted as the second curve; use the DTW distance between the first curve and the second curve as the distribution difference. 6.The AI voice wake-up method based on a multi-modal intelligent terminal controlled by voice according to claim 1, wherein, The specific steps involved in the wake-up recognition process are as follows: At each moment, a voice sample Q1 and a corresponding facial action sample Q2 are collected in real time using a microphone. The features of the voice sample Q1 are extracted using a voice model, and the features of the facial action sample Q2 are extracted using a video model. The mean F of the features of the voice sample Q1 and the facial action sample Q2 is obtained. The mean of F1 and F2 is denoted as F3. When the Euclidean distance between F and F3 is less than a first preset wake-up threshold, wake-up is initiated. When the Euclidean distance between F and F3 is greater than or equal to the first preset wake-up threshold, wake-up is not initiated. 7.The AI voice wake-up method based on the voice-controlled multi-modal intelligent terminal according to claim 6, wherein, The specific steps for obtaining the facial motion samples are as follows: When the wake-up voice is recorded or the voice sample Q1 is collected, the camera is used to capture the user's video. Facial key point detection technology is used to obtain all key points of the user's face in each frame of the video. The positions of the same key point in all frames of the face image constitute a sequence. The sequence corresponding to all key points in all frames of the face image constitutes a facial action sample.

8. A voice-controlled multimodal intelligent terminal AI voice wake-up device, the device comprising: A microphone, a camera, a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor runs the computer program, it reads the voice and video captured by the microphone and the camera, and executes all the steps of the voice-controlled multimodal intelligent terminal AI voice wake-up method according to any one of claims 1 to 7.

Images