Intelligent control method and system of Bluetooth headset and storage medium

Abstract

The invention relates to the technical field of audio processing of wearable devices, and discloses an intelligent control method and system of a Bluetooth headset and a storage medium, and the intelligent control method of the Bluetooth headset comprises the following steps: voice command signal processing, action image sequence processing, and acceleration and angular velocity data processing. Extracting Mel-frequency cepstrum coefficient features of voice data, carrying out scale invariant feature transformation on images, and extracting features of actions; the extracted physical signs are fused; and fusing the feature vectors, inputting the fused feature vectors into a pre-trained deep neural network model long-short-term memory network for instruction identification, and outputting a corresponding instruction by the model. According to the invention, multi-mode information such as voice, action images, acceleration and angular velocity can be comprehensively utilized, richer and diversified intelligent control functions of the Bluetooth earphone are realized, diversified operation requirements of a user in different scenes are met, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing of wearable devices, and more specifically, to an intelligent control method, system and storage medium of a Bluetooth headset. Background Art

[0002] With the rapid development of wireless communication technology, Bluetooth headsets have been widely used in daily life due to their convenience. At present, the control methods of most Bluetooth headsets are relatively traditional, mainly relying on physical buttons or simple touch operations. For example, when the user's hands are occupied during exercise, it becomes very difficult to accurately operate the Bluetooth headset through physical buttons to switch songs, adjust the volume and other functions. Moreover, existing Bluetooth headsets lack the ability to intelligently recognize complex user commands, and are unable to conduct comprehensive analysis and perform corresponding operations based on the user's scene, behavior or voice commands. For example, when a user issues a voice command in a noisy environment, the Bluetooth headset often cannot accurately recognize the command due to interference from environmental noise, resulting in a poor user experience. This traditional control method limits the efficient use of Bluetooth headsets in a variety of scenarios and cannot meet the user's needs for convenient and intelligent interaction. Summary of the invention

[0003] The present invention provides an intelligent control method for a Bluetooth headset, comprising the following steps: Information preprocessing; voice command signal processing, motion image sequence processing, acceleration and angular velocity data processing; Feature extraction: Mel-frequency cepstral coefficient feature extraction of speech data, scale-invariant feature transformation of images, feature extraction of actions; Multimodal information fusion: concatenate the speech feature vector, image feature vector, and action feature vector to obtain a fused feature vector; Instruction recognition and execution: The fused feature vector is input into the pre-trained deep neural network model long short-term memory network for instruction recognition, and the model outputs the corresponding instruction.

[0004] In a preferred embodiment, the voice command signal processing includes processing the voice command signal acquired through the built-in microphone of the Bluetooth headset. Frame processing is performed, and the length of each frame is ; Perform Fourier transform to obtain frequency domain signal .

[0005] In a preferred embodiment, the action image sequence processing includes obtaining the action image sequence of the user through the camera of the mobile device connected to the Bluetooth headset. , grayscale it and perform edge detection to obtain the edge image .

[0006] In a preferred embodiment, the acceleration and angular velocity data processing includes using a built-in accelerometer and gyroscope of the mobile device to obtain acceleration data when the user issues a voice command. and angular velocity data , filter it, remove noise interference, and get smoothed acceleration data and angular velocity data .

[0007] In a preferred embodiment, multimodal information fusion includes: , image feature vector , action feature vector Splice to get the fused feature vector .

[0008] In a preferred embodiment, in order to further enhance the intelligent control capability of the Bluetooth headset in a complex environment, an intelligent control method of the Bluetooth headset includes: All surrounding voice signals acquired through the built-in microphone of the Bluetooth headset Frame processing is performed, and the length of each frame is , and get a series of speech frames , , is the number of frames; The independent component analysis algorithm is used to analyze the speech frames Process and separate each independent speech source signal , , is the number of separated speech sources; For each separated speech source signal Perform Fourier transform to get , and extract the MFCC feature vector from it ; Calculate each eigenvector Information about user-specific voice templates Similarity ; The filtered feature vector The command is also input into the pre-trained deep neural network model long short-term memory network for command recognition, and the model outputs the corresponding command.

[0009] In a preferred embodiment, considering the scenario that a user may have language communication barriers when using a Bluetooth headset, a smart control method for a Bluetooth headset further includes: Ambient sound processing: Ambient sound signals acquired through the built-in microphone of the Bluetooth headset Perform bandpass filtering to remove low-frequency and high-frequency noise interference and obtain the filtered ambient sound signal , and then perform Fourier transform to obtain ; Environmental sound feature extraction: from Extract MFCC feature vector ; The user's speech feature vector and the environmental sound feature vector Splice to get the fused feature vector ; The fused feature vector Input into the pre-trained speech recognition model to recognize the text content of user voice commands and environmental sounds ; The translated text Input into the speech synthesis model to generate speech signals in the target language , play via Bluetooth headphones .

[0010] In a preferred embodiment, the speech recognition model adopts a sequence-to-sequence model with an attention mechanism, wherein the encoder encodes the input speech feature sequence into a hidden state sequence, and the decoder predicts the text output at the current moment based on the hidden state sequence and the previous output text; The speech synthesis model uses an end-to-end model based on deep learning to directly model the original speech waveform: ; in, Represents the entire speech waveform sequence The probability distribution of is the speech waveform sequence, For a given Under the condition of sampling points, The probability distribution of sampling points is modeled by a causal convolutional neural network; Represents continuous product.

[0011] In a preferred embodiment, an intelligent control system for a Bluetooth headset includes: The data acquisition module is responsible for collecting voice command signals, action image sequences, acceleration and angular velocity; The data preprocessing module is used to perform frame segmentation, windowing, Fourier transform, grayscale, edge detection, and low-pass filtering operations on the collected voice, image, and sensor data; The feature extraction module is used to obtain the Mel frequency cepstral coefficient features, the scale-invariant feature transformation of the image, and the action features from the preprocessed data; The information fusion module is used to concatenate the feature vectors of speech, image and action to obtain a fused feature vector containing comprehensive information; The instruction recognition module inputs the fused feature vector into the pre-trained long short-term memory network of the deep neural network model to identify the corresponding instruction category; The instruction execution module executes the corresponding complex instructions according to the identified instruction category to complete the operation expected by the user.

[0012] A storage medium stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the steps in the above-mentioned intelligent control method for a Bluetooth headset can be executed.

[0013] The beneficial effects of the present invention are: Multimodal intelligent interaction: Based on multimodal information fusion technology, combined with voice commands, motion images, acceleration and angular velocity data and other information, it can understand the user's intention more comprehensively and accurately. Compared with the traditional single control method, it greatly improves the accuracy and flexibility of command recognition, providing users with a more convenient and natural interactive experience.

[0014] Strong scene adaptability: Through comprehensive analysis of multiple information, it can better adapt to different scenes. Whether in a quiet indoor environment or a noisy outdoor environment, it can accurately recognize user commands. For example, in a noisy environment, combined with motion information to assist voice command recognition, it effectively avoids command recognition errors caused by environmental noise, and improves the stability and reliability of users using Bluetooth headsets in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of an intelligent control method of a Bluetooth headset of the present invention; Figure 2 is an example diagram of discrete data collected by the present invention; Figure 3 is an example of the image RGB value of the present invention; Figure 4 is an example of data of the accelerometer and gyroscope of the present invention; Figure 5 is an example of data after smoothing of the accelerometer and gyroscope of the present invention; Figure 6 is an example of the characteristic value of the frequency domain signal of the present invention; Figure 7 is an example of a characteristic value of an edge image of the present invention; Figure 8 It is an example of the smoothed acceleration and angular velocity motion characteristics of the present invention. DETAILED DESCRIPTION

[0016] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the present specification. Various examples may omit, replace, or add various processes or components as needed. In addition, the features described in some examples may also be combined in other examples.

[0017] At least one embodiment of the present invention discloses an intelligent control method for a Bluetooth headset, such as Figure 1 As shown, including: Step 100, information preprocessing: voice command signal processing, action image sequence processing, acceleration and angular velocity data processing; Voice command signal processing: Voice command signals acquired through the built-in microphone of the Bluetooth headset Frame processing is performed, and the length of each frame is ; Perform Fourier transform to obtain frequency domain signal ; In one embodiment of the present invention, Fourier transform is a mathematical method for converting a time domain signal into a frequency domain signal, and its expression is: ; in, It represents the frequency domain signal obtained after Fourier transform; Indicates that in the range from negative infinity to positive infinity Find the integral; Indicates the acquired voice command signal; represents frequency variable; t represents time; represents an imaginary unit, satisfying ; Represents a natural constant; this transformation can be used to analyze the components of the speech signal at different frequencies.

[0018] Action image sequence processing: The mobile device camera connected to the Bluetooth headset obtains the user's action image sequence , grayscale it and perform edge detection to obtain the edge image , the grayscale processing formula is as follows: ; in, Indicates The user's action image after frame image grayscale processing, , , Respectively represent Pixel values ​​of red, green, and blue channels in a frame color image; Indicates Edge image after edge detection of frame image.

[0019] Acceleration and angular velocity data processing: The built-in accelerometer and gyroscope in the mobile device obtains the acceleration data when the user issues a voice command and angular velocity data , filter it, remove noise interference, and get smoothed acceleration data and angular velocity data ; In one embodiment of the present invention, the acceleration and angular velocity filtering calculation formula is as follows: ; in, Indicates that after low-pass filtering, at time The smoothed acceleration data is represents the filter window length, Represents the index variable used to traverse the data points in the filter window; Indicates at time Acceleration data at 100 Hz; ; in, It represents the smoothed angular velocity data at time t after low-pass filtering; Indicates the filter window length; Represents the index variable used to traverse the data points in the filter window; Indicates at time Angular velocity data at .

[0020] Step 200, feature extraction; feature extraction includes: Mel-Frequency Cepstral Coefficients (MFCC) feature extraction of speech, Scale-Invariant Feature Transform (SIFT) of images, and feature extraction of actions; In one embodiment of the present invention, the MFCC feature extraction of speech includes: extracting the preprocessed speech frequency domain signal , extract the Mel frequency cepstral coefficient feature vector ; The scale-invariant feature transformation of an image includes: Extract SIFT feature vector ; The feature extraction of the action includes: from the smoothed acceleration data and angular velocity data Extract the action amplitude, action frequency and other features from the .

[0021] Step 300, multimodal information fusion; the speech feature vector , image feature vector , action feature vector Splice to get the fused feature vector ; Step 400, instruction recognition and execution; fusion feature vector Input into the pre-trained deep neural network model Long Short-Term Memory (LSTM) for command recognition, and the model outputs the corresponding command category; In one embodiment of the present invention, the LSTM structure includes an input gate, a forget gate, an output gate and a memory unit; the input gate controls new information to enter the memory unit, the forget gate determines whether to retain or discard old information in the memory unit, and the output gate determines the information output by the memory unit to the network. The calculation formula of the input gate is as follows: ; in, is the Sigmoid activation function; Represents the fused feature vector of the input ; is the first weight matrix of the input gate, , n represents the last moment, is the second weight matrix of the input gate, Indicates hidden state, It is the input gate The first bias term of ; The memory unit calculation formula is as follows: ; in, is the Sigmoid activation function; represents the first weight matrix of the memory unit; Represents the fused feature vector of the input ; is the second weight matrix of the memory unit; is the bias term of the memory unit; The calculation formula of the forget gate is as follows: ; in, is the Sigmoid activation function; Represents the first weight matrix of the forget gate; Represents the fused feature vector of the input ; Represents the second weight matrix of the forget gate; is the bias term of the forget gate; The calculation formula of the output gate is as follows: ; in, is the Sigmoid activation function; represents the first weight matrix of the output gate; Represents the fused feature vector of the input ; Represents the second weight matrix of the output gate; is the bias term of the output gate; ; in, represents the tth hidden state, represents the hyperbolic tangent function; Represents element-wise multiplication; ; in, Indicates the output instruction category, that is, after a series of processing, the model determines that the instruction the user wants to execute belongs to the category. Indicates that among all possible instruction categories, find the category index that makes the probability value the largest , is the Sigmoid activation function; represents the output weight matrix; Represents the hidden state at the nth moment; Indicates output bias.

[0022] LSTM can effectively handle long-term dependency problems in sequence data and is suitable for command recognition of time series data such as speech and action. Execute the corresponding complex instructions.

[0023] In one embodiment of the present invention, an application example of the aforementioned intelligent control method for a Bluetooth headset is provided: Assume that the user is in the gym and wants to control fitness equipment through Bluetooth headphones: Information preprocessing: Voice command signal processing: The user issues a voice command to increase the treadmill speed, and the built-in microphone of the Bluetooth headset receives the voice command signal ; Assume that within 0-1 seconds, Figure 2The discrete data shown; after framing (each frame length ), windowing and Fourier transform to obtain the frequency domain signal ; Action image sequence processing: The user makes a gesture of waving his hand upwards, and the mobile device camera acquires a sequence of action images , take one of the frames, its RGB value is as follows Figure 3 As shown; after grayscale processing: , after edge detection, we get the edge image ; Acceleration and angular velocity data processing: The data obtained by the built-in accelerometer and gyroscope of the mobile device is as follows: Figure 4 As shown; After filtering, the smoothed data is obtained as follows Figure 5 As shown; Feature extraction: Speech MFCC feature extraction: For frequency domain signals Extract MFCC feature vector , some eigenvalues ​​such as Figure 6 As shown; Image SIFT feature extraction: From edge image Extract SIFT feature vector , some data such as Figure 7 As shown; Motion feature extraction: Extract motion feature components from smoothed acceleration and angular velocity data ,like Figure 8 As shown; Multimodal information fusion: ; Instruction recognition and execution: Will Input to the pre-trained LSTM model for command recognition, and the model outputs the command category To increase the speed of the treadmill, the Bluetooth headset sends a command to the treadmill, and the treadmill speed is successfully increased.

[0024] In one embodiment of the present invention, the use environment and scenario of the Bluetooth headset are not single. Considering that there may be complex environments in actual use scenarios, in order to further improve the intelligent control capability of the Bluetooth headset in complex environments, an intelligent control method of the Bluetooth headset includes: Step 101, information preprocessing, all surrounding voice signals obtained through the built-in microphone of the Bluetooth headset Frame processing is performed, and the length of each frame is , and get a series of speech frames , , is the number of frames; Step 102, speech separation, using independent component analysis (ICA) algorithm to analyze the speech frames Process and separate each independent speech source signal , , is the number of separated speech sources; ICA achieves speech separation by maximizing the statistical independence between speech components, and its objective function is: ; in, represents the objective function; is the separation matrix; Indicates the number of separated speech sources; For the The separated speech signal, is the negative entropy, which is used to measure non-Gaussianity; represents logarithm; is the matrix determinant; by maximizing the objective function , we can obtain the optimal separation matrix , and then get the separated speech source signal: ; in, Indicates The separated speech source signal, is the optimal separating matrix No. row vector; Represents the fused feature vector of the input ; Step 103, feature extraction, for each separated speech source signal Perform Fourier transform to get , and extract the MFCC feature vector from it .

[0025] Step 104, user matching, calculate each feature vector Information about user-specific voice templates Similarity , using the cosine similarity formula: ; in, Represents the feature vector Information about user voice templates The similarity of Indicates The MFCC feature vector is extracted after Fourier transform of the separated speech source signal; Indicates that the specific user voice template information is used with Perform similarity calculation; represents the calculation norm; Get user location information , combined with location information , filter out the voice signal that is closest to the specific user's location and has the highest similarity The corresponding eigenvector .

[0026] In one embodiment of the present invention, the specific user voice template information By collecting multiple speech samples of a specific user in advance, performing Fourier transform and MFCC feature extraction operations on these samples, and then averaging or other appropriate statistical processing of the extracted multiple MFCC feature vectors, the specific user speech template information for subsequent similarity calculation is obtained. .

[0027] Step 105, instruction recognition and execution, the filtered feature vector Input into the pre-trained deep neural network model LSTM for command recognition, and the model outputs the corresponding command category The Bluetooth headset recognizes the command type. Execute the corresponding complex instructions.

[0028] In a complex environment, considering the scenario that a user may have language communication barriers when using a Bluetooth headset, an intelligent control method for a Bluetooth headset also includes: Step 201, information preprocessing, environmental sound processing: the environmental sound signal obtained through the built-in microphone of the Bluetooth headset Perform bandpass filtering to remove low-frequency and high-frequency noise interference and obtain the filtered ambient sound signal , and then perform Fourier transform to obtain .

[0029] Step 202, feature extraction, environmental sound feature extraction: from Extract MFCC feature vector ; Step 203, multi-modal information fusion, user voice feature vector and the environmental sound feature vector Splice to get the fused feature vector .

[0030] Step 204, speech recognition and translation, fusion feature vector Input into the pre-trained speech recognition model to recognize the text content of user voice commands and environmental sounds ; In one embodiment of the present invention, the speech recognition model adopts an end-to-end structure based on deep learning, such as a sequence-to-sequence model with an attention mechanism, in which the encoder inputs a speech feature sequence Encoded as a sequence of hidden states , the decoder is based on the hidden state sequence And the previous output text predicts the text output at the current moment , and its calculation formula is as follows: ; in, Indicates at time The hidden state of the encoder output; Represents the encoder, which converts the input speech feature sequence at the current moment and the hidden state at the previous moment As input, output the hidden state at the current moment ; ; in, represents the context vector at the current decoding moment, is the attention mechanism, which uses the query vector at the current decoding moment and the hidden state sequence output by the encoder As input, the context vector is calculated; ; in, Indicates at time The text output by the decoder; Represents a decoder, which converts the output text of the previous moment , the context vector at the current decoding time and the hidden state at the previous decoding moment As input, output the text at the current time .

[0031] Then the recognized text Input into the machine translation model, according to the source language set by the user and target language Translate and get the translated text ; In one embodiment of the present invention, the machine translation model also uses a sequence-to-sequence model based on an attention mechanism to perform text translation.

[0032] Step 205, speech synthesis and playback, the translated text Input into the speech synthesis model to generate speech signals in the target language , play via Bluetooth headphones The speech synthesis model uses an end-to-end model based on deep learning to directly model the original speech waveform: ; in, Represents the entire speech waveform sequence The probability distribution of is the speech waveform sequence, For a given Under the condition of sampling points, The probability distribution of sampling points is modeled by a causal convolutional neural network; Represents continuous product.

[0033] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation mode. The above-mentioned specific implementation mode is merely illustrative and not restrictive. Under the guidance of this embodiment, ordinary technicians in this field can also make more forms of equivalent embodiments, all of which are within the protection of this embodiment.

Claims

1. An intelligent control method for a Bluetooth headset, characterized in that: include: Information preprocessing; Voice command signal processing, motion image sequence processing, acceleration and angular velocity data processing; Feature extraction; Mel-frequency cepstral coefficient feature extraction of speech data, scale-invariant feature transformation of images, and feature extraction of actions; Multimodal information fusion: concatenate the speech feature vector, image feature vector, and action feature vector to obtain a fused feature vector; Instruction recognition and execution: The fused feature vector is input into the pre-trained deep neural network model long short-term memory network for instruction recognition, and the model outputs the corresponding instruction.

2. The intelligent control method of a Bluetooth headset according to claim 1, characterized in that: Voice command signal processing includes processing the voice command signal obtained through the built-in microphone of the Bluetooth headset Frame processing is performed, and the length of each frame is ; Perform Fourier transform to get frequency domain signal .

3. The intelligent control method of a Bluetooth headset according to claim 1, characterized in that: Action image sequence processing includes obtaining the user's action image sequence using the mobile device camera connected to the Bluetooth headset , grayscale it and perform edge detection to obtain the edge image .

4. The intelligent control method of a Bluetooth headset according to claim 1, characterized in that: Acceleration and angular velocity data processing includes: The built-in accelerometer and gyroscope of the mobile device obtains the acceleration data when the user issues a voice command and angular velocity data , filter it, remove noise interference, and get smoothed acceleration data and angular velocity data .

5. The intelligent control method of a Bluetooth headset according to claim 1, characterized in that: Multimodal information fusion includes speech feature vector , image feature vector , action feature vector Splice to get the fused feature vector .

6. The intelligent control method of a Bluetooth headset according to claim 1, characterized in that: The following steps are also included: All surrounding voice signals acquired through the built-in microphone of the Bluetooth headset Frame processing is performed, and the length of each frame is , and get a series of speech frames , , is the number of frames; The independent component analysis algorithm is used to analyze the speech frames Process and separate each independent speech source signal , , is the number of separated speech sources; For each separated speech source signal Perform Fourier transform to get , and extract the MFCC feature vector from it ; Calculate each eigenvector Information about user-specific voice templates Similarity ; The filtered feature vector The command is also input into the pre-trained deep neural network model long short-term memory network for command recognition, and the model outputs the corresponding command.

7. The intelligent control method of a Bluetooth headset according to claim 1, characterized in that: The following steps are also included: Ambient sound processing: Ambient sound signals acquired through the built-in microphone of the Bluetooth headset Perform bandpass filtering to remove low-frequency and high-frequency noise interference and obtain the filtered ambient sound signal , and then perform Fourier transform to obtain ; Environmental sound feature extraction: from Extract MFCC feature vector ; The user's speech feature vector and the environmental sound feature vector Splice to get the fused feature vector ; The fused feature vector Input into the pre-trained speech recognition model to recognize the text content of user voice commands and environmental sounds ; The translated text Input into the speech synthesis model to generate speech signals in the target language , play via Bluetooth headphones .

8. The intelligent control method of a Bluetooth headset according to claim 7, characterized in that: The speech recognition model uses a sequence-to-sequence model with an attention mechanism. Its encoder encodes the input speech feature sequence into a hidden state sequence, and the decoder predicts the text output at the current moment based on the hidden state sequence and the previous output text. The speech synthesis model uses an end-to-end model based on deep learning to directly model the original speech waveform: ； in, Represents the entire speech waveform sequence The probability distribution of is the speech waveform sequence, For a given Under the condition of sampling points, The probability distribution of sampling points is modeled by a causal convolutional neural network; Represents continuous product.

9. An intelligent control system for a Bluetooth headset, characterized in that: The method for intelligently controlling a Bluetooth headset according to any one of claims 1 to 8 comprises: The data acquisition module is responsible for collecting voice command signals, action image sequences, acceleration and angular velocity; The data preprocessing module is used to perform frame segmentation, windowing, Fourier transform, grayscale, edge detection, and low-pass filtering operations on the collected voice, image, and sensor data; The feature extraction module is used to obtain the Mel frequency cepstral coefficient features, the scale-invariant feature transformation of the image, and the action features from the preprocessed data; The information fusion module is used to concatenate the feature vectors of speech, image and action to obtain a fused feature vector containing comprehensive information; The instruction recognition module inputs the fused feature vector into the pre-trained long short-term memory network of the deep neural network model to identify the corresponding instruction category; The instruction execution module executes the corresponding complex instructions according to the identified instruction category to complete the operation expected by the user.

10. A storage medium storing non-transitory computer-readable instructions, characterized in that: When the non-transitory computer-readable instructions are executed by a computer, the steps in the intelligent control method for a Bluetooth headset described in any one of claims 1-8 can be executed.

Images