Custom voice instruction recognition method and device based on twin network, and electronic equipment

Abstract

The invention discloses a custom voice instruction recognition method and device based on a twin network and electronic equipment. Preprocessing the collected voice instruction data to obtain a voice training data set; constructing a twin network architecture; performing end-to-end joint training on the identification model and the discrimination model; deploying the trained identification model and discrimination model to the electronic equipment; performing feature extraction on the voice data, inputting the extracted features into the recognition model, and calculating to obtain a reference space vector; collecting a to-be-recognized voice signal in real time, and inputting the extracted voice features into the recognition model to calculate and obtain a detection space vector; and jointly inputting the detection space vector and the reference space vector into a discrimination model, calculating to obtain a discrimination value, comparing the discrimination value with a preset threshold value, and judging whether a target voice instruction is recognized or not according to a comparison result. The user-defined voice recognition instruction can be newly added on the basis of the preset voice instruction, the voice recognition effect is optimized, and the user interaction experience is improved.

Description

Technical Field

[0001] This invention relates to the field of voice command recognition technology, and in particular to a custom voice command recognition method, device and electronic device based on twin networks. Background Technology

[0002] Voice command recognition refers to users controlling electronic devices by speaking commands. Electronic products such as mobile phones, headphones, smart speakers, and smart home appliances can be operated without physical contact, increasing the convenience and enjoyment of using these devices. Offline voice recognition typically only supports recognizing pre-set voice commands. In real-world scenarios, users may want to add other voice commands beyond the pre-set ones, requiring customized voice command recognition functionality. Users may add voice commands at any time, and traditional solutions require retraining the model, making it difficult to meet dynamic update needs. Voice recognition performance is affected by the speaker's regional accent and pronunciation style, necessitating the creation of more tailored custom voice commands for specific users to achieve better recognition results.

[0003] Some existing speech command recognition methods employ Siamese networks based on metric learning. Siamese networks commonly use contrastive loss or triplet loss as loss functions, determining similarity based on the Euclidean distance between the output space vectors of the recognition model. Real-time acquired speech segments are converted into embedding vectors, and the similarity between these vectors and reference vectors is calculated. This similarity metric typically uses fixed mathematical formulas such as cosine similarity or the reciprocal of the Euclidean distance. The calculated similarity is compared to a preset global fixed threshold; if the similarity is higher than the threshold, the speech command is considered recognized.

[0004] This technical solution has the following inherent drawbacks: First, simple similarity metrics have limited expressive power. Cosine similarity or Euclidean distance are linear and relatively shallow metrics, while speech signals are highly complex and nonlinear. Simple distance metrics cannot fully capture the complex and deep correlation patterns between two embedding vectors, which fundamentally limits the upper limit of the recognition model's performance, resulting in insufficient ability to distinguish similar instructions or filter out similar pronunciation noise. Second, the fixed threshold has poor rigidity and adaptability. A globally fixed threshold cannot adapt to the inherent differences in the characteristics of different speech instructions. For example, the discrimination boundaries required for instructions with short syllables and obvious spectral features should be different from those for instructions with long syllables and complex spectra. A fixed threshold will inevitably lead to good performance on some instructions, while resulting in a large number of false positives or false negatives on others. In addition, the user's accent, speech rate, emotional changes, and fluctuations in environmental noise will all change the distribution of the speech signal in the embedding space. The fixed threshold cannot dynamically adapt to these changes, seriously affecting its robustness in real-world complex scenarios. Third, there is a mismatch between the training and inference objectives of the model. During the training phase, the model learns how to increase the distance between classes and decrease the distance within classes through methods such as triplet loss. However, during the inference phase, a fixed metric that does not directly correspond to the training objective is used for decision-making. This inconsistency between training and inference leads to the model's learned feature space not being optimally utilized. Summary of the Invention

[0005] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a custom voice command recognition method, device, and electronic device based on Siamese networks. By replacing the fixed similarity measurement function in traditional schemes with an adaptive discriminant model, it can add custom voice recognition commands on top of preset voice commands, thereby optimizing the voice recognition effect of electronic devices and improving the user's voice recognition interaction experience.

[0006] On one hand, embodiments of the present invention provide a custom voice command recognition method based on Siamese networks, including: The collected voice command data is preprocessed to obtain a voice training dataset, which contains multiple voice data and corresponding category labels; A Siamese network architecture is constructed, comprising a recognition model and a discriminant model. The recognition model is used to map the input speech features into spatial vectors, and the discriminant model is used to receive two spatial vectors and calculate a similarity discriminant value. Based on the speech training dataset, the recognition model and the discrimination model are jointly trained end-to-end to optimize the model parameters; The trained recognition model and the discrimination model are deployed to electronic devices; When customizing voice commands, feature extraction is performed on the voice data, and the extracted features are input into the recognition model to calculate the reference space vector; During speech recognition, the speech signal to be recognized is acquired in real time, and the speech data is selected by sliding at a certain step size. Speech features are extracted for each selected speech data segment, and the extracted speech features are input into the recognition model to calculate the detection space vector. The detection space vector and the reference space vector are jointly input into the discrimination model to calculate the discrimination value. The discrimination value is compared with a preset threshold, and the target speech instruction is determined based on the comparison result.

[0007] According to some embodiments of the present invention, the preprocessing of the collected voice command data includes: The acquired raw speech signal is preprocessed using a dynamic adaptive noise suppression algorithm to estimate the environmental noise power spectrum in real time and optimize speech features to obtain denoised speech features. The dynamic adaptive noise suppression algorithm includes the following steps: The speech signal is divided into frames by short-time Fourier transform, and the power spectrum of each frame is calculated. The noise power spectrum estimate is dynamically updated using a minimum-cost recursive algorithm. Based on the estimated noise power spectrum, an adaptive spectral subtraction method is applied to generate denoised speech features, where the gain function parameters are dynamically adjusted according to the real-time signal-to-noise ratio.

[0008] According to some embodiments of the present invention, end-to-end joint training of the recognition model and the discrimination model includes: Training batches are constructed from the speech training data. Each training batch contains multiple triplet samples. Each triplet sample includes a sample speech, a positive example speech of the same category as the sample speech, and a negative example speech of a different category from the sample speech. The triplet samples are constructed using a dynamic hard example sampling strategy, prioritizing the selection of sample combinations that the current model misclassifies as training data. The three speech data points in the triplet sample are respectively input into the recognition model to obtain the sample space vector, the positive example space vector, and the negative example space vector; The sample space vector and the positive example space vector are input into the discrimination model to obtain the positive example discrimination value; the sample space vector and the negative example space vector are input into the discrimination model to obtain the negative example discrimination value. Calculate the total model loss based on the positive example discriminant value and the negative example discriminant value; Based on the total loss of the model, the parameters of the recognition model and the discrimination model are updated simultaneously through the backpropagation algorithm.

[0009] According to some embodiments of the present invention, the input of the discrimination model is composed of the absolute difference vector between the detection space vector and the reference space vector, and the concatenated vector of the detection space vector and the reference space vector.

[0010] According to some embodiments of the present invention, the discrimination model includes a multilayer perceptron, the output layer of which uses a sigmoid activation function, and the discrimination value ranges from (0, 1), representing the probability that two input speech words belong to the same instruction.

[0011] According to some embodiments of the present invention, when multiple audio data of the same target voice command are acquired, the average value of the corresponding multiple reference space vectors is calculated, and the average value is used as the final reference space vector.

[0012] According to some embodiments of the present invention, the preset threshold can be dynamically adjusted based on the ambient noise level of the electronic device or the user's historical identification data.

[0013] According to some embodiments of the present invention, the twin network architecture also integrates a cross-language adaptation mechanism, which enables the network to quickly adapt to new languages ​​with a small number of samples through a meta-learning strategy; The cross-language adaptation mechanism includes: During the meta-training phase, the Siamese network is optimized using a multilingual speech dataset to enable the network's initial parameters to quickly adapt to new languages. During the inference phase, the language type of the query speech is detected by language identifiers, and the corresponding meta-learning parameters are loaded. A phoneme mapper is used to project phoneme sequences from different languages ​​into a shared vector space, reducing phonological differences.

[0014] On the other hand, embodiments of the present invention provide a custom voice command recognition device based on a Siamese network, used to implement the above-mentioned custom voice command recognition method based on a Siamese network, the device comprising: The recognition model is used to extract features from the input speech and output the corresponding spatial vector. A discriminant model is used to receive two spatial vectors output by the recognition model and calculate a discriminant value representing the speech similarity between the two spatial vectors. The deployment module is used to deploy the trained recognition model and the discrimination model to an electronic device; The recognition model and the discrimination model are jointly trained end-to-end, enabling the discrimination model to perform similarity discrimination based on the spatial vector generated by the recognition model.

[0015] In another aspect, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the above-described custom voice command recognition method based on twin networks.

[0016] In another aspect, embodiments of the present invention provide a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described custom voice command recognition method based on twin networks.

[0017] The embodiments of the present invention have at least the following beneficial effects: This embodiment provides a custom voice command recognition method and apparatus based on Siamese networks. It utilizes Siamese networks to determine the similarity of voice commands to achieve voice command recognition, and uses a discriminant model to determine the similarity of the outputs of the recognition model. Based on a voice training dataset, the recognition model and the discriminant model are jointly trained end-to-end to optimize model parameters. The parameters of the discriminant model undergo adaptive learning during training, resulting in stronger discriminative ability and better recognition performance. This allows custom voice recognition commands to be no longer limited to specific languages, but has broader applicability to regional accents and pronunciation patterns. It can overcome the limitations of fixed thresholds and simple metrics, providing a custom voice command recognition function with stronger adaptability and higher recognition accuracy. By replacing the fixed similarity measurement function in traditional schemes with an adaptive discriminant model, custom voice recognition commands can be added on top of preset voice commands, optimizing the voice recognition effect of electronic devices and improving the user's voice recognition interaction experience.

[0018] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a custom voice command recognition method based on a Siamese network according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the end-to-end joint training of the recognition model and the discrimination model in the custom voice command recognition method based on Siamese networks according to an embodiment of the present invention. Figure 3 This is a detailed flowchart of the custom voice command recognition method based on Siamese networks according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the training phase of the custom voice command recognition method based on Siamese networks according to an embodiment of the present invention. Figure 5 This is a flowchart illustrating the process of setting custom voice commands using the custom voice command recognition method based on twin networks according to an embodiment of the present invention. Figure 6 This is a flowchart illustrating the process of recognizing custom voice commands using the twin network-based custom voice command recognition method according to an embodiment of the present invention. Detailed Implementation

[0020] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0021] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0022] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," "exceeding," etc. are understood to exclude the stated number, and "above," "below," "within," etc. are understood to include the stated number. If "first," "second," etc. are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of the indicated technical features.

[0023] In the description of this invention, unless otherwise explicitly defined, the terms "setting", "installing", "connecting" and "linking" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0024] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Please see Figure 1 This embodiment provides a custom voice command recognition method based on a Siamese network, mainly including steps S101~S107: S101. Preprocess the collected voice command data to obtain a voice training dataset, which contains multiple voice data and corresponding category labels. S102. The Siamese network architecture includes a recognition model and a discriminant model. The recognition model is used to map the input speech features into spatial vectors, and the discriminant model is used to receive two spatial vectors and calculate the similarity discriminant value. S103. Based on the speech training dataset, perform end-to-end joint training on the recognition model and the discrimination model to optimize the model parameters; S104. Deploy the trained recognition and discrimination models to electronic devices; S105. When defining custom voice commands, feature extraction is performed on the voice data, and the extracted features are input into the recognition model to calculate the reference space vector. S106. When performing speech recognition, the speech signal to be recognized is collected in real time, the speech data is selected by sliding at a certain step size, the speech features are extracted for each selected speech data segment, and the extracted speech features are input into the recognition model to calculate the detection space vector. S107. Input the detection space vector and the reference space vector into the discrimination model to calculate the discrimination value, compare the discrimination value with the preset threshold, and determine whether the target speech command is recognized based on the comparison result.

[0026] The preprocessing of the collected voice command data in step S101 above includes: The acquired raw speech signal is preprocessed using a dynamic adaptive noise suppression algorithm to estimate the environmental noise power spectrum in real time and optimize speech features to obtain denoised speech features.

[0027] The dynamic adaptive noise suppression algorithm includes the following steps: The speech signal is segmented into frames using short-time Fourier transform, and the power spectrum of each frame is calculated. The input speech signal is segmented into frames (e.g., frame length 25ms, frame shift 10ms), and the power spectrum of each frame is calculated.

[0028] The noise power spectrum estimate is dynamically updated using the MCRA algorithm (Minimum Cost Recursive Algorithm).

[0029] Based on the estimated noise power spectrum, an adaptive spectral subtraction method is applied to generate denoised speech features. The gain function parameters are dynamically adjusted according to the real-time signal-to-noise ratio, and the gain function formula is as follows: In the formula, β is the over-reduction factor (dynamically adjusted), and γ is the spectral floor value.

[0030] Dynamically adaptive noise suppression is achieved by adaptively adjusting β and γ through real-time noise estimation. Integrated into the front end of the Siamese network, it directly optimizes the speech signal during the feature extraction stage, avoiding post-processing distortion. A noise-invariant loss function is introduced into the training of the Siamese network, forcing the network to output similar vectors for noisy and denoised speech, thus improving robustness.

[0031] Please see Figure 2 The end-to-end joint training of the recognition model and the discrimination model in step S103 above includes: S201. Construct training batches from the training data. Each training batch contains multiple triplet samples. Each triplet sample includes sample speech, positive example speech of the same category as the sample speech, and negative example speech of a different category from the sample speech. S202. A dynamic hard example sampling strategy is used to construct triplet samples, and the sample combinations that the current model misclassifies are selected as training data first. S203. Input the three speech data from the triplet sample into the recognition model respectively to obtain the sample space vector, positive example space vector and negative example space vector; S204. Input the sample space vector and the positive example space vector into the discrimination model to obtain the positive example discrimination value. Input the sample space vector and the negative example space vector into the discrimination model to obtain the negative example discrimination value. S205. Calculate the total loss of the model based on the positive and negative discriminant values; S206. Based on the total model loss, update the parameters of both the identification model and the discrimination model simultaneously using the backpropagation algorithm.

[0032] In some embodiments, the input to the discriminative model consists of the absolute difference vector between the detection space vector and the reference space vector, and the concatenated vector of the detection space vector and the reference space vector.

[0033] In some embodiments, the discrimination model includes a multilayer perceptron, the output layer of which uses a sigmoid activation function, and the discriminant value ranges from (0, 1), representing the probability that two input speech words belong to the same instruction.

[0034] In some embodiments, when multiple audio data of the same target voice command are acquired, the average value of the corresponding multiple reference space vectors is calculated, and the average value is used as the final reference space vector.

[0035] In some embodiments, the preset threshold can be dynamically adjusted based on the ambient noise level of the electronic device or the user's historical identification data.

[0036] In some embodiments, the Siamese network architecture also integrates a cross-language adaptation mechanism, enabling the network to quickly adapt to new languages ​​with a small number of samples through Model-Agnostic Meta-Learning (MAML).

[0037] Cross-language adaptation mechanisms include: During the meta-training phase, the Siamese network is optimized using a multilingual speech dataset to enable the network's initial parameters to quickly adapt to new languages. During the inference phase, the language type of the query speech is detected by language identifiers, and the corresponding meta-learning parameters are loaded. A phoneme mapper is used to project phoneme sequences from different languages ​​into a shared vector space, reducing phonological differences.

[0038] Based on MAML meta-learning, Siamese networks can quickly adapt to new languages ​​using a small number of samples (e.g., 5-10), reducing reliance on pre-trained models. A phoneme mapper maps phonemes from different languages ​​to a shared vector space, reducing phonological differences. Language identifiers are used to extract language features (such as fundamental frequency and phoneme duration) from speech signals, dynamically switching the branch parameters of the Siamese network. Focusing on phoneme-level cross-language generalization avoids the complexity of multimodal fusion and improves the stability of cross-language adaptation.

[0039] Please see Figure 3 The custom voice command recognition method based on Siamese networks provided in this embodiment may include the following detailed steps: S301. Collect speech data and preprocess it to obtain a speech training dataset. Multiple speech data points corresponding to the same speech command belong to the same speech class. The collected speech dataset should contain as many speech classes as possible, and each speech class should contain as many speech data points as possible. In addition to collecting speech data for each specific command, some non-class speech data that does not contain each specific command, as well as some noisy audio data, can also be collected. Divide each class of speech data, non-class speech data, and noisy audio data into a training set and a validation set in a certain ratio (e.g., 9:1).

[0040] S302. Construct a Siamese network architecture and select a recognition model and a discriminant model. The Siamese network architecture does not restrict the specific network structure type; therefore, the recognition model can be selected based on available resources such as computing power, choosing one or a combination of several network structures, such as convolutional neural networks, residual networks, and Transformers. The input to the recognition model is the speech features generated from the speech data, and the output is a spatial vector. The discriminant model selects a combination of fully connected layers and activation functions. The input to the discriminant model is the two spatial vectors corresponding to the two speech samples whose similarity needs to be determined, and the output is a discriminant value representing the similarity between the two speech samples.

[0041] S303, please refer to Figure 4 The model is trained, and each batch of data contains several sets of data, with each set containing three audio data points.

[0042] (1) Randomly select a class and randomly select a sample speech data from that class; (2) Randomly select one speech data point from the same class as the sample speech data as a positive example; (3) Randomly select a speech data from class or non-class speech data that is different from the sample speech data as a negative example.

[0043] To improve the generalization of the model, noisy audio can be added to the speech data randomly with a certain probability (e.g., 50%).

[0044] S304. Extract features from the speech data in each batch and input them into the discrimination model to calculate the positive and negative discrimination values. Commonly used speech features include MFCC (Mel-frequency cepstral coefficients) and Fbank (Mel-frequency filter bank coefficients).

[0045] The recognition model calculates a spatial vector based on each input speech feature. Three speech data points in the same group will result in three spatial vectors: a sample spatial vector, a positive example spatial vector, and a negative example spatial vector.

[0046] The sample space vector and the positive example space vector of the same group are jointly input into the discriminant model to calculate the positive example discriminant value.

[0047] The sample space vector and negative example space vector of the same group are jointly input into the discriminant model to calculate the negative example discrimination value.

[0048] S305. Select an appropriate loss function, calculate the loss function using the positive and negative discriminant values ​​of a batch of data, and then use the backpropagation algorithm to update the weights of the recognition model and the discriminant model. For example, if the output of the discriminant model is selected using the Sigmoid function, then the loss function can be the binary cross-entropy function.

[0049] The formula for the Sigmoid function is: The Sigmoid function is used to implement the output of hidden layer neurons, with a value range of (0,1), which can map a real number to the interval (0,1).

[0050] The formula for the binary cross-entropy function is: In the formula, This represents the probability that the input is a sample of the same type. This indicates that the input is a sample of the same type. This indicates that different classes of samples are being input.

[0051] The positive example judgment value Negative example judgment value Substitute the values ​​into the formula to obtain the loss value. The formula for calculating the loss value is: In the formula, This is the value for determining a positive example. This is the negative case determination value.

[0052] S306. Iteratively select each batch of data from the training set to optimize the model until the loss function calculated for the validation set data no longer decreases after several consecutive rounds of optimization, then stop training.

[0053] S307. Port and deploy the trained recognition and discrimination models to electronic devices.

[0054] S308, please refer to Figure 5 When a user sets a voice command, features are extracted from the voice data, and these extracted features are input into the recognition model to calculate a reference space vector. To improve recognition performance, users can record multiple custom voice commands at this stage, thereby generating more reference space vectors.

[0055] S309, please refer to Figure 6 The system continuously monitors and collects speech data, selecting speech segments in steps. Features are extracted from each selected segment, and these features are input into a recognition model to calculate a detection space vector. The detection space vector and a reference space vector are then jointly input into a discriminant model to calculate a discriminant value. If the discriminant value exceeds a set threshold, the user-defined voice command is considered recognized.

[0056] It should be noted that the custom voice command recognition method based on Siamese networks provided in this embodiment mainly includes two stages: the model training stage and the model deployment and recognition stage.

[0057] 1. The model training phase includes the following steps: (1) Data preparation and preprocessing: Collect large-scale, multi-class voice command datasets and perform audio preprocessing (pre-emphasis, framing and windowing) and feature extraction (such as Fbank).

[0058] (2) Model architecture construction: Construct the specific network structure of the recognition model and the discrimination model.

[0059] (3) End-to-end joint training: a. Triple Sample Construction: Construct triples (Anchor, Positive, Negative) from the dataset, where Anchor is the sample speech, Positive is the positive example speech of the same class as Anchor, and Negative is the negative example speech of a different class than Anchor or is a non-command speech.

[0060] b. Forward propagation: The three speech segments in the triplet are passed through the recognition model to obtain three spatial vectors (V... a V p V n Then (V) a V p ) and (V a V n Inputting each value into the discriminant model yields two discriminant values ​​(D). p D n ).

[0061] c. Loss Calculation and Backpropagation: The binary cross-entropy loss function is used, and the loss is minimized by a gradient descent algorithm (such as Adam). The backpropagation algorithm is then used to update all parameters of both the recognition and discrimination models simultaneously.

[0062] d. Dynamic hard example sampling: During training, prioritize samples that have been misclassified by the current model, i.e., D. p Value small or D n Training with triples of high value accelerates model convergence and improves the quality of discrimination boundaries.

[0063] 2. The model deployment and recognition phase includes the following steps: (1) Model deployment: The trained and performance-compliant recognition and discrimination models are converted into a format suitable for embedded devices, such as TFLite format, and integrated into electronic devices.

[0064] (2) User-defined instructions: The user records the target instruction voice, for example, 3 times; these voice segments are converted into multiple reference space vectors through the recognition model.

[0065] (3) Reference vector fusion: To improve robustness, the average value or centroid of multiple reference vectors for the same instruction is calculated, and the centroid vector is used as the unique reference vector V for the instruction. ref storage.

[0066] (4) Real-time speech recognition: Continuous recording is performed using electronic devices, and audio segments are extracted using a sliding window (e.g., window length 1 second, step size 0.2 seconds). For each audio segment, features are extracted, and the detection space vector V is obtained through a recognition model. det .

[0067] (5) Similarity judgment: V det and V ref Input the discriminant model and calculate the discriminant value D.

[0068] (6) Decision and execution: Compare the discrimination value D with a preset threshold T. If D>T, the target instruction is identified and the corresponding operation is triggered; otherwise, continue listening.

[0069] Example 1 This embodiment is a cloud-based training and edge-side deployment solution.

[0070] 1. Data preparation and preprocessing (1) Data preparation We used an internally collected voice command dataset containing 500 different command words, each recorded 5 times by 100 different speakers (covering various dialects and accents) in both quiet and noisy environments (signal-to-noise ratio approximately 10dB), totaling 250,000 voice data points. In addition, we included 50,000 non-command voice data points (background voices, music, etc.) and 20,000 pure noise data points.

[0071] (2) Pretreatment The sampling rate is uniformly set to 16kHz, and it is mono.

[0072] A pre-emphasis filter (coefficient 0.97) is applied to enhance high-frequency components.

[0073] The frame length is 25ms, the frame shift is 10ms, and a Hamming window is used.

[0074] Extract 80-dimensional Fbank features and perform mean-variance normalization.

[0075] The dataset was randomly divided into training, validation, and test sets in a ratio of 8:1:1.

[0076] 2. Model Building (1) Recognition Model The model employs a Transformer encoder-based architecture. Inputting 80-dimensional Fbank features, it first passes through a linear layer projecting to 256 dimensions, followed by sinusoidal positional encoding. Then, it passes through a 4-layer Transformer encoder (8 attention heads, 512 dimensions in the feedforward network). Finally, the output sequence undergoes global average pooling and is passed through another linear layer to output a 128-dimensional spatial vector. The model has approximately 5M parameters in total.

[0077] (2) Discriminant model: A multilayer perceptron is used, with an input dimension of (128 + 128) + 128 = 384 dimensions, which is the concatenation vector of 256 dimensions + the difference vector of 128 dimensions.

[0078] The hidden layers consist of two fully connected layers: the first layer has 256 neurons and the second layer has 128 neurons, both using the ReLU activation function.

[0079] The output layer has a single neuron and uses the sigmoid activation function. The model has approximately 150K parameters, making it lightweight.

[0080] 3. Model Training (1) Hardware and software: Four NVIDIA V100 GPUs were used for training based on the PyTorch framework.

[0081] (2) Optimizer and hyperparameters: The AdamW optimizer was used, with an initial learning rate of 1e-4 and cosine annealing scheduling. The batch size was set to 512, meaning each batch contained 512 triples.

[0082] (3) Difficult example sampling: At the beginning of each epoch, the current model is used to perform an inference on the training set once, and a difficulty score is calculated for each candidate triplet, defined as D. n - D p The top 20% of triplet scores with the highest difficulty scores will be prioritized for this round of training.

[0083] (4) Training process: A total of 200 epochs were trained. After each epoch, the loss was calculated on the validation set, and the model with the lowest validation loss was saved. The training lasted approximately 48 hours. The final model's equal error rate on the test set was reduced to 0.8%, which is far better than the baseline model's 1.5%.

[0084] 4. Model Deployment and Recognition (1) Model conversion: The trained PyTorch model was converted to .tflite format using the TensorFlow Lite conversion tool and dynamic range quantization was performed. The model size was compressed from about 20MB to 5MB, making it suitable for running on resource-constrained embedded devices.

[0085] (2) Integration and Deployment: The quantized model is integrated into the firmware of a smart speaker. A dedicated low-power DSP core is set up on the chip to continuously run voice activity detection and feature extraction. When a valid voice segment is detected, the main application processor is woken up to run the recognition model and the discrimination model.

[0086] (3) User-defined process: After the user presses and holds the "Settings" button, they say a custom command (such as "Starry Sky Flashing") 3 times. The device background automatically completes the recording, feature extraction, and vector centroid calculation, and calculates the centroid vector V. ref It is bound to the instruction ID and stored in non-volatile memory.

[0087] (4) Real-time recognition: The device is in standby listening mode. Microphone data is truncated in 200ms increments to capture 1-second audio segments. For each audio segment, 80-dimensional Fbank features are calculated in real time and input into the recognition model to obtain V. det Then V det and all stored V refInput the discrimination model sequentially. Set the threshold to 0.85. If the discrimination value of a certain instruction exceeds the threshold, immediately trigger the corresponding operation (such as playing specified music), and set a 200ms silence period after triggering to prevent repeated triggering.

[0088] Example 2 This embodiment is a lightweight embedded device optimization solution, applicable to electronic products with more limited computing power, such as smart headphones, and simplifies embodiment 1.

[0089] 1. Recognition Model: A one-dimensional depthwise separable convolutional network with only 3 layers was used to output a 64-dimensional spatial vector, reducing the model parameters to 50K.

[0090] 2. Discriminant Model: The MLP structure is simplified to a single hidden layer (64 neurons), and the parameters are reduced to 10K.

[0091] 3. Training data: Use a smaller but noisier dataset.

[0092] 4. Results: Although the absolute performance is slightly lower than that of Example 1, with an EER of 1.2%, it is still within an acceptable range. Moreover, the power consumption and latency are extremely low, demonstrating the good scalability of this solution.

[0093] This application embodiment also provides a custom voice command recognition device based on a Siamese network, used to implement the above-described custom voice command recognition method based on a Siamese network. The device includes: The recognition model is used to extract features from the input speech and output the corresponding spatial vector. The discriminant model is used to receive two spatial vectors output by the recognition model and calculate a discriminant value that represents the speech similarity between the two spatial vectors. The deployment module is used to deploy the trained recognition and discrimination models onto electronic devices; The recognition model and the discrimination model are jointly trained end-to-end, enabling the discrimination model to perform similarity discrimination based on the spatial vectors generated by the recognition model.

[0094] It should be noted that this embodiment of the custom voice command recognition device based on twin networks is based on the same inventive concept as the above-described embodiment of the custom voice command recognition method based on twin networks. For any content not covered in this embodiment of the custom voice command recognition device based on twin networks, please refer to the above-described embodiment of the custom voice command recognition method based on twin networks, which will not be repeated here.

[0095] The embodiments of the present invention have the following beneficial effects: 1. A qualitative leap in recognition accuracy: Because the discriminant model is a powerful nonlinear function, it can learn discriminant boundaries that are far more complex than fixed metrics such as cosine similarity, and has an extremely strong ability to capture subtle differences in high-dimensional feature spaces. Experiments show that on the same test set, this scheme can reduce the error rate by 15%-25% and significantly improve the discrimination of easily confused instructions.

[0096] 2. High adaptability and robustness: The discriminative model learns a general ability to "judge similarity" rather than a rigid standard. Therefore, it can automatically adapt to the inherent differences in the characteristics of different instructions, and shows a strong tolerance to user accents, speech rate changes, and interference from environmental background noise, significantly reducing the probability of false triggering and missed triggering in complex real-world scenarios.

[0097] 3. Highly unified training and inference objectives, resulting in superior model performance: Through end-to-end joint training, the feature extraction process of the recognition model is explicitly guided towards the ultimate goal of "generating a feature space that facilitates high-precision discrimination by the discrimination model." This consistency makes the overall system performance superior to traditional schemes that separate feature learning and similarity measurement.

[0098] 4. Improve the usability and user experience of the custom voice command function: Make the process of user-defined commands smoother, eliminate the need for tedious threshold debugging, and obtain stable and reliable recognition results with just one recording, thereby improving ease of use and user satisfaction.

[0099] This application also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the above-described custom voice command recognition method based on twin networks.

[0100] This application also provides a storage medium that, when executed by a computer's processor, enables the computer to perform the custom voice command recognition method based on a twin network provided in the above embodiments. For example, the storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, USB flash drive, or optical data storage device. It is worth noting that the storage medium mentioned in this application embodiment can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0101] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions; the computer instructions can be stored in the storage medium described above. In some embodiments, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the custom voice command recognition method based on twin networks provided in the above embodiments.

[0102] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A custom voice command recognition method based on Siamese networks, characterized in that, include: The collected voice command data is preprocessed to obtain a voice training dataset, which contains multiple voice data and corresponding category labels; A Siamese network architecture is constructed, comprising a recognition model and a discriminant model. The recognition model is used to map the input speech features into spatial vectors, and the discriminant model is used to receive two spatial vectors and calculate a similarity discriminant value. Based on the speech training dataset, the recognition model and the discrimination model are jointly trained end-to-end to optimize the model parameters; The trained recognition model and the discrimination model are deployed to electronic devices; When customizing voice commands, feature extraction is performed on the voice data, and the extracted features are input into the recognition model to calculate the reference space vector; During speech recognition, the speech signal to be recognized is acquired in real time, and the speech data is selected by sliding at a certain step size. Speech features are extracted for each selected speech data segment, and the extracted speech features are input into the recognition model to calculate the detection space vector. The detection space vector and the reference space vector are jointly input into the discrimination model to calculate the discrimination value. The discrimination value is compared with a preset threshold, and the target speech instruction is determined based on the comparison result.

2. The custom voice command recognition method based on Siamese networks according to claim 1, characterized in that, The preprocessing of the collected voice command data includes: The original speech signal is preprocessed using a dynamic adaptive noise suppression algorithm to estimate the environmental noise power spectrum in real time and optimize speech features to obtain denoised speech features. The dynamic adaptive noise suppression algorithm includes the following steps: The speech signal is divided into frames by short-time Fourier transform, and the power spectrum of each frame is calculated. The noise power spectrum estimate is dynamically updated using a minimum-cost recursive algorithm. Based on the estimated noise power spectrum, an adaptive spectral subtraction method is applied to generate denoised speech features, where the gain function parameters are dynamically adjusted according to the real-time signal-to-noise ratio.

3. The custom voice command recognition method based on Siamese networks according to claim 1 or 2, characterized in that, End-to-end joint training of the recognition model and the discrimination model includes: Training batches are constructed from the speech training data. Each training batch contains multiple triplet samples. Each triplet sample includes a sample speech, a positive example speech of the same category as the sample speech, and a negative example speech of a different category from the sample speech. The triplet samples are constructed using a dynamic hard example sampling strategy, prioritizing the selection of sample combinations that the current model misclassifies as training data. The three speech data points in the triplet sample are respectively input into the recognition model to obtain the sample space vector, the positive example space vector, and the negative example space vector; The sample space vector and the positive example space vector are input into the discrimination model to obtain the positive example discrimination value; the sample space vector and the negative example space vector are input into the discrimination model to obtain the negative example discrimination value. Calculate the total model loss based on the positive example discriminant value and the negative example discriminant value; Based on the total loss of the model, the parameters of the recognition model and the discrimination model are updated simultaneously through the backpropagation algorithm.

4. The custom voice command recognition method based on Siamese networks according to claim 3, characterized in that, The input to the discrimination model consists of the absolute difference vector between the detection space vector and the reference space vector, and the concatenated vector of the detection space vector and the reference space vector.

5. The custom voice command recognition method based on Siamese networks according to claim 4, characterized in that, The discrimination model includes a multilayer perceptron, the output layer of which uses the sigmoid activation function, and the discrimination value ranges from (0, 1), representing the probability that two input speech words belong to the same instruction.

6. The custom voice command recognition method based on Siamese networks according to claim 1, characterized in that, When multiple audio data of the same target voice command are acquired, the average value of the corresponding multiple reference space vectors is calculated, and the average value is used as the final reference space vector.

7. The custom voice command recognition method based on Siamese networks according to claim 1, characterized in that, The preset threshold is dynamically adjusted based on the ambient noise level of the electronic device or the user's historical identification data.

8. The custom voice command recognition method based on Siamese networks according to claim 1, characterized in that, The twin network architecture also integrates a cross-language adaptation mechanism, which enables the network to quickly adapt to new languages ​​with a small number of samples through a meta-learning strategy; The cross-language adaptation mechanism includes: During the meta-training phase, the Siamese network is optimized using a multilingual speech dataset to enable the network's initial parameters to quickly adapt to new languages. During the inference phase, the language type of the query speech is detected by language identifiers, and the corresponding meta-learning parameters are loaded. A phoneme mapper is used to project phoneme sequences from different languages ​​into a shared vector space, reducing phonological differences.

9. A custom voice command recognition device based on a twin network, characterized in that, The method for implementing the custom voice command recognition method based on a twin network as described in any one of claims 1 to 8 includes: The recognition model is used to extract features from the input speech and output the corresponding spatial vector. A discriminant model is used to receive two spatial vectors output by the recognition model and calculate a discriminant value representing the speech similarity between the two spatial vectors. The deployment module is used to deploy the trained recognition model and the discrimination model to an electronic device; The recognition model and the discrimination model are jointly trained end-to-end, enabling the discrimination model to perform similarity discrimination based on the spatial vector generated by the recognition model.

10. An electronic device comprising a processor and a memory, characterized in that, The memory stores a computer program, which, when executed by the processor, implements the steps of the custom voice command recognition method based on twin networks as described in any one of claims 1 to 8.

Images