Earphone instruction control method and system based on voice recognition

Abstract

The invention relates to the technical field of earphone instruction control, in particular to an earphone instruction control method and system based on voice recognition. The method comprises the following steps of performing framing processing on voice feature data, extracting a basic acoustic feature and an auxiliary feature corresponding to each frame, calculating a semantic matching degree between a candidate voice instruction and a scene state vector according to the candidate voice instruction, and constructing a joint feature vector by introducing a historical instruction consistency parameter; inputting the joint feature vector into a joint scoring model to generate a comprehensive score for instruction execution judgment; only when the comprehensive score meets a preset execution condition, the corresponding instruction is judged to be a valid instruction, otherwise, the corresponding instruction is judged to be invalid and execution of the instruction is inhibited; a frame-level validity discrimination mechanism is introduced, and frame-by-frame screening is performed on an original acoustic feature matrix, so that invalid voices are prevented from entering a recognition model, and the voice recognition capability of the whole system in multi-person conversation, noisy environment and motion scene is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to earphone instruction control technology field, specifically, earphone instruction control method and system based on speech recognition. BACKGROUND

[0002] With the development of intelligent earphones and voice interaction technology, voice-based control has gradually become an important operation method for devices such as earphones. However, existing speech recognition technology still has problems such as command misrecognition, speech ambiguity, and false triggering in complex environments (such as multiple people talking, background noise, or user rapid movement, etc.). In particular, in a multi-user environment, the system is prone to recognizing the speech of non-target users or being triggered by incidental noise. In view of this, an earphone instruction control method and system based on speech recognition are provided. SUMMARY

[0003] The present application aims to provide an earphone instruction control method and system based on speech recognition to solve the problems of command misrecognition, speech ambiguity, and false triggering in complex environments (such as multiple people talking, background noise, or user rapid movement, etc.) as mentioned in the background technology. In particular, in a multi-user environment, the system is prone to recognizing the speech of non-target users or being triggered by incidental noise.

[0004] To achieve the above-mentioned purpose, the present application aims to provide an earphone instruction control method based on speech recognition, comprising the following steps:

[0005] S1, pre-processing the collected original speech signal and behavior state data to obtain speech feature data;

[0006] S2, frame processing the pre-processed speech feature data, and extracting the corresponding basic acoustic features and auxiliary features of each frame, constructing a speech feature matrix according to the time sequence of the basic acoustic features, and performing speech activity validity discrimination on the speech feature matrix based on the auxiliary features, to generate an acoustic feature matrix after validity screening;

[0007] Wherein, the basic acoustic features include MFCC vectors, fundamental frequency and energy envelope;

[0008] The auxiliary features include signal-to-noise ratio estimate, speech activity confidence and far-near field discrimination factor;

[0009] S3, decoding and recognizing the acoustic feature matrix after validity screening based on a speech recognition model, to generate candidate speech instructions and their corresponding speech recognition confidence;

[0010] Wherein, the speech recognition model is an end-to-end recognition model based on Transformer, constructed by a convolutional neural network combined with a Transformer encoder.

[0011] S4, structurally encode the behavior state data to generate a scene state vector, and calculate a semantic matching degree between the candidate voice instruction and the scene state vector;

[0012] S5, combine the semantic matching degree with a voice recognition confidence, and construct a joint feature vector by introducing a historical instruction consistency parameter;

[0013] construct a joint scoring model based on a lightweight fully connected neural network, input the joint feature vector into the joint scoring model, and generate a comprehensive score for instruction execution decision ;

[0014] Only when the comprehensive score satisfies a preset execution condition, the corresponding instruction is determined as a valid instruction, otherwise it is determined as invalid and its execution is inhibited;

[0015] S6, generate a corresponding earphone control operation for the candidate instruction determined as valid and execute it;

[0016] discard the candidate instruction that does not satisfy the execution condition to avoid environmental voice mis-triggering.

[0017] As a further improvement of the technical solution, the original voice signal includes time sequence sampling values of the voice signal, corresponding sampling frequency, and each microphone channel signal in a multi-microphone scene;

[0018] The behavior state data includes earphone wearing state parameters, user motion state parameters, device running state parameters, and interaction state parameters.

[0019] As a further improvement of the technical solution, in S2, the acoustic feature matrix after effectiveness screening is generated, involving the following specific steps:

[0020] frame processing is performed on the preprocessed voice feature data using a sliding window method, the basic acoustic features of each frame of voice feature data are extracted, and the basic acoustic features obtained for each frame are constructed into a feature vector ;

[0021] The feature vectors of consecutive frames are spliced in time sequence to construct a preliminary voice feature matrix ;

[0022] For each frame of voice signal, auxiliary features are extracted synchronously, and auxiliary feature vectors are constructed;

[0023] Based on the auxiliary feature vector, voice activity effectiveness discrimination is performed on each frame of voice signal, and based on the result of voice activity effectiveness discrimination, the preliminary voice feature matrix Frame-level screening is performed to finally generate the acoustic feature matrix after effectiveness screening .

[0024] As a further improvement of the technical solution, the preliminary speech feature matrix The specific screening rules involved in the generation of the acoustic feature matrix after frame-level screening are:

[0025] The feature vector corresponding to the speech frame determined to be invalid is removed;

[0026] The amplitude normalization processing is performed on the retained valid speech frames;

[0027] And the time consistency constraint is performed on the continuous valid frames.

[0028] As a further improvement of the technical solution, in S3, the candidate speech instruction and its corresponding speech recognition confidence are generated by the speech recognition model, and the specific steps involved are:

[0029] Based on the acoustic feature matrix after effectiveness screening , an acoustic feature input matrix is constructed by introducing additional speech features;

[0030] Among them, the additional speech features include speaker identity label, speech separation confidence and far and near field factor;

[0031] The acoustic feature input matrix is sent into the convolutional neural network, each convolution kernel processes the input data through a sliding window, and the local features are extracted by performing convolution operation on the acoustic feature input matrix through multiple convolution kernels, and a high-dimensional feature map is outputted;

[0032] The high-dimensional feature map is sent into the Transformer encoder, and the Transformer encoder extracts global semantic information through the self-attention mechanism to output a time sequence feature matrix ;

[0033] The time sequence feature matrix is sent into the decoder, and the decoder generates the text sequence of the candidate speech instruction from the time sequence feature matrix based on the decoding algorithm;

[0034] For each candidate text sequence, the decoder calculates its corresponding posterior probability, and calculates the corresponding speech recognition confidence for each candidate text sequence.

[0035] As a further improvement of the technical solution, in S4, the semantic matching degree between the candidate speech instruction and the scene state vector is calculated, and the specific steps involved are:

[0036] The scene rule constraint is used for judging whether the candidate voice instruction meets the execution condition of the current scene.

[0037] For the candidate voice instruction that does not meet the execution condition, it is directly determined as an unexecutable instruction and its execution is inhibited.

[0038] For the candidate voice instruction that meets the execution condition, the semantic matching degree of the candidate voice instruction and the scene state vector is calculated.

[0039] As a further improvement of the technical solution, the scene rule constraint is used for judging the executability of the candidate voice instruction based on the earphone wearing state, the device running state and the task context.

[0040] The scene rule constraint includes:

[0041] When the earphone is in the wearing state, only the instruction that needs to wear the earphone is allowed to be executed.

[0042] According to the left and right ear wearing identifier, only the instruction matched with the wearing ear is allowed to be executed.

[0043] According to the user motion state, only the instruction that meets the current motion state is allowed to be executed.

[0044] According to the device running state, only the instruction matched with the current running state is allowed to be executed.

[0045] Only when the device is in the voice interaction activation state, the voice instruction is allowed to be executed.

[0046] When the candidate voice instruction meets all the above sub-rule constraints, it is determined as executable, otherwise it is determined as unexecutable and is inhibited.

[0047] As a further improvement of the technical solution, in S5, the comprehensive score is generated by the joint scoring model The specific steps are as follows:

[0048] For each candidate voice instruction , the semantic matching degree and the voice recognition confidence of the candidate voice instruction and the current scene state vector are obtained, and the historical consistency parameter is introduced for parameter splicing to construct a joint feature vector .

[0049] The historical consistency parameter is obtained based on the time and semantic stability of the last continuous candidate voice instruction for time sequence consistency constraint.

[0050] The joint feature vector The layer-by-layer processing is performed, and after the activation function output of each hidden layer, a Dropout operation is introduced to shield part of the neurons to inhibit model overfitting.

[0051] The comprehensive score is generated through linear transformation and Sigmoid mapping of the last hidden layer .

[0052] In another aspect, the application provides a voice recognition-based earphone instruction control system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the voice recognition-based earphone instruction control method.

[0053] Compared with the prior art, the application has the following beneficial effects:

[0054] 1. In the voice recognition-based earphone instruction control method and system, the scene matching degree parameter and the historical instruction consistency parameter are introduced, so that the false triggering suppression is no longer dependent on a single voice recognition confidence threshold, but is jointly determined through multi-dimensional semantic establishment conditions, thereby reducing the false triggering probability in a complex environment; and based on the scene rule constraint and the semantic matching degree calculation, the scene executability of the candidate voice instruction is determined, so that the voice instruction is not executed in the case that the earphone is not worn, the device state is not matched, or the task context is not established, and the false triggering problem caused by the inconsistency of the scene is effectively reduced.

[0055] 2. In the voice recognition-based earphone instruction control method and system, a frame-level effectiveness determination mechanism based on the signal-to-noise ratio, the voice activity confidence, and the far-field discrimination factor is introduced in the front end of the voice recognition, the original acoustic feature matrix is frame-by-frame filtered, and only the acoustic features meeting the effective voice condition are retained to participate in the subsequent recognition process, thereby suppressing abnormal input caused by environmental noise, background conversation, non-directional voice, or non-target user voice from the source, avoiding invalid voice from entering the recognition model, and improving the voice recognition capability of the overall system in the multi-person conversation, noisy environment, and motion scene. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 The overall method flowchart of the application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0058] Embodiment 1: Please refer to Figure 1 As shown in the figure, the embodiment provides a headset instruction control method based on voice recognition, comprising the following steps:

[0059] S1, pre-process the collected original voice signal and behavior state data (the pre-processing includes band-pass filtering, amplitude normalization and end-side noise reduction processing, to weaken the influence of environmental noise, background conversation and sudden interference on the quality of the voice signal), to obtain voice feature data for subsequent voice recognition;

[0060] In this embodiment, the original voice signal includes time series sampling values of the voice signal, corresponding sampling frequency and each microphone channel signal in a multi-microphone scene;

[0061] The behavior state data includes headset wearing state parameters, user motion state parameters, device running state parameters and interaction state parameters.

[0062] In this embodiment, the behavior state parameters at least include one or more of the following:

[0063] In-ear detection state, left and right ear wearing identification; still, walking or motion state identification; playing state, talking state or standby state; whether in a voice interaction activation window;

[0064] Specifically, the behavior state parameters are discretized or normalized, and are converted into low-dimensional or medium-low-dimensional behavior state feature vectors through one-hot encoding, numerical mapping or embedding encoding, so as to eliminate the differences in dimension and value range of different state parameters.

[0065] Further, the voice frame and the behavior data are aligned according to the time stamp.

[0066] S2, frame processing is performed on the pre-processed voice feature data, and the corresponding basic acoustic features and auxiliary features for voice recognition and false trigger suppression of each frame are extracted, the basic acoustic features are constructed into a voice feature matrix in time sequence as the input of the voice recognition model, and the voice activity effectiveness of the voice feature matrix is discriminated based on the auxiliary features (and noise robustness processing, to filter out abnormal inputs obviously caused by environmental noise, background conversation or non-directional voice), to generate an acoustic feature matrix after effectiveness screening;

[0067] Wherein, the basic acoustic features include MFCC vector, fundamental frequency and energy envelope;

[0068] The auxiliary features include signal-to-noise ratio estimate, voice activity confidence and far-near field discrimination factor;

[0069] The basic acoustic feature parameters are used to determine the validity of speech activity and to handle noise robustness, thereby avoiding abnormal inputs caused by environmental speech or non-command speech from participating in the subsequent command recognition process.

[0070] Specifically, based on auxiliary features, speech validity is determined, and abnormal inputs obviously caused by environmental noise, background conversations, or non-directional speech are filtered out; the basic acoustic feature parameters after validity screening are output as input for step S3 (the acoustic feature parameters are constructed based on time series to form a speech feature matrix, and the acoustic feature parameters are acoustic feature parameters after validity screening).

[0071] In this embodiment, the specific steps involved in generating the acoustic feature matrix after validity screening are as follows:

[0072] The preprocessed speech feature data is divided into frames using a sliding window approach. Basic acoustic features are extracted from each frame of speech feature data, and the basic acoustic features obtained from each frame are used to construct a feature vector. In the formula, Indicates the first The Mel frequency cepstral coefficient vector corresponding to the frame speech signal is used to characterize the spectral envelope characteristics of the speech signal in the frequency domain (its dimension is preferably 12-20, more preferably 13, which may or may not include the 0th order cepstral coefficients). Indicates the first The fundamental frequency parameter corresponding to the frame speech signal is used to characterize the fundamental frequency characteristics of the speech signal (calculated by autocorrelation method, cepstral method, YIN algorithm, PYIN algorithm or fundamental frequency estimation algorithm based on spectrum peak detection; in this embodiment, YIN algorithm is preferred for calculation). Indicates the first The energy envelope parameter corresponding to the frame speech signal is used to characterize the energy intensity characteristics of the frame speech signal; The frame index represents the position of the current speech frame in the time series after the speech signal is framed. Indicates the first The basic acoustic feature vector corresponding to the frame speech signal;

[0073] By concatenating the feature vectors of consecutive frames in chronological order, a preliminary speech feature matrix is ​​constructed. In the formula, This indicates the total number of audio frames corresponding to the current audio segment after frame processing;

[0074] For each frame of speech signal, auxiliary features are extracted synchronously, and an auxiliary feature vector is constructed (auxiliary feature vector). (Including signal-to-noise ratio estimate, speech activity confidence, and near-field discrimination factor).

[0075] Based on the auxiliary feature vector, a speech activity validity determination is performed on each frame of speech signal, and based on the result of the speech activity validity determination, the preliminary speech feature matrix is... Frame-level filtering is performed, and finally, an acoustic feature matrix with validity filtered is generated. In the formula, Indicates that the index is The basic acoustic feature vector corresponding to the speech frame, This indicates the number of valid speech frames retained after the speech activity validity determination, and Acoustic feature matrix It only includes acoustic feature parameters that meet the criteria for determining the validity of speech activity.

[0076] Furthermore, the validity of the speech activity is determined for each frame of the speech signal, specifically including:

[0077] Calculate the signal-to-noise ratio for each frame. and compared with the preset signal-to-noise ratio threshold Comparison:

[0078] like If the frame is not identified as a noise frame, it will not be included in subsequent speech recognition.

[0079] Based on the confidence level of speech activity in each frame Compared with the preset confidence threshold Comparison:

[0080] like If the frame is not valid, it will be considered an invalid speech frame and will not participate in subsequent speech recognition.

[0081] Near and far field factors for each frame With preset near and far field factor thresholds Compare:

[0082] like If the frame is not a target user's voice frame, it will be determined as such and will not participate in subsequent voice recognition.

[0083] Among them, the preliminary speech feature matrix The specific filtering rules involved in frame-level filtering are as follows:

[0084] Remove the feature vectors corresponding to speech frames that are judged to be invalid (including noise frames, invalid speech frames, and non-target user speech frames);

[0085] Amplitude normalization or spectral smoothing enhancement is performed on the retained valid speech frames;

[0086] Furthermore, time consistency constraints are applied to consecutive valid frames to prevent stray noise frames from entering the frame.

[0087] The time consistency constraint is used to eliminate sporadic noise frames caused by environmental noise or accidental interference, and is realized in the following ways: based on a voice activity duration constraint, a voice activity smoothing constraint, a frame similarity constraint, or a time window-based sliding window constraint.

[0088] In this embodiment, a voice activity duration-based constraint is adopted:

[0089] A constraint threshold is set (i.e., a minimum length of continuous valid frames or a number of continuous frames), and only when the sequence of valid frames is continuous for more than a certain number of frames or length of time, is it considered to be a valid voice sequence, which is used to avoid occasional environmental noise or transient interference signals being misjudged as valid voice;

[0090] That is, a minimum number of continuous valid frames is set If the number of continuous valid frames is less than this value, it is determined to be a sporadic noise frame, and this part is removed.

[0091] S3, decoding and recognizing the acoustic feature matrix after the effectiveness screening based on a voice recognition model to generate a candidate voice instruction and a corresponding voice recognition confidence;

[0092] The voice recognition model is an end-to-end recognition model based on Transformer, which is constructed by combining a convolutional neural network with a Transformer encoder.

[0093] The voice recognition model outputs a plurality of candidate text sequences, and based on the decoding probability or posterior probability inside the model, calculates the corresponding voice recognition confidence for each candidate text sequence, thereby forming a candidate voice instruction set.

[0094] In this embodiment, the candidate voice instruction and the corresponding voice recognition confidence are generated by the voice recognition model, and the specific steps involved are:

[0095] Based on the acoustic feature matrix after the effectiveness screening By introducing additional voice features, the acoustic feature matrix and the additional voice features are aligned on the time axis, and are spliced and constructed into an acoustic feature input matrix column by column, with a size of , wherein is the number of time frames, is the dimension of the basic acoustic features, is the dimension of the additional voice features.

[0096] Among them, the additional voice features include a speaker identity label (determined whether it is the voice of the target user through voiceprint recognition technology (extracting feature vectors such as MFCC, formant, etc. from the collected voice signal; comparing the extracted voiceprint features with the pre-stored target user template; confirming whether the voice signal is from the target user by calculating the matching degree)), a voice separation confidence (evaluating whether the voice signal contains information of multiple speakers through speaker separation technology, and quantifying whether it is valid voice of the target user), and a far-field factor (confirming whether the voice comes from the target wearer by calculating the spatial position of the voice and the microphone through the sound source positioning technology (such as microphone array));

[0097] In this embodiment, the speaker separation technology is used to separate individual voice signals from mixed voice signals of multiple speakers, especially in a multi-user or noisy environment, it can identify and extract the voice of the target user and eliminate other non-target user voice interference; based on blind source separation (BSS) and independent component analysis (ICA), different sound sources are separated from the mixed signal by analyzing the statistical characteristics of each independent source in the sound signal, or end-to-end speaker separation is performed in combination with deep learning (such as convolutional neural network (CNN) or recurrent neural network (RNN)); Specifically, a deep neural network (such as CNN) is trained by signals received by a multi-microphone array to separate mixed voice signals; then, based on the confidence evaluation of the separated voice signals, it is determined whether the voice signal contains the voice of the target user; finally, a voice separation confidence is generated, which quantifies the relative proportion of target voice and other non-target voice, thereby effectively suppressing interference;

[0098] The acoustic feature input matrix is input into the convolutional neural network, each convolution kernel processes the input data through a sliding window, and local features are extracted through convolution operations of multiple convolution kernels on the acoustic feature input matrix, outputting a high-dimensional feature map , the size of which is , wherein is the remaining time step after pooling, is the number of convolution kernels, i.e. the number of extracted feature channels;

[0099] The convolution kernel extracts local features in the time domain and the frequency domain, respectively, thereby capturing the time sequence pattern and the frequency domain pattern in the input features. After the convolution layer, the activation function (such as ReLU) performs a nonlinear transformation on the convolution result to enhance the expression ability of the model. The pooling layer (such as maximum pooling or average pooling) receives the convolution output and retains the most significant features through dimension reduction operation, reduces the computational complexity, and enhances the robustness of the model to local features;

[0100] Through convolution operations, the following key information can be extracted: time-domain and frequency-domain features of the speech signal, whether the speech signal comes from the target user (based on voiceprint features), noise interference information in the speech signal (through speech separation confidence), and whether the speech signal comes from the wearer (through near-far field factors).

[0101] High-dimensional feature maps The data is fed into a Transformer encoder, which extracts global semantic information through a self-attention mechanism and outputs a temporal feature matrix. Temporal feature matrix It contains global semantic information in the time dimension;

[0102] Time series feature matrix The data is fed into the decoder, which uses a decoding algorithm (such as CTC decoding or an autoregressive decoder) to analyze the temporal feature matrix. Generate a text sequence of candidate voice commands;

[0103] For each candidate text sequence, the decoder calculates its corresponding posterior probability and calculates the corresponding speech recognition confidence score for each candidate text sequence (used to represent the probability that the candidate instruction is a correct instruction; the specific speech recognition confidence score is used as the basis for sorting and filtering candidate instructions, and the candidate instruction with a higher confidence score is more likely to be an effective recognition result).

[0104] In this embodiment, the speech recognition model preferably adopts the Conformer (Convolution-augmented Transformer) model, which combines the local feature extraction capability of convolutional networks with the global modeling capability of Transformers, making it suitable for complex noisy environments and speech recognition tasks.

[0105] Specifically, the Conformer model is constructed by combining a convolutional neural network (CNN) and a Transformer. The input to the Conformer model is a preprocessed acoustic feature matrix and additional speech features.

[0106] The Conformer model at least includes an audio input layer, a convolutional neural network layer, a Transformer encoder layer, a decoder layer, and an output layer; in this embodiment, by adding multiple input heads to the Conformer model, the collection and input of additional speech features are realized, the basic acoustic feature matrix (such as MFCC) and the additional speech features (such as the speaker identity label, the speech separation confidence, and the far-near field factor) are input into different input heads as two-way input, and after processing of the basic acoustic features and the additional speech features (the additional features (such as the speaker identity, the speech separation confidence, and the far-near field factor) are encoded (such as through one-hot encoding, embedding layer, normalization, etc.), and are sent to the additional input head of the model for processing), the two are spliced (or other fusion methods) in the model to generate a composite feature matrix for subsequent convolutional neural network (CNN) and Transformer encoder processing.

[0107] The Transformer encoder is used to capture long-range global dependencies, further optimize the modeling capability of the speech signal, and introduce a self-attention mechanism to enable the model to weightedly pay attention to features at different time steps, effectively improving the modeling capability for long-time dependencies; the decoder layer generates candidate speech instructions according to the encoded feature sequence, outputs the probability distribution of each candidate speech instruction through a Softmax layer, outputs several candidate text sequences, and calculates the speech recognition confidence for each candidate text according to the decoded probability; the output layer calculates the recognition confidence based on the output probability of the Conformer model (the probability of each candidate instruction is calculated using a softmax activation function, and then the speech recognition confidence corresponding to each candidate text sequence is generated according to the posterior probability), and finally generates a candidate speech instruction set.

[0108] S4, structurally encode the behavior state data to generate a scene state vector (usage environment state, user motion state, running state of the earphone or external device, and current task context state) for representing the current usage environment and task context;

[0109] For the candidate speech instruction, the semantic matching degree with the scene state vector is calculated (based on a rule constraint to determine whether the instruction is executable in the current scene, and finally obtain a matching degree parameter corresponding to each candidate speech instruction, which is used to quantify the semantic establishment degree of the instruction in the current scene); by explicitly encoding the behavior state data and introducing it into the scene matching evaluation process, the execution determination of the candidate speech instruction is no longer dependent on the speech content itself, but is jointly constrained by the current wearing state, the user behavior state, and the device running state, thereby effectively eliminating the semantic ambiguity of the instruction caused by the change of the usage scene;

[0110] In this embodiment, the matching degree of the candidate voice instruction and the scene state vector is calculated, and the specific steps are as follows:

[0111] The behavior state data is structured and coded:

[0112] The in-ear detection data is obtained through a sensor (such as an accelerometer or a gyroscope): if the earphone is in a wearing state, the value is 1, otherwise, the value is 0; the left and right ear wearing indicators can be coded in binary, such as [1, 0] indicating left ear wearing and [0, 1] indicating right ear wearing;

[0113] The motion state is obtained through sensor data (such as an accelerometer): the motion state is represented by a discrete identifier: 0 represents static, 1 represents walking, 2 represents running, and 3 represents cycling;

[0114] The current earphone running state is obtained through a device interface, for example, 0 represents standby, 1 represents playing, 2 represents calling, and the like;

[0115] If the device is in a voice interaction activation window, the value is 1, otherwise, the value is 0;

[0116] Further, One-Hot Encoding is used for vectorization processing to obtain a corresponding high-dimensional sparse representation for processing discrete state data, such as device running state and user motion state;

[0117] In the coding process, all behavior state data is aligned in time (for example, through a sliding window method) and spliced by column to finally generate a fixed-length scene state vector ;

[0118] The scene rule constraint is used to determine the scene executability of the candidate voice instruction, which is used to determine whether the candidate voice instruction meets the execution condition of the current scene (to exclude instructions that are obviously impossible to execute);

[0119] For the candidate voice instruction that does not meet the execution condition, it is directly determined as an unexecutable instruction and its execution is suppressed;

[0120] For the candidate voice instruction that meets the execution condition, the semantic matching degree thereof and the scene state vector is calculated, which is used to quantify and screen the candidate voice instruction that meets the scene semantics;

[0121] Among them, the scene rule constraint is used to eliminate candidate instructions that obviously do not have execution legality, and the semantic matching degree is used to further distinguish the degree of fit of the candidate instructions that meet the rule constraint and the current scene semantics, and the two are used in different decision levels;

[0122] In this embodiment, the candidate voice instruction mapped into a semantic vector by a semantic encoding model , represents the th candidate voice instruction, represents the candidate voice instruction mapped into a semantic vector by a semantic encoding model, with a dimension of ;

[0123] a scene state vector is nonlinearly mapped to obtain a scene semantic vector , wherein, represents a nonlinear function or model (such as a multilayer perception or an embedding mapping function) for mapping the scene state vector to a semantic space;

[0124] based on a cosine similarity algorithm, a semantic matching degree between the semantic vector and the scene semantic vector is calculated as follows:

[0125] ;

[0126] , wherein, represents a norm (usually a Euclidean norm) of the candidate instruction semantic vector, for normalization; represents a norm (usually a Euclidean norm) of the scene semantic vector, for normalization;

[0127] Further, by a preset matching degree threshold , the semantic matching degree is compared with the preset matching degree threshold:

[0128] when , the candidate voice instruction is semantically matched with the current scene state;

[0129] when , the candidate voice instruction is not matched with the current scene state, and is suppressed or down-weighted.

[0130] Further, the scene rule constraint is based on an earphone wearing state, a device running state, and a task context, to judge the executability of the candidate voice instruction;

[0131] wherein, the scene rule constraint includes:

[0132] when the earphone is in a wearing state, only instructions requiring the earphone to be worn are allowed to be executed;

[0133] In this embodiment, the earphone wearing state parameter is acquired through the in-ear detection sensor, and when the voice instruction involves an earphone operation that can be executed in the wearing state, it is determined that the candidate voice instruction meets the wearing state constraint; otherwise, it is determined that it does not meet and is inhibited (for example, when it is detected that the earphone is in the wearing state, and the candidate voice instruction is to increase the volume, the action type label is parsed as an earphone playback control class operation, and it is determined that the wearing state constraint is met; when the candidate voice instruction is to turn off the power of the earphone box, the action type does not match the current wearing state, and it is determined that the wearing state constraint is not met and is inhibited);

[0134] According to the left and right ear wearing marks, only instructions matching the wearing ear are allowed to be executed;

[0135] In this embodiment, when the target ear side is inconsistent with the voice instruction, execution is inhibited, and if the target ear side is consistent with the voice instruction or the instruction does not contain ear side pointing information, it is determined that it meets the constraint (for example, when it is detected that only the left ear is worn, and the candidate voice instruction is to turn off the noise reduction of the right ear, the target ear side parameter is parsed, and if the two are inconsistent, it is determined that the instruction cannot be executed; when the candidate voice instruction is to turn on the noise reduction mode and does not specify a specific ear side, it is determined that the ear side consistency constraint is met);

[0136] According to the user motion state, only instructions that conform to the current motion state are allowed to be executed;

[0137] In this embodiment, the motion state parameter (such as a state of being still, walking, or moving) is acquired through an inertial sensor, and when the user is in a motion state, only voice instructions with a low complexity interaction complexity label are allowed to be executed; for voice instructions that require fine operations or long-time interaction, it is determined that they conflict with the current motion state and are inhibited (for example, when it is detected that the user is in a running state, and the candidate voice instruction is “next song”, the interaction complexity is parsed as low complexity, and it is determined that it can be executed; when the candidate voice instruction is “enter settings and modify equalization parameters”, the interaction complexity is high, and it is determined that the motion state constraint is not met);

[0138] According to the device running state, only instructions that match the current running state are allowed to be executed;

[0139] In this embodiment, according to the device running state, only voice instructions whose instruction semantics are logically consistent with the current device running state are allowed to be executed, when the device is in a call state, the candidate voice instruction is “pause music”, the target running state is inconsistent with the current call state, and it is determined that it cannot be executed; when the candidate voice instruction is “hang up the phone”, the target running state is consistent with the current call state, and it is determined that it can be executed;

[0140] Only when the device is in a voice interaction activated state, voice instructions are allowed to be executed;

[0141] If the candidate voice instruction meets all the above sub-rule constraints, it is determined to be executable (the candidate voice instruction is a scene executable instruction, and the subsequent semantic matching degree calculation or joint scoring determination process is entered), otherwise it is determined to be unexecutable and is suppressed;

[0142] Further, the above wearing constraint based on the earphone wearing state is used to limit the execution of a preset type of voice control instruction in the unwearing state; the state constraint based on the device running state is used to limit the legality between the action semantics of the candidate voice instruction and the current device running state; the task constraint based on the task context is used to limit whether the candidate voice instruction belongs to the instruction set allowed in the current task context.

[0143] S5, the semantic matching degree is combined with the voice recognition confidence, and a joint feature vector is constructed by introducing a historical instruction consistency parameter (obtained through time sequence consistency constraint; obtained through time sequence consistency constraint calculation on the stability of the latest continuous candidate instructions in time and semantics, used to quantify the matching degree of the current candidate instruction and the historical instruction sequence, so as to suppress the false triggering caused by accidental voice or abnormal input);

[0144] A joint scoring model is constructed based on a lightweight fully connected neural network, and the joint feature vector is input into the joint scoring model (the joint scoring model can be a multi-layer perception (MLP), a lightweight fully connected neural network, or other scoring models suitable for end-side operation, and in the embodiment, the lightweight fully connected neural network is preferred), to generate a comprehensive score for instruction execution determination , the specific steps are:

[0145] For each candidate voice instruction , the semantic matching degree and the voice recognition confidence with the current scene state vector are obtained, and the historical consistency parameter is introduced for parameter splicing. The above parameters are normalized according to a unified dimension to construct a joint feature vector , wherein, the semantic matching degree is represented by, the voice recognition confidence is represented by, the historical instruction consistency parameter is represented by; specifically, the historical instruction consistency parameter is modeled by the consistency of the continuous candidate instructions in the time interval and the semantic similarity, and participates in the joint scoring calculation, so that the low consistency candidate instruction appearing in isolation is naturally de-weighted in the comprehensive score, thereby suppressing the ambiguous triggering caused by transient recognition deviation in the multi-candidate instruction decision process, improving the stability of instruction selection, and reducing the number of false executions and passive backtracking;

[0146] wherein, the historical instruction consistency parameter is based on the latest The time sequence consistency constraint of the stability of the sub-continuous candidate voice instruction in time and semantics is used to suppress the false triggering of accidental voice or abnormal input, and the history instruction consistency parameter is:

[0147] ;

[0148] In the formula, is the current time th candidate instruction, indicates the index number of the candidate voice instruction in the time sequence; is the instruction sequence composed of the last history candidate voice instruction before the current candidate instruction, wherein is the history instruction index; the function is a history consistency calculation function, which is used to quantify the consistency of the current candidate instruction and the history instruction sequence in time interval and semantic similarity, wherein the function is one or more of a time interval decay function, a semantic vector similarity function, or an instruction category consistency constraint function, and the history consistency parameter is generated by a weighted fusion manner to suppress the false triggering caused by accidental voice or abnormal input; indicates the number of history candidate instructions for history consistency analysis;

[0149] The joint feature vector is processed layer by layer based on the full connection hidden layer (specifically, the joint feature vector is first input to the first hidden layer, and after linear transformation and bias weighting processing of the hidden layer, the first layer hidden state is obtained through the mapping of the non-linear activation function (ReLU / Leaky ReLU / Swish); then, the first hidden state is sequentially transmitted to the subsequent hidden layers as input, and each hidden layer performs the same linear transformation and non-linear activation mapping operation to generate the corresponding hidden state ), and after the output of each hidden layer activation function, the Dropout operation is introduced to randomly shield part of the neurons to suppress model overfitting, while maintaining the original dimension structure of the hidden state for the next hidden layer to continue processing;

[0150] wherein the Dropout operation is applied after each layer activation output, and the output is still a vector of the same dimension, except that part of the elements are set to 0 during training (Dropout is usually not enabled or is adjusted by scaling in the prediction stage), therefore, after Dropout is still the hidden layer state for subsequent layer input;

[0151] The comprehensive score is generated by linear transformation and Sigmoid mapping through the last hidden layer ;

[0152] In the embodiment, the lightweight fully connected neural network at least includes an input layer, a fully connected hidden layer, and an output layer;

[0153] The input dimension of the input layer is equal to the length of the joint feature vector, and each feature is normalized to ;

[0154] The fully connected hidden layer is provided with at least 3 layers (the number of layers is adjusted according to the end-side computing capability), and a part of neurons is randomly shielded after the activation output of each layer (in the training stage) by introducing Dropout, the original dimension of the hidden state is maintained, and the prediction stage can be closed or adjusted by scaling;

[0155] The lightweight of the lightweight fully connected neural network adopts parameter pruning and low-precision quantization, the weights of the fully connected hidden layer are pruned to generate a sparse weight matrix, and then the sparse weights are quantized at low precision to realize double compression of model parameters and calculation; specifically, the weight matrix of each fully connected hidden layer is processed by sparsification, the proportion of the retained weights to the original weights after pruning is adjustable, and 50% to 80% is preferably retained to reduce the model parameter quantity and the calculation complexity, at the same time, the full connection topology of each layer is maintained during the pruning process, and only the number of effective parameters participating in the calculation is reduced; further, the non-zero weights after pruning are quantized by using 8-bit or 16-bit fixed-point representation to replace the original 32-bit floating-point numbers; in the quantization process, a symmetric or asymmetric quantization method is used, and a scaling factor is used for numerical mapping to map the quantized values to the approximate floating-point range, which is used to ensure the numerical accuracy and stability of the network output, wherein the scaling factor is calculated by using the maximum absolute value method or the dynamic range method;

[0156] In the training stage of the lightweight fully connected neural network, mean square error (MSE) or cross-entropy loss (if the score is mapped to a binary classification probability) is used as the loss function, and Adam or SGD is used as the optimizer;

[0157] According to the preset execution condition (i.e. a preset score threshold, which is adaptively adjusted according to the use scene, user habit or historical execution result), only when the comprehensive score result meets the preset execution condition, the corresponding candidate voice instruction is determined as a valid instruction and enters the execution step; otherwise, the candidate instruction is determined as an invalid instruction and its execution is inhibited, so that the ambiguity of the voice instruction in the complex use scene is eliminated and the mis-triggering behavior is inhibited;

[0158] only when the comprehensive score When the preset execution condition is met, the corresponding instruction is determined as a valid instruction, otherwise, it is determined as invalid and its execution is inhibited;

[0159] Among them, for the continuous candidate instruction, when the comprehensive score Sliding window smoothing or exponential weighted average is performed to enhance the execution stability (to inhibit the false triggering of short-time abnormal instructions, and to improve the stability and robustness of continuous instruction execution):

[0160] ;

[0161] Among them, is a smoothing coefficient, used to adjust the weight ratio of the current score and the historical score in the smoothing calculation; is the smoothed comprehensive score of the last candidate voice instruction; represents the original comprehensive score corresponding to the th candidate voice instruction at the th moment; represents the smoothed comprehensive score obtained after smoothing the original comprehensive score of the th candidate voice instruction at the th moment, which is used for instruction execution determination, represents the time index of the candidate voice instruction in the time sequence.

[0162] S6, for the candidate instruction determined to be valid, the corresponding earphone control operation is generated and executed;

[0163] The candidate instruction that does not meet the execution condition is discarded, so as to avoid the semantic invalid instruction or the environmental voice false triggering.

[0164] Embodiment 2: The embodiment provides an earphone instruction control system based on voice recognition, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the earphone instruction control method based on voice recognition.

[0165] The basic principles, main features and advantages of the present application are shown and described above. It should be understood by those skilled in the art that the present application is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A headphone command control method based on voice recognition, characterized in that, Includes the following steps: S1. Preprocess the collected raw speech signals and behavioral state data to obtain speech feature data; S2. Perform frame segmentation on the preprocessed speech feature data, extract the basic acoustic features and auxiliary features corresponding to each frame, construct a speech feature matrix by time sequence of the basic acoustic features, and perform speech activity validity discrimination on the speech feature matrix based on the auxiliary features to generate an acoustic feature matrix after validity screening. The fundamental acoustic features include the MFCC vector, fundamental frequency, and energy envelope; Auxiliary features include signal-to-noise ratio estimates, speech activity confidence, and near-far discriminant factors; S3. Based on the speech recognition model, decode and recognize the acoustic feature matrix after validity screening to generate candidate speech commands and their corresponding speech recognition confidence scores. Among them, the speech recognition model is an end-to-end recognition model based on Transformer, which is constructed by combining a convolutional neural network with a Transformer encoder; S4. Perform structured encoding on the behavioral state data to generate a scene state vector. For candidate voice commands, calculate their semantic matching degree with the scene state vector. S5. The semantic matching degree is combined with the speech recognition confidence degree, and a joint feature vector is constructed by introducing a historical instruction consistency parameter. A joint scoring model is constructed based on a lightweight fully connected neural network, and the joint feature vector is input into the joint scoring model to generate a comprehensive score for instruction execution determination. ; Only when the overall score When the preset execution conditions are met, the corresponding instruction is determined to be a valid instruction; otherwise, it is determined to be invalid and its execution is suppressed. S6. For candidate instructions that are determined to be valid, generate the corresponding headphone control operation and execute it; Candidate instructions that do not meet the execution conditions are discarded to avoid accidental triggering by environmental voice.

2. The headphone command control method based on voice recognition according to claim 1, characterized in that: The original speech signal includes the time-series sampled values ​​of the speech signal, the corresponding sampling frequency, and the signals of each microphone channel in a multi-microphone scenario; The behavioral state data includes headphone wearing status parameters, user movement status parameters, device operation status parameters, and interaction status parameters.

3. The headphone command control method based on voice recognition according to claim 1, characterized in that: In step S2, generating the acoustic feature matrix after validity screening involves the following specific steps: The preprocessed speech feature data is divided into frames using a sliding window approach, and the basic acoustic features are extracted for each frame of speech feature data. The basic acoustic features obtained from each frame are used to construct a feature vector. ; By concatenating the feature vectors of consecutive frames in chronological order, a preliminary speech feature matrix is ​​constructed. ; For each frame of speech signal, auxiliary features are extracted synchronously, and an auxiliary feature vector is constructed. Based on the auxiliary feature vector, a speech activity validity determination is performed on each frame of speech signal, and based on the result of the speech activity validity determination, the preliminary speech feature matrix is... Frame-level filtering is performed, and finally, an acoustic feature matrix with validity filtered is generated. .

4. The headphone command control method based on voice recognition according to claim 3, characterized in that: The preliminary speech feature matrix The specific filtering rules involved in performing frame-level filtering to ultimately generate the acoustic feature matrix are as follows: Remove the feature vectors corresponding to the speech frames that are deemed invalid; Amplitude normalization is performed on the retained valid speech frames; And time consistency constraints are applied to consecutive valid frames.

5. The headphone command control method based on voice recognition according to claim 1, characterized in that: In step S3, the specific steps involved in generating candidate speech commands and their corresponding speech recognition confidence scores through a speech recognition model are as follows: Based on the acoustic feature matrix after effectiveness screening By introducing additional speech features, an acoustic feature input matrix is ​​constructed; The additional speech features include speaker identity labels, speech separation confidence, and near-far field factors; The acoustic feature input matrix is ​​fed into a convolutional neural network. Each convolutional kernel processes the input data through a sliding window. By performing convolution operations on the acoustic feature input matrix using multiple convolutional kernels, local features are extracted, and a high-dimensional feature map is output. ; High-dimensional feature maps The data is fed into a Transformer encoder, which extracts global semantic information through a self-attention mechanism and outputs a temporal feature matrix. ; Time series feature matrix The data is fed into the decoder, which, based on a decoding algorithm, extracts the temporal feature matrix. Generate a text sequence of candidate voice commands; For each candidate text sequence, the decoder calculates its corresponding posterior probability and calculates the corresponding speech recognition confidence for each candidate text sequence.

6. The headphone command control method based on voice recognition according to claim 1, characterized in that: In step S4, the specific steps involved in calculating the semantic matching degree between the candidate voice command and the scene state vector are as follows: Based on scene rule constraints, the executability of candidate voice commands is determined to determine whether the candidate voice commands meet the execution conditions of the current scene. For candidate voice commands that do not meet the execution conditions, they are directly determined as non-executable commands and their execution is suppressed; For candidate voice commands that meet the execution conditions, calculate their semantic matching degree with the scene state vector.

7. The headphone command control method based on voice recognition according to claim 6, characterized in that: The scenario rule constraints are based on the headphone wearing status, device operating status, and task context to determine the executability of candidate voice commands; The scenario rule constraints include: When the headphones are being worn, only commands requiring the headphones to be worn are allowed. Based on the left and right ear markings, only commands matching the ear being worn are permitted. Based on the user's movement status, only instructions that match the current movement status are allowed to be executed; Based on the device's operating status, only instructions that match the current operating status are allowed to be executed; Voice commands are allowed to be executed only when the device is in a voice interaction active state; Furthermore, if a candidate voice command simultaneously satisfies all of the above sub-rules, it is determined to be executable; otherwise, it is determined to be non-executable and suppressed.

8. The headphone command control method based on voice recognition according to claim 1, characterized in that: In step S5, a comprehensive score is generated by the joint scoring model. The specific steps involved are as follows: For each candidate voice command The semantic matching degree and speech recognition confidence of the vector with the current scene state vector are obtained, and a joint feature vector is constructed by concatenating the parameters by introducing a historical consistency parameter. ; Among them, the historical consistency parameter is based on the most recent The stability of successive candidate speech commands in terms of time and semantics is obtained by constraining the temporal consistency. Based on fully connected hidden layers for joint feature vectors The process is performed layer by layer, and after the activation function output of each hidden layer, a Dropout operation is introduced to screen out some neurons to suppress model overfitting. A comprehensive score is generated by applying a linear transformation and a Sigmoid mapping through the last hidden layer. .

9. A voice recognition-based headphone command control system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the voice recognition-based headphone command control method as described in any one of claims 1-8.

Images