Speech recognition method and system for a voice assistant

Abstract

The application is suitable for the technical field of audio processing, and provides a speech recognition method and system of a voice assistant, the speech recognition method of the voice assistant comprising: acquiring a speech signal, extracting an acoustic feature sequence, pronunciation constraint information and prosodic constraint information from the speech signal respectively; constructing a joint constraint optimization problem by taking acoustic likelihood of the acoustic feature sequence as a hard constraint, and taking pronunciation consistency of the pronunciation constraint information and prosodic structure of the prosodic constraint information as soft constraints; obtaining a recognition text sequence by solving the joint constraint optimization problem through constraint optimization; performing constraint satisfaction degree verification on the recognition text sequence, and when the constraint satisfaction degree is lower than a preset threshold, adaptively adjusting a soft constraint weight according to reliability of each constraint and re-solving. By introducing pronunciation constraint and prosodic constraint, the recognition accuracy is improved, and the adaptability to environmental changes is enhanced.

Description

Technical Field

[0001] This invention belongs to the technical field of audio processing, and particularly relates to a voice recognition method and system for a voice assistant. Background Technology

[0002] With the rapid development of smart homes and digital entertainment, set-top boxes, as crucial devices connecting users to multimedia content, are constantly expanding their functions, and the application of voice assistants is becoming increasingly widespread. Voice assistants can use natural language processing technology to help users conveniently perform operations such as content searching, playback control, and system settings. However, existing set-top box voice assistants still face many challenges in terms of voice recognition accuracy, affecting the user experience.

[0003] Current speech recognition technologies primarily rely on a combination of acoustic and language models; however, in practical applications, the accuracy of these technologies remains unsatisfactory. Firstly, environmental noise, the speaker's accent, and speech rate significantly affect the quality of the speech signal, leading to reduced accuracy in the recognition results. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a speech recognition method and system for a voice assistant to solve the technical problem that factors such as environmental noise, the speaker's accent and speaking speed can significantly affect the quality of the speech signal, leading to a decrease in the accuracy of the recognition results.

[0005] A first aspect of this invention provides a speech recognition method for a voice assistant, the speech recognition method for the voice assistant comprising: Acquire a speech signal, and extract acoustic feature sequences, articulation constraint information, and prosodic constraint information from the speech signal, respectively; Using the acoustic likelihood of the acoustic feature sequence as a hard constraint, and the phonetic consistency of the phonetic constraint information and the prosodic structure of the prosodic constraint information as soft constraints, a joint constraint optimization problem is constructed. The joint constraint optimization problem is solved by constraint optimization to obtain the recognized text sequence; The constraint satisfaction of the identified text sequence is verified. When the constraint satisfaction is lower than a preset threshold, the soft constraint weights are adaptively adjusted according to the reliability of each constraint and the solution is re-solved. The steps of acquiring the speech signal and extracting the acoustic feature sequence, articulation constraint information, and prosodic constraint information from the speech signal include: The speech signal is subjected to framing and pre-emphasis processing to obtain a framed speech signal; Mel frequency cepstral coefficients are extracted from the framed speech signal to form an acoustic feature sequence; The acoustic feature sequence is input into a pre-trained acoustic-to-speech inversion model, which outputs a sequence of articulatory organ position parameters. The acoustic-to-speech inversion model takes the acoustic feature sequence as input and maps the acoustic features to the articulatory organ position parameter space through a deep neural network. The articulatory organ position parameters include tongue height, tongue anteroposterior position, lip rounding, and oral cavity opening. Calculate the positional change between adjacent frames in the articulatory organ position parameter sequence, and use the positional change as the articulation constraint information; Extract the fundamental frequency trajectory and short-time energy trajectory from the speech signal; Align the fundamental frequency trajectory and the short-time energy trajectory along the time axis, and calculate the fundamental frequency change rate and energy change rate for each frame; Frames whose absolute value of the fundamental frequency change rate is greater than a first preset threshold and whose absolute value of the energy change rate is greater than a second preset threshold are marked as prosodic boundary candidate frames. Merging is performed on adjacent prosodic boundary candidate frames to determine the prosodic boundary position; The speech signal is divided into prosodic unit sequences based on the prosodic boundary positions, and the prosodic unit sequences and their corresponding fundamental frequency trajectories and short-time energy trajectories constitute the prosodic constraint information.

[0006] Furthermore, the steps of acquiring the speech signal and extracting the acoustic feature sequence, articulation constraint information, and prosodic constraint information from the speech signal include: The speech signal is subjected to framing and pre-emphasis processing to obtain a framed speech signal; Mel frequency cepstral coefficients are extracted from the framed speech signal to form an acoustic feature sequence; The acoustic feature sequence is input into a pre-trained acoustic-to-speech inversion model, which outputs a sequence of articulatory organ position parameters. The acoustic-to-speech inversion model takes the acoustic feature sequence as input and maps the acoustic features to the articulatory organ position parameter space through a deep neural network. The articulatory organ position parameters include tongue height, tongue anteroposterior position, lip rounding, and oral cavity opening. Calculate the positional change between adjacent frames in the articulatory organ position parameter sequence, and use the positional change as the articulation constraint information; Extract the fundamental frequency trajectory and short-time energy trajectory from the speech signal; Align the fundamental frequency trajectory and the short-time energy trajectory along the time axis, and calculate the fundamental frequency change rate and energy change rate for each frame; Frames whose absolute value of the fundamental frequency change rate is greater than a first preset threshold and whose absolute value of the energy change rate is greater than a second preset threshold are marked as prosodic boundary candidate frames. Merging is performed on adjacent prosodic boundary candidate frames to determine the prosodic boundary position; The speech signal is divided into prosodic unit sequences based on the prosodic boundary positions, and the prosodic unit sequences and their corresponding fundamental frequency trajectories and short-time energy trajectories constitute the prosodic constraint information.

[0007] Furthermore, the step of constructing a joint constraint optimization problem using the acoustic likelihood of the acoustic feature sequence as a hard constraint and the phonetic consistency of the phonetic constraint information and the prosodic structure of the prosodic constraint information as soft constraints includes: The acoustic feature sequence is input into the acoustic model, which takes the acoustic feature sequence as input and outputs the posterior probability distribution of each phoneme category in each frame through a deep neural network. Based on the posterior probability distribution, multiple candidate text sequences are generated through decoding search, and each candidate text sequence is a sequence obtained by mapping phoneme sequences to text. For each candidate text sequence, the product of the posterior probabilities of each phoneme in its corresponding phoneme sequence is calculated as the acoustic likelihood value of the candidate text sequence. Candidate text sequences with acoustic likelihood values ​​lower than a preset likelihood threshold are excluded from the search space, and the remaining candidate text sequences form a hard-constrained candidate set. For each candidate text sequence in the hard constraint candidate set, the pronunciation transfer cost between its corresponding articulatory organ position sequence and the pronunciation constraint information is calculated as the pronunciation consistency score. For each candidate text sequence in the hard constraint candidate set, the overlap rate between its prosodic boundary position and the prosodic boundary position in the prosodic constraint information is calculated as the prosodic structure score. Using the hard-constrained candidate set as the feasible region, and the product of the pronunciation consistency score and the first weight coefficient plus the product of the prosodic structure score and the second weight coefficient as the objective function value, a joint constraint optimization problem is constructed to maximize the objective function value within the feasible region.

[0008] Further, the step of calculating the pronunciation transfer cost between the corresponding articulatory organ position sequence and the pronunciation constraint information as a pronunciation consistency score for each candidate text sequence in the hard constraint candidate set includes: Based on the pre-stored phoneme-to-articulation organ position mapping table, each phoneme in the candidate text sequence in the hard constraint candidate set is converted into the corresponding expected articulation organ position parameter to form the expected articulation position sequence; Calculate the Euclidean distance between the expected articulation position sequence and the actual articulation organ position parameter sequence in the articulation constraint information on each frame to obtain the frame-by-frame position deviation sequence; The mean of the frame-by-frame positional deviation sequence is calculated as the pronunciation transfer cost; The complement of the pronunciation transfer cost is used as the pronunciation consistency score; wherein the pronunciation consistency score ranges from 0 to 1.

[0009] Further, the step of calculating the overlap rate between the prosodic boundary position and the prosodic boundary position in the prosodic constraint information for each candidate text sequence in the hard constraint candidate set as the prosodic structure score includes: Based on the pre-stored text-to-prosodic mapping rules, the expected prosodic boundary position of each word in the candidate text sequence in the hard constraint candidate set is determined, thus forming the expected prosodic boundary set; Calculate the number of intersections between the expected prosodic boundary set and the actual prosodic boundary positions in the prosodic constraint information; The overlap rate is defined as the ratio of the number of intersections to the total number of boundaries in the expected prosodic boundary set. The overlap rate is used as the prosodic structure score.

[0010] Furthermore, the step of performing constraint satisfaction verification on the identified text sequence, and when the constraint satisfaction is lower than a preset threshold, adaptively adjusting the soft constraint weights according to the reliability of each constraint and resolving the problem, includes: The weighted sum of the acoustic likelihood value, pronunciation consistency score, and prosodic structure score of the identified text sequence is used as the constraint satisfaction. When the constraint satisfaction is lower than the preset threshold, the signal-to-noise ratio estimate of the speech signal is calculated; When the estimated signal-to-noise ratio is lower than the preset signal-to-noise ratio threshold, the weight of the pronunciation consistency constraint is reduced by a preset percentage, and the weight of the prosodic structure constraint is increased by the preset percentage. When the estimated signal-to-noise ratio is higher than the preset signal-to-noise ratio threshold, the weight of the pronunciation consistency constraint is increased by the preset proportion, and the weight of the prosodic structure constraint is decreased by the preset proportion. The joint constraint optimization problem is solved again using the adjusted weights.

[0011] Furthermore, after the step of performing constraint satisfaction verification on the identified text sequence, and adaptively adjusting the soft constraint weights according to the reliability of each constraint and resolving when the constraint satisfaction is lower than a preset threshold, the method further includes: Based on the pre-stored phoneme-to-speech organ position mapping table, each phoneme in the identified text sequence is converted into the corresponding expected speech organ position parameter; Calculate the positional deviation between the expected articulatory organ position parameters and the actual articulatory organ position parameters of the corresponding frame in the articulation constraint information; Phonemes with a positional deviation greater than a preset deviation threshold are marked as abnormal phonemes; When the aberrant phoneme is in a non-critical position in the recognized text sequence, the aberrant phoneme is deleted from the recognized text sequence; wherein, non-critical positions include unstressed syllables at the end of words, filler syllables, and ambiguous syllables near prosodic boundaries; When the abnormal phoneme has a missing position in the recognized text sequence, the phoneme with the smallest positional deviation is searched in the phoneme-to-speech organ position mapping table according to the position parameters of the articulatory organs of the adjacent phonemes to complete the missing position.

[0012] Furthermore, the step of solving the joint constraint optimization problem through constraint optimization to obtain the recognized text sequence includes: Construct an objective function, which is the weighted sum of the pronunciation consistency score and the prosodic structure score for each candidate text sequence in the hard constraint candidate set; wherein the first weight coefficient corresponding to the pronunciation consistency score and the second weight coefficient corresponding to the prosodic structure score are determined based on the prior reliability of each soft constraint; Perform a beam search in the search space corresponding to the hard constraint candidate set in a frame-synchronized manner; In each frame, expand all legal phoneme transfers for each candidate path currently retained, update the pronunciation consistency score and prosodic structure score of the candidate path based on the expanded phoneme sequence, and calculate the cumulative objective function value; All expanded candidate paths are sorted in descending order of cumulative objective function value, and the top K candidate paths are retained as surviving paths, while the remaining candidate paths are pruned; where K is the preset bundle width. When all frames have been processed, the path with the largest cumulative objective function value is selected from the candidate survival paths as the optimal path. The phoneme sequence corresponding to the optimal path is obtained by backtracking, and the phoneme sequence is mapped to a text sequence as the recognition text sequence.

[0013] A second aspect of the present invention provides a voice recognition device for a voice assistant, comprising: The constraint extraction module is used to acquire speech signals and extract acoustic feature sequences, articulation constraint information, and prosodic constraint information from the speech signals, respectively. An optimization construction module is used to construct a joint constraint optimization problem by using the acoustic likelihood of the acoustic feature sequence as a hard constraint and the pronunciation consistency of the pronunciation constraint information and the prosodic structure of the prosodic constraint information as soft constraints. The solution module is used to solve the joint constraint optimization problem through constraint optimization to obtain the recognized text sequence; The verification and adjustment module is used to perform constraint satisfaction verification on the identified text sequence. When the constraint satisfaction is lower than a preset threshold, the soft constraint weights are adaptively adjusted according to the reliability of each constraint and the solution is re-solved. The steps of acquiring the speech signal and extracting the acoustic feature sequence, articulation constraint information, and prosodic constraint information from the speech signal include: The speech signal is subjected to framing and pre-emphasis processing to obtain a framed speech signal; Mel frequency cepstral coefficients are extracted from the framed speech signal to form an acoustic feature sequence; The acoustic feature sequence is input into a pre-trained acoustic-to-speech inversion model, which outputs a sequence of articulatory organ position parameters. The acoustic-to-speech inversion model takes the acoustic feature sequence as input and maps the acoustic features to the articulatory organ position parameter space through a deep neural network. The articulatory organ position parameters include tongue height, tongue anteroposterior position, lip rounding, and oral cavity opening. Calculate the positional change between adjacent frames in the articulatory organ position parameter sequence, and use the positional change as the articulation constraint information; Extract the fundamental frequency trajectory and short-time energy trajectory from the speech signal; Align the fundamental frequency trajectory and the short-time energy trajectory along the time axis, and calculate the fundamental frequency change rate and energy change rate for each frame; Frames whose absolute value of the fundamental frequency change rate is greater than a first preset threshold and whose absolute value of the energy change rate is greater than a second preset threshold are marked as prosodic boundary candidate frames. Merging is performed on adjacent prosodic boundary candidate frames to determine the prosodic boundary position; The speech signal is divided into prosodic unit sequences based on the prosodic boundary positions, and the prosodic unit sequences and their corresponding fundamental frequency trajectories and short-time energy trajectories constitute the prosodic constraint information.

[0014] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the voice recognition method of the voice assistant described in the first aspect.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the voice recognition method for a voice assistant described in the first aspect.

[0016] The beneficial effects of this invention compared to existing technologies are as follows: By introducing joint optimization of acoustic features, pronunciation constraint information, and prosodic constraint information, this embodiment can effectively reduce the impact of background noise and accent differences on recognition results. The acoustic likelihood of the acoustic feature sequence, as a hard constraint, provides a strong foundation, while pronunciation consistency and prosodic structure, as soft constraints, further enhance the system's ability to understand speech input, thereby significantly improving recognition accuracy. This embodiment, through joint optimization of information from different constraints, can maintain high recognition accuracy under various environmental conditions (such as noisy environments, different speakers, etc.). When faced with diverse speech inputs, the system can adaptively adjust the soft constraint weights, thereby enhancing its adaptability to pronunciation changes and prosodic features, and improving the robustness of the voice assistant in practical applications. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic flowchart of a speech recognition method for a voice assistant provided by the present invention is shown; Figure 2 This diagram illustrates a speech recognition device for a voice assistant according to an embodiment of the present invention. Figure 3 A schematic diagram of a terminal device provided in an embodiment of the present invention is shown. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0020] This invention provides a speech recognition method and system for a voice assistant to address the technical problem that factors such as environmental noise, the speaker's accent, and speech rate can significantly affect the quality of the speech signal, leading to a decrease in the accuracy of the recognition results.

[0021] First, this invention provides a speech recognition method for a voice assistant. Please refer to [link / reference]. Figure 1 , Figure 1A schematic flowchart of a speech recognition method for a voice assistant provided by the present invention is shown. Figure 1 As shown, the voice recognition method of this voice assistant may include the following steps: Step 101: Acquire a speech signal, and extract acoustic feature sequences, articulation constraint information, and prosodic constraint information from the speech signal respectively; After receiving the user's voice command, the first step is to convert the raw sound wave signal into three types of structured information that can be processed by subsequent algorithms. The input to this step is a single-channel speech waveform captured by the microphone, and the output is three parallel information channels: ① Acoustic feature sequence: a parameterized representation of the change of the spectral envelope of a speech signal over time, which is the basic information upon which traditional speech recognition systems rely to work; ② Pronunciation constraint information: depicts the movement trajectory of the user's speech organs such as tongue, lips, and jaw during pronunciation, and constrains the pronunciation feasibility of the recognition results from a physiological perspective; ③ Prosodic constraint information: describes the temporal structure of suprasegmental features such as fundamental frequency, energy, and pauses in speech signals, providing a basis for prosodic boundary localization in text.

[0022] These three types of information correspond to the three linguistic levels of acoustics, pronunciation, and prosody, respectively. This application jointly models these three layers of information under a unified constraint optimization framework, so that the recognition results are not only acoustically likely and pronunciation feasible, but also prosodically natural.

[0023] Specifically, step 101 includes steps A1 to A9: Step A1: Perform frame segmentation and pre-emphasis processing on the speech signal to obtain a framed speech signal; The raw speech signal is a variable-length time series, unsuitable for direct input into subsequent feature extraction modules. Frame segmentation divides the speech signal into short, overlapping analysis frames of fixed duration, typically 20 to 40 milliseconds in length, with a frame shift generally half the frame length. For example, a frame length of 25 milliseconds and a frame shift of 10 milliseconds provide approximately 60% overlap between adjacent frames, ensuring both quasi-stationarity of the signal within a frame (the physical motion of the vocal tract is approximately constant within 25 milliseconds) and sufficient temporal resolution to track rapid movements of the vocal organs.

[0024] The pre-emphasis processing aims to compensate for the 6dB / octave attenuation of high-frequency components of the speech signal during radiation due to the lip radiation effect. It is implemented using a first-order high-pass digital filter, with the difference equation y[n]=x[n]-α·x[n-1]; where x[n] is the original speech signal value at the nth sampling point, x[n-1] is the value at the previous sampling point, and α is the pre-emphasis coefficient. For example, α is set to 0.97, which provides approximately 6dB of boost in the frequency band above 4kHz, effectively equalizing the speech spectrum and bringing the energy of high-frequency consonants (such as / s / or / f / ) closer to that of mid-to-low-frequency vowels, preventing them from being submerged by noise in subsequent feature extraction.

[0025] As an optional embodiment of this application, the pre-emphasis processing can also employ an adaptive pre-emphasis method—dynamically adjusting the α value based on the signal spectrum slope of the current speech frame instead of taking a fixed value. Specifically, the spectral tilt of the current frame is estimated in the linear predictive analysis, and an appropriate pre-emphasis coefficient is derived based on the tilt, making the processed spectrum approximately flat across all frequency bands. Adaptive pre-emphasis can provide a more balanced spectral output in scenarios where there are significant differences in the speaker's vocal tract characteristics (such as different spectral slopes for male and female voices).

[0026] Step A2: Extract Mel frequency cepstral coefficients from the framed speech signal to form an acoustic feature sequence; Mel frequency cepstral coefficients are one of the most widely used acoustic features in the field of speech recognition. Their design simulates the nonlinear perception characteristics of the human auditory system at frequencies.

[0027] For each frame of speech signal, the extraction process is as follows: a Hamming window is added to the intra-frame signal to suppress spectral leakage; the power spectrum of the frame is calculated by fast Fourier transform; the power spectrum is passed through a set of triangular Mel filter banks, which are densely distributed in the low-frequency band and sparsely distributed in the high-frequency band to simulate the nonlinear frequency analytical characteristics of the basilar membrane of the human ear; the logarithm of the energy value output by each filter is taken; finally, the logarithmic energy vector of the filter bank is transformed into cepstral coefficients by discrete cosine transform, and the first few coefficients (usually 13-dimensional) are taken as the MFCC feature vector of the frame.

[0028] For example, this embodiment uses 13-dimensional MFCC features as the basic static acoustic feature vector. Furthermore, to capture the dynamic characteristics of the speech signal, first-order and second-order differences are calculated for the static MFCC, each supplemented with 13-dimensional features, ultimately resulting in a 39-dimensional acoustic feature vector. The first-order difference reflects the rate of spectral change, and the second-order difference reflects the acceleration of spectral change; together, they characterize the temporal dynamics of the acoustic features.

[0029] After processing all framed speech signals according to the above procedure, the acoustic feature sequence O=[o1,o2,...,o] is obtained. T], where T is the total number of frames, and each frame o_t∈R 39 This is a 39-dimensional real vector. This sequence will serve as the base input for subsequent acoustic and phonation models.

[0030] As an optional embodiment of this application, acoustic feature extraction can also use perceptual linear prediction coefficients instead of MFCC. PLP introduces equal loudness curve pre-emphasis and intensity-loudness power-law compression on the basis of Mel frequency scale, which is slightly more robust to noisy environments than MFCC, and is suitable for use in voice assistant usage scenarios with low signal-to-noise ratio (such as in-vehicle environments and open office areas).

[0031] Step A3: Input the acoustic feature sequence into the pre-trained acoustic-to-speech inversion model and output the articulatory organ position parameter sequence; wherein, the acoustic-to-speech inversion model takes the acoustic feature sequence as input and maps the acoustic features to the articulatory organ position parameter space through a deep neural network, and the articulatory organ position parameters include tongue height, tongue anteroposterior position, lip rounding and oral cavity opening degree; Acoustic-to-articulation inversion is a core problem in speech science, aiming to infer the unobservable trajectories of articulatory organs from observable acoustic signals. The acoustic-to-articulation inversion model used in this step is a feedforward deep neural network. The input is a 39-dimensional acoustic feature vector for each frame, and the output is a four-dimensional vector of articulatory organ position parameters for that frame: p_t=[tongue_height, tongue_frontness, lip_rounding, jaw_opening]∈[0,1]. 4 .

[0032] Tongue height describes the position of the tip or back of the tongue in the vertical direction of the oral cavity, ranging from 0 (low position, such as the tongue surface descending when pronouncing / a / ) to 1 (high position, such as the tongue surface rising when pronouncing / i / ). Tongue anterior / posterior describes the position of the tongue body in the anterior / posterior direction of the oral cavity, ranging from 0 (anterior position, such as the tongue surface protruding when pronouncing / i / ) to 1 (posterior position, such as the tongue root retracting when pronouncing / u / ). Lip roundness describes the degree of roundness of the lips, ranging from 0 (spread lips, such as the corners of the mouth spreading outwards when pronouncing / i / ) to 1 (rounded lips, such as the lips rounded when pronouncing / u / ). Oral cavity opening describes the distance between the upper and lower jaws, ranging from 0 (closed, such as the lips tightly closed when pronouncing plosives like / p / and / b / ) to 1 (fully open, such as the lower jaw descending when pronouncing / a / ).

[0033] This deep neural network is trained offline using supervised training on the motion trajectories of the articulatory organs and corresponding acoustic features acquired synchronously by a large-scale electromagnetic vocalization instrument or real-time magnetic resonance imaging. The training dataset contains speech materials read by multiple speakers with various phoneme combinations, and each frame is simultaneously labeled with acoustic features and articulatory organ position parameters. The network structure is a 5-layer fully connected network with 512 neurons in each hidden layer, using the ReLU activation function.

[0034] For the acoustic feature sequence O=[o1,o2,...,o T By inputting the model frame by frame, the sequence of articulatory organ position parameters P=[p1,p2,...,p] is obtained. T ], where p_t∈[0,1] 4 .

[0035] Step A4: Calculate the positional change between adjacent frames in the articulatory organ position parameter sequence, and use the positional change as the articulation constraint information; The speech organs possess physical inertia; the mass of organs such as the tongue and jaw prevents them from making large positional jumps in a very short time. This physiological constraint is reflected in the time series, meaning that the change in the position of the speech organs between adjacent frames should remain within a reasonable range. If the trajectory of the speech organs corresponding to a candidate text sequence exhibits a drastic jump (e.g., the tongue height suddenly jumps from 0.1 to 0.9 between two adjacent frames), then the candidate sequence is difficult to achieve in actual pronunciation and should be penalized.

[0036] This step calculates the positional changes between adjacent frames in the sequence of positional parameters of the articulatory organs, P. For each frame t, the Euclidean distance between its four-dimensional positional parameters and the previous frame t-1 is calculated, resulting in the frame-by-frame positional change sequence ΔP=[Δp2,Δp3,...,Δp...]. T The articulation constraint information is defined as the sequence of positional changes, together with the original articulatory organ position parameter sequence P itself, as complete information describing the user's articulation trajectory.

[0037] Step A5: Extract the fundamental frequency trajectory and short-time energy trajectory from the speech signal; The prosodic information of speech is mainly reflected in the fundamental frequency F0 and the pattern of energy change over time. The fundamental frequency is determined by the vibration frequency of the vocal cords and is the physical basis for perceiving intonation and stress; energy reflects the change of sound loudness over time. Together, they determine the prosodic structure of speech.

[0038] Fundamental frequency extraction employs the autocorrelation method. For each frame of the speech signal, the autocorrelation function is calculated. Within a reasonable search range of F0 (exemplarily 50Hz to 400Hz, covering the typical fundamental frequency ranges of male, female, and children's voices), the period corresponding to the autocorrelation peak is found, and the reciprocal of the period is taken as the estimated F0 value for that frame. Short-time energy extraction is performed by summing the squares of the signal values ​​at all sampling points within the frame to obtain the energy value of that frame.

[0039] After calculating F0 and short-time energy for all speech frames sequentially, the fundamental frequency trajectory F0=[F01,F02,...,F0] aligned with the acoustic feature sequence frames is obtained. T and the short-time energy trajectory E=[E1,E2,...,E T ].

[0040] Step A6: Align the fundamental frequency trajectory and the short-time energy trajectory along the time axis, and calculate the fundamental frequency change rate and energy change rate for each frame; The fundamental frequency trajectory and short-time energy trajectory are naturally aligned with the acoustic feature sequence frames in step A5 (both based on the same framing parameters), so alignment along the time axis is essentially naturally satisfied. For each frame t, the fundamental frequency change rate is defined as ΔF0_t = |F0_t - F0_{t-1}| / F0_{t-1}, and the energy change rate is defined as ΔE_t = |E_t - E_{t-1}| / E_{t-1}. The fundamental frequency change rate reflects the degree of change in tone or accent at that frame, and the energy change rate reflects the degree of abrupt change in loudness.

[0041] For the first frame of the sequence, since there is no preceding frame, its fundamental frequency change rate and energy change rate are both set to 0.

[0042] Step A7: Mark frames whose absolute value of the fundamental frequency change rate is greater than a first preset threshold and whose absolute value of the energy change rate is greater than a second preset threshold as prosodic boundary candidate frames; Prosodic boundaries are the dividing points between intonation phrases or prosodic words in speech. Acoustically, they are typically characterized by a fundamental frequency reset (the overall level of F0 returns to a lower value and begins to rise again) and a short-term energy decrease (the loudness naturally decays before a pause). This means that near prosodic boundaries, both the fundamental frequency change rate and the energy change rate may exhibit large absolute values ​​simultaneously.

[0043] This step iterates through all frames, checking for two conditions simultaneously for each frame: the absolute value of the fundamental frequency change rate ΔF0_t is greater than a first preset threshold, and the absolute value of the energy change rate ΔE_t is greater than a second preset threshold. For example, the first preset threshold is set to 0.08 (i.e., the relative change in fundamental frequency exceeds 8%), and the second preset threshold is set to 0.15 (i.e., the relative change in energy exceeds 15%). These thresholds are determined based on statistical analysis of prosodic boundary annotations from a large volume of standard Mandarin speech corpora—at the annotated prosodic boundary positions, the 95th percentile of the relative change in fundamental frequency is approximately 8%, and the 95th percentile of the relative change in energy is approximately 15%. Frames that simultaneously meet both conditions are marked as prosodic boundary candidate frames, indicating that this frame is highly likely to represent a prosodic boundary.

[0044] Step A8: Perform a merging process on adjacent prosodic boundary candidate frames to determine the prosodic boundary positions; Since acoustic changes near the prosodic boundary often span multiple consecutive frames (during fundamental frequency rise or fall, the rate of change in several consecutive frames may exceed the threshold), candidate frames marked in step A7 often have multiple consecutive frames marked simultaneously. This step merges adjacent candidate frames. If the time indices of multiple candidate frames are consecutive (frames t, t+1, t+2, ..., t+k are all marked), the frame with the largest absolute value of the fundamental frequency change rate among these frames is taken as the position of the merged prosodic boundary.

[0045] The merged data results in a list of prosodic boundary locations, arranged chronologically. Each prosodic boundary location corresponds to a frame index, indicating the presence of a prosodic segmentation point in the speech within that frame.

[0046] Step A9: Divide the speech signal into a prosodic unit sequence based on the prosodic boundary position, and construct the prosodic constraint information by combining the prosodic unit sequence and its corresponding fundamental frequency trajectory and short-time energy trajectory.

[0047] Using the prosodic boundary positions determined in step A8 as the segmentation points, the speech signal is divided into several non-overlapping prosodic units along the time axis. Each prosodic unit contains acoustic content within a continuous frame interval. Prosotically, the intonation and stress distribution within the unit are relatively coherent, while there are obvious transitions in intonation or loudness between units.

[0048] For each prosodic unit, its corresponding fundamental frequency trajectory segment and short-time energy trajectory segment are extracted, along with the unit's temporal boundary information (start frame index and end frame index), to form the prosodic description of that unit. The entire sequence of prosodic units and their associated information constitutes the prosodic constraint information. This information defines the speech segmentation scheme from a prosodic perspective. In subsequent recognition processes, if the prosodic boundaries of candidate text sequences do not conform to this, they will be considered prosodically incoherent recognition results and will be downweighted.

[0049] In the embodiments corresponding to steps A1 to A9, through the cascading processing of the above steps, a piece of original speech signal is decomposed into three parallel information channels. The acoustic feature sequence encodes spectral envelope information in the form of 39-dimensional MFCC feature vectors, the articulation constraint information describes the physiological trajectory of the articulation process in the form of articulator position parameters and their inter-frame variation, and the prosody constraint information characterizes intonation and pause-duration patterns in the form of a prosody unit segmentation scheme and related fundamental frequency / energy trajectories. The frame rates of these three channels are completely consistent, all based on the same framing parameters, which provides an input basis at three linguistic levels from acoustics, articulation to prosody for the multi-layer constraint joint optimization framework constructed in subsequent steps 102 to 105. All model parameters and thresholds are determined through large-scale corpus training and statistical analysis in the offline stage, and only forward calculation and threshold decision are required during online operation, without relying on online adaptive training.

[0050] Step 102: Taking the acoustic likelihood of the acoustic feature sequence as a hard constraint, and taking the articulation consistency of the articulation constraint information and the prosodic structure of the prosody constraint information as soft constraints, construct a joint constraint optimization problem; Speech recognition is essentially a search problem, which seeks the text that best matches the input speech signal in a huge text sequence space. Traditional speech recognition only relies on the single optimization target of acoustic likelihood, and is prone to making mistakes on candidate sequences that are acoustically likely but linguistically unreasonable. For example, misrecognizing "I want to go to Beijing today" as "I want to go to the background today", the two candidate sequences are both acoustically likely, but only the former is reasonable in the movement trajectory of articulators: / j / → / b / requires the tongue position to jump rapidly from the hard palate to the lips, which is physiologically difficult to achieve, and it is natural in prosody: the prosodic segmentation of "go / background" does not conform to the prosodic pattern of the commonly used verb-object structure in Chinese. The core idea of this step is to take the acoustic likelihood as the access threshold for entering the candidate set, that is, the hard constraint, and take the articulation consistency and prosodic structure as preference items in the optimization objective, that is, the soft constraints, to make a comprehensive decision under a unified mathematical optimization framework.

[0051] Specifically, step 102 specifically includes steps B1 to B7: Step B1: Input the acoustic feature sequence into an acoustic model, the acoustic model takes the acoustic feature sequence as input, and outputs the posterior probability distribution of each phoneme category in each frame through a deep neural network; The acoustic model is the core module in the speech recognition system that is responsible for mapping acoustic features to phonemes. The acoustic model adopted in this step is a deep bidirectional long short-term memory network, which takes the acoustic feature sequence O=[o1,o2,...,o output in step 101 as input T as input, and outputs a posterior probability vector y_t=[P(s1|o with dimension S for each frame t t), P(s2|o t ), ..., P(s s |o t ]], where S is the total number of phoneme categories. For Mandarin Chinese, including initials, finals, and silences, S is exemplarily taken as 68, P(s i |o t ) represents the acoustic features of the current frame. t This frame corresponds to the phoneme s i The posterior probability.

[0052] The temporal modeling capability of bidirectional LSTM allows the network to utilize contextual information from past and future frames simultaneously when estimating the posterior probability of each frame. The output layer of the network uses the Softmax activation function to ensure that all components of the output posterior probability vector are non-negative and sum to 1.

[0053] As an optional embodiment of this application, the acoustic model can also use a Transformer encoder structure based on a self-attention mechanism instead of a bidirectional LSTM. The Transformer's fully parallel self-attention mechanism is more advantageous than LSTM in handling long-distance phoneme dependencies, but the computational cost and memory usage are also correspondingly increased, making it suitable for use in server-side voice assistant solutions with sufficient computing power.

[0054] Step B2: Based on the posterior probability distribution, generate multiple candidate text sequences through decoding search. Each candidate text sequence is a sequence obtained by mapping phoneme sequences to text. After obtaining the posterior probability distribution of phonemes for each frame, a decoding search is needed to convert the frame-level phoneme probabilities into sequence-level text hypotheses. This step employs a decoding search framework based on a weighted finite-state converter, which pre-integrates knowledge from the acoustic model, pronunciation dictionary, and language model to form a compact search graph from phoneme units to text word sequences.

[0055] The decoding process unfolds in a frame-synchronized manner within the search graph, expanding along the legal arcs in the search graph from the current active state in each frame. Initialization begins from the initial state. For frame t, for each active search path, all legal phoneme transitions are considered, and the cumulative score after expansion is calculated by combining the posterior probability distribution of the current frame and the cost on the transition arc. During the search, the top K optimal candidate paths are retained in descending order of cumulative score, where K is a preset bundle width, exemplarily taken as K=200. After all frames are processed, for the final K retained candidate paths, the phoneme sequence corresponding to each path is mapped to a word sequence via a pronunciation dictionary, and then post-processed by a language model (such as Chinese word segmentation and word selection) to obtain a complete candidate text sequence. Finally, a candidate set consisting of K candidate text sequences is output.

[0056] Step B3: For each candidate text sequence, calculate the product of the posterior probabilities of each phoneme in its corresponding phoneme sequence as the acoustic likelihood value of the candidate text sequence. For each candidate text sequence generated in step B2, the sequence can be uniquely back-derived as the corresponding phoneme sequence, that is, the phoneme expansion of the candidate text in the pronunciation dictionary. Let the phoneme sequence corresponding to a certain candidate text sequence be π=[π1,π2,...,π]. T ], where π t Let L be the phoneme category corresponding to frame t. The acoustic likelihood of this candidate text sequence is defined as the product of the posterior probabilities of the corresponding phonemes in each frame: L = ∏_{t=1} {T} P(π t |o t ).

[0057] To avoid numerical underflow, since there's a low probability that consecutive multiplication might result in zero with floating-point precision, the log-likelihood value logL=∑_{t=1} is used in actual calculations. {T} log P(π t |o t The higher the acoustic likelihood value, the better the candidate text sequence matches the input speech acoustically.

[0058] Step B4: Eliminate candidate text sequences with acoustic likelihood values ​​lower than a preset likelihood threshold from the search space, and form a hard-constrained candidate set from the remaining candidate text sequences; Acoustic likelihood, as a hard constraint, means that candidate text sequences must reach a minimum quality threshold at the acoustic level. Otherwise, no matter how ideal their pronunciation consistency or prosodic structure is, they cannot be used as the final recognition result—because a text sequence that is completely acoustically mismatched cannot have any substantial correspondence with the input speech.

[0059] This step sets a preset likelihood threshold θ_L to exclude candidates from the K candidate text sequences generated in step B2 whose acoustic likelihood values ​​are lower than θ_L. For example, θ_L is the maximum acoustic likelihood value of all candidate sequences minus 15 dB (i.e., retaining all candidates whose likelihood values ​​are within 15 dB of the optimal candidate). This value is based on the N-best list retention strategy commonly used in speech recognition: the optimal candidate is usually reliable, while candidates that are more than 15 dB worse than the optimal candidate often contain a large number of erroneous syllable substitutions and are therefore unnecessary to retain.

[0060] The remaining candidate text sequences after elimination constitute a hard-constrained candidate set H = {h1, h2, ..., h...} M}, where M≤K is the number of candidates retained after hard constraint screening. The hard constraint candidate set is the feasible region for subsequent soft constraint optimization, and the final identification result must be selected from H.

[0061] Step B5: For each candidate text sequence in the hard constraint candidate set, calculate the pronunciation transfer cost between its corresponding articulatory organ position sequence and the pronunciation constraint information as a pronunciation consistency score; The pronunciation consistency score measures the degree of matching between the implicit articulatory organ movement trajectories of a candidate text sequence and the actual articulatory organ movement trajectories extracted in step 101. The calculation of this score involves a deeper articulation-acoustic mapping. This step provides the overall process, and the detailed calculations within it are carried out in subsequent steps B51 to B54.

[0062] Specifically, step B5 includes steps B51 to B54: Step B11: Based on the pre-stored phoneme-to-speech organ position mapping table, convert each phoneme in the candidate text sequence in the hard constraint candidate set into the corresponding expected speech organ position parameter to form the expected speech position sequence; The phoneme-to-articulation organ position mapping table is an offline lookup table that records the typical articulation organ position parameters corresponding to each phoneme (or more refined phoneme variants). This mapping table is built based on EMA data collection and statistical analysis from multiple standard Mandarin speakers: for each speaker's reading corpus, each frame of phoneme is labeled, and the corresponding articulation organ position parameters are extracted, including but not limited to tongue height, tongue posterior / posterior position, lip rounding, and oral cavity opening. The median of all samples for the same phoneme is taken as the representative value of that phoneme in the mapping table.

[0063] For example, sample values ​​for the phoneme-to-speech organ location mapping table are shown in Table 1 below: Table 1. Examples of phoneme-to-articulation organ location mappings The tongue height and mouth opening in this mapping table jointly reflect the degree of opening during articulation: the mouth opening for the vowel / a / is 0.92 and the tongue height is 0.95, indicating a fully open mouth and a low tongue position; the mouth opening for the plosive / p / is 0.02 and the tongue height is 0.15, indicating closed lips and blocked airflow. Lip rounding is 0.95 (highly rounded lips) at / u / and only 0.02 (fully spread lips) at / i / .

[0064] For each candidate text sequence h in the hard-constrained candidate set H, the phonemes corresponding to each frame of its corresponding phoneme sequence are looked up in the mapping table to obtain the expected articulatory organ position parameter vector q_t∈[0,1] for that frame. 4 After frame-by-frame search, the expected pronunciation position sequence Q(h) = [q1, q2, ..., q] is obtained. T ].

[0065] Step B12: Calculate the Euclidean distance between the expected articulation position sequence and the actual articulation organ position parameter sequence in the articulation constraint information on each frame to obtain the frame-by-frame position deviation sequence; For candidate h, its expected articulation position sequence Q(h) is compared with the actual articulation organ position parameter sequence P=[p1,p2,...,p] extracted in step 101. T Frame-by-frame alignment. Since Q(h) and P use the same frame rate (both based on the framing parameters of step A1), frame alignment is naturally achieved.

[0066] For frame t, calculate the four-dimensional Euclidean distance between q_t and p_t. The superscripts 1-4 correspond to the four dimensions of tongue height, tongue anteroposterior position, lip roundness, and oral cavity opening, respectively. The resulting frame-by-frame positional deviation sequence is D(h) = [d1, d2, ..., d...]. T ].

[0067] Step B13: Calculate the mean of the frame-by-frame positional deviation sequence as the pronunciation transfer cost; The pronunciation transfer cost C(h) is defined as the mean of the frame-by-frame positional deviation sequence: C(h) = (1 / T)·∑_{t=1} {T} d_t represents the average per-frame deviation between the expected articulatory organ trajectory and the actual trajectory of candidate h. The lower the cost, the closer the pronunciation of the candidate sequence matches the actual pronunciation.

[0068] Step B14: Use the complement of the pronunciation transfer cost as the pronunciation consistency score; wherein the pronunciation consistency score ranges from 0 to 1.

[0069] The articulation consistency score is defined as S_artic(h) = exp(-C(h)). This exponential transformation maps the articulation transfer cost to the interval (0,1): S_artic(h) = 1 when C(h) = 0 (perfect match); S_artic(h) approaches 0 when C(h) approaches infinity, but in reality, C(h) rarely exceeds 1 (each dimension of the articulatory organ position parameter is in the range [0,1], and the maximum Euclidean distance is about 2), so S_artic(h) is usually above 0.13.

[0070] In the embodiments corresponding to steps B11 to B14, through the cascaded processing of the above steps, the mapping of phonemes to articulatory organ positions, frame-by-frame Euclidean distance calculation, and exponential score transformation constitute a complete quantitative evaluation process for articulation consistency. The phoneme-to-articulatory organ position mapping table is established based on statistical analysis of large-scale EMA data, ensuring the physiological rationality of the mapping; the score after exponential transformation naturally falls into the 0 to 1 interval, consistent with the subsequent prosodic structure score dimension, facilitating weighted combination in a unified objective function.

[0071] Step B6: For each candidate text sequence in the hard constraint candidate set, calculate the overlap rate between its prosodic boundary position and the prosodic boundary position in the prosodic constraint information as the prosodic structure score. The prosodic structure score measures whether the implicit prosodic boundary segmentation scheme of the candidate text sequence is consistent with the actual prosodic boundary extracted in step 101. If the text content of the candidate sequence has a linguistically natural prosodic segmentation that is also the location of the acoustically detected prosodic boundary, then the candidate sequence is highly matched with the input speech at the prosodic level.

[0072] Specifically, step B6 includes steps B61 to B64: Step B21: Based on the pre-stored text-to-prosodic mapping rules, determine the expected prosodic boundary position of each word in the candidate text sequence in the hard constraint candidate set, and form the expected prosodic boundary set; The text-to-prosodic mapping rules are a set of rules based on Chinese linguistics, used to infer the prosodic boundary positions during natural reading from any sequence of Chinese text. This set of rules includes: inserting prosodic boundaries at punctuation marks (commas, periods, semicolons, question marks, etc.); inserting optional prosodic boundaries at subject-verb boundaries; inserting optional prosodic boundaries at long modifier-headword boundaries; and inserting optional prosodic boundaries between items in a parallel structure.

[0073] Taking candidate text h as an example, the system scans it word by word and records a frame-level number (the frame number of the starting frame after the word is expanded on the time axis) for each word position that meets the above rules, thus forming the expected prosodic boundary set B_exp(h).

[0074] Step B22: Calculate the number of intersections between the expected prosodic boundary set and the actual prosodic boundary positions in the prosodic constraint information; Let the set of actual prosodic boundary positions output in step 101 be B_real={b1,b2,...,b_R} (R actual prosodic boundary frame numbers), and the set of expected prosodic boundaries for candidate h be B_exp(h)={e1,e2,...,e_N} (N expected prosodic boundary frame numbers). Since the frame numbers of the actual and expected prosodic boundaries may not be exactly the same due to slight differences in frame alignment, this step introduces a tolerance window Δt during matching: as long as the frame number difference between an expected boundary frame e_i and an actual boundary frame b_j does not exceed Δt (for example, Δt is 2 frames, i.e., about 20 milliseconds of tolerance), it is considered a successful match.

[0075] After matching is completed, the number of successful matches is counted as N_match=|B_exp(h)∩_{Δt} B_real|, where ∩_{Δt} represents the approximate intersection within the tolerance Δt.

[0076] Step B23: The ratio of the number of intersections to the total number of boundaries in the expected prosodic boundary set is taken as the overlap rate; The overlap rate is defined as R_overlap(h) = N_match / N, where N = |B_exp(h)| is the total number of expected prosodic boundaries for candidate h. The overlap rate ranges from 0 to 1; the closer it is to 1, the more consistent the natural prosodic structure of the candidate text is with the prosodic segmentation of the actual speech.

[0077] Step B24: Use the overlap rate as the prosodic structure score.

[0078] The prosodic structure score is directly taken as the overlap rate value: S_pros(h) = R_overlap(h). Since the overlap rate is naturally in the range [0,1], no additional transformation is required.

[0079] In the embodiments corresponding to steps B21 to B24, the expected prosodic boundaries of candidate texts are inferred through text-to-prosodic mapping rules, approximate matching based on tolerance windows, and overlap rate calculation, thereby achieving a quantitative evaluation of the prosodic naturalness of candidate texts. The tolerance window mechanism avoids strict matching that is overly sensitive to frame segmentation accuracy, improving the robustness of prosodic evaluation in real-world scenarios.

[0080] Step B7: Using the hard-constrained candidate set as the feasible region, and the product of the pronunciation consistency score and the first weight coefficient plus the product of the prosodic structure score and the second weight coefficient as the objective function value, construct a joint constraint optimization problem that maximizes the objective function value within the feasible region.

[0081] The mathematical form of the joint constraint optimization problem is: find a candidate h* in the feasible region H (the set of hard constraint candidates) such that the objective function J(h) = w1·S_artic(h) + w2·S_pros(h) is maximized.

[0082] Wherein, w1 is the first weighting coefficient (pronunciation consistency weight), and w2 is the second weighting coefficient (prosodic structure weight). For example, both w1 and w2 are initially set to 0.5, meaning that it is initially assumed that the pronunciation constraints and prosodic constraints have equal reliability. These two weights will be adaptively adjusted in step 104 based on speech quality.

[0083] This optimization problem aims to select the best-performing sequence in terms of both pronunciation feasibility and prosodic naturalness from candidate text sequences that meet acoustic likelihood criteria. Hard constraints ensure that the result does not deviate from acoustic reality, while soft constraints guide the result towards optimization towards reasonable pronunciation and natural prosody.

[0084] In the embodiments corresponding to steps B1 to B7, a hierarchical constraint optimization framework is established through the cascaded processing of the above steps, with acoustic likelihood as the threshold and pronunciation consistency and prosodic structure as optimization objectives. The acoustic model uses a bidirectional LSTM deep neural network to provide frame-by-frame phoneme posterior probabilities, WFST decoding searches to generate multiple candidate hypotheses, and acoustic likelihood thresholding filters out a set of hard constraint candidates that meet the quality standards. Based on this, steps B51 to B54 quantify the pronunciation feasibility of each candidate, and steps B61 to B64 quantify its prosodic naturalness. The weighted sum of the two constitutes the objective function, providing a clear search objective for the subsequent optimization solution in step 103.

[0085] Step 103: Solve the joint constraint optimization problem through constraint optimization to obtain the recognized text sequence; Based on the hard-constrained candidate set H and the objective function J(h) = w1·S_artic(h) + w2·S_pros(h) constructed in step 102, this step is responsible for executing the actual optimization search process, finding the candidate text sequence in H that maximizes J(h) as the final output of speech recognition. Since the number of candidates in H is limited, typically not exceeding 200 after acoustic likelihood hard constraint screening, the global optimum can theoretically be found by traversing all candidates in H. However, in actual implementation, to seamlessly integrate with the decoding search process and avoid performing an independent traversal after decoding is complete, which would increase latency, this step adopts a method of directly embedding the soft constraint score into the frame-synchronized calculation during the decoding search process.

[0086] Specifically, step 103 includes steps C1 to C5: Step C1: Construct an objective function, which is the weighted sum of the pronunciation consistency score and the prosodic structure score for each candidate text sequence in the hard constraint candidate set; wherein, the first weight coefficient corresponding to the pronunciation consistency score and the second weight coefficient corresponding to the prosodic structure score are determined based on the prior reliability of each soft constraint; The objective function is defined as J(h) = w1·S_artic(h) + w2·S_pros(h). The initial values ​​of w1 and w2 are preset based on prior reliability—which reflects the reliability of pronunciation constraints or prosodic constraints in general usage scenarios. In general voice assistant scenarios, the movement of the vocal organs is strictly constrained by physiological laws, and pronunciation consistency information is relatively stable and reliable. However, prosodic boundary detection is greatly affected by noise, especially in noisy environments where the accuracy of fundamental frequency extraction decreases. Therefore, the initial value of w1 is slightly higher than w2. For example, w1 is set to 0.6, and w2 to 0.4. This ratio is determined based on an offline validation experiment containing 2000 speech samples with different noise levels—scanning the combination of w1∈[0.3,0.7] and w2=1-w1 with a step size of 0.05 on the validation set, the optimal performance with the lowest character error rate (8.7%) was achieved when w1=0.6.

[0087] Step C2: Perform a beam search in the search space corresponding to the hard constraint candidate set in a frame-synchronized manner; Beam search is a search strategy that progresses frame by frame in the state space, retaining the top K optimal candidate paths. The beam search in this step is conceptually similar to the decoding search in step B2, but the key difference is that the search objective in step B2 is only acoustic likelihood, while the search objective in this step integrates articulation consistency score and prosodic structure score.

[0088] The search starts from the acoustic posterior probability distribution generated in steps B1-B4 and processes each frame in a frame-synchronized manner. Due to the constraints of the hard-constrained candidate set H, the acoustic likelihood is not lower than the threshold, and paths with excessively low acoustic likelihood are automatically pruned during the search.

[0089] Step C3: In each frame, expand all legal phoneme transfers for each candidate path currently retained, update the pronunciation consistency score and prosodic structure score of the candidate path according to the expanded phoneme sequence, and calculate the cumulative objective function value; sort all expanded candidate paths in descending order of cumulative objective function value, retain the top K candidate paths as surviving paths, and prune the remaining candidate paths; where K is the preset bundle width; The processing flow for frame t is as follows: For each surviving path (at most K) retained from the previous frame, consider all legal phoneme transitions in the pronunciation dictionary. After selecting a legal phoneme for each path in frame t, extend the phoneme sequence of that path forward one frame. At this point, recalculate the pronunciation consistency score and prosodic structure score for the latest phoneme sequence of that path. Weight these two scores according to the weights in step C1, and sum them with the objective function value accumulated by that path in previous frames to obtain the cumulative objective function value up to frame t.

[0090] For all candidate paths expanded in frame t, they are sorted in descending order of cumulative objective function value. The top K paths are retained as surviving paths to enter the next frame, and the remaining paths are pruned and discarded. For example, K is set to 150, which strikes a balance between search accuracy and computational efficiency. Experimental statistics on 2000 test voice samples show that when K=150, the probability of missing the global optimum is about 2.7%, while when K=100, this probability rises to 5.3%. The further improvement (1.8%) when K=200 is not significant, and the search time increases by about 30%.

[0091] Step C4: When all frames have been processed, select the path with the largest cumulative objective function value from the surviving candidate paths as the optimal path, and obtain the phoneme sequence corresponding to the optimal path by backtracking; After all T frames have been processed, the set of surviving paths containing no more than K candidate paths is the optimal candidate set after comprehensive evaluation of all frames. The path with the largest cumulative objective function value J(h) is selected as the globally optimal path.

[0092] By using the frame-level phoneme records maintained within the optimal path, backtracking from frame 1 to frame T, the decoded phonemes of each frame are restored sequentially, resulting in the optimal phoneme sequence π*=[π*1,π*2,...,π*_T].

[0093] Step C5: Map the phoneme sequence to a text sequence, which is then used as the recognized text sequence.

[0094] The optimal phoneme sequence π* is derived by reverse lookup in a pronunciation dictionary, restoring consecutive phoneme combinations to their corresponding word units. For Mandarin Chinese, the phonemes in π* are first divided into syllable sequences according to syllable boundary rules (such as the combination pattern of initial consonant + final vowel), and then these syllable sequences are converted into Chinese character sequences using a Chinese syllable-character mapping table. For polyphonic characters, the language model determines the most likely Chinese character based on the context.

[0095] The final output sequence of recognized text is the text content that the voice assistant recognizes based on the input speech.

[0096] In the embodiments corresponding to steps C1 to C5, the beam search directly embeds the articulation consistency score and prosodic structure score into the frame-by-frame search process, avoiding the two-stage pipeline of performing pure acoustic decoding first and then independent soft constraint evaluation, thus reducing end-to-end latency. The soft constraint score is updated frame-by-frame synchronously during the search process, allowing articulation and prosodic information to take effect early in the search. For example, if the expected articulator position of a candidate path deviates significantly from its actual position in the early stages, the articulation consistency score of that path will be continuously suppressed, leading to its early elimination in beam pruning and freeing up beam width resources for more reasonable candidate paths. The computational complexity of the entire solution process mainly depends on the beam width K and the total number of frames T. Under a typical configuration (K=150, T approximately 500 frames, i.e., 5 seconds of speech), the time taken for a single solution is less than 50 milliseconds, meeting the real-time requirements of voice assistants.

[0097] Step 104: Perform constraint satisfaction verification on the identified text sequence. When the constraint satisfaction is lower than the preset threshold, adaptively adjust the soft constraint weights according to the reliability of each constraint and solve the problem again. Speech recognition systems face various acoustic environments in practical deployments, ranging from quiet indoor spaces to noisy streets. Under these different environments, the reliability of the three types of constraint information exhibits systematic differences: in high signal-to-noise ratio (SNR) environments, acoustic features are clear and prosodic parameters are accurately extracted; in low SNR environments, noise superposition can lead to MFCC feature distortion, while the error rate of the fundamental frequency extraction algorithm increases significantly, and the fundamental frequency is easily misled by the randomness of noise. However, the articulation constraint information, due to its reliance on the noise-insensitive structure of the acoustic-to-articulation inversion model (ATI model), and the inclusion of noise-enhanced data in offline training, actually shows increased relative reliability.

[0098] When the initial recognition results are not ideal, instead of adjusting the acoustic model or decoder parameters, we simply reallocate the priority weights of the two soft constraints and use the more reliable constraint information in the current environment to correct the bias caused by the less reliable information.

[0099] Specifically, step 104 includes steps D1 to D5: Step D1: The weighted sum of the acoustic likelihood value, pronunciation consistency score, and prosodic structure score of the identified text sequence is used as the constraint satisfaction. The constraint satisfaction Q is a comprehensive indicator that measures the performance of the output of step 103 across various dimensions. Q is defined as: Q = α·L_norm + β·S_artic* + γ·S_pros*. Here, L_norm is the normalized acoustic likelihood value (mapping the log-likelihood value calculated in step B3 to the [0,1] interval), S_artic* is the articulation consistency score of the optimal path, and S_pros* is the prosodic structure score of the optimal path. For example, α, β, and γ are all taken as 1 / 3, meaning the three constraint dimensions are treated equally.

[0100] Step D2: When the constraint satisfaction is lower than the preset threshold, calculate the signal-to-noise ratio estimate of the speech signal; The preset threshold θ_Q is the threshold value that triggers weight adjustment. For example, θ_Q is set to 0.55. This value is determined based on offline experiments on 1000 speech samples containing different noise levels. When Q is lower than 0.55, the median character error rate (CER) of the recognition results exceeds 15%, indicating that the initial recognition results are unreliable and it is necessary to trigger weight reallocation for a second solution.

[0101] When Q < θ_Q, calculate the signal-to-noise ratio (SNR) estimate of the current speech signal, SNR_est. The SNR estimation uses a speech activity detection-based method. First, all frames of the speech signal are divided into speech frames and silence frames. The average energy E_speech of the speech frames and the average energy E_silence of the silence frames are calculated separately. The SNR estimate is SNR_est = 10·log 10 ((E_speech-E_silence) / E_silence), in dB.

[0102] Step D3: When the estimated signal-to-noise ratio is lower than the preset signal-to-noise ratio threshold, the weight of the pronunciation consistency constraint is reduced by a preset percentage, and the weight of the prosodic structure constraint is increased by the preset percentage. In low signal-to-noise ratio (SNR_est < θ_SNR) environments, the accuracy of fundamental frequency extraction and energy trajectory calculation decreases significantly, and the reliability of prosodic constraint information is reduced. In such cases, the weight of prosodic constraints should be weakened to avoid misguidance. However, since the acoustic-to-phonological inversion model has already undergone noise enhancement during the training phase, and additive noise of different SNR levels is intentionally added to the training set, its output under low SNR conditions remains basically reliable, and the general trend of the articulatory organ position parameters does not undergo a fundamental reversal under noise interference.

[0103] Therefore, under low signal-to-noise ratio (SNR) conditions, the strategy is to increase the weight of the articulation consistency constraint to leverage its noise robustness, and decrease the weight of the prosodic constraint to reduce its unreliability. For example, the preset SNR threshold θ_SNR is 12dB, and the preset adjustment ratio Δ is 0.15. After adjustment: w1' = w1 + Δ, w2' = w2 - Δ. With initial values ​​of w1 = 0.6 and w2 = 0.4, after adjustment, w1' = 0.75 and w2' = 0.25.

[0104] Step D4: When the estimated signal-to-noise ratio is higher than the preset signal-to-noise ratio threshold, the weight of the pronunciation consistency constraint is increased by the preset ratio, and the weight of the prosodic structure constraint is decreased by the preset ratio. In high signal-to-noise ratio (SNR) environments, fundamental frequency extraction and energy calculation are reliable, and prosodic constraint information is effective. Furthermore, speech in high SNR environments typically originates from relatively ideal usage scenarios, and the user's pronunciation is generally standard—in which case the position parameters of the articulatory organs should also be close to standard values. However, fundamental frequency and energy provide valuable intonation information in this scenario and should be given higher weight. Meanwhile, although the acoustic-to-articulation inversion model has high mapping accuracy in quiet scenarios, prosodic information has unique value in eliminating homophonic ambiguity in quiet environments.

[0105] For example, after adjustment, w1'=0.5 and w2'=0.5 are distributed proportionally.

[0106] Step D5: Solve the joint constraint optimization problem again using the adjusted weights.

[0107] Replace the original weights with the adjusted weights (w1', w2') obtained in step D3 or D4, and re-execute the bundle search solution process in step 103. The recognized text sequence after the second solution is the final output after adaptive weight adjustment.

[0108] In the embodiments corresponding to steps D1 to D5, constraint satisfaction verification and adaptive weight adjustment constitute a self-correcting feedback path. Compared with a single solution with fixed weights, the mechanism in this embodiment automatically allocates preferences between pronunciation constraints and prosodic constraints based on the actual quality of the input speech. In noisy environments, it leverages the noise robustness of the pronunciation model to ensure quality control, while in quiet environments, it utilizes the disambiguation capabilities of prosodic information. The total time for the initial and secondary solutions does not exceed 100 milliseconds, and the additional latency increment is within an acceptable range for the user experience of the voice assistant.

[0109] Step 105: Based on the pre-stored phoneme-to-articulation organ position mapping table, convert each phoneme in the identified text sequence into a corresponding expected articulation organ position parameter; calculate the positional deviation between the expected articulation organ position parameter and the actual articulation organ position parameter of the corresponding frame in the articulation constraint information; mark phonemes with positional deviations greater than a preset deviation threshold as abnormal phonemes; when the abnormal phoneme is in a non-critical position in the identified text sequence, delete the abnormal phoneme from the identified text sequence; when the abnormal phoneme has a missing position in the identified text sequence, search for the phoneme with the smallest positional deviation in the phoneme-to-articulation organ position mapping table based on the articulation organ position parameters of adjacent phonemes to complete the missing position. While the adaptive weight adjustment in step 104 can optimize the recognition results at the macro level, it may still leave individual phoneme-level errors at the micro level, especially those phonemes whose acoustic likelihood and pronunciation consistency are both moderate and that have barely passed the constraint verification under the current weight combination. This step performs a phoneme-by-phoneme post-processing based on pronunciation physiological consistency on the final recognized text sequence after step 104, which is equivalent to adding a pronunciation-level review in the last stage of recognition.

[0110] Specifically, step 105 includes steps E1 to E5: Step E1: Based on the pre-stored phoneme-to-speech organ position mapping table, convert each phoneme in the identified text sequence into the corresponding expected speech organ position parameter; The same phoneme-to-articulation organ position mapping table as in step B51 is used. For the phoneme sequence π*=[π*1,π*2,...,π*_T] corresponding to the identified text sequence, the four-dimensional expected articulation organ position parameters of each phoneme in the mapping table are searched frame by frame to obtain the expected sequence Q_full=[q1,q2,...,q_T].

[0111] Step E2: Calculate the positional deviation between the expected articulatory organ position parameters and the actual articulatory organ position parameters of the corresponding frame in the articulation constraint information; For each frame t, calculate the four-dimensional Euclidean distance between q_t and the actual vocal organ position parameter p_t output in step 101. The physical meaning of this distance is the difference between the standard pronunciation posture of the phoneme corresponding to that frame in the recognition result and the actual observed pronunciation posture. When d*_t is zero, it indicates a perfect match; the larger d*_t is, the less the phoneme recognized in that frame matches the phoneme actually pronounced by the user in terms of pronunciation posture.

[0112] Step E3: Mark phonemes with positional deviations greater than a preset deviation threshold as abnormal phonemes; A preset deviation threshold θ_d is used to determine whether the positional deviation is abnormal. For example, θ_d is set to 0.35. This value is based on the statistical results of the fluctuation of the actual articulatory organ position of each phoneme in normal speech around the standard value of the mapping table. In a large number of normal speech samples, the 95th percentile of the positional deviation is about 0.32. A value slightly higher than 0.35 is set to leave a moderate fluctuation space, while avoiding oversensitivity to real pronunciation variations.

[0113] Phonemes with a positional deviation d*_t > θ_d are marked as anomalous phonemes, indicating that the phoneme's articulation posture in the current frame is significantly different from the expected one, and there is reason to suspect that the phoneme has been misidentified.

[0114] Step E4: When the abnormal phoneme is in a non-critical position in the recognized text sequence, the abnormal phoneme is deleted from the recognized text sequence; wherein, non-critical positions include unstressed syllables at the end of words, filler syllables, and ambiguous syllables near prosodic boundaries; For each anomalous phoneme marked in step E3, the system determines whether the phoneme is in a critical position in the identified text sequence. A critical position refers to a location where deleting the phoneme would change the meaning or grammatical structure of the word, such as the stem syllable of a core verb, the syllable of a negation word, or the central syllable of a key noun. Non-critical positions include: unstressed syllables at the end of words, filler syllables, and ambiguous syllables near prosodic boundaries.

[0115] Abnormal phonemes identified as being in non-critical positions are directly removed from the recognized text sequence. The removal operation also requires updating the corresponding time alignment information.

[0116] Step E5: When the abnormal phoneme has a missing position in the recognized text sequence, search for the phoneme with the smallest position deviation in the phoneme-to-speech organ position mapping table according to the position parameters of the adjacent phonemes to complete the missing position. For certain anomalous phonemes, although their pronunciation deviates from expectations, their position in the text sequence is crucial; deleting this position would lead to grammatical fragmentation or semantic loss in the sentence. For these phonemes missing from the text sequence, the system attempts to complete them through pronunciation constraints.

[0117] The completion method is as follows: Take the articulatory organ position parameters p_{t-1} and p_{t+1} of the normal phonemes in the frames immediately before and after the missing position, and calculate their mean p_mid = (p_{t-1} + p_{t+1}) / 2, which serves as the interpolation estimate of the articulatory posture at the missing position. Then, traverse all phonemes in the phoneme-to-articulatory organ position mapping table, calculate the four-dimensional Euclidean distance between the standard articulatory organ position parameter of each phoneme and p_mid, and select the phoneme with the smallest distance as the completion phoneme for the missing position. After replacing the original abnormal phoneme with the completion phoneme, the corresponding position in the identified text sequence is repaired.

[0118] In the embodiments corresponding to steps E1 to E5, through the cascaded processing of the above steps, a further round of fine post-processing is performed on the second solution result after weight adjustment. The threshold detection of the position deviation of the articulatory organs acts as a physical examination mechanism for erroneous phonemes. The deletion of non-critical abnormal phonemes avoids the interference of redundant or erroneous syllables on the fluency of reading. The interpolation and completion of critical missing phonemes relies on the continuity of the articulation posture of adjacent phonemes (the movement of the articulatory organs is smooth, and the posture of adjacent frames should not change drastically) to infer the correct phoneme. Unlike error correction strategies that rely solely on acoustic posterior probability, the error detection and repair in this embodiment are based on the articulation physiological dimension that is completely independent of the acoustic model, and have the complementary advantage of being less susceptible to noise interference.

[0119] To verify the technical effectiveness of this embodiment, a comparative experiment was conducted on a test dataset containing 5000 real-world voice assistant usage scenarios. The test set covered voice samples with different signal-to-noise ratio levels (5dB to 30dB), different speakers (50 males and 50 females), different speech rates (slow, normal, and fast, each accounting for approximately one-third), and different command types (weather query, navigation control, device control, information search, etc.). The experiment compared three methods: ① the pure acoustic likelihood method, which only uses the acoustic model decoding in steps B1-B2, without introducing articulation and prosodic constraints; ② the acoustic + articulation constraint method, which introduces soft articulation consistency constraints but not prosodic constraints; and ③ the complete method of this embodiment, which combines hard acoustic constraints, soft articulation constraints, soft prosodic constraints, adaptive weights, and post-processing for articulation. The evaluation metric was the character error rate, i.e., the percentage of the total number of characters with insertion, deletion, and replacement errors in the identified text relative to the total number of characters in the standard answer. The experimental results are shown in Table 2 below.

[0120] For example, Table 2 shows a comparison of character error rates for different methods under different signal-to-noise ratio conditions: Table 2 Comparison of character error rates for different methods The experimental results show that, under all signal-to-noise ratio (SNR) conditions, the complete method of this embodiment achieved the lowest character error rate, with the advantage becoming more significant at lower SNRs. Under the extreme low SNR condition of 5dB, the method of this embodiment reduced the character error rate from 42.3% to 28.1% compared to the purely acoustic method, and also reduced it by approximately 7.6 percentage points compared to the method that only introduces pronunciation constraints. Under high SNR conditions, the method of this embodiment further reduced the error rate from 8.1% to 4.3%, demonstrating the added value of prosodic information in eliminating homonyms in quiet environments.

[0121] In the ablation experiment, after disabling adaptive weight adjustment, the CER under 15dB conditions rebounded from 9.5% to 11.8%; after removing the post-pronunciation processing in step 105 with adaptive adjustment already disabled, the CER rebounded to 13.5%. Adaptive weight adjustment is particularly crucial for improving low to medium signal-to-noise ratio scenarios, while post-pronunciation processing provides a stable gain of approximately 2 percentage points for recognition results under all signal-to-noise ratio conditions.

[0122] In the embodiments corresponding to steps 101 to 104, by introducing joint optimization of acoustic features, pronunciation constraint information, and prosodic constraint information, this embodiment can effectively reduce the impact of background noise and accent differences on the recognition results. The acoustic likelihood of the acoustic feature sequence, as a hard constraint, provides a strong foundation, while pronunciation consistency and prosodic structure, as soft constraints, further enhance the system's ability to understand speech input, thereby significantly improving recognition accuracy. This embodiment, through joint optimization of information from different constraints, can maintain high recognition accuracy under various environmental conditions (such as noisy environments, different speakers, etc.). When faced with diverse speech inputs, the system can adaptively adjust the soft constraint weights, thereby enhancing its adaptability to pronunciation changes and prosodic features, and improving the robustness of the voice assistant in practical applications.

[0123] like Figure 2 This invention provides a voice recognition device for a voice assistant; please refer to [link / reference]. Figure 2 , Figure 2 A schematic diagram of a voice recognition device for a voice assistant provided by the present invention is shown, such as... Figure 2 The voice recognition device for a voice assistant shown includes: The constraint extraction module 21 is used to acquire a speech signal and extract acoustic feature sequences, pronunciation constraint information and prosodic constraint information from the speech signal, respectively. The optimization construction module 22 is used to construct a joint constraint optimization problem by using the acoustic likelihood of the acoustic feature sequence as a hard constraint and the pronunciation consistency of the pronunciation constraint information and the prosodic structure of the prosodic constraint information as soft constraints. Solving module 23 is used to solve the joint constraint optimization problem through constraint optimization to obtain the recognized text sequence; The verification and adjustment module 24 is used to perform constraint satisfaction verification on the identified text sequence. When the constraint satisfaction is lower than a preset threshold, the soft constraint weights are adaptively adjusted according to the reliability of each constraint and the solution is recalculated.

[0124] This invention provides a speech recognition device for a voice assistant. By introducing joint optimization of acoustic features, pronunciation constraints, and prosodic constraints, this embodiment effectively reduces the impact of background noise and accent differences on the recognition results. The acoustic likelihood of the acoustic feature sequence, as a hard constraint, provides a strong foundation, while pronunciation consistency and prosodic structure, as soft constraints, further enhance the system's ability to understand speech input, thereby significantly improving recognition accuracy. This embodiment, through joint optimization of information from different constraints, can maintain high recognition accuracy under various environmental conditions (such as noisy environments, different speakers, etc.). When faced with diverse speech inputs, the system can adaptively adjust the weights of the soft constraints, thereby enhancing its adaptability to pronunciation changes and prosodic features, and improving the robustness of the voice assistant in practical applications.

[0125] Figure 3 This is a schematic diagram of a terminal device provided in an embodiment of the present invention. Figure 3 As shown, a terminal device 3 in this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a voice recognition program for a voice assistant. When the processor 30 executes the computer program 32, it implements the steps in the various embodiments of the voice recognition method for a voice assistant described above, for example... Figure 1 Steps 101 to 104 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each unit in the above-described device embodiments, for example... Figure 2 The function of the unit shown.

[0126] For example, the computer program 32 can be divided into one or more units, which are stored in the memory 31 and executed by the processor 30 to complete the present invention. The one or more units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 32 in the terminal device 3. For example, the specific functions of each unit of the computer program 32 can be divided as follows: The constraint extraction module is used to acquire speech signals and extract acoustic feature sequences, articulation constraint information, and prosodic constraint information from the speech signals, respectively. An optimization construction module is used to construct a joint constraint optimization problem by using the acoustic likelihood of the acoustic feature sequence as a hard constraint and the pronunciation consistency of the pronunciation constraint information and the prosodic structure of the prosodic constraint information as soft constraints. The solution module is used to solve the joint constraint optimization problem through constraint optimization to obtain the recognized text sequence; The verification and adjustment module is used to perform constraint satisfaction verification on the identified text sequence. When the constraint satisfaction is lower than a preset threshold, the soft constraint weights are adaptively adjusted according to the reliability of each constraint and the solution is re-solved.

[0127] The terminal device includes, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of a terminal device 3 and does not constitute a limitation on a terminal device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0128] The processor 30 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0129] The memory 31 can be an internal storage unit of the terminal device 3, such as a hard disk or memory of the terminal device 3. The memory 31 can also be an external storage device of the terminal device 3, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 3. Furthermore, the memory 31 can include both internal and external storage units of the terminal device 3. The memory 31 is used to store the computer program and other programs and data required by the roaming control device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0130] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0131] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0133] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0134] This invention provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0136] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0138] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units.

[0140] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0141] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0142] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0143] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0144] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0145] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech recognition method for a voice assistant, characterized in that, The voice recognition method of the voice assistant includes: Acquire a speech signal, and extract acoustic feature sequences, articulation constraint information, and prosodic constraint information from the speech signal, respectively; Using the acoustic likelihood of the acoustic feature sequence as a hard constraint, and the phonetic consistency of the phonetic constraint information and the prosodic structure of the prosodic constraint information as soft constraints, a joint constraint optimization problem is constructed. The joint constraint optimization problem is solved by constraint optimization to obtain the recognized text sequence; The constraint satisfaction of the identified text sequence is verified. When the constraint satisfaction is lower than a preset threshold, the soft constraint weights are adaptively adjusted according to the reliability of each constraint and the solution is re-solved. The steps of acquiring the speech signal and extracting the acoustic feature sequence, articulation constraint information, and prosodic constraint information from the speech signal include: The speech signal is subjected to framing and pre-emphasis processing to obtain a framed speech signal; Mel frequency cepstral coefficients are extracted from the framed speech signal to form an acoustic feature sequence; The acoustic feature sequence is input into a pre-trained acoustic-to-speech inversion model, which outputs a sequence of articulatory organ position parameters. The acoustic-to-speech inversion model takes the acoustic feature sequence as input and maps the acoustic features to the articulatory organ position parameter space through a deep neural network. The articulatory organ position parameters include tongue height, tongue anteroposterior position, lip rounding, and oral cavity opening. Calculate the positional change between adjacent frames in the articulatory organ position parameter sequence, and use the positional change as the articulation constraint information; Extract the fundamental frequency trajectory and short-time energy trajectory from the speech signal; Align the fundamental frequency trajectory and the short-time energy trajectory along the time axis, and calculate the fundamental frequency change rate and energy change rate for each frame; Frames whose absolute value of the fundamental frequency change rate is greater than a first preset threshold and whose absolute value of the energy change rate is greater than a second preset threshold are marked as prosodic boundary candidate frames. Merging is performed on adjacent prosodic boundary candidate frames to determine the prosodic boundary position; The speech signal is divided into prosodic unit sequences based on the prosodic boundary positions, and the prosodic unit sequences and their corresponding fundamental frequency trajectories and short-time energy trajectories constitute the prosodic constraint information. The step of performing constraint satisfaction verification on the identified text sequence, and adaptively adjusting the soft constraint weights based on the reliability of each constraint and resolving the problem when the constraint satisfaction is lower than a preset threshold, includes: The weighted sum of the acoustic likelihood value, pronunciation consistency score, and prosodic structure score of the identified text sequence is used as the constraint satisfaction. When the constraint satisfaction is lower than the preset threshold, the signal-to-noise ratio estimate of the speech signal is calculated; When the estimated signal-to-noise ratio is lower than the preset signal-to-noise ratio threshold, the weight of the pronunciation consistency constraint is reduced by a preset percentage, and the weight of the prosodic structure constraint is increased by the preset percentage. When the estimated signal-to-noise ratio is higher than the preset signal-to-noise ratio threshold, the weight of the pronunciation consistency constraint is increased by the preset proportion, and the weight of the prosodic structure constraint is decreased by the preset proportion. The joint constraint optimization problem is solved again using the adjusted weights.

2. The voice recognition method for a voice assistant as described in claim 1, characterized in that, The steps of constructing a joint constraint optimization problem, using the acoustic likelihood of the acoustic feature sequence as a hard constraint and the phonetic consistency of the phonetic constraint information and the prosodic structure of the prosodic constraint information as soft constraints, include: The acoustic feature sequence is input into the acoustic model, which takes the acoustic feature sequence as input and outputs the posterior probability distribution of each phoneme category in each frame through a deep neural network. Based on the posterior probability distribution, multiple candidate text sequences are generated through decoding search, and each candidate text sequence is a sequence obtained by mapping phoneme sequences to text. For each candidate text sequence, the product of the posterior probabilities of each phoneme in its corresponding phoneme sequence is calculated as the acoustic likelihood value of the candidate text sequence. Candidate text sequences with acoustic likelihood values ​​lower than a preset likelihood threshold are excluded from the search space, and the remaining candidate text sequences form a hard-constrained candidate set. For each candidate text sequence in the hard constraint candidate set, the pronunciation transfer cost between its corresponding articulatory organ position sequence and the pronunciation constraint information is calculated as the pronunciation consistency score. For each candidate text sequence in the hard constraint candidate set, the overlap rate between its prosodic boundary position and the prosodic boundary position in the prosodic constraint information is calculated as the prosodic structure score. Using the hard-constrained candidate set as the feasible region, and the product of the pronunciation consistency score and the first weight coefficient plus the product of the prosodic structure score and the second weight coefficient as the objective function value, a joint constraint optimization problem is constructed to maximize the objective function value within the feasible region.

3. The voice recognition method for a voice assistant as described in claim 2, characterized in that, The step of calculating the pronunciation transfer cost between the corresponding articulatory organ position sequence and the pronunciation constraint information as a pronunciation consistency score for each candidate text sequence in the hard constraint candidate set includes: Based on the pre-stored phoneme-to-articulation organ position mapping table, each phoneme in the candidate text sequence in the hard constraint candidate set is converted into the corresponding expected articulation organ position parameter to form the expected articulation position sequence; Calculate the Euclidean distance between the expected articulation position sequence and the actual articulation organ position parameter sequence in the articulation constraint information on each frame to obtain the frame-by-frame position deviation sequence; The mean of the frame-by-frame positional deviation sequence is calculated as the pronunciation transfer cost; The complement of the pronunciation transfer cost is used as the pronunciation consistency score; wherein the pronunciation consistency score ranges from 0 to 1.

4. The voice recognition method for a voice assistant as described in claim 2, characterized in that, The step of calculating the overlap rate between the prosodic boundary position and the prosodic boundary position in the prosodic constraint information for each candidate text sequence in the hard constraint candidate set as the prosodic structure score includes: Based on the pre-stored text-to-prosodic mapping rules, the expected prosodic boundary position of each word in the candidate text sequence in the hard constraint candidate set is determined, thus forming the expected prosodic boundary set; Calculate the number of intersections between the expected prosodic boundary set and the actual prosodic boundary positions in the prosodic constraint information; The overlap rate is defined as the ratio of the number of intersections to the total number of boundaries in the expected prosodic boundary set. The overlap rate is used as the prosodic structure score.

5. The voice recognition method for a voice assistant as described in claim 1, characterized in that, After the step of performing constraint satisfaction verification on the identified text sequence, and adaptively adjusting the soft constraint weights according to the reliability of each constraint and resolving when the constraint satisfaction is lower than a preset threshold, the method further includes: Based on the pre-stored phoneme-to-speech organ position mapping table, each phoneme in the identified text sequence is converted into the corresponding expected speech organ position parameter; Calculate the positional deviation between the expected articulatory organ position parameters and the actual articulatory organ position parameters of the corresponding frame in the articulation constraint information; Phonemes with a positional deviation greater than a preset deviation threshold are marked as abnormal phonemes; When the aberrant phoneme is in a non-critical position in the recognized text sequence, the aberrant phoneme is deleted from the recognized text sequence; wherein, non-critical positions include unstressed syllables at the end of words, filler syllables, and ambiguous syllables near prosodic boundaries; When the abnormal phoneme has a missing position in the recognized text sequence, the phoneme with the smallest positional deviation is searched in the phoneme-to-speech organ position mapping table according to the position parameters of the articulatory organs of the adjacent phonemes to complete the missing position.

6. The speech recognition method for a voice assistant as described in claim 2, characterized in that, The step of solving the joint constraint optimization problem through constraint optimization to obtain the recognized text sequence includes: Construct an objective function, which is the weighted sum of the pronunciation consistency score and the prosodic structure score for each candidate text sequence in the hard constraint candidate set; wherein the first weight coefficient corresponding to the pronunciation consistency score and the second weight coefficient corresponding to the prosodic structure score are determined based on the prior reliability of each soft constraint; Perform a beam search in the search space corresponding to the hard constraint candidate set in a frame-synchronized manner; In each frame, expand all legal phoneme transfers for each candidate path currently retained, update the pronunciation consistency score and prosodic structure score of the candidate path based on the expanded phoneme sequence, and calculate the cumulative objective function value; All expanded candidate paths are sorted in descending order of cumulative objective function value, and the top K candidate paths are retained as surviving paths, while the remaining candidate paths are pruned; where K is the preset bundle width. When all frames have been processed, the path with the largest cumulative objective function value is selected from the candidate survival paths as the optimal path. The phoneme sequence corresponding to the optimal path is obtained by backtracking, and the phoneme sequence is mapped to a text sequence as the recognition text sequence.

7. A voice recognition device for a voice assistant, characterized in that, The voice recognition device of the voice assistant includes: The constraint extraction module is used to acquire speech signals and extract acoustic feature sequences, articulation constraint information, and prosodic constraint information from the speech signals, respectively. An optimization construction module is used to construct a joint constraint optimization problem by using the acoustic likelihood of the acoustic feature sequence as a hard constraint and the pronunciation consistency of the pronunciation constraint information and the prosodic structure of the prosodic constraint information as soft constraints. The solution module is used to solve the joint constraint optimization problem through constraint optimization to obtain the recognized text sequence; The verification and adjustment module is used to perform constraint satisfaction verification on the identified text sequence. When the constraint satisfaction is lower than a preset threshold, the soft constraint weights are adaptively adjusted according to the reliability of each constraint and the solution is re-solved. The steps of acquiring the speech signal and extracting the acoustic feature sequence, articulation constraint information, and prosodic constraint information from the speech signal include: The speech signal is subjected to framing and pre-emphasis processing to obtain a framed speech signal; Mel frequency cepstral coefficients are extracted from the framed speech signal to form an acoustic feature sequence; The acoustic feature sequence is input into a pre-trained acoustic-to-speech inversion model, which outputs a sequence of articulatory organ position parameters. The acoustic-to-speech inversion model takes the acoustic feature sequence as input and maps the acoustic features to the articulatory organ position parameter space through a deep neural network. The articulatory organ position parameters include tongue height, tongue anteroposterior position, lip rounding, and oral cavity opening. Calculate the positional change between adjacent frames in the articulatory organ position parameter sequence, and use the positional change as the articulation constraint information; Extract the fundamental frequency trajectory and short-time energy trajectory from the speech signal; Align the fundamental frequency trajectory and the short-time energy trajectory along the time axis, and calculate the fundamental frequency change rate and energy change rate for each frame; Frames whose absolute value of the fundamental frequency change rate is greater than a first preset threshold and whose absolute value of the energy change rate is greater than a second preset threshold are marked as prosodic boundary candidate frames. Merging is performed on adjacent prosodic boundary candidate frames to determine the prosodic boundary position; The speech signal is divided into prosodic unit sequences based on the prosodic boundary positions, and the prosodic unit sequences and their corresponding fundamental frequency trajectories and short-time energy trajectories constitute the prosodic constraint information. The step of performing constraint satisfaction verification on the identified text sequence, and adaptively adjusting the soft constraint weights based on the reliability of each constraint and resolving the problem when the constraint satisfaction is lower than a preset threshold, includes: The weighted sum of the acoustic likelihood value, pronunciation consistency score, and prosodic structure score of the identified text sequence is used as the constraint satisfaction. When the constraint satisfaction is lower than the preset threshold, the signal-to-noise ratio estimate of the speech signal is calculated; When the estimated signal-to-noise ratio is lower than the preset signal-to-noise ratio threshold, the weight of the pronunciation consistency constraint is reduced by a preset percentage, and the weight of the prosodic structure constraint is increased by the preset percentage. When the estimated signal-to-noise ratio is higher than the preset signal-to-noise ratio threshold, the weight of the pronunciation consistency constraint is increased by the preset proportion, and the weight of the prosodic structure constraint is decreased by the preset proportion. The joint constraint optimization problem is solved again using the adjusted weights.

8. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and a voice recognition program for a voice assistant stored in the memory and executable on the processor, the voice recognition program for the voice assistant being configured to implement the steps in the voice recognition method of the voice assistant as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the speech recognition method of the voice assistant as described in any one of claims 1 to 6.

Images