A method for processing speech signals of educational robots

Abstract

The present invention discloses a method for processing speech signals for educational robots, which relates to the field of speech signal processing technology. To address the problems of multiple people speaking in the classroom and environmental noise interference, the method relies on a microphone array and a camera to acquire multi-channel data. In step one, an environmental model is constructed to determine the noise baseline and student distribution. In step two, overlapping speech is detected and the sound source is localized using arrival time difference and lip shape data. In step three, directional gain is applied to the target direction and a deep network is used to separate the overlapping speech. In step four, the separated speech is input into a child-customized recognition engine and combined with confidence assessment to achieve high-precision recognition and interaction. This method can significantly improve recognition accuracy and interaction efficiency in complex scenarios with classroom reverberation and multiple people speaking simultaneously. At the same time, a global environmental model is used to track noise changes, enabling the educational robot to maintain stable recognition performance in diverse teaching interactions, significantly enhancing teaching effectiveness.

Description

Technical Field

[0001] The present invention relates to the technical field of voice signal processing, and in particular to a voice signal processing method for an educational robot. Background Art

[0002] In modern education, more and more schools are experimenting with using educational robots to assist in classroom instruction, reducing teacher workload and stimulating student learning interest. However, real classrooms often present diverse interactive needs: for example, multiple students answering questions simultaneously during group discussions, teachers communicating with students from different locations, and even remote communication in large lecture halls. The classroom noise environment is highly complex, encompassing not only the sounds of traffic and hallway noise outside, but also the whirring of desks and chairs, the ringing of the class bell, and spontaneous student conversations. Furthermore, the classroom's spatial structure can cause sound waves to reflect multiple times off the walls and ceiling, creating significant reverberation. These combined factors pose significant challenges to real-time voice interaction. Especially in large classes with active and vocal students, if one or more robots cannot accurately recognize the target student's voice, they will struggle to answer questions or provide learning feedback. To address this, multimodal perception technology based on microphone arrays, cameras, and deep learning algorithms has emerged. It aims to extract valid voice signals from extremely noisy and multi-person environments, ensuring that the robot's responses are timely and accurate, meeting the high standards of voice interaction in real-world classrooms.

[0003] A search revealed a Chinese invention patent application with publication number CN112863538A, which provides a multimodal speech separation method and apparatus based on an audio-visual network. The method is used to separate the speech of a speaker in a scene from the audio and video to be separated. The method comprises the following steps: processing first audio and video training data using a first preprocessing method to obtain first preprocessed data; constructing a multimodal network model; inputting the first preprocessed data into the multimodal network model for training to obtain an audio-video alignment judgment model; processing second audio and video training data using a second preprocessing method to obtain second preprocessed data; constructing a Wave-U-Net segmentation model and combining it with the audio-video alignment judgment model to form an audio-visual model; inputting the second preprocessed data into the audio-visual model for training to obtain an audio-visual segmentation model; and inputting the audio and video to be separated into the audio-visual segmentation model to obtain the speech of the speaker in the scene. Subsets of the second preprocessed data are then fed into the audio-visual model for training in ascending order of speaker number.

[0004] However, based on the above application scenarios and the existing technologies retrieved:

[0005] Existing technologies still face significant bottlenecks when dealing with these high-speech and high-noise environments, particularly inadequate processing of overlapping speech. When two or more students ask questions simultaneously or consecutively, and their distances and directions are dispersed throughout the classroom, simple directional microphones or traditional speech recognition methods often fail to quickly track and isolate the correct speech signals.

[0006] On the one hand, the frequent and irregular noise sources in the classroom can invalidate the originally stable background noise model, resulting in a significant decrease in the accuracy of voice activity detection and beamforming algorithms. On the other hand, even if certain classroom reflection characteristics or common noise spectra are pre-modeled, it is difficult to maintain accurate separation when students suddenly engage in group discussions or answer in unison. The resulting consequences include: the target student's speech is severely drowned out or aliased, making it impossible for the robot to recognize the actual content of the question; when multiple people speak, the system may frequently miss or misdetect speakers, disrupting the fluency of teaching; and during multiple rounds of interaction, students and teachers will significantly reduce their interest in using the robot due to low recognition accuracy or interaction delays. In summary, how to effectively identify interference from multiple sound sources in crowded classroom environments and ensure that the target student's speech can be accurately extracted and recognized in real time is a technical challenge that urgently needs to be overcome in the field of educational robot voice interaction.

[0007] To this end, the present invention provides a method for processing speech signals of an educational robot. Summary of the Invention

[0008] (1) Technical problems solved

[0009] To address the shortcomings of existing technologies, the present invention provides a method for processing speech signals for educational robots. In step 1, an environmental model is constructed to determine the noise baseline and student distribution. In step 2, overlapping speech is detected and localized using arrival time differences and lip shape data. In step 3, directional gain is applied to the target direction and a deep network is used to separate the overlapping speech. In step 4, the separated speech is input into a customized recognition engine for children, combined with confidence assessment to achieve high-precision recognition and interaction. This method significantly improves recognition accuracy and interaction efficiency in complex classroom scenarios with reverberation and multiple people speaking simultaneously. Furthermore, a global environmental model is used to track noise changes, enabling the educational robot to maintain stable recognition performance during diverse teaching interactions. This method thus addresses the technical issues discussed in the background art.

[0010] (2) Technical solution

[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for processing speech signals of an educational robot, comprising: when entering a classroom and needing to initialize the current space environment, enabling a microphone array and a camera to collect multi-channel audio and video signals, and performing noise energy measurement and student distribution modeling to obtain a global environment model;

[0012] When multi-channel speech is detected with a cross peak and it is suspected that multiple people are speaking at the same time, overlapping speech detection is called and the sound source is localized based on the arrival time difference combined with the lip shape data, and the sound source direction information and student posture mapping are output;

[0013] When the sound source direction information is obtained and the target speech needs to be enhanced, the beamforming vector is called to perform gain processing on the corresponding directional signal and the deep separation network is used to remove background noise to produce the separated target speech;

[0014] When the separated target speech is input into the child speech customized recognition engine and a sound source recognition result is generated, the recognition reliability is determined based on the confidence evaluation and the noise mismatch information is recorded in the global environment model. If the result is low, multi-source correction or repeated inquiry is triggered, and the recognition result is obtained for real-time teaching interaction and continuous feedback.

[0015] Using directional information and clean speech, we extract multi-scale voiceprint vectors and store them in additional fields of the global environment model. Using a projection function, we improve the subsequent recognition accuracy of the student and fine-tune the acoustic model.

[0016] Preferably, a camera is used to capture and cache images of various locations inside the classroom, and the original time domain signal corresponding to each microphone is obtained, and a short-time Fourier transform is performed to obtain a multi-channel speech in complex form;

[0017] After introducing the weighted cross-power spectrum function to measure the joint energy characteristics between different microphone channels at discrete angular frequencies, the full-band weighted cross-power spectrum set and time-aligned camera image information are output.

[0018] Preferably, energy clustering analysis is performed on the full-band weighted cross-power spectrum set in frequency bands to distinguish stable noise, transient noise and potential speech peak areas, and the direction of the suspected noise source in the classroom is preliminarily marked based on the clustering results;

[0019] The student gathering areas and suspicious noise locations in the video frames are mapped to obtain the student distribution map and noise direction map in the classroom plane coordinate system; the global environment model is formed by integrating weighted cross-power spectrum analysis, video labeling information and classroom geometric dimensions.

[0020] Preferably, multi-channel speech is collected in real time within a time segment to obtain spectrum data of each microphone in the current time window; an overlapping speech index function is introduced to measure whether there are multiple speakers concurrently in the time window;

[0021] If the overlapping speech index function exceeds a preset overlapping threshold, it is determined that at least two or more speaking sources are active simultaneously in the current time window, and the overlapping speech period is identified and the subsequent sound source direction localization in step 2 is triggered.

[0022] Preferably, in the time interval determined to be the overlapping speech, direction estimation is performed on the multi-channel signal, and an arrival delay mapping function is calculated to obtain an arrival delay estimate;

[0023] Based on the geometric layout of the microphone array and the results of all arrival delay estimates, the direction vectors of multiple sound sources are analyzed through triangulation or specific beamforming, and then corrected based on the reflection characteristics. The peaks in different directions are mapped to different sound source IDs, and a list of target directions is output.

[0024] Preferably, the camera image is used to detect the student's lip shape or upper body posture to obtain the image coordinates of multiple potential speakers;

[0025] The neighboring metric is calculated after transforming each image coordinate into the same reference coordinate system as the target direction based on the previously calibrated in-camera microphone array geometry in the global environment model.

[0026] If the proximity metric exceeds a preset proximity threshold, it is determined that the student image coordinates and the sound source direction come from the same speaker;

[0027] For the successfully matched sound source direction, a label consistent with the student ID detected in the image is assigned to form a final dynamically updateable sound source-student binding list; after aggregation, sound source localization information is generated, including the sound source direction, the corresponding student ID and its spatial coordinates.

[0028] Preferably, the target direction of each active speaker is obtained from the sound source localization information, and a beam weight vector for the target direction is constructed at each center frequency;

[0029] When there are multiple active sound sources, a set of beam weight vectors is calculated in parallel for each sound source, allowing the robot to simultaneously amplify or suppress voice signals from different directions. The beam weight vectors are combined with multi-channel voice to generate a voice signal stream with directional gain.

[0030] Preferably, the speech signal stream after directional gain is fed into a multi-channel deep speech separation network, and the purified target speech is output;

[0031] Based on the student identity mapping or lip matching information, the separation quality can be evaluated and the stream with the highest confidence can be selected as the main separated target speech.

[0032] Preferably, a fusion metric function is defined to achieve synchronous analysis of the separated target speech and camera lip timing in two dimensions: time and frequency:

[0033] If the fusion metric function reaches or exceeds the threshold, the separated speech stream is identified as the separated target speech, and the corresponding time domain waveform is generated for speech recognition. The fusion metric function and lip synchronization information are then fed back for online correction. If the fusion metric function continues to be low, the input features or beam parameters of the multi-channel deep speech separation network are updated.

[0034] Preferably, a pre-trained children's speech customized recognition engine is called to perform recognition for each target source:

[0035] If it is found that multiple target sources meet certain audio and video fusion metrics or teacher instructions, the recognition process will be performed on the separated target speech sets one by one in parallel and the transcribed sound source recognition results will be temporarily stored. If it is detected that the sound source recognition results do not match the global environment model or classroom scene, an update prompt will be issued immediately.

[0036] Preferably, the text transcription in each sound source recognition result is evaluated for confidence using the internal score and audio matching output by the recognition engine and a comprehensive confidence function. If the comprehensive confidence function is lower than the confidence threshold, the recognition result is determined to be unreliable and a correction strategy is triggered:

[0037] Call other retained candidate sources and verify whether there is mis-separation or source ID confusion by comparing lip sync or context; if the suspected error is still not clearly corrected, prompt the student or teacher to briefly repeat the current question or speech to obtain a new voice segment and re-perform recognition;

[0038] If the comprehensive confidence function is not lower than the confidence threshold, the sound source recognition result will be marked as passed. If corrections or repeated inquiries are made, new voice data will be generated and recognized again. If it still does not meet the standards, it may be reported to the teacher interface for manual intervention.

[0039] Preferably, the final sound source recognition result determined by the confidence level is input into the dialogue management module, which parses the question command or feedback tag to generate the robot's answer or action instruction;

[0040] Record new information in the global environment model about key conversation content and current noise conditions;

[0041] If it is detected that a student asks questions multiple times in a row or speaks frequently, the targeted tracking of the student's voice can be strengthened in the next period based on pre-defined priority rules and resource allocation strategies.

[0042] Preferably, after recording the original speech waveform, the target direction and the separated target speech are used to locate and extract the clean speech segment of the student within the required time period; if there is still background interference, the multi-channel deep speech separation network is called to further de-alias the original speech waveform, and the clean speech segment is subjected to multi-scale time-frequency analysis, and the 2D wavelet transform is introduced to construct the following mapping to obtain the voiceprint vector.

[0043] Preferably, the voiceprint vector is written into the additional recording area of ​​the global environment model to provide fast correction and introduce matching for registered students in the recognition engine;

[0044] When the matching degree is greater than expected, it is determined that the student's voiceprint matching degree is high, and the recognition engine model parameters are preferentially retrieved, or the recognition engine is partially fine-tuned.

[0045] (3) Beneficial effects

[0046] The present invention provides a method for processing speech signals of an educational robot, which has the following beneficial effects:

[0047] Utilizing the global environment model Throughout all steps, we ensure that the speech separation of noise energy measured from environment initialization and multimodal data acquisition is fully utilized. As a result, the robot can not only accurately detect the period when multiple students speak at the same time, but also accurately detect the period when multiple students speak at the same time based on the beam weight vector. The deep separation network performs directional gain and de-aliasing on each target sound source, outputting the separated target speech. This synergistic mechanism of multimodal data and deep learning significantly reduces the speech error rate in noisy classrooms and ensures the system's timely response to the target speaker.

[0048] Adaptive error correction and comprehensive confidence function are proposed , dynamically corrects unstable recognition results, and allows backtracking to step 3 to call the retained separation flow or prompt the student to repeat the speech, thus forming an evolvable closed-loop interaction. In this way, the robot can still maintain good recognition results in extremely high noise levels or in the presence of children with diverse accents, and write the latest dialogue or noise information back to the global environment model. In order to accumulate more abundant data support for subsequent lessons;

[0049] By collecting the original speech waveform , and extract the voiceprint vector based on the two-dimensional wavelet transform operator W2D , and then use the projection function and recognition, the target speech after current separation The comparison function Real-time matching allows for rapid identification of specific students' speech, even in extreme scenarios with multiple people overlapping. Local parameter adjustments within the recognition engine are then made to achieve even better transcription accuracy. This collaboration allows registered students to repeatedly benefit from efficient recognition, gradually building a personalized database and continuously improving overall system performance.

[0050] In summary, the technical features of each step complement each other: environmental modeling and directional positioning ensure accurate front-end acquisition, directional beamforming and depth separation enable precise separation of multiple sound sources, child-customized ASR and adaptive dialogue management form a closed-loop error correction mechanism, and personalized voiceprint registration provides exclusive optimization for specific student interactions. This layered approach and the advanced integration of multimodal data demonstrate the solution's creativity and synergistic benefits in noise reduction, multi-person interaction, customized recognition, and dynamic updates, enabling educational robots to achieve greater adaptability and high-quality interactive performance in noisy classrooms. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of the method for processing speech signals of an educational robot according to the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0053] See also Figure 1 The present invention provides a method for processing speech signals of an educational robot, comprising:

[0054] Step 1: When the background noise and student distribution need to be initialized and analyzed, based on the microphone signal The multimodal feature fusion is used with the camera image to measure the possible reflective surface position and noise energy and extract the student group coordinate information. At the same time, the acquired multi-channel data is subjected to short-time Fourier transform to estimate the energy distribution within the typical bandwidth, and the noise baseline is stored in the environment description field to generate a global environment model. ;

[0055] The step 1 includes the following:

[0056] Step 101: Multi-channel signal acquisition and weighted cross-power spectrum generation

[0057] The original audio and video signals are collected simultaneously using several microphones and cameras, and the spatial and frequency characteristics of multi-channel audio are measured through the weighted cross-power spectrum function, providing a quantitative basis for the subsequent construction of the environmental model.

[0058] In the time interval Internal for each microphone Get the corresponding original time domain signal , perform short-time Fourier transform on each channel signal to obtain multi-channel speech in complex form :

[0059]

[0060] in represents the discrete angular frequency;

[0061] To measure the difference between different microphone channels at discrete angular frequencies The joint energy characteristics at , introduce the weighted cross power spectrum function , used to quantify the Hedi Frequency between channels The instantaneous coupling at , the larger the value, the stronger the coherence between the two in this frequency band:

[0062]

[0063] in, Indicates different microphone channels, * indicates complex conjugate operation;

[0064] is the normalization coefficient, which is used to constrain the overall amplitude scale and standardize the overall amplitude of the original cross-power spectrum to improve the comparability of the algorithm under different environments, different time windows or different signal-to-noise ratio conditions; is the time attenuation factor, which gives more weight to the data at the most recent moment to highlight the characteristics of the current environment and attenuate past interference;

[0065] Use cameras to capture and cache images from various locations inside the classroom to identify the approximate distribution and posture of students. Here, only the original images and time-series markers are recorded;

[0066] For the Road Microphone at the Moment The original time domain sampling of , the value range is determined by the hardware sampling bit width;

[0067] For the Road Microphone at the Moment and angular frequency The complex amplitude on , which can be positive or negative, depends on the strength of the speech signal; is the time decay factor, which can be ,The specific value can be adjusted according to the length of the speech window and the size of the classroom;

[0068] Finally, the full-band weighted cross-power spectrum set is output , and time-aligned camera image information;

[0069] When in use, by defining the cross-power spectrum with time decay terms, the effective information at the current moment is highlighted. Compared with the traditional method of simple accumulation or averaging, it is more sensitive to transient noise and rapid sound source changes. The video frames are recorded at the same timestamp and can be combined with the cross-power spectrum analysis results to achieve a consistent mapping from audio to video coordinates, laying the foundation for the subsequent construction of higher-precision environmental models.

[0070] Step 102: Environment model generation and multimodal fusion labeling

[0071] According to the obtained full-band weighted cross-power spectrum set Generate a global environment model based on the image information captured by the camera , and uniformly mark possible noise source blocks, wall reflection areas, and student distribution locations; among which:

[0072] The weighted cross power spectrum of the entire frequency band is set Perform energy cluster analysis by frequency band to distinguish stable noise, transient noise, and potential speech peak areas. Based on the clustering results, make preliminary markings on the direction of suspected noise sources in the classroom.

[0073] Combining the camera calibration parameters with the actual position of the microphone array, the student gathering area and the suspicious noise location in the video frame are mapped to obtain the student distribution map and noise direction map in the classroom plane coordinate system, achieving video and audio coordinate alignment;

[0074] Combining weighted cross-power spectrum analysis, video tagging information, and classroom geometry to form a global environment model :

[0075]

[0076] Where: Used to record the coordinates of the students' approximate location and the number of people identified; Used to indicate frequency Next, a certain location The spatial spectrum distribution function of the noise energy relative to the microphone array; Used to describe the reflection characteristics of sound waves by walls or objects here;

[0077] When used, the student distribution, noise energy and reflection coefficient are packaged into a unified global environment model It provides a high-dimensional benchmark for subsequent multi-speaker detection, beamforming, speech separation and other links. With the help of the clustering results of the weighted cross-power spectrum, it can locate the problem area in both the frequency domain and the spatial domain, and creatively integrate traditional audio analysis with classroom coordinate mapping, which facilitates the precise calling of subsequent steps (especially sound source direction estimation).

[0078] Step 2: When the global environment model When the noise baseline is obtained and the multi-channel speech in the current time window is detected to have significant cross energy, the overlapping speech detection is triggered and the arrival delay estimation is calculated. And combine the camera mouth shape or posture image for multi-source positioning, while referring to the noise energy in the environment description To reduce multi-source confusion and suppress known interference, and finally infer the target direction of each active sound source , and then generate sound source localization information;

[0079] The second step includes the following:

[0080] Step 201: Overlapping speech detection

[0081] In time segments For multi-channel speech Perform real-time acquisition to obtain the channel of each microphone in the current time window Spectrum data of

[0082] Introduce the following overlapping speech index function To measure whether there are multiple speakers concurrently within the time window, where:

[0083]

[0084] Where: For in time , angular frequency The multi-channel complex vector collected by the microphone array is defined as: ,in, Indicates the The complex frequency domain amplitude of the microphone, that is, multi-channel speech;

[0085] express The Hermitian conjugate of (that is, transpose and take the complex conjugate), so is a The inter-channel autocorrelation matrix can characterize the energy coupling between different microphone signals;

[0086] Same dimension The weighted matrix, whose elements are determined by the global environment model The decision includes mapping of information such as microphone position and noise energy distribution. Typically, the diagonal elements can be Set as a single channel gain coefficient, non-diagonal element The noise energy spatial spectrum distribution function and Combined to highlight or suppress coherent features in certain directions;

[0087] It is the Hadamard product, that is, element-by-element multiplication, ensuring that each channel at the matrix level Apply different environmental weights; is the Frobenius norm, The matrix A is defined as:

[0088]

[0089] here, , further amplifying the energy coupling when there is overlapping speech between multiple channels;

[0090] is the joint attenuation factor of time and frequency, represents the global adjustment coefficient, It is the dynamic attenuation factor of the environmental noise, which can be determined by the noise dynamics of the previous environmental model or the sudden interference information detected in the latest frame, and is used to attenuate the intervals with historical or excessively strong noise frequency bands;

[0091] , are the upper and lower bounds of the frequency for speech analysis, Indicates integration over the specified bandwidth;

[0092] The overlapping threshold is determined as follows: If the overlapping speech index function Exceeding a pre-set overlap threshold , determining that there are at least two or more speaking sources active simultaneously in this time window, identifying it as an overlapping speech period, and triggering the subsequent sound source direction localization unit in step 2;

[0093] When using, with the help of noise energy spatial spectrum distribution function The existing environmental noise distribution information in the system can dynamically attenuate the frequency band with large fixed noise when weighting, thereby reducing the false alarm rate. After detecting the specific time window of overlapping speech, the time window label (with overlap or no overlap) and the current time will be added. The information is transmitted to the subsequent direction positioning unit (step 202) so that the unit can perform multi-sound source direction estimation only when the overlapping state is confirmed, thereby reducing useless calculations and improving real-time performance.

[0094] Step 202: Multiple sound source direction estimation

[0095] In step 201, the time interval of overlapping speech is determined, and direction estimation is performed on the multi-channel signal. First, the following arrival delay mapping function is calculated to obtain the arrival delay estimation :

[0096]

[0097] in, For the With the Channels at time The arrival time delay corresponding to the maximum cross-correlation is estimated, thereby knowing the relative arrival time difference of the sound source relative to the two microphones; Indicates the The complex frequency domain amplitude of the microphone, that is, multi-channel speech; is the angular frequency; is the frequency domain phase compensation factor, represents an imaginary unit;

[0098] According to the microphone array geometry (given by the global environment model Provided coordinates), combining all arrival delay estimates The direction vectors of multiple sound sources are analyzed by triangulation or specific beamforming. If the three-dimensional angle needs to be estimated, the reflection characteristics are further combined. Make corrections to avoid false directions caused by highly reflective areas;

[0099] Correspond the peaks in different directions to different sound source IDs and output the target direction list , where each target direction Corresponding to the approximate position of a speaker, which may include polar coordinates or Cartesian coordinates, indicating the The direction or coordinate vector of a sound source in the classroom environment coordinate system can be expressed as or etc., depending on the array arrangement; the target direction list output in step 202 It is directly passed to step 203 for matching with the student's posture information in the camera image;

[0100] When used, it can be combined with the frequency domain phase compensation factor Frequency domain scanning enables direction estimation to maintain high resolution when there are multiple sources and some frequency bands overlap, and to fuse reflection features. The information can effectively reduce the false peak interference caused by reflections from walls or obstacles. This feature is relatively missing in ordinary pure microphone array algorithms.

[0101] Step 203: Camera image matching and direction calibration

[0102] In the time window parallel to steps 201 and 202, the camera image is used to detect the student's lip shape or upper body posture to obtain the image coordinates of multiple potential speakers. , For the camera The detected coordinates of the student's upper body or mouth position are determined by the image resolution (e.g. );

[0103] According to the global environment model The geometric position of the microphone array in the previously calibrated camera is used to convert each image coordinate into Transform to the target direction Same reference coordinate system, then calculate the adjacent metrics :

[0104]

[0105] in, To convert the image coordinates The vector mapped to the world coordinate system;

[0106] Represents vector distance; is the mapping convergence factor, which is used to penalize pairings with large positioning differences;

[0107] If the adjacent metric If the adjacent threshold is exceeded, the student image coordinates are determined. Direction of sound source From the same speaker; for the successfully matched sound source direction, give it a label consistent with the student ID detected in the image ,form the final sound source-student binding list so that the corresponding sound sources can be enhanced and separated preferentially in the subsequent steps;

[0108] Step 203 finally generates the sound source localization information, including: the direction of the sound source, the corresponding student ID and its spatial coordinates.

[0109] If a new student is detected entering the screen later, the coordinate mapping and matching can be re-executed, and the sound source-student binding list of this step can be dynamically updated to ensure accurate tracking in complex scenarios where multiple students speak frequently.

[0110] When used, by adjacent metric The audio direction estimation result is bidirectionally verified with the visual detection result, which can effectively eliminate interference such as no sound source or people who are not speaking. After completing the accurate projection of the image plane to the environment coordinates, the target direction is matched with the target direction. A one-to-one correspondence is established to improve the accuracy of identifying the real speaker in a noisy classroom. The visually recognized student ID is assigned the direction of the sound source. Subsequent steps can perform fine noise reduction or separation on a specific student's sound source, further improving the contextual continuity of speech recognition.

[0111] Step 3: Receive sound source positioning information and lock the target direction When the beam weight vector is called Directional gain is performed on multi-channel speech in the corresponding direction and background noise is effectively suppressed to improve the signal-to-noise ratio. A multi-channel deep speech separation network is introduced. Time-frequency signal characteristics after gain Perform multi-source demixing to generate separated target speech Combined with camera lip detection, it prioritizes enhancement and outputs a clean voice stream;

[0112] The step three includes the following:

[0113] Step 301: Beamforming initialization and environment adaptive directional gain

[0114] Get the target direction of each active speaker from the sound source localization information output in step 2 ,in Indicates the number of multiple identified sound sources, at each center frequency Build in the target direction The beam weight vector , which can be calculated using the following formula:

[0115]

[0116] Where: Represents the spatial covariance matrix of the microphone array, or the multi-channel autocorrelation information accumulated from the previous time windows, with a size of ,in is the number of microphones;

[0117] is the direction vector The corresponding steering vector can be derived based on a typical array manifold model using parameters such as microphone array geometry and sound velocity;

[0118] is a regularization term used to suppress gain divergence when the noise is too high or the target direction is unclear; : Regularization coefficient, usually a small positive number, used to avoid numerical instability when the matrix is ​​inverted. is an identity matrix with the same dimensions ;

[0119] When there are multiple active sound sources, Compute a set of beam weight vectors in parallel , so that the robot can simultaneously amplify or suppress speech signals from different directions; beam weight vector Multiply the multi-channel signal of the microphone to generate a speech signal stream after directional gain ;

[0120] When used, the direction positioning result produced in the previous step 2 is combined for precise pointing, which significantly improves the signal strength of the target voice and suppresses external interference. The inverse of can remain stable when the noise energy is high or the combinatorial calculation has the risk of singular values, taking into account both practicality and robustness.

[0121] Step 302: Deep learning multi-source speech separation

[0122] For each sound source number , the speech signal stream after directional gain generated in step 301 is Feed it into a deep learning separation model, which can also receive enhanced signals from multiple directions simultaneously to achieve collaborative separation when there are overlapping sources;

[0123] Use a multi-channel deep speech separation network trained specifically for children and multi-speaking scenarios , its output can be expressed as:

[0124]

[0125] Where: Indicates that step 301 is Time-frequency signal characteristics after source-directed enhancement; To separate the training parameter set of the model, including network structure, weights and biases; Represents the purified target speech output by the network;

[0126] If multiple sources are Separation, multi-channel deep speech separation network Will output the corresponding According to the student identity mapping or lip matching information provided in step 2, the separation quality can be evaluated and the stream with the highest credibility can be selected as the main separated target speech. After the separation is completed, several separated speech streams will be obtained. In step 303, the camera lip detection confidence or student ID mapping will be combined for priority selection or merging, and a unique or few high-quality voice streams will be output for use in the next step.

[0127] When in use, the environmental prior and directional information established in steps one and two can be fully utilized to make the deep network more focused on separating the target student's voice rather than large-scale mixed signals. In the case of multiple people speaking in parallel, the corresponding input gain can be configured for multiple sources respectively, and the unified separation network can be used for parallel processing to greatly improve the separation efficiency and accuracy. A two-layer strategy combining directional gain + deep separation is adopted. Beamforming is used to reduce interference first, and then the deep network is used to finely distinguish the overlapping parts. It is especially suitable for situations where multiple parties are speaking in the classroom and the noise is complex. The multi-channel deep speech separation network The network input features can be modified in real time based on the noise information continuously updated by the environmental model, forming a more adaptive separation process.

[0128] Step 303: Multimodal priority fusion and target speech output

[0129] Matching with camera detection results: Based on the sound source direction-student ID binding completed in step 2, compare the multiple separated voice streams output by depth separation If a separated stream is found to have the best lip-sync with the speaker ID and lip-sync timing detected by the current camera, its output priority is increased:

[0130] In order to achieve synchronous analysis of the separated target speech and camera lip timing in the two dimensions of time and frequency, the following fusion metric function is defined: :

[0131]

[0132] Where: Indicates that for A speaking source at time and frequency The time-frequency domain audio feature matrix on the y-axis can be obtained by the depth separation result output by step 302. Obtained through appropriate processing (such as framing and feature extraction);

[0133] Indicates that the camera is facing the same speaking source In time The lip shape / facial feature vector captured during the period can include multiple dimensions such as lip opening and closing degree, key point detection results, etc. If the feature vector length is set to ,but and The ranks in the Hadamard product It can achieve element-by-element correspondence or broadcast expansion. The specific implementation requires dimension mapping at the beginning of the system. is the Frobenius norm, which is the square root of the sum of the squares of the matrix elements and measures the quadratic energy of the entire matrix;

[0134] is a power exponent, which is used to amplify the contribution of peak response or strong matching area and improve the sensitivity of multimodal consistency. is the attenuation coefficient;

[0135] Indicates this moment The possible environmental noise or mismatch degree. The larger the value, the stronger the noise or the audio and video are temporarily out of sync. Indicates that in a given time interval and frequency bandwidth On the other hand, the above fusion terms are double integrated;

[0136] If the fusion metric function If the threshold is reached or exceeded, the separated voice stream Identified as the target speech after separation , the corresponding time domain waveform can be generated and output to step 4 for speech recognition. If multiple sources meet the standards, they can be retained for subsequent parallel recognition or further strategy judgment. At the same time, the fusion metric function The lip sync information can be passed back to step 302 for online correction. If the value is continuously low, it means that the current separation network has a poor separation effect on the source, and the input features or beam parameters of the multi-channel deep speech separation network need to be updated.

[0137] When in use, dual verification of audio and video can be achieved, reducing the possible probability of misclassification in the output of a simple deep model and enhancing the availability and accuracy of the final output voice. In extreme scenarios where multiple people are speaking at the same time and the voice similarity is high, the image port type can be used to assist in making the final judgment to ensure that there will be no confusion in separating student A's voice into student B's ID. This achieves the dual protection of directional gain and deep separation, first suppressing non-target direction noise at the physical level, and then finely distinguishing the remaining overlapping voices at the algorithm level. The audio separation results are verified and weighted through lip timing, significantly reducing the risk of misjudgment caused by voice similarity and noise aliasing.

[0138] Step 4: When the separated target speech is received and needs to be converted into text or semantic information, the child speech customized recognition engine is called to perform feature matching on the audio and generate the sound source recognition result. To improve the adaptation to the diversity of children's pronunciation, combined with the comprehensive confidence function Determine the recognition reliability and record the recognition difficulty information into the global environment model If the number of qualified texts is insufficient, multi-source correction or repeated inquiry will be triggered, and finally the qualified texts will be used for real-time teaching interaction and model writing back;

[0139] The step 4 includes the following contents:

[0140] Step 401: Calling the children's voice customized recognition engine

[0141] The separated target speech output from step 3 is recorded as , is the target time domain audio or feature matrix obtained after beamforming and depth separation in step 3, where Identify one or more possible target sound sources. The signal is usually a time domain waveform or a time-frequency matrix that has been characterized.

[0142] Call the pre-trained children's voice custom recognition engine ,in The parameter set of the engine, including acoustic model, language model and dictionary, for each target source implement:

[0143]

[0144] in, The sound source recognition result, which represents the text transcription and basic confidence description (such as probability distribution or scoring sequence) obtained by the source recognition within a given time period, may include metadata such as timestamp and speaker ID (if voiceprint matching has been performed);

[0145] If multiple target sources are found to meet certain audio and video fusion metrics or teacher instructions (for example, two students speak one after another), the separated target speech sets can be analyzed in parallel. Execute the above recognition process one by one and temporarily store the transcribed sound source recognition results;

[0146] If the sound source recognition result is detected and the global environment model Or if the classroom scenario does not match, this step can also immediately send an update prompt to step three or step two.

[0147] When in use, the separated target speech is mapped to the text and basic semantic level, completing the key step of speech-to-text; the use of a child-customized engine can significantly reduce the missed recognition or misrecognition rate caused by children's pronunciation characteristics, and is more suitable for actual classroom needs than conventional adult recognition engines.

[0148] Step 402: Confidence Adaptive Error Correction and Multi-Source Correction

[0149] Perform confidence evaluation, where each sound source recognition result Transcribe text in , using the recognition engine Output internal scoring and audio matching, define the comprehensive confidence function ,For example:

[0150]

[0151] in: Indicates time The corresponding acoustic matching degree (such as the acoustic model to the frame pronunciation confidence);

[0152] This can come from quantification of noise levels, noise bursts, or speaker anomalies in a model of the environment, such as output by a trained neural network; A normalized and adjustable nonlinear function for scoring acoustics;

[0153] , represents the attenuation coefficient for environmental mismatch;

[0154] If the comprehensive confidence function Below a certain confidence threshold , the recognition result is determined to be unreliable, triggering the following correction strategy:

[0155] Secondary alignment: Call other candidate sources retained in step 3 And verify whether there is mis-separation or source ID confusion through lip sync or context comparison;

[0156] Repeated inquiry: If the suspected error is still not clearly corrected, the student or teacher is prompted to briefly restate the current question or speech to obtain a new voice segment and re-execute step 401.

[0157] If the confidence level is high enough, step 402 will be used to identify the sound source. The result is marked as passed and the conversation is managed in step 403. If a correction or repeated question is asked, the new voice data is sent to step 401 again for secondary recognition. If the result still does not meet the requirements, the result may be reported to the teacher interface for manual intervention.

[0158] When used, it can improve the overall accuracy of speech recognition: combining confidence with environmental factors and making corrections at the output layer of the recognition engine can help to detect errors in a timely manner and make adaptive adjustments. It can be compatible with multiple people speaking: it can compare and correct similar or intertwined sounds to minimize the probability of mistakenly identifying student A's speech as student B's; it organically combines acoustic model scoring with environmental model noise mapping, and suppresses misleading scoring in the event of severe mismatch, with high adaptability.

[0159] Step 403: Adaptive dialogue management and environment model writing back

[0160] The final sound source recognition result with confidence level determined The input includes a dialogue management module (DM) that can contain a teaching knowledge base, a student FAQ database, and question-and-answer templates. It parses question commands or feedback tags (such as students answering questions or asking for help). The dialogue management module (DM) combines the preset teaching scenario logic with the knowledge base to generate the robot's answers or action instructions.

[0161] For key conversation content and current noise conditions, in the global environment model Record new information in the session, such as the type of students' questions, identification difficulty, and environmental interference level during this period;

[0162] Here, you can define additional attribute sets of the environment model EnvAttr and mark them with timestamps for subsequent classroom analysis or long-term performance optimization, including noise peaks, recognition difficulty, speaker abnormality indicators, and voiceprint matching records;

[0163] The robot selects voice responses, facial expressions or body movements based on the results of the dialogue management, and can inform the teacher of the current recognition and interaction status. If it detects that a student asks questions or speaks frequently, it can strengthen the directional tracking of the student's voice in the next period according to the pre-defined priority rules and resource allocation strategy (send instructions to steps 2 and 3 to increase the priority). The additional attribute EnvAttr represents the global environment model. The storage method of the expanded record, such as the current noise peak, the difficulty of the student's accent recognition, etc., can be refined according to needs.

[0164] When in use, a closed-loop interaction can be formed from the recognized text to the robot's response, truly realizing the natural human-computer dialogue function in the classroom. Through the backwriting of the environmental model, it can continuously accumulate data from multiple students and multiple scenarios in long-term operation, providing a historical basis for the next recognition optimization or beam parameter tuning; through dialogue management, the algorithm parameters that are more suitable for the student's language style can be automatically called in subsequent classes or the next process (such as directly enhancing the base student's speech recognition model or custom vocabulary).

[0165] When the recognition confidence is low or possible multi-source intersection is detected, the retained separated audio can be retroactively called for secondary processing, or a re-separation request can be made to the beamforming and deep network. Integration with steps one and two: the latest environment model additional attribute set for noise, reflection and student position will be updated to the global environment model , making subsequent classroom acoustic modeling and direction estimation more realistic. The introduction of adaptive dialogue management and environment writeback mechanisms to build a continuous cycle of classroom teaching not only greatly improves recognition success rates in complex scenarios but also enables educational robots to have interactive capabilities that are scalable and evolve.

[0166] Step 5: Extract the original speech waveform based on the output direction information and the clean speech formed And perform two-dimensional wavelet transform W2D to generate voiceprint vector , and then store it in the additional field of the environment model and pass it through the projection function Target speech after separation from the current Comparison to identify matching in multi-person speech situations And perform local fine-tuning on the ASR recognition engine;

[0167] The step five includes the following:

[0168] Step 501: Voiceprint sample collection and feature extraction

[0169] For students who voluntarily register their voiceprints, they are guided to read a text or answer simple questions in a short quiet window. The original waveform of the recorded voice is recorded as ,in Indicates the student's identity mark;

[0170] Using the target direction in step 2 And the separated target speech output in step 3 , locate and extract the clean speech segment of the student within the required time period of this step; if there is still background interference, the multi-channel deep speech separation network DNet can be called to separate the original speech waveform Further anti-aliasing is performed to obtain high-purity voiceprint samples;

[0171] Perform multi-scale time-frequency analysis on the clean speech segment and introduce 2D wavelet transform to construct the following mapping to obtain the voiceprint vector :

[0172]

[0173] Where: Indicates that students Original waveform during the acquisition period Convert to frequency Dimensional time-frequency spectrum matrix;

[0174] It is a two-dimensional wavelet transform operator used to extract multi-level time-frequency texture and speech formant distribution;

[0175] Recorded as the student's voiceprint vector or matrix representation, the length or size can be customized according to subsequent matching requirements;

[0176] When in use, with the help of the sound source localization and depth separation mechanism in steps two and three, the voiceprint recording process can be made more accurate, reducing the interference of environmental noise or other students' overlapping words. Through two-dimensional wavelet transform, the characteristics of children's voice in multiple frequency bands and multiple time scales can be captured, providing refined features for the subsequent distinction of different students' timbres.

[0177] Step 502: Voiceprint registration and adaptive recognition integration

[0178] The voiceprint vector generated in step 501 Writing to the global environment model Additional record area;

[0179] In step 4, the children's voice custom recognition engine Provide quick corrections for enrolled students, such as in the recognition engine The matching degree is introduced at the acoustic model level :

[0180]

[0181] Where: Represents the time domain or feature vector of the currently detected target speech, that is, the separated target speech;

[0182] Indicates that students The projection of the voiceprint mapped to the same feature space;

[0183] is the convergence factor, which is used to measure the difference between the actual speech and the registration recording;

[0184] When the matching When it is greater than expected, the student is judged The voiceprint matching degree is high, and the recognition engine can be called first Model parameters, or partially fine-tune the recognition engine to recognize the student's speech with higher accuracy.

[0185] When using, when a new speech is detected in step 4, the voiceprint vector can be quickly compared Automatic speaker recognition and directional enhancement are achieved to reduce the rate of false attribution or misjudgment. As more students enter their voiceprints, the robot can form personalized acoustic models for common speakers in frequent interactions, thereby continuously improving recognition efficiency and accuracy. The global environment model and recognition engine Seamless connection, rapid identity recognition and acoustic adaptation capabilities in the case of multiple people speaking, using and Matching degree Real-time collaboration can more accurately identify the student's pronunciation characteristics in high noise environments or when multiple people are speaking at the same time.

[0186] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0187] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only for some logical functions. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0190] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for processing speech signals of an educational robot, characterized in that: include, When entering the classroom and initializing the current space environment, the microphone array and camera are activated to collect multi-channel audio and video signals, and the environmental noise energy is measured and the student distribution model is modeled to obtain a global environmental model; When multi-channel speech is detected with a cross peak and it is suspected that multiple people are speaking at the same time, overlapping speech detection is called and the target sound source is located based on the arrival time difference combined with the lip shape data, and the target sound source direction information and student posture mapping are output; When the target sound source direction information is obtained and the separated target speech needs to be enhanced, the beamforming vector is called to perform gain processing on the corresponding directional signal and a multi-channel deep speech separation network is used to remove environmental noise to produce the separated target speech; When the separated target speech is input into the child speech customized recognition engine and the target sound source recognition result is generated, the recognition confidence is determined based on the recognition confidence evaluation and the environmental noise mismatch information is recorded in the global environment model. If the result is low, multi-source correction or repeated inquiry is triggered, and the recognition result is obtained for real-time teaching interaction and continuous feedback; The multi-scale voiceprint vector is extracted using the directional information and the separated target speech and stored in an additional field of the global environment model. The subsequent recognition accuracy of the student is improved through the projection function and the acoustic model is fine-tuned.

2. The method for processing speech signals of an educational robot according to claim 1, wherein: Use cameras to capture and cache images from various locations inside the classroom, and obtain the corresponding original time domain signal from each microphone. After performing short-time Fourier transform, multi-channel speech in complex form is obtained. After introducing the weighted cross-power spectrum function to measure the joint energy characteristics between different microphone channels at discrete angular frequencies, the full-band weighted cross-power spectrum set and time-aligned camera image information are output.

3. The method for processing speech signals of an educational robot according to claim 2, wherein: Perform energy clustering analysis on the full-band weighted cross-power spectrum set by frequency band to distinguish stable ambient noise, transient ambient noise, and potential speech peak areas. Based on the clustering results, make preliminary markings on the direction of suspicious ambient noise in the classroom. The student gathering areas and suspicious environmental noise locations in the video frames are mapped to obtain the student distribution map and environmental noise direction map in the classroom plane coordinate system; the global environmental model is formed by integrating weighted cross-power spectrum analysis, video tag information and classroom geometric dimensions.

4. The method for processing speech signals of an educational robot according to claim 3, wherein: The multi-channel speech is collected in real time within a time segment to obtain the spectrum data of each microphone in the current time window. The overlapping speech index function is introduced to measure whether there are multiple speakers in the time window. If the overlapping speech index function exceeds a preset overlapping threshold, it is determined that at least two target sound sources are active simultaneously in the current time window, which is identified as an overlapping speech period and triggers the subsequent target sound source direction positioning in step 2.

5. The method for processing speech signals of an educational robot according to claim 4, wherein: In the time interval determined to be the overlapping speech, performing direction estimation on the multi-channel signal, calculating the arrival delay mapping function to obtain the arrival delay estimation; Based on the geometric layout of the microphone array and the results of all arrival delay estimates, the direction vectors of multiple target sound sources are analyzed through triangulation or specific beamforming. These are corrected based on the reflection characteristics, and the peaks in different directions are mapped to different target sound source IDs. A list of target directions is then output.

6. The method for processing speech signals of an educational robot according to claim 5, wherein: Detect students' lip shapes or upper body postures on the camera image to obtain the image coordinates of multiple potential speakers; The neighboring metric is calculated after transforming each image coordinate into the same reference coordinate system as the target direction based on the previously calibrated in-camera microphone array geometry in the global environment model. If the proximity metric exceeds a preset proximity threshold, it is determined that the student image coordinates and the target sound source direction come from the same speaker; The successfully matched target sound source direction is assigned a label consistent with the student ID detected in the image to form a final dynamically updateable target sound source-student binding list. After aggregation, the target sound source positioning information is generated, including the target sound source direction, the corresponding student ID and its spatial coordinates.

7. The method for processing speech signals of an educational robot according to claim 6, wherein: Obtain the target direction of each active speaker from the target sound source localization information, and construct a beam weight vector for the target direction at each center frequency; When there are multiple active target sound sources, a set of beam weight vectors is calculated in parallel for each target sound source, allowing the robot to simultaneously amplify or suppress speech signals from different directions. The beam weight vectors are combined with multi-channel speech to generate a speech signal stream with directional gain.

8. The method for processing speech signals of an educational robot according to claim 7, wherein: The speech signal stream after directional gain is fed into the multi-channel deep speech separation network, and the output is the purified and separated target speech; Based on the student identity mapping or lip matching information, the separation quality can be evaluated and the stream with the highest recognition confidence can be selected as the main separated target speech.

9. The method for processing speech signals of an educational robot according to claim 8, wherein: Define the fusion metric function to synchronize the separated target speech and camera lip timing in both time and frequency dimensions: If the fusion metric function reaches or exceeds the threshold, the separated speech stream is identified as the separated target speech, and the corresponding time domain waveform is generated for speech recognition. The fusion metric function and lip synchronization information are then fed back for online correction. If the fusion metric function continues to be low, the input features or beam parameters of the multi-channel deep speech separation network are updated.

10. The method for processing speech signals of an educational robot according to claim 9, wherein: Call the pre-trained children's speech custom recognition engine to perform recognition for each target sound source: If it is found that multiple target sound sources meet certain audio and video fusion metrics or teacher instructions, the recognition process will be performed on the separated target speech sets one by one in parallel and the transcribed target sound source recognition results will be temporarily stored. If it is detected that the target sound source recognition results do not match the global environment model or classroom scene, an update prompt will be issued immediately.

11. The method for processing speech signals of an educational robot according to claim 10, wherein: For each target sound source recognition result, the text transcription is evaluated using the internal score and audio matching output by the recognition engine and the comprehensive recognition confidence function. If the comprehensive recognition confidence function is lower than the recognition confidence threshold, the recognition result is deemed unreliable and a correction strategy is triggered: Recall other candidate target sound sources and verify whether there is any mis-separation or target sound source ID confusion through lip synchronization or context comparison; if the suspected error is still not clearly corrected, prompt the student or teacher to briefly repeat the current question or speech to obtain a new voice segment and re-perform recognition; If the comprehensive recognition confidence function is not lower than the recognition confidence threshold, the target sound source recognition result will be marked as passed. If corrections or repeated inquiries occur, the new voice data generated will be recognized again. If it still does not meet the standards, it may be reported to the teacher interface for manual intervention.

12. The method for processing speech signals of an educational robot according to claim 11, wherein: The final target sound source recognition result determined by the recognition confidence is input into the dialogue management module, which parses the question command or feedback label and generates the robot's answer or action instruction; Record new information in the global environment model for key conversation content and current ambient noise conditions; If it is detected that a student asks questions multiple times in a row or speaks frequently, the targeted tracking of the student's voice can be strengthened in the next period based on pre-defined priority rules and resource allocation strategies.

13. The method for processing speech signals of an educational robot according to claim 12, wherein: After recording the original speech waveform, the target direction and the separated target speech are used to locate and extract the separated target speech segment of the student within the required time period. If there is still environmental noise, the multi-channel deep speech separation network is called to further de-alias the original speech waveform, and the separated target speech segment is subjected to multi-scale time-frequency analysis. The 2D wavelet transform is introduced to construct the following mapping to obtain the voiceprint vector.

14. The method for processing speech signals of an educational robot according to claim 13, wherein: The voiceprint vector is written into the additional record area of ​​the global environment model to provide fast correction and introduce matching for registered students in the recognition engine; When the matching degree is greater than expected, it is determined that the student's voiceprint matching degree is high, and the recognition engine model parameters are preferentially retrieved, or the recognition engine is partially fine-tuned.

Images