AI voice assistant semantic enhancement method oriented to education scene

Abstract

The invention discloses an AI voice assistant semantic enhancement method oriented to an education scene, and the method comprises the following steps: S1, collecting classroom voice signals in real time through an annular multi-microphone array which is installed at the edge of a blackboard in an embedded manner, and obtaining the real-time three-dimensional coordinates of a teacher through a UWB positioning module; s2, constructing a spatial sound source mapping model based on the three-dimensional coordinates of the teacher; s3, acquiring blackboard writing character content through a visual sensor mounted on the side edge of the blackboard; s4, injecting the extracted feature keywords into a dynamic term library generation module to generate a standardized teaching instruction, and synchronizing the standardized teaching instruction to student terminal equipment; and S5, when detecting that the confidence of the voice instruction is lower than a preset threshold value, jointly analyzing the time sequence characteristics of the lip movement track and the blackboard writing handwriting to perform semantic completion. According to the method, an intelligent semantic analysis system with acoustic enhancement, term adaptation and multimode verification is constructed, and the semantic distortion problem caused by environmental interference in traditional speech recognition is eliminated through directional pickup and noise suppression of an intelligent array.

Description

Technical Field

[0001] The present invention belongs to the field of semantic communication, and more specifically, relates to a semantic enhancement method for an AI voice assistant in educational scenarios. Background Art

[0002] With the development of semantic communication, its application in the field of education has also become popular. However, many problems still exist in its actual application. For example, the patent CN117972112A proposes a method for enhancing the robustness of semantic communication based on knowledge graphs. It calculates the semantic similarity of triples by constructing a BERT_KG model and uses a dynamic threshold and mean joint decision algorithm to suppress noise interference. Although this technology significantly improves the reliability of text semantic transmission, it faces the following bottlenecks in the complex acoustic environment of educational scenarios:

[0003] Its decision algorithm relies on a static knowledge graph collection and is unable to cope with the multipath reflections, blackboard writing noise, and student interference that exist in the classroom. When the teacher moves while teaching, the fixed microphone array generates spatial reverberation, which distorts the time-frequency characteristics of the speech signal and directly reduces the accuracy of semantic symbol encoding. Furthermore, the document does not include any beamforming or acoustic noise reduction methods. When the signal-to-noise ratio falls below 10dB, the text recovery error rate surges.

[0004] The semantic similarity calculation using a fixed threshold and the maximum judgment of the historical mean cannot adapt to the subject-specific terms generated in real time during teaching;

[0005] Relying solely on the semantic associations of text symbols, when voice commands are drowned out by ambient noise, it is impossible to integrate and verify heterogeneous data such as lip movement features and blackboard writing trajectories. Test data shows that when the audio channel bit error rate exceeds 5%, the command loss rate increases due to the lack of multimodal compensation.

[0006] Based on the above, we propose a semantic enhancement method for AI voice assistants in educational scenarios to specifically solve the above problems. Summary of the Invention

[0007] This invention aims to address the shortcomings of existing technologies by proposing a semantic enhancement method for AI voice assistants in educational settings. This method addresses the complex acoustic environments and dynamic teaching requirements of educational settings by constructing an intelligent semantic parsing system integrating acoustic enhancement, terminology adaptation, and multi-mode verification. Through the directional sound pickup and noise suppression of an intelligent array, the semantic distortion caused by environmental interference in traditional speech recognition is eliminated.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The semantic enhancement method for AI voice assistants in educational scenarios includes the following steps:

[0010] S1, uses a circular multi-microphone array embedded on the edge of the blackboard to collect classroom voice signals in real time, and uses the UWB positioning module to obtain the teacher's real-time three-dimensional coordinates;

[0011] S2. Construct a spatial sound source mapping model based on the teacher's three-dimensional coordinates, use an adaptive neural network to control the phase of the microphone array, generate a directional beam, and attenuate the sound source in the non-teacher direction by more than 20 dB. Input the beamformed teacher's voice signal into a multi-layer residual convolutional network for acoustic feature enhancement.

[0012] S3, using the visual sensor installed on the side of the blackboard to obtain the text content on the blackboard, and using the OCR recognition module to extract subject feature keywords;

[0013] S4. Inject the extracted feature keywords into the dynamic terminology library generation module, create a subject-specific dictionary tree, and perform online weight fusion with the general speech recognition model to form a domain-adaptive language model for the current classroom scenario. The enhanced speech signal is input into the domain-adaptive language model for semantic analysis, and standardized teaching instructions are generated and synchronized to the student terminal device.

[0014] S5. When the confidence level of the voice command is detected to be lower than the preset threshold, the multimodal verification mechanism is automatically triggered to jointly analyze the lip movement trajectory and the temporal characteristics of the blackboard handwriting for semantic completion.

[0015] Preferably, the adaptive neural network in step S2 includes the following processing mechanisms:

[0016] S21, using the acoustic transfer function matrix to construct a transfer delay model including the teacher coordinates to each microphone path;

[0017] S22. Optimize the beam weight vector using an online gradient descent algorithm to maximize the gain in the target direction and suppress interference signals in directions other than 45°.

[0018] S23, using dual recurrent neural networks to model short-term sound field disturbances and long-term environmental noise characteristics respectively, and dynamically adjust the beamforming parameters of the frequency sub-band;

[0019] The multi-layer residual convolutional network contains 12 dilated convolutional layers, which separates speech harmonic components from transient noise in the time-frequency domain and compensates for high-frequency information loss through cross-layer residual connections.

[0020] Preferably, the optimization calculation formula of the beam weight vector is:

[0021] Among them, W (t) is the optimized beam weight vector at the current time t, is the acoustic transfer function matrix in the direction d, calculated by the transfer delay model, is the actual delay of microphone m receiving the UWB positioning signal, is the theoretical sound wave propagation delay in the target direction d, is the instantaneous channel response of frequency f at the mth microphone, calculated by a dual recurrent neural network, N(f, t-1) is the energy proportion of the non-target frequency band at the previous moment, obtained from the interference signal statistics, α is the sub-band suppression coefficient, used to enhance the directivity accuracy of the high-frequency band, β is the delay compensation factor, used to reduce the impact of coordinate positioning error on phase control, λ is the residual learning coefficient, taken from the cross-layer connection characteristics of the multi-layer residual convolutional network, is the high-frequency compensation of the beamforming signal at the previous moment after being processed by the residual network; this optimization formula is optimized through multi-physical field coupling to reduce the main lobe width of the beam and improve the sidelobe suppression ratio.

[0022] Preferably, the processing of the OCR recognition module in step S3 includes:

[0023] S31, using wide-angle cameras installed on the left and right sides of the blackboard to capture images of writing on the blackboard at a frame rate of 30fps, and using image stitching algorithms to generate a panoramic image;

[0024] S32, using the bicubic interpolation algorithm to perform super-resolution reconstruction on the stitched image, increasing the resolution to 800dpi;

[0025] S33. Perform semantic partitioning of the reconstructed image into different disciplines. The mathematics domain uses the LaTeX symbol recognition engine, and the chemistry domain uses a molecular formula parser based on the topological structure of the bond line graph.

[0026] S34. Perform contextual semantic correction on the recognition results through a bidirectional LSTM network to generate a feature keyword sequence containing a timestamp.

[0027] Preferably, the online weight fusion in step S4 is achieved by:

[0028] S41. Replace the output layer of the general speech recognition model with a scalable hybrid output layer.

[0029] S42, dynamically adjusting the node weights of the dedicated dictionary tree according to the word frequency statistics of the currently extracted feature keywords;

[0030] S43. Use the multi-head attention mechanism to calculate the similarity weights between general word vectors and professional term vectors;

[0031] S44. The fused probability distribution is calibrated through the temperature scaling function to improve the recognition confidence of professional terminology; standardized teaching instructions are structured and coded according to the IEEE 1873 standard for educational information exchange.

[0032] Preferably, the temperature scaling function is calculated as follows:

[0033] Among them, T k represents the confidence adjustment value of term k, is the frequency of occurrence of term k in the current class time period, is the initial node weight of term k in the subject-specific dictionary tree, ·Γ (t) is the time decay factor, η is the subject adaptation coefficient, and through nonlinear temperature control, the recognition probability improvement curve of professional terms conforms to the Weibull distribution characteristics, which reduces the false recognition rate compared with the traditional softmax temperature scaling method.

[0034] Preferably, the multimodal verification mechanism in step S5 includes the following processing steps:

[0035] S51, extracting a fundamental frequency trajectory curve from the enhanced speech signal, and matching the key frame sequence of the lip movement using a dynamic time warping algorithm;

[0036] S52. Perform quantile regression analysis on the writing speed of blackboard handwriting and establish a correlation model between pen tip pressure and speech intensity;

[0037] S53: When the speech confidence is lower than 0.7, the optical flow method is activated to track the teacher's hand movements and the missing instructions are reconstructed based on the spatial distribution characteristics of the writing trajectory.

[0038] S54. Using a multi-source information fusion strategy, the output results of the three modalities of speech, lip movement and blackboard writing are synthesized using the D-S evidence theory to finally generate a semantic completion result.

[0039] Preferably, in step S1: the annular multi-microphone array includes 32 evenly distributed digital microphone units, and the distance between adjacent microphone units is in a logarithmic relationship with the height of the blackboard;

[0040] The UWB positioning module establishes a three-dimensional coordinate system with the center of the blackboard as the coordinate origin. By calculating the time delay difference between the pulse signal emitted by the active beacon worn by the teacher and reaching each receiving antenna, the generalized cross-correlation algorithm is used to solve the spatial coordinates.

[0041] The speech signals collected by the microphone array are synchronized at a 96kHz sampling rate to form a 32-channel data stream aligned in the time domain.

[0042] Preferably, the data synchronization method between the annular multi-microphone array and the UWB positioning module includes:

[0043] 1) Align the UWB positioning pulse with the microphone sampling clock through a hardware trigger signal;

[0044] 2) Using a sliding window mechanism to establish a mapping relationship between the sound source direction estimation and the spatial coordinates;

[0045] 3) When the teacher's movement speed is detected to exceed 0.5m / s, it automatically switches to motion compensation mode and uses acceleration sensor data to correct the pointing hysteresis of the beamforming;

[0046] 4) The working mode of the microphone array is dynamically adjusted through the information entropy evaluation mechanism, and the spectral component of chalk friction noise is preferentially suppressed during the blackboard writing period.

[0047] Technical effects and advantages of the present invention: Compared with the existing technology, the semantic enhancement method of AI voice assistant in educational scenarios provided by the present invention has the following effects:

[0048] Directional acoustic enhancement: Through intelligent beamforming of a multi-microphone array, focused enhancement of the teacher's voice and spatial filtering of interfering signals are achieved, effectively suppressing blackboard writing noise and student discussion noise, ensuring pure voice signal input.

[0049] The system has strong semantic adaptability for interdisciplinary teaching. By leveraging real-time blackboard visual analysis and dynamic terminology fusion, it automatically identifies and adapts to the specific expression paradigms of mathematics, chemistry, and physics, eliminating ambiguity in the parsing of cross-disciplinary terminology.

[0050] The multimodal fault-tolerance and recovery capabilities are outstanding. In scenarios with strong noise or speech interruption, the multi-source information of lip movement features and writing trajectories is used to complement each other, enabling intelligent reconstruction and integrity assurance of missing semantic instructions.

[0051] The advantage of intelligent tracking of teaching behavior, combined with teacher position positioning and sound field modeling, realizes voice enhancement that adapts to the teaching route, supports teachers' free movement and explanation in front of the blackboard, and maintains stable sound pickup quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of the semantic enhancement method of the AI ​​voice assistant in the educational scenario of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0054] The present invention provides Figure 1 The semantic enhancement method for AI voice assistants in educational scenarios, shown here, addresses the complex acoustic environments and dynamic teaching needs of educational scenarios by building an intelligent semantic parsing system that integrates acoustic enhancement, terminology adaptation, and multi-mode verification.

[0055] The directional sound pickup and noise suppression of the intelligent array eliminates the semantic distortion caused by environmental interference in traditional speech recognition. The dynamic terminology mechanism breaks through disciplinary barriers and achieves a seamless integration of professional expression and common language. Multimodal verification technology innovatively integrates audio-visual features to maintain the consistency of instruction delivery when voice quality deteriorates. The teacher's positioning tracking and adaptive processing capabilities ensure a natural interactive experience during the teaching process. The entire solution effectively solves the core problems of existing technologies such as high environmental sensitivity, weak subject adaptability, and lack of fault-tolerant mechanisms. It significantly improves the semantic understanding accuracy and universality of teaching scenarios of educational AI assistants, providing reliable technical support for the construction of smart classrooms.

[0056] The method specifically includes the following steps:

[0057] S1. Real-time classroom speech signals are collected through a circular multi-microphone array embedded in the edge of the blackboard, and the teacher's real-time three-dimensional coordinates are obtained through the UWB positioning module. In step S1: the circular multi-microphone array includes 32 evenly distributed digital microphone units, which are installed at a distance of 5 cm ± 0.2 mm from the edge of the blackboard plane. The distance between adjacent microphone units is logarithmically related to the height of the blackboard.

[0058] The UWB positioning module establishes a three-dimensional coordinate system with the center of the blackboard as the coordinate origin. By calculating the time delay difference between the pulse signal emitted by the active beacon worn by the teacher and reaching each receiving antenna, the generalized cross-correlation algorithm is used to solve the spatial coordinates, and the positioning error is controlled within ±1.5cm; the voice signal collected by the microphone array is synchronized at a 96kHz sampling rate to form a 32-channel data stream aligned in the time domain.

[0059] The data synchronization method between the annular multi-microphone array and the UWB positioning module includes:

[0060] The UWB positioning pulse is aligned with the microphone sampling clock through the hardware trigger signal;

[0061] A sliding window mechanism is used to establish the mapping relationship between the sound source direction estimation and the spatial coordinates, with a window length of 5 seconds and an overlap rate of 75%;

[0062] When the teacher's movement speed is detected to exceed 0.5m / s, it automatically switches to motion compensation mode and uses acceleration sensor data to correct the pointing hysteresis of the beamforming;

[0063] The working mode of the microphone array is dynamically adjusted through the information entropy evaluation mechanism, and the spectral component of chalk friction noise is preferentially suppressed during the blackboard writing period.

[0064] Specifically, the UWB positioning error compensation formula is:

[0065] Where Δτ m is the delay compensation of microphone m, is the signal attenuation factor of the nth reflection path, extracted from the UWB channel impulse response in S1, is the arrival angle of the nth reflection path, is the azimuth angle of microphone m, c is the speed of sound, is the maximum moving speed of the teacher, Represents the position prediction residual based on Kalman filter, F s is the sampling rate; through multipath effect modeling and kinematic analysis, the positioning error is reduced from ±1.5cm to ±0.8cm, which improves the beam pointing accuracy.

[0066] S2. Construct a spatial sound source mapping model based on the teacher's three-dimensional coordinates, use an adaptive neural network to control the phase of the microphone array, generate a directional beam, and attenuate the sound source in the non-teacher direction by more than 20dB. Input the beamformed teacher's speech signal into a multi-layer residual convolutional network for acoustic feature enhancement. The adaptive neural network in step S2 includes the following processing mechanisms:

[0067] The acoustic transfer function matrix is ​​used to construct a transfer delay model including the teacher coordinates to each microphone path;

[0068] The beam weight vector is optimized through an online gradient descent algorithm to maximize the gain in the target direction and suppress interference signals in directions other than 45°.

[0069] A dual recurrent neural network is used to model short-term sound field disturbances and long-term environmental noise characteristics respectively, and dynamically adjust the beamforming parameters of the frequency sub-band;

[0070] The multi-layer residual convolutional network contains 12 dilated convolutional layers, which separates speech harmonic components from transient noise in the time-frequency domain and compensates for high-frequency information loss through cross-layer residual connections.

[0071] The optimization calculation formula of the beam weight vector is:

[0072] Among them, W (t) is the optimized beam weight vector at the current time t, is the acoustic transfer function matrix in the direction d, calculated by the transfer delay model, is the actual delay of microphone m receiving the UWB positioning signal, is the theoretical sound wave propagation delay in the target direction d, is the instantaneous channel response of frequency f at the mth microphone, calculated by a dual recurrent neural network, N(f, t-1) is the energy proportion of the non-target frequency band at the previous moment, obtained from the interference signal statistics, α is the sub-band suppression coefficient, used to enhance the directivity accuracy of the high-frequency band, β is the delay compensation factor, used to reduce the impact of coordinate positioning error on phase control, λ is the residual learning coefficient, taken from the cross-layer connection characteristics of the multi-layer residual convolutional network, is the high-frequency compensation of the beamforming signal at the previous moment after being processed by the residual network; this optimization formula is optimized through multi-physical field coupling to reduce the main lobe width of the beam and improve the sidelobe suppression ratio.

[0073] It should be noted that this step also includes transient noise suppression, and the noise suppression calculation formula is:

[0074] Among them, γ (f，t) is the suppression coefficient of frequency f at time t, is the speech presence probability output by the voice activity detector, D KL The noise spectrum With phonetic spectrum The KL divergence of is the historical average signal-to-noise ratio, is the spectral entropy of the current frame signal, and ζ is the suppression strength adjustment factor. By combining information entropy with acoustic statistical characteristics, the chalk noise suppression during blackboard writing reaches more than 10 dB and the speech distortion is reduced.

[0075] S3. Obtain the text content on the blackboard through the visual sensor installed on the side of the blackboard, and use the OCR recognition module to extract subject feature keywords; the processing process of the OCR recognition module in step S3 includes:

[0076] The wide-angle cameras installed on the left and right sides of the blackboard capture the images of the writing on the blackboard at a frame rate of 30fps, and the image stitching algorithm is used to generate a panoramic image;

[0077] The bicubic interpolation algorithm is used to perform super-resolution reconstruction on the stitched images, increasing the resolution to 800 dpi;

[0078] Perform disciplinary semantic partitioning on the reconstructed image. For the mathematical area, use a LaTeX symbol recognition engine, and for the chemical area, use a molecular formula parser based on the topological structure of bond-line diagrams;

[0079] Perform context semantic error correction on the recognition results through a bidirectional LSTM network, and generate a feature keyword sequence containing timestamps;

[0080] Furthermore, through a 4K wide-angle visual sensor array installed on both the left and right sides of the blackboard, synchronously capture the full view of the blackboard writing at 30 frames per second. The coverage range of a single-side image reaches 120°. The specific processing process includes the following key technical links:

[0081] Multi-source image preprocessing and fusion. Use an adaptive trapezoidal correction algorithm to eliminate the perspective deformation of the blackboard border, and detect the edges of chalk marks through the Hough transform to generate a reference correction grid;

[0082] Trigger the intelligent exposure control module, dynamically adjust the shutter speed of the camera according to the ambient light intensity, with a range of 1 / 60s to 1 / 2000s and the ISO value with a range of 100 - 3200, to ensure the recognizability of the blackboard writing in backlight or strong reflection scenarios;

[0083] Apply multi-scale image stitching technology to seamlessly fuse the local images captured by the bilateral cameras based on SIFT feature points, and generate an ultra-clear blackboard writing panorama with a resolution of 8192×4320;

[0084] Deploy a lightweight super-resolution network with a 12-layer residual structure to enhance the handwriting of the writing content of mixed chalk and whiteboard pens, make the stroke width uniform, and repair the broken point areas.

[0085] Multi-disciplinary content partitioning and recognition. Based on a semantic segmentation model improved from YOLOv5, divide the blackboard writing picture into a mathematical formula area, a chemical equation area, a physical diagram area, and a text description area, and set the classification confidence threshold to 0.95;

[0086] For the mathematical area, use a bidirectional parsing engine based on the LaTeX syntax tree to real-time recognize special mathematical symbols such as integral symbols, matrix expressions, and Greek letters, and perform spatial relationship modeling on the subscript / superscript positions;

[0087] For the chemical area, start the bond-line diagram topological analysis module, identify the benzene ring structure through the ring detection algorithm, and combine atomic connectivity rules, such as the carbon tetravalence principle, to correct the molecular formula recognition results and support the stereochemical configuration analysis of hand-drawn isomers;

[0088] For the physical area, deploy a dimensional unit knowledge base to perform intelligent unit conversion on the recognized physical quantities, such as automatically annotating "kg·m² / s²" as "W", and analyze the mechanical diagrams through the vector arrow direction detection algorithm.

[0089] Dynamic semantic library construction and optimization, establishing a dual buffer mechanism: the original OCR results are stored in a temporary buffer area, and the semantically verified feature keywords are stored in a stable storage area. The two are quickly synchronized through hash indexes;

[0090] For interdisciplinary keywords that appear repeatedly, such as "speed" and "molecule", a multi-dimensional semantic vector is generated. A hierarchical clustering algorithm is used to construct a discipline-specific context map, and the node weights are dynamically updated according to the co-occurrence frequency of the context.

[0091] Deploy an incremental term merging strategy to automatically create independent child nodes when the semantic differences between the same term in different subject areas exceed a preset threshold, such as the semantic separation of "group" in mathematics and "atomic group" in chemistry;

[0092] The updated dictionary tree is deeply coupled with the general speech model through a real-time injection protocol, and subject feature gating units are implanted in the hidden layer of the LSTM network to achieve seamless switching of domain language models.

[0093] Exception handling and self-learning mechanism: For handwritten content that cannot be parsed, a distributed crowdsourcing annotation mechanism is launched. Image slices of the area are sent to the teacher terminal for manual annotation, and the feedback results are used as reinforcement learning samples.

[0094] Build a dynamic discrimination model for confusing characters, such as distinguishing the number "0" from the letter "O," and reduce ambiguity through writing stroke tracking and contextual semantic analysis, with an error rate of less than 0.3%;

[0095] Record the log data of each OCR correction operation, regularly generate the misrecognition heat map of feature keywords, and optimize the region division strategy of the segmentation model accordingly;

[0096] When new subject symbols are detected, such as newly added biological genetic map markers, the online learning mode is automatically triggered, and the coverage of the recognition engine is expanded through small sample transfer learning.

[0097] S4. Inject the extracted feature keywords into the dynamic terminology library generation module, create a subject-specific dictionary tree, and perform online weight fusion with the general speech recognition model to form a domain-adaptive language model for the current classroom scenario. The enhanced speech signal is input into the domain-adaptive language model for semantic analysis, and standardized teaching instructions are generated and synchronized to the student terminal device. Online weight fusion is achieved through the following methods:

[0098] Replace the output layer of the general speech recognition model with a scalable hybrid output layer;

[0099] Dynamically adjust the node weights of the dedicated dictionary tree according to the word frequency statistics of the currently extracted feature keywords;

[0100] A multi-head attention mechanism is used to calculate the similarity weights between general word vectors and professional term vectors;

[0101] The fused probability distribution is calibrated using a temperature scaling function, which improves the recognition confidence of professional terms by more than 35%.

[0102] Standardized teaching instructions are structured and coded according to the IEEE 1873 standard for educational information interchange;

[0103] The temperature scaling function is calculated as:

[0104] Among them, T k represents the confidence adjustment value of term k, is the frequency of occurrence of term k in the current class time period, is the initial node weight of term k in the subject-specific dictionary tree, ·Γ (t) is the time decay factor, η is the subject adaptation coefficient, and through nonlinear temperature control, the recognition probability improvement curve of professional terms conforms to the Weibull distribution characteristics, which reduces the false recognition rate compared with the traditional softmax temperature scaling method.

[0105] S5. When the confidence level of the voice command is detected to be lower than a preset threshold, the multimodal verification mechanism is automatically triggered to jointly analyze the lip movement trajectory and the temporal features of the blackboard handwriting for semantic completion. The multimodal verification mechanism in step S5 includes the following processing steps:

[0106] Extract the fundamental frequency trajectory curve from the enhanced speech signal and match it with the key frame sequence of lip movement using the dynamic time warping algorithm;

[0107] Quantile regression analysis was conducted on the writing speed of blackboard handwriting to establish a correlation model between pen tip pressure and speech intensity;

[0108] When the speech confidence is lower than 0.7, the optical flow method is activated to track the teacher's hand movements and the missing instructions are reconstructed by combining the spatial distribution characteristics of the writing trajectory;

[0109] A multi-source information fusion strategy is adopted to synthesize the output results of the three modalities of speech, lip movement and blackboard writing using the D-S evidence theory, and finally generate the semantic completion result.

[0110] Optionally, when the confidence level of a voice command is detected to be lower than a preset threshold, the system automatically activates the multimodal verification mechanism. The specific processing flow is as follows:

[0111] Multimodal data is collected synchronously. A binocular vision sensor mounted on the top of the blackboard captures a high-definition video stream of the teacher's lips at a frame rate of 60fps. Simultaneously, piezoelectric writing track sensors deployed on both sides of the blackboard record the coordinate sequence of the handwriting in real time with a spatial resolution of 1mm. The video stream and handwriting data are aligned using hardware-level timestamps to ensure timing skew less than 5ms. After triggering verification, the system immediately caches the previous 5 seconds of speech signals and establishes a synchronization index with the multimodal data for the corresponding period.

[0112] Lip motion trajectory analysis uses an adaptive optical flow algorithm to extract 12 biometric features of the lips, including the motion vectors of the mouth corners and peaks, and calculates the displacement rate and directional consistency index for each point. A dynamic time warping algorithm nonlinearly aligns the lip motion trajectory with the fundamental frequency variation of the speech signal to identify possible areas of lip-reading-speech asynchrony. Spatial interpolation compensation is applied to areas of abnormal displacement exceeding 10 consecutive frames to eliminate feature point loss due to head deflection.

[0113] Dynamic modeling of blackboard handwriting: Based on the three-dimensional coordinate sequence x, y, t output by the piezoelectric sensor, a velocity-acceleration model of the pen tip motion is constructed. A Fourier descriptor of the writing rhythm is calculated using a sliding window with a window length of 2 seconds. A spatial clustering algorithm is used to identify logical divisions of the blackboard content, such as the formula derivation area and the diagram area, and a correlation map is established between the mean handwriting pressure and speech intensity in each area. For the special structure of chemical equations, a topological feature extractor is activated to analyze the bond-line connection relationship.

[0114] A multimodal feature analysis combines the time-frequency characteristics of lip movement trajectories, fundamental frequency harmonics, mouth shape, timing characteristics of the writing on the blackboard, writing pause duration, and line curvature into a bidirectional gated recurrent network to generate a fused semantic enhancement vector. The network dynamically assigns modal weights through an attention mechanism, prioritizing the semantic features of the writing on the blackboard when the speech confidence deviation exceeds 0.3. For segments with significant conflicting data from multiple sources, such as when the lip movement indicates "sodium chloride" while the writing on the blackboard displays "NaCl," the domain ontology library is used to verify term valence.

[0115] Semantic completion and feedback verification: The fused semantic hypothesis is used through an incremental decoder to generate a set of candidate commands. A negative sampling strategy is used to select the top three candidates with the smallest edit distance to the original speech hypothesis. After the candidate commands are reviewed by the acoustic-visual consistency assessment module, a gentle beep is issued through the classroom sound system to request teacher confirmation. If there is no human intervention within 10 seconds, the system corrects the command based on the output of the modality with the highest historical accuracy, and simultaneously updates the weight parameters of the dynamic terminology library.

[0116] A self-optimization mechanism automatically generates a multimodal alignment quality assessment report after each verification, recording the lip movement-speech correlation coefficient and handwriting-semantic matching metrics. If the triggering reasons for three consecutive verifications are similar, the online learning module is activated to adjust the critical threshold of the acoustic model and optimize the exposure parameters of the visual sensor to adapt to changes in ambient lighting. All successful correction cases are archived as reinforcement learning samples to improve the response efficiency of subsequent verifications.

[0117] In summary, the present invention has the following effects:

[0118] Directional acoustic enhancement: Through intelligent beamforming of a multi-microphone array, focused enhancement of the teacher's voice and spatial filtering of interfering signals are achieved, effectively suppressing blackboard writing noise and student discussion noise, ensuring pure voice signal input.

[0119] The system has strong semantic adaptability for interdisciplinary teaching. By leveraging real-time blackboard visual analysis and dynamic terminology fusion, it automatically identifies and adapts to the specific expression paradigms of mathematics, chemistry, and physics, eliminating ambiguity in the parsing of cross-disciplinary terminology.

[0120] The multimodal fault-tolerance and recovery capabilities are outstanding. In scenarios with strong noise or speech interruption, the multi-source information of lip movement features and writing trajectories is used to complement each other, enabling intelligent reconstruction and integrity assurance of missing semantic instructions.

[0121] The advantage of intelligent tracking of teaching behavior, combined with teacher position positioning and sound field modeling, realizes voice enhancement that adapts to the teaching route, supports teachers' free movement and explanation in front of the blackboard, and maintains stable sound pickup quality.

[0122] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. The semantic enhancement method of AI voice assistant in educational scenarios is characterized by: The steps include: S1, uses a circular multi-microphone array embedded on the edge of the blackboard to collect classroom voice signals in real time, and uses the UWB positioning module to obtain the teacher's real-time three-dimensional coordinates; S2. Construct a spatial sound source mapping model based on the teacher's three-dimensional coordinates, use an adaptive neural network to control the phase of the microphone array, generate a directional beam, and attenuate the sound source in the non-teacher direction by more than 20 dB. Input the beamformed teacher's voice signal into a multi-layer residual convolutional network for acoustic feature enhancement. S3, using the visual sensor installed on the side of the blackboard to obtain the text content on the blackboard, and using the OCR recognition module to extract subject feature keywords; S4. Inject the extracted feature keywords into the dynamic terminology library generation module, create a subject-specific dictionary tree, and perform online weight fusion with the general speech recognition model to form a domain-adaptive language model for the current classroom scenario. The enhanced speech signal is input into the domain-adaptive language model for semantic analysis, and standardized teaching instructions are generated and synchronized to the student terminal device. S5. When the confidence level of the voice command is detected to be lower than the preset threshold, the multimodal verification mechanism is automatically triggered to jointly analyze the lip movement trajectory and the temporal characteristics of the blackboard handwriting for semantic completion.

2. The method for semantic enhancement of AI voice assistants in educational scenarios according to claim 1 is characterized in that: The adaptive neural network in step S2 includes the following processing mechanisms: S21, using the acoustic transfer function matrix to construct a transfer delay model including the teacher coordinates to each microphone path; S22. Optimize the beam weight vector using an online gradient descent algorithm to maximize the gain in the target direction and suppress interference signals in directions other than 45°. S23, using dual recurrent neural networks to model short-term sound field disturbances and long-term environmental noise characteristics respectively, and dynamically adjust the beamforming parameters of the frequency sub-band; The multi-layer residual convolutional network contains 12 dilated convolutional layers, which separates speech harmonic components from transient noise in the time-frequency domain and compensates for high-frequency information loss through cross-layer residual connections.

3. The method for semantic enhancement of AI voice assistants in educational scenarios according to claim 2, characterized in that: The optimization calculation formula of the beam weight vector is: Among them, W (t) is the optimized beam weight vector at the current time t, is the acoustic transfer function matrix in the direction d, calculated by the transfer delay model, is the actual delay of microphone m receiving the UWB positioning signal, is the theoretical sound wave propagation delay in the target direction d, is the instantaneous channel response of frequency f at the mth microphone, calculated by a dual recurrent neural network, N(f, t-1) is the energy proportion of the non-target frequency band at the previous moment, obtained from the interference signal statistics, α is the sub-band suppression coefficient, used to enhance the directivity accuracy of the high-frequency band, β is the delay compensation factor, used to reduce the impact of coordinate positioning error on phase control, λ is the residual learning coefficient, taken from the cross-layer connection characteristics of the multi-layer residual convolutional network, is the high-frequency compensation of the beamforming signal at the previous moment after being processed by the residual network; this optimization formula is optimized through multi-physical field coupling to reduce the main lobe width of the beam and improve the sidelobe suppression ratio.

4. The method for semantic enhancement of AI voice assistants in educational scenarios according to claim 1, characterized in that: The processing of the OCR recognition module in step S3 includes: S31, using wide-angle cameras installed on the left and right sides of the blackboard to capture images of writing on the blackboard at a frame rate of 30fps, and using image stitching algorithms to generate a panoramic image; S32, using the bicubic interpolation algorithm to perform super-resolution reconstruction on the stitched image, increasing the resolution to 800dpi; S33. Perform semantic partitioning of the reconstructed image into different disciplines. The mathematics domain uses the LaTeX symbol recognition engine, and the chemistry domain uses a molecular formula parser based on the topological structure of the bond line graph. S34. Perform contextual semantic correction on the recognition results through a bidirectional LSTM network to generate a feature keyword sequence containing a timestamp.

5. The method for semantic enhancement of AI voice assistants in educational scenarios according to claim 1, characterized in that: The online weight fusion in step S4 is achieved by: S41. Replace the output layer of the general speech recognition model with a scalable hybrid output layer. S42, dynamically adjusting the node weights of the dedicated dictionary tree according to the word frequency statistics of the currently extracted feature keywords; S43. Use the multi-head attention mechanism to calculate the similarity weights between general word vectors and professional term vectors; S44. The fused probability distribution is calibrated through the temperature scaling function to improve the recognition confidence of professional terminology; standardized teaching instructions are structured and coded according to the IEEE 1873 standard for educational information exchange.

6. The method for semantic enhancement of AI voice assistants in educational scenarios according to claim 5, characterized in that: The temperature scaling function is calculated as: Among them, T k represents the confidence adjustment value of term k, is the frequency of occurrence of term k in the current class time period, is the initial node weight of term k in the subject-specific dictionary tree, ·Γ (t) is the time decay factor, η is the subject adaptation coefficient, and through nonlinear temperature control, the recognition probability improvement curve of professional terms conforms to the Weibull distribution characteristics, which reduces the false recognition rate compared with the traditional softmax temperature scaling method.

7. The method for semantic enhancement of AI voice assistants in educational scenarios according to claim 1, characterized in that: The multimodal verification mechanism in step S5 includes the following processing steps: S51, extracting a fundamental frequency trajectory curve from the enhanced speech signal, and matching the key frame sequence of the lip movement using a dynamic time warping algorithm; S52. Perform quantile regression analysis on the writing speed of blackboard handwriting and establish a correlation model between pen tip pressure and speech intensity; S53: When the speech confidence is lower than 0.7, the optical flow method is activated to track the teacher's hand movements and the missing instructions are reconstructed based on the spatial distribution characteristics of the writing trajectory. S54. Using a multi-source information fusion strategy, the output results of the three modalities of speech, lip movement and blackboard writing are synthesized using the D-S evidence theory to finally generate a semantic completion result.

8. The method for semantic enhancement of AI voice assistants in educational scenarios according to claim 1, characterized in that: In step S1: the annular multi-microphone array includes 32 evenly distributed digital microphone units, and the distance between adjacent microphone units is logarithmically related to the height of the blackboard; The UWB positioning module establishes a three-dimensional coordinate system with the center of the blackboard as the coordinate origin. By calculating the time delay difference between the pulse signal emitted by the active beacon worn by the teacher and reaching each receiving antenna, the generalized cross-correlation algorithm is used to solve the spatial coordinates. The speech signals collected by the microphone array are synchronized at a 96kHz sampling rate to form a 32-channel data stream aligned in the time domain.

9. The method for semantic enhancement of AI voice assistants in educational scenarios according to claim 8, characterized in that: The data synchronization method between the annular multi-microphone array and the UWB positioning module includes: 1) Align the UWB positioning pulse with the microphone sampling clock through a hardware trigger signal; 2) Using a sliding window mechanism to establish a mapping relationship between the sound source direction estimation and the spatial coordinates; 3) When the teacher's movement speed is detected to exceed 0.5m / s, it automatically switches to motion compensation mode and uses acceleration sensor data to correct the pointing hysteresis of the beamforming; 4) The working mode of the microphone array is dynamically adjusted through the information entropy evaluation mechanism, and the spectral component of chalk friction noise is preferentially suppressed during the blackboard writing period.

Images