An end-cloud collaborative voice interaction method and system suitable for a smart class board

Abstract

The application discloses a terminal-edge-cloud collaborative voice interaction method and system suitable for a smart class board, relates to the technical field of intelligent voice interaction, and comprises a three-level collaborative architecture of a terminal layer, an edge / cloud layer and a service layer. The class board of the terminal layer is provided with an embedded chip and a customized wake-up word engine, and a large model is arranged in a private integrated machine of the edge / cloud layer. The embedded chip and the customized wake-up word engine are used to run a MUSIC algorithm and the terminal layer is provided with an intelligent sensing system to lock the main speaker source opposite the class board. According to the design scheme, the terminal layer is provided with a multi-channel sound source positioning algorithm and an inertial sensing system, spatial orientation prior information is constructed, and the multi-sound source reverberation, far-field reflection and sidelobe interference in a classroom scene are inhibited, so that reliable spatial criteria are obtained for voice activity detection and front-end noise reduction, the audio quality of a voice recognition link is ensured, and the locking precision and collection integrity of a target speaker source in a noisy environment are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent voice interaction technology, and in particular to an edge-cloud collaborative voice interaction method and system suitable for smart class signs. Background Technology

[0002] Currently, smart class signs, as core interactive terminals in campus scenarios, generally adopt traditional interaction methods such as touch screens, card swiping, or facial recognition. With the popularization of artificial intelligence voice interaction technology, introducing it into smart class signs to improve operational convenience and service initiative has become a development trend. However, the campus environment has strict requirements for data security and tends to deploy privately to avoid the leakage of sensitive information. At the same time, the hardware computing power of class sign terminals is limited, making it difficult to handle high-load local voice inference tasks.

[0003] In existing technologies, in noisy classroom scenarios for smart class signs, the superposition of reverberation from multiple sound sources and far-field reflections leads to incomplete echo cancellation. Traditional linear filtering struggles to distinguish between the main speaker's voice facing the device and sidelobe interference, resulting in insufficient accuracy in locating the target sound source during the wake-up phase. Furthermore, the location deviation directly impacts the Voice Activity Detection (VAD) and front-end noise reduction stages, causing the audio stream acquired in a truncated manner to lack key semantic segments. This, in turn, leads to blurred recognition of dialect accents and sentence boundaries during streaming ASR (speech-to-text) decoding. Additionally, the lack of reliable prior knowledge of sound source location makes it difficult to accurately remove interjections and complete omitted subjects. Moreover, when calling multiple business systems such as campus attendance and timetables, the response latency of each business API fluctuates at the second level, and the generation of large model responses and data retrieval are difficult to maintain atomic consistency, potentially resulting in the final broadcast content being based on outdated or non-current-day valid data. Therefore, how to achieve collaborative alignment of sound source, semantic, and business data in the voice interaction of smart class signs under multi-source interference environments is the problem that this invention aims to solve. Summary of the Invention

[0004] In order to overcome the above-mentioned defects in the existing technology, the present invention provides an edge-cloud collaborative voice interaction method and system suitable for smart class signs.

[0005] The technical solution adopted by this invention to solve its technical problem is as follows: Firstly, this invention provides an edge-cloud collaborative voice interaction method suitable for smart class signs, comprising the following steps:

[0006] Step 1: Deploy a three-tier collaborative architecture of terminal layer - edge / cloud layer - business layer. The class sign in the terminal layer is equipped with an embedded chip and a custom wake word engine, and the QWEN-7B / 14B large model is deployed on the private all-in-one machine (equipped with NVIDIA L40S GPU) in the edge / cloud layer.

[0007] Step 2: At the terminal layer, the MUSIC algorithm (multi-signal classification algorithm) and a customized wake word engine are run on the embedded chip. The intelligent sensor system data is integrated to lock the main speaker's voice source facing the class sign. The Speex / WebRTC NNNoise algorithm is used to optimize noise reduction and VAD (voice activity detection) to complete echo cancellation and audio compression, filter out side lobe interference, and output a high signal-to-noise ratio audio stream.

[0008] Step 3: Locally identify simple commands and stream complex audio to the edge. At the same time, use the deployed QWEN-7B / 14B large model to perform parallel ASR (speech-to-text) decoding on the streaming audio, and introduce a BERT-based error correction model to re-rank candidates, remove interjections and complete omitted subjects, and restore sentence boundaries and dialect accent semantics.

[0009] Step 4: Use Natural Language Understanding (NLU) to identify intent, and integrate entity links from the campus knowledge graph through the DIFY workflow to asynchronously prefetch business data from structured business instructions such as campus attendance and class schedules, ensuring that the semantic understanding results are accurately bound to the valid entities of the day;

[0010] Step 5: The QWEN-7B / 14B big model generates a response based on the intent and prefetched data. In simple dialogues, the big model generates the answer directly. In structured business instructions, the prefetched data is integrated to generate accurate broadcast content, and a response is generated based on the intent type branch.

[0011] Step 6: The response content is pre-segmented into TTS audio segments and cached. The edge devices send these segments in sequence, and the terminal layer plays and retrieves them simultaneously. Combined with local simple command pass-through processing, the overall end-to-end response latency is compressed to within 800ms, improving the smoothness of real-time interaction. When the user interrupts the TTS playback, the terminal layer immediately stops the current segment and sends back a new command. The wake-up positioning, ASR correction, data prefetching, and generation links are quickly reset and re-entered, forming a closed-loop collaboration to ensure continuous alignment of sound source-semantic-business data under multi-source interference.

[0012] Furthermore, step 1 specifically includes:

[0013] The terminal layer is configured with the Feiling embedded RK3568 chip as the main control computing unit, equipped with a low-power custom wake-up word engine and set wake-up words, with "Hello Tianzhi" as the wake-up trigger identifier. The custom wake-up word engine is built on a deep convolutional neural network, uses quantized 8-bit weight parameters stored in on-chip SRAM, and sets a dual threshold wake-up decision mechanism. Through on-chip storage and dual threshold mechanism, it can achieve low power consumption and standby at all times, effectively reducing false wake-ups and improving wake-up response reliability.

[0014] The private all-in-one machine is deployed at the edge / cloud layer. It has an integrated NVIDIA L40S GPU and partitions the video memory. The first partition is used for resident inference of the Qwen-7B large model, the second partition is used for dynamic loading of the Qwen-14B large model on demand, and the third partition serves as a model hot-switching buffer. The video memory partition management enables the coexistence of on-demand model switching and resident inference, meeting the different needs of complex semantics and high-frequency dialogue, and improving the efficiency of inference resource utilization.

[0015] In the three-tier collaborative architecture, the terminal layer and the edge layer maintain a long-term connection heartbeat through the MQTT protocol, and the edge layer and the business layer connect through a RESTful API gateway. All layers use the Protobuf serialization format for data exchange, with timestamp and version number fields added. The business layer connects to the campus attendance system, timetable system and notification publishing system through a unified API gateway. The protocol layering and unified serialization format ensure the real-time performance and data consistency of cross-layer communication, facilitating flexible access and collaboration of multiple business systems.

[0016] Furthermore, step 2 specifically includes:

[0017] The MUSIC algorithm runs in real time on the embedded chip. The terminal layer relies on the NEON coprocessor of the embedded chip to accelerate the eigenvalue decomposition operation in the MUSIC algorithm. Taking the phase difference matrix of the six-microphone ring array as input, combined with the three-axis attitude angle of the device provided by the inertial sensing system, the horizontal azimuth and pitch angles of each sound source are calculated. The NEON coprocessor accelerates the positioning operation and combines the inertial navigation data to eliminate the influence of device shaking and improve the accuracy of sound source orientation perception in complex sound fields.

[0018] The customized wake word engine extracts energy peaks from the azimuth spectrum output by the MUSIC algorithm that exceed a preset threshold and whose azimuth angle is within ±25 degrees directly in front of the signboard. o Sound sources within the cone region are used as candidate wake-up sound sources. The positional confidence is weighted and fused based on the acoustic feature matching degree of the wake-up word "Hello Tianzhi". The azimuth cone region screening and acoustic matching weighted fusion effectively suppress sidelobe interference and ensure that only sound sources in the target direction trigger wake-up.

[0019] The intelligent sensing system includes an accelerometer, a gyroscope, and a proximity sensor. The data collected is smoothed by Kalman filtering and then timestamped with the spatial spectrum output of the MUSIC algorithm. The system jointly determines that the main speaker's voice source, which is continuously facing the class sign and is less than 3 meters away, is the target. The timestamping and joint decision of the multi-sensor data ensures that the main speaker's voice source is continuously locked and eliminates interference sources that are far away or briefly intrusive.

[0020] Furthermore, step 2 also includes:

[0021] The Speex algorithm is used to perform linear adaptive filtering on the locked target sound source to eliminate the acoustic echo generated by the device's own speaker. The filter coefficients are updated iteratively with normalized minimum mean square error based on the correlation between the far-end reference signal and the near-end acquired signal.

[0022] The WebRTC NNNoise algorithm takes frequency domain features as input, performs mask estimation of environmental noise in non-target directions through a trained recurrent neural network, multiplies the estimated gain factor with the output signal of the Speex algorithm band by band, and performs residual noise suppression after spatial filtering. The neural network mask estimation accurately distinguishes speech from noise, suppresses residual interference band by band, and improves the clarity of the target speech.

[0023] VAD (Voice Activity Detection) is based on a joint decision of azimuth stability criterion and short-time energy zero-crossing rate for locking onto the target sound source. The azimuth drift must be less than 5° within a continuous 200ms window. o The system activates valid voice markers. Valid voice frames are compressed by the Opus encoder and encapsulated into audio data packets with azimuth metadata. Azimuth stability and energy zero-crossing rate are used for joint judgment to ensure that only directionally stable voice is collected, avoiding accidental triggering by interference sources.

[0024] Furthermore, step 3 specifically includes:

[0025] When recognizing simple commands locally, the terminal layer calls the built-in lightweight ASR engine to perform end-to-end decoding of the audio frames output by the VAD. This lightweight ASR engine is trained based on the CTC loss function, and its recognition results are matched with the preset campus shortcut command whitelist to determine whether local pass-through processing is required.

[0026] Complex audio streams are uploaded to the edge device in real time via a WebSocket full-duplex channel. After receiving the audio stream, the edge device starts the speech encoder of the Qwen-7B / 14B large model to perform block-based (2-second block) parallel CTC / Attention joint decoding, outputting the n-best candidate text sequence and corresponding confidence score. Full-duplex real-time upload and block-based parallel decoding ensure efficient transcription of continuous speech streams and provide multiple candidate support for subsequent accurate semantic understanding.

[0027] The perplexity of the n-best candidate text sequence is calculated using a masked language model based on a BERT-based error correction model. The perplexity is then reordered according to the principle of lowest perplexity. Redundant interjections are removed and omitted subjects are filled in based on the contextual information cached from the previous round of dialogue. A punctuation recovery model is called to restore the sentence boundaries. The corrected text is then output. Intelligent error correction is performed based on contextual information and perplexity ranking, removing redundancy and filling in omissions to restore complete semantics and improve the readability and accuracy of the recognition results.

[0028] Furthermore, step 4 specifically includes:

[0029] The NLU module of the QWEN-7B / 14B large model receives the error-corrected text and performs intent classification through an intent classifier. This intent classifier is based on the fine-tuning of the prompt words of the QWEN-7B / 14B and outputs intent labels and corresponding slot information. The intent labels cover four types of business: timetable query, attendance check-in, notification broadcast, and device control. The fine-tuning of prompt words achieves accurate intent classification and slot extraction, clearly defining the four types of campus business scenarios and providing clear semantic guidance for subsequent instruction execution.

[0030] A campus knowledge graph is constructed based on the DIFY workflow. This campus knowledge graph includes course entities, teacher entities, class entities, classroom entities, and equipment entities, as well as the relationships between entities. The slot values ​​output by NLU are linked and matched with the graph entities to achieve referential resolution. The knowledge graph entity links effectively eliminate ambiguity of slots with the same name or synonym. Combined with the relationships, accurate referential resolution is achieved, avoiding mismatches.

[0031] Based on the intent and entity parameters determined after resolution, the unified API gateway asynchronously and in parallel calls are made to the campus attendance system to query today's attendance status, the timetable system to query today's timetable information, and the notification system to query unread notifications. The prefetched results are stored in the edge Redis cache with the data version number and the valid time period label for the day. Data from multiple business systems is prefetched asynchronously and in parallel with the version and time period labels to ensure that subsequent responses are based on the latest valid data and reduce waiting delays.

[0032] Furthermore, step 4 also includes:

[0033] In the Redis cache at the edge, an expiration period field based on NTP synchronization timestamp is set for the prefetched business data. This expiration period field has an expiration boundary of 23:59:59 of the current day. When the terminal layer initiates a request, the edge first checks the comparison result between the expiration period of the cached data and the current NTP time. The expiration boundary of the current day ensures that the cached data is strictly updated on a daily basis, avoiding business errors caused by cross-day data confusion.

[0034] When the verification finds that the cached data has crossed the day or exceeded the validity period, it forces a callback to the source to call the business system API to pull the latest data, updates the Redis cache content and validity period field, and refreshes the version number marker of the corresponding entity in the knowledge graph based on the timestamp of the newly pulled data. When the cache expires, it forces a callback to the source to refresh and synchronously updates the graph version number to ensure the consistency of multi-source data versions.

[0035] The slot values ​​identified by NLU, the graph entity linking results, and the business data version number are bound into triples to generate a semantic parsing result with entity fingerprints. This semantic parsing result serves as a data reference anchor throughout the subsequent large model generation and TTS broadcasting process.

[0036] Furthermore, step 5 specifically includes:

[0037] The QWEN-7B / 14B large model generates branch routes based on the intent tags output by NLU. If the intent tag belongs to the unstructured dialogue category, it directly drives the large model to generate free text. During generation, length constraints and factual verification rules are enabled. Based on the intent branch routes, the model achieves differentiated generation of dialogue and instructions. Factual verification ensures that the unstructured response content is true and credible.

[0038] If the intent tag belongs to the structured business instruction class, then construct a JSON structured prompt word containing business data fields, inject the pre-fetched Redis cache data into the prompt word context in the form of key-value pairs, require the large model to generate the broadcast text based on the injected data, and force the model to generate the structured prompt word based on the injected data to avoid fabricating business information and ensure that the broadcast content strictly conforms to the actual data.

[0039] The semantic hash value of the response text generated by the large model is calculated for caching and deduplication. At the same time, the entity fingerprint of the response text is associated with the generated triple binding result and stored to ensure that the data source version referenced by the broadcast content is traceable.

[0040] Furthermore, step 6 specifically includes:

[0041] The generated response text is divided into sequentially numbered TTS audio segments at the edge end with a fixed duration (200ms). Each TTS audio segment is independently synthesized using the VITS accelerated acoustic model and stored in the edge buffer. The edge end sends the segments to the terminal layer in sequence through the low-latency RTP protocol. The segment synthesis and buffer sending reduce the broadcast waiting time, and the low-latency protocol ensures real-time continuous output of the audio stream.

[0042] While playing the current segment, the terminal layer fetches the next segment to the local circular buffer in advance to realize the simultaneous playback and fetching mechanism. Simple instructions recognized locally are executed directly without going through the edge end. The combined local passthrough and segment prefetching strategy compresses the response latency of the first segment to within 800ms. Simultaneous playback and fetching and local passthrough are processed in parallel to reduce the interaction waiting time and improve the smoothness of user dialogue.

[0043] The terminal layer VAD continuously listens to the user's voice input. When a new voice triggers an interruption TTS broadcast event, it immediately stops the current segment delivery and clears the circular buffer. It then sends the interruption event and the azimuth angle of the new voice back to the edge end. The edge end cancels the current ASR decoding and large model generation tasks and resets the state machine to the idle state, triggering a re-entry into the wake-up positioning and ASR correction link. The interrupt response quickly clears the old tasks and resets the link, ensuring that the processing of new instructions is not interfered with by residual data, thus improving the continuous dialogue experience.

[0044] Secondly, the present invention also provides an edge-cloud collaborative voice interaction system suitable for smart class signs, used to implement the aforementioned edge-cloud collaborative voice interaction method for smart class signs, comprising a three-level collaborative architecture of terminal layer, edge / cloud layer and business layer, wherein the edge / cloud layer has the following communication connection modules:

[0045] The terminal perception module, deployed on the class sign terminal, is based on an embedded chip and a customized wake word engine. It runs the MUSIC algorithm and integrates inertial navigation data to lock the location of the speaker's voice source. It also works with the Speex / WebRTC NNNoise algorithm to complete echo cancellation and noise reduction, outputting a high signal-to-noise ratio audio stream to enable direct access to simple local commands. It integrates sound source localization and noise reduction algorithms to accurately lock the speaker and enables fast response to high-frequency simple commands.

[0046] The edge inference module is used to perform parallel ASR decoding, semantic correction and intent recognition on the uploaded audio through the deployed Qwen-7B / 14B large model and BERT-based error correction model, to complete high-precision speech-to-semantics conversion. The large model parallel decoding and intelligent correction improve the accuracy of complex speech transcription and restore the complete semantic expression.

[0047] The business collaboration module is used to asynchronously prefetch business data such as attendance and class schedules through entity links in the campus knowledge graph and the DIFY workflow, generate entity fingerprints and bind semantic parsing results to ensure data version traceability. The knowledge graph eliminates semantic ambiguity, and asynchronous prefetching and fingerprint binding ensure business data version traceability.

[0048] The playback response module is used to divide the response into TTS segments and buffer them for delivery. The terminal can play and retrieve them simultaneously. It also supports VAD interrupt detection and fast state machine reset to achieve continuous alignment of sound source, semantics, and data. Segment delivery and fast interrupt reset enable low-latency broadcasting and ensure that new instructions are not contaminated by old data to achieve full-link alignment.

[0049] The beneficial effects of this invention are:

[0050] 1. The design scheme of this invention integrates multi-channel sound source localization algorithm and inertial sensing system at the terminal layer to construct spatial orientation prior information, suppress multi-source reverberation, far-field reflection and sidelobe interference in classroom scene, so that the speech activity detection and front-end noise reduction links obtain reliable spatial criteria, ensure the audio quality of the speech recognition link, and improve the locking accuracy and acquisition integrity of the target speaker's sound source in noisy environment.

[0051] 2. The design scheme of this invention adopts an edge-end collaborative speech recognition mechanism. A large-model speech encoder is deployed at the edge and parallel decoding is performed. Combined with a context-aware semantic error correction model, the recognition candidates are reordered. Historical dialogue context information is used to automatically supplement the omitted subject components in spoken expression and restore the sentence boundaries. This improves the problems of fuzzy dialect accent recognition and redundant interjections, and achieves high-quality conversion from noisy signals to complete semantic text.

[0052] 3. The design scheme of this invention binds the semantic parsing results with the business layer data version number and generates a unique entity fingerprint. It establishes a version verification mechanism in the entire chain of data retrieval, large model generation and voice broadcasting. When data version inconsistency is detected, it automatically triggers rollback and re-prefetching, avoiding the problem of large model replies based on outdated or non-valid data due to business system response delay fluctuations, and ensuring the timeliness and accuracy of broadcast content. Attached Figure Description

[0053] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] Figure 1 This is a schematic diagram illustrating the workflow of an edge-cloud collaborative voice interaction method and system applicable to smart class signs according to the present invention.

[0055] Figure 2 This is a schematic diagram of the method flow for an edge-cloud collaborative voice interaction method applicable to smart class signs according to the present invention. Detailed Implementation

[0056] To more clearly illustrate the technical solution of the present invention, the present invention will be further described below with reference to the accompanying drawings. Obviously, the drawings described below are only one embodiment of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings and embodiments without creative effort, and all of them fall within the protection scope of the present invention.

[0057] Example 1, according to Figure 1 , Figure 2 As shown, an edge-cloud collaborative voice interaction method suitable for smart class signs includes the following steps:

[0058] Step 1: Deploy a three-level collaborative architecture consisting of a terminal layer - edge / cloud layer - service layer. The class signboard in the terminal layer is equipped with an embedded chip and a customized wake word engine, and the QWEN-7B / 14B large model is deployed on the private all-in-one machine (equipped with NVIDIA L40S GPU) in the edge / cloud layer. The three-level collaborative architecture realizes layered computing power, avoids the limitation of terminal computing power, and private deployment ensures that campus data is not leaked externally. The QWEN-7B / 14B large model is an open-source large language model (LLM) developed by Tongyi Laboratory of Alibaba Group, belonging to the Qwen (Tongyi Qianwen) series, where the numbers 7B and 14B represent that the parameter quantities of the model are 7 billion and 14 billion respectively. The terminal layer is configured with the Forlinx embedded RK3568 chip as the main control computing power unit, equipped with a low-power customized wake word engine and set with a wake word, with "Ni Hao Tian Zhi" as the wake-up trigger identifier. This customized wake word engine is built based on a deep convolutional neural network, adopts quantized 8bit weight parameters stored in on-chip SRAM, and is set with a dual-threshold wake-up decision mechanism. Through on-chip storage and the dual-threshold mechanism, low-power on-call readiness is realized, false wake-ups are effectively reduced, and wake-up response reliability is improved. The private all-in-one machine is deployed in the edge / cloud layer, with a built-in NVIDIA L40S GPU and divided video memory partitions, where the first partition is used for resident inference of the Qwen-7B large model, the second partition is used for on-demand dynamic loading of the Qwen-14B large model, and the third partition serves as a model hot-switching buffer. Video memory partition management realizes the coexistence of on-demand model switching and resident inference, meets the different needs of complex semantics and high-frequency dialogue, and improves the utilization efficiency of inference resources. In the three-level collaborative architecture, the terminal layer and the edge layer maintain long-connection heartbeats through the MQTT protocol, the edge layer and the service layer are interconnected through a RESTful API gateway, and all levels uniformly adopt the Protobuf serialization format for data exchange, with additional timestamp and version number fields. The service layer is connected to the campus attendance system, class schedule system and notification release system through a unified API gateway. Protocol layering and unified serialization format ensure the real-time performance and data consistency of cross-level communication, and facilitate the flexible access and collaboration of multi-service systems;

[0059] Specifically, the Feiling embedded RK3568 chip integrates a quad-core ARM Cortex-A55 processor and a NEON coprocessor. The customized wake-up word engine's network structure adopts a deep separable convolutional design. After training, the weight parameters are stored in the on-chip SRAM in 8-bit quantization format, with an allocated capacity of 256KB. To address the issue of false wake-ups in classroom environments, a dual-threshold wake-up decision mechanism is set: the first threshold is a low threshold, used to activate the audio front-end's data acquisition and buffer filling, putting the system into a pre-wake state, but without triggering the main control interrupt, only continuously buffering audio frames; the second threshold is a high threshold, only when the acoustic feature matching degree and spatial position confidence both exceed this threshold is the wake-up state formally triggered, sending a wake-up interrupt signal to the system main control, and starting the subsequent VAD and ASR pipelines. At the same time, the terminal layer integrates a six-microphone ring array and an inertial sensing system; the video memory is managed in three levels of partitions: the first partition is fixedly allocated 16GB for the permanent resident Qwen-7B large model. The inference service handles high-frequency simple dialogues and real-time ASR-assisted decoding tasks, residing in memory without loading latency. The second partition, dynamically allocated 24GB, is used for on-demand loading of the Qwen-14B large model. When the NLU recognizes complex semantics or structured business instructions, the system loads model weights from storage to this partition for inference. Cold start loading latency is ≤3.2 seconds, and hot loading (with cached weights) is ≤800ms. The third partition, allocated 8GB, serves as a model hot-switching buffer. When switching between the 7B and 14B models, zero-copy weight migration is achieved through unified memory management technology, keeping the switching latency within 200ms. In this three-tier collaborative architecture, the terminal layer and edge layer maintain a long-term connection heartbeat using the MQTT protocol, with a heartbeat interval set to 30 seconds. In the event of a network interruption, an exponential backoff reconnection mechanism is triggered (initial backoff 1s, multiplied by 2, maximum backoff 60s). The edge layer and business layer communicate via RESTful... API gateway integration, the gateway is configured with a circuit breaker (circuit breaker when error rate > 50%, recovery wait 5s) and a timeout retry policy (connection timeout 3s, read timeout 5s, retries 2 times, interval 500ms) to prevent downstream business system anomalies from affecting the main voice interaction link. All layers uniformly use Protobuf serialization format for data exchange, and each frame of data is forcibly appended with NTP synchronization timestamp and protocol version number fields.

[0060] Step 2: At the terminal layer, the MUSIC algorithm (multi-signal classification algorithm) and a customized wake-up word engine are run on the embedded chip. Data from the intelligent sensing system is integrated to locate the main speaker's voice facing the class sign, and Speex / WebRTC is used in conjunction with the system. The NNNoise algorithm optimizes noise reduction and VAD (Voice Activity Detection), completing echo cancellation and audio compression, filtering out sidelobe interference, and outputting a high signal-to-noise ratio audio stream. Multi-algorithm fusion accurately locates the main speaker's voice source, filtering out sidelobe interference and echoes, providing high-quality audio input for subsequent recognition. The MUSIC algorithm runs in real-time on the embedded chip. The terminal layer relies on the NEON coprocessor of the embedded chip to accelerate the eigenvalue decomposition calculation in the MUSIC algorithm. Using the phase difference matrix of a six-microphone ring array as input, combined with the device's three-axis attitude angles provided by the inertial sensing system, it calculates the horizontal azimuth and pitch angles of each sound source. The NEON coprocessor accelerates the positioning calculation, combining inertial navigation data to eliminate the influence of device shaking, improving the accuracy of sound source orientation perception in complex sound fields. A customized wake-up word engine extracts energy peaks exceeding a preset threshold and whose azimuth angles are located within ±25 degrees directly in front of the signboard from the azimuth spectrum output by the MUSIC algorithm. o Sound sources within the cone area are used as candidate wake-up sound sources. Based on the acoustic feature matching degree of the wake-up word "Hello Tianzhi", the positional confidence is weighted and fused. The cone area screening and acoustic matching weighted fusion effectively suppress sidelobe interference and ensure that only the sound source in the target direction triggers wake-up. The intelligent sensing system includes an accelerometer, a gyroscope and a proximity sensor. The data collected by the system is smoothed by Kalman filtering and then time-stamped with the spatial spectrum output of the MUSIC algorithm. The main speaker sound source that is continuously facing the class sign and is less than 3 meters away is jointly determined as the target. The time-stamp alignment and joint decision of the multi-sensor data ensure that the main speaker sound source is continuously locked and eliminates interference sound sources that are far away or briefly intruded.

[0061] Specifically, at the terminal layer, after the Feiling embedded RK3568 chip is powered on and initialized, the sampling parameters of the six-microphone ring array are configured via the I2C interface. Six-channel audio data is read in real time. After Hanning windowing, a 4096-point FFT is used to convert the time-domain signal to the frequency domain, constructing a 6×6 cross-power spectral density matrix as input to the MUSIC algorithm. The NEON coprocessor executes the eigenvalue decomposition operation of this matrix in parallel using a single instruction multiple data stream (SID) method, decomposing the covariance matrix into signal and noise subspaces. Then, the horizontal azimuth and pitch angles are calculated through spectral peak search. Meanwhile, the inertial sensing system outputs the device's three-axis attitude angle data at a sampling rate of 200Hz. After quaternion attitude calculation, this data is hard-synchronized with the azimuth output of the MUSIC algorithm to eliminate coordinate system deviations introduced by device shaking. The MUSIC algorithm has an angle resolution of 5. o The spatial spectrum scanning step size is set to 1. o The horizontal azimuth output range is 0. o Up to 360o The pitch angle output range is -90°. o Up to +90 o The custom wake-up word engine extracts sound source locations whose energy peaks exceed the adaptive noise floor plus a 6dB threshold from the azimuth spectrum output by the MUSIC algorithm in each frame, and then filters those with azimuth angles within ±25 degrees of the signboard. o The effective peak value within the cone region determines the sound source that meets the criteria and is included in the candidate wake-up sound source list. For each candidate sound source, a customized wake-up word engine runs a deep separable convolutional network to calculate the cosine similarity between the 80-dimensional log-Mel spectrum features of the audio frame and the locally stored "Hello Tianzhi" template features, obtaining the acoustic feature matching degree. Simultaneously, the peak sharpness of the sound source in the MUSIC spatial spectrum is normalized to the positional confidence. Furthermore, the system sets a dual-threshold decision mechanism, where the first threshold is a combination of an acoustic matching degree of 0.35 and a positional confidence of 0.30, used to activate the audio front-end to enter the pre-wake state and activate the audio circular buffer (starting the holding...). (Continue filling, retaining the audio from 1.2 seconds before wake-up); the second threshold is a combined threshold of acoustic matching degree 0.72 and azimuth confidence 0.80. Only when both reach this high threshold simultaneously will the system send a wake-up interrupt signal to the main controller. The raw data from the accelerometer, gyroscope, and proximity sensor are smoothed by Kalman filtering, and the output device's three-axis attitude angle and target distance estimate are obtained. The process noise covariance Q of the Kalman filter is set to 0.01, and the observation noise covariance R is set to 0.10. The filtered sensor data and the MUSIC spatial spectrum output are aligned with the same timestamp reference. The system jointly determines that the azimuth angle is within ±25°. o Within the cone region, the azimuth drift is <5 within a continuous 200ms window. o A sound source that meets three conditions—a proximity sensor reading of less than 3 meters—is the target speaker;

[0062] It should also be noted that step 2 further includes: using the Speex algorithm to perform linear adaptive filtering on the locked target sound source to eliminate acoustic echoes generated by the device's own speaker. The filter coefficients are iteratively updated based on the correlation between the far-end reference signal and the near-end acquired signal using normalized minimum mean square error. Through adaptive filtering, the echo path changes are tracked in real time, effectively eliminating the device's own broadcast echo and ensuring the purity of the acquired signal. The WebRTC NNNoise algorithm takes frequency domain features as input and performs mask estimation on environmental noise in non-target directions through a trained recurrent neural network. The estimated gain factor is multiplied with the output signal of the Speex algorithm band by band to suppress residual noise after spatial filtering. The neural network mask estimation accurately distinguishes speech from noise, suppresses residual interference band by band, and improves the clarity of the target speech. VAD (Voice Activity Detection) is based on the azimuth stability criterion of the locked target sound source and the short-time energy zero-crossing rate joint decision, where the azimuth drift is less than 5 rpm within a continuous 200ms window. o The valid voice marker is activated at the time. The valid voice frame is compressed by the Opus encoder and encapsulated into an audio data packet with azimuth metadata. The azimuth stability and energy zero-crossing rate are jointly determined to ensure that only azimuth-stable voice is collected, avoiding false triggering by interference sources.

[0063] Specifically, the Speex algorithm, or Speex acoustic echo cancellation algorithm, uses the correlation between the far-end reference signal and the near-end acquired signal as a basis to iteratively update the filter coefficients using normalized minimum mean square error. The filter order is set to 512, the step size factor is 0.3, the leakage factor is 0.001, and the delay compensation range is configured to 0 to 200 milliseconds. When the signboard loudspeaker broadcasts TTS audio, the system captures the broadcast signal in real time as a reference input. Simultaneously, a six-microphone array acquires the near-end signal containing echo components. Through frame-by-frame iteration of the adaptive filter, the filter coefficients converge to a steady state within approximately 300 milliseconds. The estimated echo signal is then subtracted from the near-end acquired signal, reducing the residual echo power to below 0.3% of the original echo. The echo cancellation amount stably reaches over 25 dB, ensuring the purity of the locked target sound source. WebRTC The NNNoise algorithm, also known as the WebRTC neural network noise suppression algorithm, performs secondary suppression on residual non-target direction environmental noise in the Speex output signal. Using 20-millisecond audio frames as processing units, it extracts amplitude spectrum features across 128 frequency bands after Fast Fourier Transform (FFT), which are then input into a pre-trained recurrent neural network for mask estimation. A two-layer gated recurrent unit structure is employed, with a hidden layer dimension of 128 and approximately 0.8M parameters. After 8-bit quantization, the features are deployed at the edge for inference. The network outputs gain factors for each frequency band, representing the retention ratio of the target speech at each frequency point. These gain factors are multiplied band-by-band by the frequency domain amplitude spectrum of the Speex algorithm output signal to finely suppress residual noise after spatial filtering. The denoised audio stream then enters the speech activity detection module. The system comprehensively judges based on the azimuth stability criterion and the short-time energy zero-crossing rate. The azimuth stability criterion requires that the horizontal azimuth drift of the target sound source be less than 5 degrees within a continuous 200-millisecond time window, ensuring that the detected voice activity originates from a stable speaker in the locked direction, rather than a briefly intruding interference source. The short-time energy zero-crossing rate detection uses a frame length of 10 milliseconds and calculates the number of times the signal crosses the zero level within the frame. The judgment threshold is set to an energy value that is 3 dB higher than the background noise and a zero-crossing rate between 20 and 80 times per 10 milliseconds. Only when both the azimuth stability condition and the energy zero-crossing rate condition are met simultaneously, a valid high signal-to-noise ratio audio stream is output. The valid high signal-to-noise ratio audio stream is compressed by the Opus encoder with a bit rate set to 16 kilobits per second and a frame length of 20 milliseconds. After compression, it is encapsulated into an audio data packet.

[0064] Step 3: Locally recognize simple commands, and stream complex audio to the edge. Simultaneously, utilize the deployed QWEN-7B / 14B large model to perform parallel ASR (speech-to-text) decoding on the streaming audio. Introduce a BERT-based error correction model for candidate reordering, remove interjections and complete omitted subjects, restore sentence boundaries and dialectal semantics. Parallel decoding and error correction reordering can improve the accuracy of complex speech recognition, complete omitted components, and restore complete semantic expression. When recognizing simple commands locally, the terminal layer calls the built-in lightweight ASR engine to perform end-to-end decoding of the audio frames output by the VAD. This lightweight ASR engine is trained based on the CTC loss function. Its recognition results are fuzzy matched with a pre-set whitelist of campus shortcut commands to determine whether to perform local pass-through processing. Local lightweight decoding and fuzzy matching enable fast command response. High-frequency and simple operations do not need to be uploaded to the cloud, significantly reducing interaction latency. Complex audio streams are uploaded to the edge terminal in real time via a WebSocket full-duplex channel. After receiving the audio stream, the edge terminal starts the Qwen-7B / 14B large model speech encoder to perform CTC / Attenti parallel processing on the audio stream in blocks (2 seconds each). The system uses joint decoding to output the n-best candidate text sequence and its corresponding confidence score. Full-duplex real-time uploading and block-parallel decoding ensure efficient transcription of continuous speech streams, providing multiple candidate support for subsequent accurate semantic understanding. The system uses a BERT-based error correction model to perform masked language model perplexity calculation on the n-best candidate text sequence, re-sorts it according to the principle of lowest perplexity, removes redundant interjections and completes omitted subjects based on the contextual information cached from the previous round of dialogue, calls the punctuation recovery model to restore sentence boundaries, and outputs the corrected text. Intelligent error correction is performed based on contextual information and perplexity ranking, removing redundancy and completing omissions to restore complete semantics and improve the readability and accuracy of the recognition results.

[0065] Specifically, the lightweight ASR engine built into the terminal layer adopts an end-to-end speech recognition architecture, trained based on the connection-time classification loss function. The model has 18M parameters, which are quantized to 8 bits and deployed on the NPU coprocessor of the RK3568 chip. This lightweight ASR engine decodes the high signal-to-noise ratio audio frames output by the speech activity detection module in real time. The decoding frame length is 20 milliseconds, and the output is the recognized text and the corresponding confidence score. The terminal layer has a pre-set whitelist of campus shortcut commands, which includes fixed command sets such as "open class schedule", "check-in", "page turn", "return to main interface", and "refresh notification". The total number of entries in the whitelist does not exceed 50. Each command is stored in the local Flash in plain text form. After the recognition result is output, the system calls the string fuzzy matching algorithm to calculate the edit distance between the recognized text and each entry in the whitelist. The edit distance threshold is set to 2, that is, the matching tolerance is based on a maximum of two character insertion, deletion or replacement operations. When the match is successful and the recognition confidence score is higher than 0, the system will perform a matching operation.At 85%, simple instructions deemed locally processable are executed directly at the terminal layer without uploading to the edge. When matching fails or the confidence level falls below a threshold, the system marks the current audio stream as a complex audio stream, triggering subsequent upload processing. For complex audio streams that cannot be matched locally, the terminal layer uploads them to the edge in real-time via a WebSocket full-duplex channel. The upload frame length is consistent with the encoding frame length at 20 milliseconds. Every 100 frames (2 seconds) of accumulated audio data are packaged into a transmission block for transmission to reduce network overhead. Upon receiving the audio transmission block, the edge layer activates the Qwen-7B large-scale model's speech encoder module, which is based on W... The hisper architecture is adapted to handle 2-second audio blocks, which are then framed with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. 80-dimensional log-Mel spectrum features are extracted. After the feature sequence is fed into the encoder, two parallel paths, CTC decoding and Attention decoding, are executed simultaneously. The CTC decoding path outputs character-level alignment results based on the independence hypothesis, while the Attention decoding path outputs sequence-level results based on the global context. The two paths are probabilistically fused using a joint decoding algorithm to output an n-best candidate text sequence, where n is set to 5. Each candidate text is accompanied by a corresponding confidence score, calculated based on the normalized log-likelihood of the path probabilities. The n-best candidate text sequence is then further processed. The n-best candidate text sequences and their confidence scores are fed into subsequent error correction processing. The error correction process loads a BERT-based error correction model, employing the Chinese RoBERTa-wwm-ext architecture with 6 layers and a hidden layer dimension of 384. After FP16 precision quantization, it is deployed at the edge. The perplexity of the masking language model is calculated for each of the n-best candidate text sequences. Each word in each candidate text is replaced sequentially with a [MASK] marker. The BERT-based error correction model predicts the probability distribution of the masked positions, using the negative logarithmic mean of the predicted probabilities as the perplexity measure. The candidate with the lowest perplexity is selected as the initial text for further processing. The contextual information from the previous round of dialogue is extracted. This cache is stored in Redis at the edge, indexed by the session identifier, and saves the complete semantic units of the most recent three rounds of dialogue. Based on the subject entity, time adverbial, and place adverbial in the contextual information, the initially selected text is supplemented with omitted components. That is, when a sentence is detected to be missing a noun subject at the beginning and there is a unique active subject in the context, the subject is automatically inserted at the beginning of the sentence. At the same time, the punctuation recovery model is invoked. The punctuation recovery model is a bidirectional long short-term memory network structure with a hidden layer dimension of 256. After training, it outputs the predicted position and type of four punctuation marks: comma, period, question mark, and exclamation mark for text without punctuation. After processing, the corrected complete semantic text is output.

[0066] Step 4: Use Natural Language Understanding (NLU) to identify intent. Through the DIFY workflow, integrate entity links from the campus knowledge graph and asynchronously prefetch business data from structured business instructions such as campus attendance and class schedules. This ensures that the semantic understanding results are accurately bound to the valid entities of the day, avoiding the introduction of ambiguity and expired information. Knowledge graph entity links eliminate semantic ambiguity, and asynchronous prefetching ensures the timeliness of business data and avoids misleading information from expired information.

[0067] Step 5: The QWEN-7B / 14B large model generates responses based on intent and prefetched data. Simple dialogues are answered directly by the large model. Structured business instructions are integrated with prefetched data to generate accurate broadcast content. Responses are generated based on intent type branches. Unstructured dialogues are flexible and natural. Structured instructions are integrated with data to ensure accurate and reliable broadcast content.

[0068] Step 6: The response content is pre-segmented into TTS audio segments and cached. These segments are then sent sequentially from the edge and retrieved while playing at the terminal. Combined with local simple command pass-through processing, the overall end-to-end response latency is compressed to within 800ms, improving the smoothness of real-time interaction. When the user interrupts the TTS playback, the terminal immediately stops the current segment and sends back a new command. The wake-up positioning, ASR correction, data prefetching, and generation links are quickly reset and re-entered, forming a closed-loop collaboration. This ensures the continuous alignment of sound source-semantic-business data under multi-source interference. The segmentation caching and simultaneous playback and retrieval mechanism effectively reduce response latency, and the rapid interruption reset ensures that new commands are not contaminated by old data. The end-to-end alignment guarantees the continuity of interaction.

[0069] Example 2, according to Figure 1 , Figure 2As shown, based on Embodiment 1, this invention provides a technical solution: Step 4 specifically includes: the NLU module of the QWEN-7B / 14B large model receives the corrected text, performs intent classification through an intent classifier, which is implemented based on the QWEN-7B / 14B prompt word fine-tuning, and outputs intent tags and corresponding slot information. The intent tags cover four business types: timetable query, attendance check-in, notification broadcast, and device control. Prompt word fine-tuning achieves accurate intent classification and slot extraction, clearly defining the four types of campus business scenarios, providing clear semantic guidance for subsequent instruction execution, and constructing a campus knowledge graph based on the DIFY workflow. This campus knowledge graph includes course entities, teacher entities, class entities, and Classroom entities, equipment entities, and their interrelationships are linked and matched with NLU output slot values ​​to achieve referential resolution. Knowledge graph entity links effectively eliminate ambiguity for slots with the same name or synonyms. Combined with the interrelationships, accurate referential resolution is achieved, avoiding mismatches. Based on the intent and entity parameters determined after resolution, the unified API gateway asynchronously and in parallel calls are made to the campus attendance system to query today's attendance status, the timetable system to query today's timetable information, and the notification system to query unread notifications. The prefetched results are stored in the edge Redis cache with a data version number and a valid time period label for the day. Data from multiple business systems is prefetched asynchronously and in parallel, with version and time period labels, to ensure that subsequent responses are based on the latest valid data and reduce waiting delays.

[0070] Specifically, the NLU module of the Qwen-7B / 14B large model receives the corrected complete semantic text output from step 3 and performs intent classification. In the fine-tuning stage of the Qwen-7B / 14B prompt words, 20 Few-shot examples cover four business scenarios: timetable query, attendance check-in, notification broadcast, and device control. The intent classifier outputs intent labels and corresponding slot information. Slot extraction uses a sequence labeling method based on BIO annotation, and the extracted slot types include time (formatted as YYYY-MM-DD). The system uses the following tokens for each class: HH:MM, class name (matching the campus class code table), teacher name (matching the teacher ID database), classroom number (matching the three-level code of teaching building-floor-room number), and class sign ID. During model inference, the maximum output sequence length is limited to 128 tokens. When the intent classification confidence score is below 0.80, the system will trigger a proactive clarification dialogue, generating a confirmatory question from the large model. After user confirmation, the subsequent process continues. The campus knowledge graph is stored using the Neo4j graph database, containing five types of entity nodes: courses, teachers, classes, classrooms, and equipment, and four types of relationship edges: "teaching," "belonging," "located in," and "associated with." The slot values ​​output by NLU are matched through graph entity links, and the slot strings are normalized. The system executes a fuzzy matching algorithm (with the edit distance threshold set to 1) to compare the entity names in the graph. When multiple candidate entities are matched, disambiguation is performed based on the class or teacher information cached in the current session context. After the entity link is completed, the system uses the unified API gateway to asynchronously and in parallel call the campus attendance system to query today's attendance status (signed / not signed / leave / late), the timetable system to query today's timetable information (period, course name, teacher, classroom), and the notification system to query unread notifications (title, content, publication time, publishing department) based on the determined intent label and entity parameters. The prefetched results are stored in the edge Redis cache, along with the data version number and validity period fields. The validity period is based on the NTP synchronization time.

[0071] It should be noted that step 4 further includes: setting an expiration date field based on NTP synchronization timestamp for the pre-fetched business data in the edge Redis cache. This expiration date field has an expiration boundary of 23:59:59 of the current day. When the terminal layer initiates a request, the edge first verifies the comparison result between the expiration date of the cached data and the current NTP time, using the current day as the expiration boundary to ensure that the cached data is strictly updated daily, avoiding business errors caused by cross-day data confusion. When the verification finds that the cached data has crossed the day or exceeded the expiration date, it forces a callback to the source to call the business system API to pull the latest data and update the Redis cache. The system caches the content and expiration date fields, and refreshes the version number marker of the corresponding entity in the knowledge graph based on the timestamp of the newly fetched data. When the cache expires, it forces a refresh from the source and updates the graph version number synchronously to ensure consistency of multi-source data versions. It binds the slot value identified by NLU, the graph entity link result, and the business data version number into a triplet to generate a semantic parsing result with entity fingerprint. This semantic parsing result serves as the data reference anchor point throughout the subsequent large model generation and TTS broadcasting. It generates a unique fingerprint as the end-to-end data anchor point to ensure that the referenced version in subsequent generation and broadcasting is traceable and avoids data mismatch.

[0072] Specifically, the edge device first calls the NTP client to obtain the current synchronization timestamp and compares it with the cache expiration field in the cached data. The comparison logic uses a Unix timestamp second-level comparison. If the current timestamp is greater than the cache expiration field, the cache is considered invalid. If the cache is valid, the cached data is directly read for subsequent processing. If the cache is invalid, the edge device triggers a forced origin pull process. During the origin pull, the edge device initiates HTTP requests in parallel to the campus attendance system, timetable system, and notification system through the unified API gateway based on the intent label and entity parameters identified by NLU. GET requests have a timeout of 3000 milliseconds and a retry policy of retrying every 500 milliseconds, with a maximum of two retries. The JSON responses returned by each business system are parsed by the gateway layer, extracting business data fields and recording the NTP timestamp of the data retrieval as the time base for the new data version number. After a successful forced origin pull, the edge device writes the newly retrieved business data to Redis using a hash structure. The write operation uses an atomic combination of HSET and EXPIRE commands, setting the cache lifetime to the remaining seconds of the day plus a 60-second buffer to prevent instantaneous cache avalanche across days. Simultaneously, the edge device calls the Neo4j graph database update interface to refresh the business data version number field of the corresponding entity node in the knowledge graph based on the timestamp of the newly retrieved data. The version number is stored as a long integer, consistent with the cached version number in Redis. The entity node version number update uses pessimistic locking. The system uses the MATCH clause to lock nodes and then executes the SET operation to ensure the atomicity and consistency of version number updates in concurrent scenarios. Triple binding is completed in the memory buffer at the edge. During the binding process, the fields such as time, class, teacher, and classroom in the slot are mapped one-to-one with the entity UUID returned by the graph. At the same time, the version number field cached in Redis is attached. The entity fingerprint is calculated using the SHA-256 hash algorithm, and the output 256-bit hash value is converted into a 64-bit hexadecimal string as the unique identifier of the semantic parsing result. When the large model generates a response, the system embeds the entity fingerprint into the metadata field of the generation context. The TTS broadcast module also writes the fingerprint into the user-defined field of the audio segment when synthesizing speech. All modules in the entire chain use this fingerprint to verify whether the version of the referenced data source is consistent. When a fingerprint mismatch is detected, the system triggers a data consistency alarm and rolls back to the re-prefetch process.

[0073] Step 5 specifically includes: The QWEN-7B / 14B large model generates branch routes based on the intent tags output by NLU. If the intent tag belongs to the unstructured dialogue category, the large model is directly driven to generate free text. During generation, length constraints and factual verification rules are enabled. Based on the intent branch routes, the dialogue and instructions are differentiated and generated. Factual verification ensures that the unstructured response content is true and credible. If the intent tag belongs to the structured business instruction category, a JSON structured prompt word containing business data fields is constructed. The pre-fetched Redis cached data is injected into the prompt word context in the form of key-value pairs. The large model is required to generate broadcast text based on the injected data. The structured prompt word forces the model to generate based on the injected data to avoid fabricating business information and ensure that the broadcast content strictly conforms to the actual data. The semantic hash value of the response text generated by the large model is calculated for cache deduplication. At the same time, the entity fingerprint of the response text and the generated triplet binding result is associated and stored to ensure that the data source version referenced by the broadcast content is traceable. Semantic hash deduplication reduces the overhead of repeated synthesis. Fingerprint association storage ensures that each broadcast can trace its data source version.

[0074] Specifically, after the edge-end NLU module outputs the intent label, the generation and routing module of the QWEN-7B / 14B large model performs branch decisions based on the label. For unstructured dialogue intents, it directly drives the large model to generate free text, with length constraints enabled during the generation process, setting the maximum output length to 64 tokens. Simultaneously, factual verification rules are enabled, cross-validating numerical information in the generated content based on an external knowledge base. When fictitious values ​​are detected, regeneration is automatically triggered. For structured business instruction intents, a JSON structured prompt containing business data fields is constructed. For each word, pre-fetched and stored Redis cached timetables, attendance, and notification data are injected into the context of the prompt word as key-value pairs. During decoding, the attention mechanism of the large model prioritizes the injected data fields. After the large model generates the response text, it calls semantic hashing to perform a similarity hash operation on the response text, outputting a 64-bit binary hash value. When the Hamming distance between the newly generated response hash value and the existing response hash value in Redis is less than 3, it is determined to be a duplicate response, and the cached content is directly reused without re-performing TTS synthesis. At the same time, the response text is associated with the entity fingerprint of the generated triple binding result. The storage structure consists of a response record table in an SQLite database at the edge. When subsequent requests arrive within the same session, fingerprint comparison quickly determines whether the current business data version matches the version used when generating the response. If the fingerprints match, the existing response is used directly; otherwise, regeneration is triggered to ensure that the broadcast content is always based on the latest data. When the terminal layer's TTS broadcast module synthesizes speech, the entity fingerprint corresponding to the current response is written into the extended header field of the RTP audio segment. The entity fingerprint forms a one-to-one correspondence with the knowledge graph node version number and the Redis cache version number. The monitoring module periodically executes fingerprint consistency verification tasks. After each complete dialogue round, the verification program reads the entity fingerprint field stored in SQLite and compares it with the business data version number field of the corresponding node in Neo4j and the data version number field of the cached data in Redis. When all three are completely consistent, the data link is considered to be normal. If any version number is found to be inconsistent, an alarm event is sent to the operation and maintenance system, and the rollback mechanism of the current session is triggered, the Redis temporary cache of the session is cleared, and the business data retrieval, graph version refresh and response generation process is forced to be re-executed.

[0075] Step 6 specifically includes: dividing the generated response text into sequentially numbered TTS audio segments at the edge end with a fixed duration (200ms); each TTS audio segment is independently synthesized using a VITS accelerated acoustic model and stored in the edge buffer; the edge end sequentially sends the segments to the terminal layer via a low-latency RTP protocol; segment synthesis and buffer sending reduce playback waiting time; the low-latency protocol ensures real-time continuous output of the audio stream; the terminal layer, while playing the current segment, pre-fetches the next segment to the local circular buffer to achieve a simultaneous playback and fetching mechanism; simple instructions recognized locally are executed directly without going through the edge end; and a combination of local passthrough and segment prefetching strategies is implemented. The initial response latency is slightly compressed to within 800ms. Simultaneous playback and retrieval are processed in parallel with local passthrough, reducing interaction waiting time and improving the smoothness of user dialogue. The terminal layer VAD continuously listens to the user's voice input. When a new voice triggers an interruption TTS playback event, the current segment distribution is immediately stopped and the circular buffer is cleared. The interruption event and the azimuth angle of the new voice are sent back to the edge end. The edge end cancels the current ASR decoding and large model generation tasks and resets the state machine to the idle state, triggering the re-entry of the wake-up positioning and ASR correction link. The interruption response quickly clears the old tasks and resets the link to ensure that the processing of new instructions is not interfered with by residual data, thus improving the continuous dialogue experience.

[0076] Specifically, after the response text is generated at the edge, the system immediately initiates the TTS synthesis process. The edge segment divides the response text into 200ms duration units, with each segment corresponding to approximately 3 to 5 Chinese characters of speech duration. Each segment independently calls the VITS accelerated acoustic model for speech synthesis. The VITS accelerated acoustic model uses forward TTA inference acceleration technology, and the synthesis time for a single segment is controlled within 35ms. The synthesized audio segments are temporarily stored in the edge buffer in PCM format, with a buffer capacity of 200 segments and a lifespan of 60 seconds for each segment. The edge sends the segments sequentially to the terminal layer via the low-latency RTP protocol, with each RTP session using a 50ms jitter buffer configuration. The system includes additional segment sequence number, total segment count, and timestamp information. Upon receiving the first segment, the terminal layer initiates speaker playback. Simultaneously, a background thread continues receiving subsequent segments and filling them into a local circular buffer. The local circular buffer has a capacity of 8 segments. A dual-pointer mechanism ensures seamless transition between playback and prefetching. For simple commands recognized locally by the terminal layer, the corresponding functional module is directly invoked for execution, bypassing edge processing paths. During TTS playback, the terminal layer continuously runs the VAD detection module, performing zero-crossing rate analysis on the microphone signal at 10ms frames. Simultaneously, it combines the azimuth stability criterion output by the MUSIC algorithm to monitor new user voice input in real time. When the VAD detects valid voice activity and the azimuth angle of the sound source is within ±25 degrees directly in front of the signboard... oWhen the terminal is within the cone region, it determines that the user has initiated an interrupt request. The terminal layer stops receiving and decoding the current RTP fragment, clears all cached but unplayed fragments in the local circular buffer, and sends an interrupt event message back to the edge terminal via the MQTT channel. This message includes the azimuth angle of the new speech, the timestamp, and the current session identifier. Upon receiving the interrupt event, the edge terminal calls the task cancellation token to terminate the currently executing ASR decoding task and large model generation task, and resets the state machine of the corresponding session from the broadcast state to the idle state. At the same time, it clears the temporary cached data of the session at the edge terminal, including the queue of unfinished fragments and the generation context to be processed. After the state machine is reset, the edge terminal retains the historical context cache of the session, including the semantic units and active subject information of the last three rounds of dialogue, and then re-triggers the data. The wake-up localization and ASR correction link is initiated. The terminal layer re-executes MUSIC sound source localization and dual-threshold wake-up decision to lock the speaker's location corresponding to the new voice. The edge end re-executes parallel CTC / Attention decoding and BERT-based error correction on the newly uploaded audio stream. At the same time, based on the intent label and entity parameters extracted after the reset, the business data cache verification and forced return-to-source process are triggered again. This ensures that the subsequent large model generation and TTS broadcast are based on the latest data. Event logs are recorded for each state switch. Through the rapid interruption response and link reset mechanism, continuous alignment of sound source location, semantic parsing and business data is achieved in scenarios where users continuously interrupt and ask questions again. This avoids broadcast errors caused by old instruction residues or inconsistent data versions.

[0077] Example 3, according to Figure 1 As shown, based on Embodiments 1 and 2, the present invention also provides an edge-cloud collaborative voice interaction system suitable for smart class signs, used to implement the above-mentioned edge-cloud collaborative voice interaction method suitable for smart class signs, including a three-level collaborative architecture of terminal layer, edge / cloud layer and business layer, wherein the edge / cloud layer has the following communication connection modules:

[0078] The terminal sensing module, deployed on the class sign terminal, is based on an embedded chip and a custom wake-up word engine. It uses a MUSIC algorithm fused with inertial navigation data to locate the speaker's voice source and combines it with the Speex / WebRTC NNNoise algorithm for echo cancellation and noise reduction. This outputs a high signal-to-noise ratio audio stream for simple local command pass-through. The module is also deployed on the class sign terminal, using the Feiling embedded RK3568 chip as the main control computing unit. It integrates a six-microphone ring array and an inertial sensing system, and features a custom wake-up word engine. This engine is built on a deep separable convolutional network, uses 8-bit quantized weights stored in on-chip SRAM, and employs a dual-threshold wake-up decision mechanism. Using "Hello Tianzhi" as the wake-up trigger, it runs a MUSIC algorithm fused with inertial navigation data to locate the speaker's voice source facing the class sign, and combines the Speex acoustic echo cancellation algorithm with WebRTC... The NNNoise neural network noise suppression algorithm completes echo cancellation and noise reduction. The VAD module outputs a high signal-to-noise ratio audio stream through the joint judgment of azimuth stability criterion and short-time energy zero crossing rate. The built-in lightweight ASR engine decodes the audio stream in real time and performs fuzzy matching with the preset campus shortcut command whitelist to achieve local simple command pass-through processing. Complex audio streams are uploaded in real time through WebSocket full-duplex channel.

[0079] The edge inference module is used to perform parallel ASR decoding, semantic correction, and intent recognition on uploaded audio using the deployed Qwen-7B / 14B large model and BERT-based error correction model, achieving high-precision speech-to-semantics conversion. Deployed on a private all-in-one machine at the edge / cloud layer, it is equipped with an NVIDIA L40S GPU with three-level partitioning of the video memory, which is used for resident inference of the Qwen-7B large model, on-demand loading of the Qwen-14B large model, and hot switching of the model. It is used to perform parallel streaming ASR decoding of uploaded audio using the speech encoder of the deployed Qwen-7B / 14B large model, outputting the n-best candidate text sequence. The candidate text is then processed by the BERT-based error correction model to calculate perplexity and reorder the masked language model. Based on the context information cached from the previous round of dialogue, it removes interjections, completes omitted subjects, and calls the punctuation recovery model to restore sentence boundaries. At the same time, the NLU module of the Qwen-7B / 14B large model is used for intent classification and slot extraction, achieving high-precision speech-to-semantics conversion.

[0080] The business collaboration module is used to asynchronously prefetch business data from structured business instructions such as attendance and timetables through entity links in the campus knowledge graph and the DIFY workflow. It generates entity fingerprints and binds them to semantic parsing results. The campus knowledge graph is stored in the Neo4j graph database and includes five types of entity nodes: courses, teachers, classes, classrooms, and equipment, as well as the relationships between entities. Fuzzy matching and disambiguation are performed on the slot values ​​output by NLU. Based on the determined intent labels and entity parameters, business data from the campus attendance system, timetable system, and notification system are asynchronously and in parallel prefetched through the unified API gateway. The data is stored in the edge Redis cache and a validity period field based on the NTP synchronization timestamp is set. When the cache expires, a forced return process is triggered to refresh the data. At the same time, the slot value, graph entity UUID, and business data version number are bound as a triple. A 64-bit hexadecimal entity fingerprint is generated by the SHA-256 hash algorithm as a unique identifier for the semantic parsing results, ensuring that the data version is traceable.

[0081] The playback response module is used to segment the response into TTS segments and cache them for delivery. The terminal can retrieve the audio while playing the response, and it supports VAD interruption detection and fast state machine reset. The response text generated by the Qwen-7B / 14B large model is segmented into TTS audio segments with a duration of 200ms. Each segment is independently synthesized by the VITS accelerated acoustic model and cached in PCM format at the edge. It is then sent to the terminal layer in sequence via the low-latency RTP protocol. The terminal layer uses a dual-pointer circular buffer to implement the playback-while-retrieve mechanism, compressing the end-to-end response latency to within 800ms. The playback response module includes an interruption detection submodule, which continuously monitors the user's voice input through the terminal layer VAD during TTS playback. Combined with the MUSIC azimuth stability criterion, when valid voice activity is detected and the azimuth angle of the sound source is within ±25 degrees in front of the signboard, the interruption detection is performed. o When an interrupt request is detected within the cone region, the terminal layer stops the current RTP fragment reception and clears the circular buffer. The interrupt event is sent back to the edge end via MQTT. The edge end calls the task cancellation token to terminate the current ASR decoding and large model generation task, resets the state machine to the idle state and clears the temporary buffer. After retaining the historical context buffer, the wake-up positioning, ASR correction, business data prefetching and response generation chain are retried to achieve continuous alignment of sound source-semantics-business data.

[0082] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A collaborative voice interaction method between the edge and cloud for smart class signs, characterized in that, Includes the following steps: Step 1: Deploy a three-tier collaborative architecture of terminal layer - edge / cloud layer - business layer. The class sign in the terminal layer is equipped with an embedded chip and a customized wake word engine, and a large model is deployed on the private all-in-one machine in the edge / cloud layer. Step 2: At the terminal layer, the MUSIC algorithm and custom wake word engine are run on the embedded chip. The intelligent sensor system data is integrated to lock the main speaker's voice source facing the class sign. The Speex / WebRTC NNNoise algorithm is used to optimize noise reduction and VAD, complete echo cancellation and audio compression, and output a high signal-to-noise ratio audio stream. Step 3: Locally identify simple commands, and upload complex audio streams to the edge. Utilize the deployed large model to perform parallel ASR decoding on the streaming audio, and introduce a BERT-based error correction model for candidate reordering, removing interjections and completing omitted subjects. Step 4: Use natural language understanding to identify intent, integrate entity links from the campus knowledge graph through the DIFY workflow, and asynchronously prefetch business data of structured business instructions; Step 5: The big model generates a response based on the intent and prefetched data. For simple dialogues, the big model generates the answer directly, while for structured business instructions, the prefetched data is integrated to generate accurate broadcast content. Step 6: The response content is pre-segmented into TTS audio segments and cached. The edge end sends them out in sequence, and the terminal layer plays and retrieves them simultaneously. When the user interrupts the TTS playback, the terminal layer immediately stops the current segment and sends back a new instruction. The wake-up positioning, ASR correction, data prefetching and generation links are quickly reset and re-entered to form a closed-loop collaboration.

2. The edge-cloud collaborative voice interaction method for smart class signs according to claim 1, characterized in that, Step 1 specifically includes: The terminal layer is configured with an embedded RK3568 chip as the main control computing unit, and is equipped with a low-power custom wake-up word engine and a wake-up word is set. The custom wake-up word engine is built based on a deep convolutional neural network and a dual threshold wake-up decision mechanism is set. The private all-in-one machine is deployed at the edge / cloud layer. It has an NVIDIA L40S GPU built in and the video memory is divided into partitions. The first partition is used for resident inference of the Qwen-7B large model, the second partition is used for dynamic loading of the Qwen-14B large model on demand, and the third partition is used as a model hot-swapping buffer. The terminal layer and the edge layer maintain a long-term connection heartbeat through the MQTT protocol, and the edge layer and the business layer connect through a RESTful API gateway. Data is exchanged between the layers using the Protobuf serialization format, with timestamp and version number fields added. The business layer connects to the campus attendance system, timetable system and notification publishing system through a unified API gateway.

3. The edge-cloud collaborative voice interaction method for smart class signs according to claim 1, characterized in that, Step 2 specifically includes: The MUSIC algorithm runs in real time on the embedded chip. The terminal layer relies on the NEON coprocessor of the embedded chip to accelerate the eigenvalue decomposition operation in the MUSIC algorithm. The phase difference matrix of the six-microphone ring array is used as input, and combined with the three-axis attitude angles of the device provided by the inertial sensing system, the horizontal azimuth and pitch angles of each sound source are calculated. The customized wake word engine extracts energy peaks from the azimuth spectrum output by the MUSIC algorithm that exceed a preset threshold and whose azimuth angle is within ±25 degrees directly in front of the signboard. o Sound sources within the cone region are used as candidate wake-up sound sources, and the positional reliability of the fusion method is weighted based on the acoustic feature matching degree of the wake-up word. The intelligent sensing system includes an accelerometer, a gyroscope, and a proximity sensor. The data it collects is smoothed by Kalman filtering and then timestamped with the spatial spectrum output of the MUSIC algorithm. Together, they determine that the main speaker who is continuously facing the class sign and is less than 3 meters away is the target.

4. The edge-cloud collaborative voice interaction method for smart class signs according to claim 3, characterized in that, Step 2 also includes: The Speex algorithm is used to perform linear adaptive filtering on the locked target sound source to eliminate acoustic echoes generated by the device's own speaker broadcast. The WebRTC NNNoise algorithm takes frequency domain features as input, performs mask estimation of environmental noise in non-target directions through a trained recurrent neural network, multiplies the estimated gain factor with the output signal of the Speex algorithm band by band, and performs residual noise suppression after spatial filtering. VAD is based on a joint decision of azimuth stability criterion for locking onto the target sound source and short-time energy zero-crossing rate, wherein the azimuth drift is less than 5° within a continuous 200ms window. o When the valid voice marker is activated, the valid voice frame is compressed by the Opus encoder and encapsulated into an audio data packet with location metadata.

5. The edge-cloud collaborative voice interaction method for smart class signs according to claim 1, characterized in that, Step 3 specifically includes: When recognizing simple commands locally, the terminal layer calls the built-in lightweight ASR engine to perform end-to-end decoding of the audio frames output by the VAD. The recognition result is then matched with the preset campus shortcut command whitelist to determine whether local pass-through processing is required. The complex audio stream is uploaded to the edge terminal in real time via a WebSocket full-duplex channel. After receiving the audio stream, the edge terminal starts the speech encoder of the large model to perform block-parallel joint decoding and output the n-best candidate text sequence and the corresponding confidence score. The perplexity of the n-best candidate text sequence is calculated using a masked language model based on a BERT-based error correction model. The perplexity is then reordered according to the principle of lowest perplexity. Redundant interjections are removed and omitted subjects are filled in based on the contextual information cached from the previous round of dialogue. The punctuation recovery model is called to restore the sentence boundaries and output the corrected text.

6. The edge-cloud collaborative voice interaction method for smart class signs according to claim 1, characterized in that, Step 4 specifically includes: The large model's NLU module receives the corrected text, classifies the intents through an intent classifier, and outputs intent labels and corresponding slot information. The intent labels cover four types of services: timetable query, attendance check-in, notification broadcast, and device control. A campus knowledge graph is constructed based on the DIFY workflow. This campus knowledge graph includes course entities, teacher entities, class entities, classroom entities, and equipment entities, as well as the relationships between entities. The slot values ​​output by NLU are linked and matched with the graph entities to achieve referential resolution. Based on the intent and entity parameters determined after resolution, the unified API gateway asynchronously and in parallel calls are made to the campus attendance system to query today's attendance status, the timetable system to query today's timetable information, and the notification system to query unread notifications. The prefetched results are stored in the edge Redis cache with the data version number and the valid time period tag for the day.

7. The edge-cloud collaborative voice interaction method for smart class signs according to claim 6, characterized in that, Step 4 also includes: In the Redis cache at the edge, set an expiration period field based on NTP synchronization timestamp for the prefetched business data. When the terminal layer initiates a request, the edge first checks the comparison result between the expiration period of the cached data and the current NTP time. When the verification finds that the cached data has crossed the day or exceeded the validity period, it forces a callback to the source to call the business system API to fetch the latest data, updates the Redis cache content and validity period field, and refreshes the version number marker of the corresponding entity in the knowledge graph based on the timestamp of the newly fetched data. The slot values ​​identified by NLU, the graph entity linking results, and the business data version number are bound into triples to generate semantic parsing results with entity fingerprints.

8. The edge-cloud collaborative voice interaction method for smart class signs according to claim 7, characterized in that, Step 5 specifically includes: The large model generates branch routes based on the intent labels output by NLU. If the intent label belongs to the unstructured dialogue category, it directly drives the large model to generate free text, and length constraints and factual verification rules are enabled during generation. If the intent tag belongs to the structured business instruction class, then construct a JSON structured prompt word containing business data fields, inject the pre-fetched Redis cache data into the prompt word context in the form of key-value pairs, and require the large model to generate the broadcast text based on the injected data; The semantic hash value of the response text generated by the large model is calculated for caching and deduplication, and the entity fingerprint of the response text is associated with the generated triplet binding result and stored.

9. A method for edge-cloud collaborative voice interaction suitable for smart class signs according to claim 8, characterized in that, Step 6 specifically includes: The generated response text is divided into sequentially numbered TTS audio segments at the edge end according to a fixed duration. Each TTS audio segment is independently synthesized using the VITS accelerated acoustic model and stored in the edge buffer. The edge end sends the segments to the terminal layer in sequence through the low-latency RTP protocol. While playing the current segment, the terminal layer fetches the next segment to the local circular buffer in advance to achieve the simultaneous playback and fetching mechanism. Simple instructions recognized locally are executed directly without passing through the edge end. The local passthrough and segment prefetching strategies are combined to compress the response delay of the first segment. The terminal layer VAD continuously listens to the user's voice input. When a new voice is detected that triggers an interruption TTS broadcast event, it immediately stops the current segment distribution and clears the circular buffer. It then sends the interruption event and the azimuth angle of the new voice back to the edge end. The edge end cancels the current ASR decoding and large model generation tasks and resets the state machine to the idle state, triggering a re-entry into the wake-up positioning and ASR correction link.

10. An edge-cloud collaborative voice interaction system for smart class signs, used to implement the edge-cloud collaborative voice interaction method for smart class signs as described in any one of claims 1-9, characterized in that, The architecture comprises a three-tiered collaborative structure consisting of a terminal layer, an edge / cloud layer, and a service layer. The edge / cloud layer communication connection includes the following modules: The terminal perception module, deployed on the class sign terminal, is based on an embedded chip and a customized wake-up word engine. It runs the MUSIC algorithm and integrates inertial navigation data to lock the location of the speaker's voice source. It also works with the Speex / WebRTC NNNoise algorithm to complete echo cancellation and noise reduction, so as to output a high signal-to-noise ratio audio stream. The edge inference module is used to perform parallel ASR decoding, semantic correction and intent recognition on uploaded audio by deploying a large model and a BERT-based error correction model, so as to complete high-precision speech-to-semantics conversion. The business collaboration module is used to asynchronously prefetch business data of structured business instructions through entity links in the campus knowledge graph and the DIFY workflow, generate entity fingerprints, and bind semantic parsing results. The broadcast response module is used to divide the response into TTS fragments and send them to the terminal for simultaneous broadcasting and retrieval, and supports VAD interrupt detection and fast state machine reset.

Images