Real-time knowledge completion prompting method and device, equipment and storage medium

Abstract

The application discloses a real-time knowledge completion prompting method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring user physiological behavior data collected by a plurality of heterogeneous sensors on a wearable device in real time; detecting cognitive lag based on the user physiological behavior data; when it is detected that the user has cognitive lag, acquiring current dialogue context text; querying the current dialogue context text to determine corresponding candidate completion information; and outputting the candidate completion information to the user through a preset prompting channel. The method can perceive the cognitive lag state in real time and provide non-intrusive knowledge completion prompts in real time without interrupting the user interaction process, thereby improving the fluency of expression and the interaction experience.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to methods, apparatus, devices and storage media for real-time knowledge completion and prompting. Background Technology

[0002] In real-time interactive scenarios such as conference presentations, public speeches, academic defenses, and social conversations, speakers frequently experience the "tip-of-the-tongue" (TOT) phenomenon—knowing a word, phrase, data, or technical term but being unable to recall it accurately at the crucial moment of expression. Cognitive psychology research indicates that the TOT phenomenon occurs approximately 1-2 times per week in adults, and increases significantly with age and cognitive load. This cognitive block not only disrupts the fluency and coherence of expression but also easily leads to social embarrassment, reduces the persuasiveness of speeches, and in severe cases, even affects the effectiveness of business negotiations and academic presentations.

[0003] Currently, the main solution to the aforementioned cognitive lag problem is to use speech recognition to align real-time speech with a pre-prepared text in time, enabling the teleprompter's content to scroll and follow automatically. However, this solution only works in scenarios with pre-prepared text and cannot adapt to scenarios without pre-prepared text. Most interactive scenarios in daily life and work do not have pre-prepared complete texts, so this type of solution cannot function in such scenarios. Summary of the Invention

[0004] The main purpose of this application is to provide a real-time knowledge completion prompt method, device, equipment and storage medium, which aims to solve the technical problem that current solutions for prompting cognitive lag cannot be adapted to different scenarios.

[0005] To achieve the above objectives, this application proposes a real-time knowledge completion suggestion method, which includes: Acquire real-time user physiological behavior data from multiple heterogeneous sensors on wearable devices; Cognitive lag detection is performed based on the user's physiological behavior data. When a user is detected to have a cognitive lapse, obtain the current conversation context text; The current dialogue context text is queried to determine the corresponding candidate completion information; The candidate completion information is output to the user through a preset prompt channel.

[0006] In one embodiment, the step of detecting cognitive lag based on the user's physiological behavior data includes: The user's physiological behavior data is preprocessed to obtain a multimodal feature vector; The cognitive lag of users is detected by using the multimodal feature vectors.

[0007] In one embodiment, the user's physiological behavior data includes: the user's audio data, gaze data, motion data, and facial data; The step of preprocessing the user's physiological behavior data to obtain multimodal feature vectors includes: Mel frequency cepstral coefficients feature extraction is performed on the audio data to obtain acoustic hesitation features; Based on the gaze data, the user's gaze variance and gaze features are extracted to obtain visual wandering features; Smoothness analysis is performed on the motion data to obtain motion pause characteristics; Spectral analysis was performed on the facial data to obtain anxiety micro-expression features; The acoustic hesitation feature, the visual wandering feature, the action pause feature, and the anxiety micro-expression feature are time-synchronized and aligned to construct a multimodal feature vector.

[0008] In one embodiment, the step of detecting cognitive lag in users using the multimodal feature vector includes: The multimodal feature vectors are input into a pre-trained cross-modal attention fusion model to calculate the correlation weights between each modality feature vector and output the current cognitive lag probability value. If the probability value of cognitive lag is greater than a preset detection threshold, it is determined that the user has cognitive lag. If the probability value of cognitive lag is less than or equal to the preset detection threshold, it is determined that the user does not have cognitive lag.

[0009] In one embodiment, after the step of outputting the candidate completion information to the user through a preset prompt channel, the method further includes: If the probability of cognitive lag is less than a preset recovery threshold, the duration is obtained; If the duration exceeds a preset time threshold, the candidate completion information is controlled to fade out and disappear within a preset time period.

[0010] In one embodiment, the step of querying the current dialogue context text to determine the corresponding candidate completion information includes: Encode the current dialogue context text into a semantic query vector; Based on the semantic query vector, a search is performed in the local knowledge base to obtain the corresponding candidate completion information.

[0011] In one embodiment, the step of retrieving corresponding candidate completion information from a local knowledge base based on the semantic query vector includes: Get the search parameters and search width; Based on the semantic query vector, an approximate nearest neighbor retrieval is performed in the local knowledge base using the retrieval parameters and the search width to obtain multiple candidate completion entries that meet the preset conditions; The corresponding candidate completion information is obtained based on multiple candidate completion entries.

[0012] In one embodiment, the step of outputting the candidate completion information to the user through a preset prompt channel includes: Obtain the current environment scene mode; Determine the preset prompt channel based on the described environmental scene mode; Based on the aforementioned environmental scenario mode, the candidate completion information is output to the user through the preset prompt channel.

[0013] In one embodiment, the environmental scene mode includes a speech mode, and the preset prompt channel includes a visual display channel; The step of outputting the candidate completion information to the user through the preset prompt channel based on the environmental scene mode includes: When the environment scene mode is speech mode, configure preset contrast and semi-transparent visual rendering parameters. The candidate completion information is displayed in the user's field of vision edge area using the visual rendering parameters through the visual display channel.

[0014] In one embodiment, the environmental scene mode includes a conference mode, and the preset prompt channel includes a bone conduction audio channel; The step of outputting the candidate completion information to the user through the preset prompt channel based on the environmental scene mode includes: When the environment scene mode is conference mode, configure a preset volume that is lower than the ambient background volume; The candidate completion information is broadcast through the bone conduction audio channel at the preset volume.

[0015] In one embodiment, the environmental scene mode includes a private conversation mode, and the preset prompt channel includes a haptic feedback channel; The step of outputting the candidate completion information to the user through the preset prompt channel based on the environmental scene mode includes: When the environment scene mode is private conversation mode, and the vibration parameters of the preset rhythm are configured; The haptic feedback channel sends preset rhythmic vibration parameters to the user to prompt for the candidate completion information.

[0016] In one embodiment, the multi-channel heterogeneous sensor includes at least a bone conduction microphone, an eye-tracking sensor, an inertial measurement unit, and a facial electromyography sensor; The steps for acquiring real-time user physiological behavior data collected by multiple heterogeneous sensors on wearable devices include: The system acquires audio data collected from the user via the bone conduction microphone, gaze data collected from the user via the eye-tracking sensor, motion data collected from the user via the inertial measurement unit, and facial data collected from the user via the facial electromyography sensor. The user's physiological behavior data is obtained by summarizing the audio data, the gaze data, the motion data, and the facial data.

[0017] In one embodiment, the step of acquiring the user's audio data collected by the bone conduction microphone, the user's gaze data collected by the eye-tracking sensor, the user's motion data collected by the inertial measurement unit, and the user's facial data collected by the facial electromyography sensor includes: Acquire audio signals of the user's throat vibrations through the bone conduction microphone, and obtain audio data based on the audio signals of the throat vibrations; The coordinates of the user's gaze point and the change in pupil diameter are acquired through the eye-tracking sensor, and gaze data is obtained based on the coordinates of the gaze point and the change in pupil diameter. The user's head posture angular velocity and limb acceleration data are acquired through the inertial measurement unit, and motion data is obtained based on the head posture angular velocity and limb acceleration data. The micro-expression electrical signals of the user's facial muscles are acquired through the facial electromyography sensor, and facial data is obtained based on the micro-expression electrical signals.

[0018] Furthermore, to achieve the above objectives, this application also proposes a real-time knowledge completion and prompting device, which includes: The acquisition module is used to acquire real-time user physiological behavior data collected by multiple heterogeneous sensors on the wearable device; The detection module is used to detect cognitive lag based on the user's physiological behavior data; The acquisition module is also used to acquire the current dialogue context text when it detects that the user has a cognitive lag. The query module is used to query the current dialogue context text to determine the corresponding candidate completion information; The output module is used to output the candidate completion information to the user through a preset prompt channel.

[0019] In one embodiment, the detection module is further configured to preprocess the user's physiological behavior data to obtain a multimodal feature vector; The cognitive lag of users is detected by using the multimodal feature vectors.

[0020] In one embodiment, the user's physiological behavior data includes: the user's audio data, gaze data, motion data, and facial data; the detection module is further configured to extract Mel-frequency cepstral coefficient features from the audio data to obtain acoustic hesitation features; Based on the gaze data, the user's gaze variance and gaze features are extracted to obtain visual wandering features; Smoothness analysis is performed on the motion data to obtain motion pause characteristics; Spectral analysis was performed on the facial data to obtain anxiety micro-expression features; The acoustic hesitation feature, the visual wandering feature, the action pause feature, and the anxiety micro-expression feature are time-synchronized and aligned to construct a multimodal feature vector.

[0021] In one embodiment, the detection module is further configured to input the multimodal feature vectors into a pre-trained cross-modal attention fusion model to calculate the correlation weights between each modal feature vector and output the current cognitive lag probability value. If the probability value of cognitive lag is greater than a preset detection threshold, it is determined that the user has cognitive lag. If the probability value of cognitive lag is less than or equal to the preset detection threshold, it is determined that the user does not have cognitive lag.

[0022] In one embodiment, the detection module is further configured to obtain the duration when the probability of cognitive lag is less than a preset recovery threshold; If the duration exceeds a preset time threshold, the candidate completion information is controlled to fade out and disappear within a preset time period.

[0023] In addition, to achieve the above objectives, this application also proposes a real-time knowledge completion and prompting device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the real-time knowledge completion and prompting method described above.

[0024] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the real-time knowledge completion and prompting method described above.

[0025] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the real-time knowledge completion and prompting method described above.

[0026] This application proposes one or more technical solutions that acquire real-time user physiological behavior data collected by multiple heterogeneous sensors on a wearable device; perform cognitive lag detection based on the user physiological behavior data; when a cognitive lag is detected, acquire the current dialogue context text; query the current dialogue context text to determine corresponding candidate completion information; and output the candidate completion information to the user through a preset prompt channel. Through the above steps, this application can rely on multiple heterogeneous sensors on a wearable device to capture the user's physiological behavior characteristics when a cognitive lag occurs from multiple dimensions. It can achieve automatic detection and knowledge completion of cognitive lag without relying on a pre-prepared complete text, and can adapt to various real-time interactive scenarios such as daily speech and social conversations without a preset text. At the same time, it can achieve early proactive detection of cognitive lag through multimodal perception, without the need for the user to manually trigger a request for help. While ensuring privacy and security, it improves the timeliness and naturalness of knowledge completion and interaction, effectively solving the technical problem that existing technologies cannot adapt to different scenarios. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating the real-time knowledge completion suggestion method of this application in Implementation Example 1. Figure 2 This is a flowchart illustrating Embodiment 2 of the real-time knowledge completion suggestion method for this application. Figure 3 This is a flowchart illustrating Embodiment 3 of the real-time knowledge completion suggestion method for this application. Figure 4 A simplified flowchart is provided for one embodiment of the real-time knowledge completion suggestion method of this application; Figure 5 This is a schematic diagram of the module structure of the real-time knowledge completion and prompting device according to an embodiment of this application; Figure 6This is a schematic diagram of the device structure of the hardware operating environment involved in the real-time knowledge completion and prompting method in the embodiments of this application.

[0030] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0031] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0032] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0033] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as real-time knowledge completion prompts. The following description uses real-time knowledge completion prompts as an example to illustrate this embodiment and the subsequent embodiments. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the corresponding device.

[0034] Based on this, the embodiments of this application provide a real-time knowledge completion and prompting method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the real-time knowledge completion and prompting method of this application.

[0035] In this embodiment, the real-time knowledge completion and prompting method includes steps S10 to S40: Step S10: Acquire real-time user physiological behavior data collected by multiple heterogeneous sensors on the wearable device.

[0036] It should be noted that multi-channel heterogeneous sensors refer to a combination of sensors with different physical sensing principles and data sampling rates integrated into wearable devices, such as smart glasses, headphones, watches, or lavalier devices. Examples include acoustic sensors for collecting voice or laryngeal vibrations, optical sensors for collecting gaze trajectories, inertial sensors for collecting limb or head movements, and bioelectrical sensors for collecting facial muscle electrical signals. Other sensors capable of characterizing a user's speech, vision, movement, or emotional state can also be selected based on actual needs, as long as they can achieve parallel acquisition of multimodal data.

[0037] Real-time acquisition means that data acquisition is triggered continuously or frequently, rather than manually by the user. The system typically uses data structures such as a circular buffer to cache and time-align the high-speed inflow of raw signals, ensuring that the data used for subsequent analysis is strictly synchronized in time. This passive and continuous data acquisition mechanism allows the system to complete status monitoring without the user's awareness, fundamentally avoiding the interaction interruptions and social awkwardness caused by the need for active user operation in existing technologies.

[0038] Understandably, the multi-channel heterogeneous sensors include at least an IMU (Inertial Measurement Unit), an EMG (Electromyography Sensor), a bone conduction microphone, and an eye-tracking infrared sensor. The bone conduction microphone is used to capture infrasound-level laryngeal vocalization delay signals, which are unaffected by environmental noise and are suitable for detecting subtle pauses in speech output. The inertial measurement unit integrates sensor modules of accelerometers and gyroscopes to detect subtle pauses in hand writing movements and gestures. The electromyography sensor is a sensor that reflects micro-expressions and changes in muscle tension by detecting electrical signals from facial muscles, which can help determine the user's emotional state such as anxiety and confusion. The eye-tracking infrared sensor is used to capture gaze movement patterns.

[0039] User physiological behavior data may include the user's audio data, gaze data, motion data, and facial data, and may also include other physiological behavior data, which are not limited in this embodiment.

[0040] In one feasible implementation, step S10 may include steps A11-A12: The multi-channel heterogeneous sensor includes at least a bone conduction microphone, an eye-tracking sensor, an inertial measurement unit, and a facial electromyography sensor; Step A11: Acquire the user's audio data collected by the bone conduction microphone, the user's gaze data collected by the eye-tracking sensor, the user's motion data collected by the inertial measurement unit, and the user's facial data collected by the facial electromyography sensor; It should be noted that the system deploys four heterogeneous sensor acquisition channels on the wearable device, enabling it to collect user audio data via bone conduction microphones. This audio data can include raw audio signals generated by laryngeal vibrations. Compared to traditional air conduction microphones, this acquisition method avoids environmental noise interference and more clearly captures subtle delay signals at the beginning of pronunciation, providing a more accurate data foundation for extracting acoustic hesitation features. The audio data can also include the user's real-time speech stream, which can be stored in the dialogue text buffer maintained by the end-side streaming ASR (Automatic Speech Recognition) engine. User gaze data is collected via eye-tracking sensors, including the coordinates of the user's gaze point, pupil diameter changes, and saccade duration, used to analyze the user's visual search behavior characteristics when cognitive pauses occur. User motion data includes angular velocity and acceleration data of the user's head and hands collected via inertial measurement units, used to extract features of motion pauses and posture adjustments. User facial data includes micro-electrical signal data of muscles such as the zygomaticus major and corrugator supercilii muscles collected via facial electromyography sensors.

[0041] In one feasible implementation, step A11 may include: acquiring a user's throat vibration audio signal collected by the bone conduction microphone, and obtaining audio data based on the throat vibration audio signal; acquiring the user's gaze fixation point coordinates and pupil diameter change data collected by the eye-tracking sensor, and obtaining gaze data based on the gaze fixation point coordinates and pupil diameter change data; acquiring the user's head posture angular velocity and limb acceleration data collected by the inertial measurement unit, and obtaining motion data based on the head posture angular velocity and limb acceleration data; acquiring the user's facial muscle micro-expression electrical signals collected by the facial electromyography sensor, and obtaining facial data based on the micro-expression electrical signals.

[0042] Bone conduction microphones are typically attached behind the ear near the mastoid process or in the neck near the Adam's apple. They directly sense the mechanical vibrations of the skull and soft tissues through piezoelectric ceramics or MEMS accelerometers, rather than airborne sound waves. This physical mechanism naturally gives them extremely high environmental noise suppression capabilities, allowing them to clearly capture the user's voice signal even in noisy streets or conference rooms. More importantly, they can detect infrasonic laryngeal vocalization delay signals that traditional air conduction microphones cannot detect—the subtle physical traces of vocal cord incomplete closure or abnormal vibration frequency fluctuations caused by the user attempting to speak but struggling to extract words. Compared to semantic analysis based solely on speech recognition and text transcription, they offer lower latency and higher specificity. Furthermore, because bone conduction signals do not travel through the air, the user's conversation will not be heard by those around them, ensuring privacy at the hardware level. Eye-tracking sensors typically use near-infrared light to illuminate the eyeball and a high-speed camera to capture the relative position of the corneal reflection spot and the center of the pupil, thereby calculating the two-dimensional or three-dimensional coordinates of the gaze point in space in real time at a sampling rate of 120Hz or higher. Simultaneously, by analyzing pixel changes at the pupil edge, the dynamic fluctuations in pupil diameter can be accurately measured. When users experience difficulty recalling information, it is often accompanied by a momentary dilation of the pupil; while an increase in the variance of the fixation point or staring into space are typical behavioral markers of attention shifting from the external environment to internal memory retrieval. By combining these two types of data, the system can effectively distinguish between introverted visual searches caused by cognitive lapses and extroverted attention shifts caused by external stimuli, avoiding misjudging normal observation behavior as a lapse.

[0043] The inertial measurement unit (IMU) integrates a three-axis accelerometer and a three-axis gyroscope, enabling continuous measurement of the linear acceleration and rotational angular velocity of the device carrier at frequencies above 200Hz. When the wearable device is smart glasses, this data primarily reflects the stability of head posture and rotational intention; when the device is a smartwatch or bracelet, it primarily reflects the fluency of handwriting, gestures, or body language. When cognitive lag occurs, previously continuous motion output often exhibits unnatural micro-pauses, trajectory jitter, or repetitive, stereotyped movements. By performing smoothness analysis or frequency domain transformation on the raw acceleration and angular velocity data, feature vectors characterizing the interruption of motion control can be extracted. Facial electromyography (EMG) sensors, using dry or wet electrodes attached to the skin surface, detect the weak bioelectrical signals generated when key facial muscle groups contract. Introducing this modality allows the system to sense changes in the user's internal emotional tension, compensating for potential omissions in psychological state information that may be missed by relying solely on external behavioral observations. Especially when the user deliberately suppresses lag or is in a state of silent contemplation, the EMG signals still maintain effective state indication capabilities.

[0044] Step A12: Summarize the audio data, the gaze data, the motion data, and the facial data to obtain the user's physiological behavior data.

[0045] The selection of the four types of sensors to construct a multi-channel heterogeneous sensing array is based on a deep deconstruction of the multidimensional representation of cognitive lapses in humans. Cognitive lapses are not simply a pause in speech, but a comprehensive result of the coordinated dysfunction of four subsystems: speech production, visual attention, motor planning, and emotion regulation. Bone conduction microphones correspond to the speech production dimension, capable of penetrating environmental noise to directly capture the underlying physical vibrations of the vocal organs; eye-tracking sensors correspond to the visual attention dimension, capable of quantifying the allocation of attentional resources during memory retrieval; inertial measurement units correspond to the motor planning dimension, capable of keenly sensing the momentary stiffness or tremors at the limb control level when thinking is blocked; and facial electromyography sensors correspond to the emotion regulation dimension, capable of revealing the subconscious anxiety or confusion accompanying memory failure. This four-dimensional orthogonal sensor combination design ensures that the system can observe the same cognitive event from multiple independent and complementary physical channels. Even if the signal-to-noise ratio of one channel decreases due to wearing position deviation or environmental interference, other channels can still provide effective state evidence, thus significantly improving the robustness of the detection algorithm in real-world complex scenarios. It should be understood that although the combination of the above four sensors is preferred in this embodiment, in other embodiments, the types of sensors may be added or removed depending on the device form or application scenario. For example, a skin electromyography sensor may be added to monitor arousal, or a camera may be used to replace the electromyography sensor for visual expression analysis.

[0046] Step S20: Perform cognitive lag detection based on the user's physiological behavior data.

[0047] Understandably, cognitive lag detection refers to the process of automatically determining whether a user is currently experiencing recall difficulties, such as "having something on the tip of their tongue" or "thought block," by analyzing multimodal data. This step does not rely on a single threshold judgment, but rather comprehensively considers the coordinated changes in various physiological and behavioral characteristics, such as hesitation and pauses in speech output, aimless eye movement, stiff gestures, or anxious and tense micro-expressions.

[0048] When these characteristics exhibit a stuttering pattern that conforms to the definition of cognitive psychology within a specific time window, the system determines that the user has a cognitive stutter. This automatic triggering mechanism is the key difference between this application and traditional voice assistants or search engines. It moves the activation time of the assistance system from "the user realizes they have forgotten and actively seeks help" to "the system senses signs of forgetting and actively responds," thereby intervening to assist before the user has completely left the current conversational context, significantly improving the timeliness and naturalness of knowledge completion.

[0049] Step S30: When a user is detected to have cognitive lag, obtain the current dialogue context text.

[0050] It should be noted that the current dialogue context text refers to the most recent linguistic information fragments most relevant to the user's current expressive intent. After detecting a stuttering event, the system does not blindly search the entire network, but first identifies the semantic environment that caused the stuttering. For example, the most recent speech stream can be transcribed into text using the on-device automatic speech recognition module, and a sliding window strategy can be used to extract the effective dialogue content within a preset duration (such as 30 seconds) as the context.

[0051] This time-window-based context extraction method filters out outdated and irrelevant historical information while retaining sufficient semantic clues to support accurate subsequent retrieval.

[0052] In practice, if a user experiences cognitive delays, the system enters a retrieval preparation phase. First, semantic cues are extracted from the dialogue text buffer at the edge-side ASR engine level. The edge-side ASR engine continuously runs on the DSP core, transing the real-time speech stream into text segments, each with a timestamp, and storing them in a ConcurrentLinkedDeque structured dialogue buffer. Expired segments exceeding a 30-second time window are cleaned up starting from the head of the buffer. Then, the remaining segments are concatenated into a complete context string, i.e., the current dialogue context text. This sliding window mechanism ensures that the retrieval input always focuses on the most recent dialogue content, reducing interference from irrelevant information and controlling the computational load of subsequent encoders.

[0053] Step S40: Query the current dialogue context text to determine the corresponding candidate completion information.

[0054] Understandably, the current dialogue context text can be parsed and processed, and then relevant content can be queried in the corresponding database to obtain candidate completion information. The content format of the candidate completion information can be diverse, including but not limited to professional terms, famous lines of poetry, historical data, names of people and places, or factual knowledge. Through deep semantic matching rather than simple keyword matching, the system can understand the user's underlying intent and provide accurate prompts even when the user's expression is vague or incomplete.

[0055] For example, the context text is first encoded into a high-dimensional semantic vector, and then an approximate nearest neighbor search is performed in a local or cloud-based knowledge base to find several entries with the highest semantic similarity as candidate completion information.

[0056] As a preferred implementation, the query process can be completed offline on the device side. This not only eliminates the latency caused by network transmission and enables a response speed in the millisecond range, but more importantly, it ensures that the user's conversation privacy data does not leave the device.

[0057] In one feasible implementation, step S40 may include steps A21-A22: Step A21: Encode the current dialogue context text into a semantic query vector; It should be noted that the current dialogue context text can be input into the lightweight Sentence-Encoder on the device side for encoding, thereby encoding it into a 128-dimensional semantic query vector. The Sentence-Encoder is a neural network model that encodes natural language text into a fixed-dimensional dense vector to measure the semantic similarity between texts. In this solution, the INT8 quantized version is deployed on the device side. The Sentence-Encoder uses INT8 quantization deployment, and the model size is about 1 / 4 of the standard FP32 version. The single encoding latency on the device CPU is less than 20ms.

[0058] Step A22: Based on the semantic query vector, perform a search in the local knowledge base to obtain the corresponding candidate completion information.

[0059] In practical implementation, the local knowledge base refers to a vector index library pre-built and stored in the wearable device's local storage. Its content covers vector embeddings of structured knowledge graphs, including commonly used poems, idioms, professional terminology, historical data, and geographical information. Since the retrieval process is entirely completed on the device side, there is no need to upload the user's voice or transcribed text to the cloud server, fundamentally eliminating the risk of sensitive privacy data leakage. Simultaneously, device-side retrieval eliminates the uncertainty and latency of network transmission, providing stable knowledge completion services even in environments with no network coverage or weak network conditions, ensuring the real-time nature and reliability of auxiliary prompts. This edge-cloud collaborative architecture design—where heavy computation and sensitive data processing are done on the device side, and incremental updates to the knowledge base are done in the cloud—constitutes a significant technical advantage over purely cloud-based retrieval solutions.

[0060] In one feasible implementation, step A22 may include: obtaining retrieval parameters and search width; performing approximate nearest neighbor retrieval in a local knowledge base based on the semantic query vector using the retrieval parameters and search width to obtain multiple candidate completion entries with matching degrees meeting preset conditions; and obtaining corresponding candidate completion information based on the multiple candidate completion entries.

[0061] Specifically, the encoded query vector is fed into the HNSW (Hierarchical Navigable SmallWorld) algorithm. HNSW is a high-dimensional vector approximate nearest neighbor retrieval algorithm based on hierarchical navigable small world graphs. It has the characteristics of fast retrieval speed and low memory consumption, and is suitable for edge deployment.

[0062] The retrieval parameter M represents the maximum number of connections for each node in the graph, and the search width ef_search represents the number of candidate nodes retained in the priority queue during the retrieval process. These two parameters directly determine the upper and lower limits of retrieval performance. Their values ​​are not arbitrarily chosen, but rather optimal values ​​determined through extensive experimental verification based on edge hardware constraints and user experience requirements.

[0063] In this embodiment, the retrieval parameter M can be set to 16. The value of M controls the connectivity and memory usage of the index graph: if the value of M is too small (e.g., less than 8), the connectivity of the graph is insufficient, and the retrieval path is prone to getting trapped in local optima, resulting in a significant decrease in recall; if the value of M is too large (e.g., greater than 32), although the recall rate is slightly improved, the memory usage increases linearly, and the graph traversal overhead of a single retrieval increases, which is not worthwhile for wearable devices with limited memory.

[0064] The search width ef_search can be set to 64. ef_search controls the breadth of exploration during retrieval: when ef_search is too small (e.g., below 32), the algorithm converges prematurely, easily missing the true nearest neighbor vector, leading to inaccurate or even completely wrong completion information, which is unacceptable in knowledge completion scenarios; when ef_search is too large (e.g., above 128), although the recall rate approaches the exact search, the retrieval time will increase dramatically, possibly exceeding the 200-millisecond real-time interaction tolerance threshold, resulting in a noticeable delay in prompts.

[0065] The preset conditions refer to the threshold requirements for matching degree. Typically, the search results ranking in the top preset positions of similarity are used as candidate completion entries that meet the conditions. This ensures that the candidate range is neither too large, causing excessive selection costs for users, nor too small, missing the completion content of the true target. The text content of the matched candidate completion entries is extracted and organized to obtain the final candidate completion information output to the user.

[0066] Specifically, the Top-K entries returned by the search can be post-processed. For example, they can be re-ranked based on category tags, timeliness weights, or user historical preferences to select one or more pieces of information that best fit the current context as the final output. The entire search process is completed locally on the device without requiring an internet connection, ensuring both user privacy (voice content does not leave the device) and low latency (total on-device search latency is less than 50ms). The index is incrementally updated from the cloud via Wi-Fi / Bluetooth when the device is idle, using a differential transmission strategy to download only newly added and changed knowledge entries.

[0067] Step S50: Output the candidate completion information to the user through a preset prompt channel.

[0068] A preset prompt channel refers to a pre-configured sensory feedback path used to deliver supplementary information to the user. To minimize disruption to the user's current interaction flow, this channel is typically non-intrusive and can be dynamically adjusted according to the environment.

[0069] For example, in public speaking scenarios, semi-transparent text display in the visual edge area can be selected; in business meeting scenarios, low-volume voice broadcasting of bone conduction headphones can be selected; and in private conversations or quiet environments, tactile vibration rhythm encoding can be selected.

[0070] It should be understood that the selection of the prompt channel is not fixed. The system can assess the user's visual load, auditory load, and ambient noise level in real time, and adaptively switch to the optimal channel or degrade to the backup channel when the main channel is unavailable. This adaptive output strategy ensures that the prompt information is effectively received by the user without interrupting their train of thought or attracting the attention of those around them, truly achieving seamless intelligent assistance throughout the entire process.

[0071] This embodiment provides a real-time knowledge completion prompt method. It acquires real-time user physiological behavior data collected by multiple heterogeneous sensors on a wearable device; performs cognitive pause detection based on the user's physiological behavior data; when a cognitive pause is detected, it acquires the current dialogue context text; queries the current dialogue context text to determine corresponding candidate completion information; and outputs the candidate completion information to the user through a preset prompt channel. Through these steps, this application can rely on multiple heterogeneous sensors on a wearable device to capture the user's physiological behavior characteristics when a cognitive pause occurs from multiple dimensions. It can automatically detect and complete knowledge during cognitive pauses without relying on a pre-prepared complete text, adapting to various real-time interactive scenarios such as daily speech and social conversations without pre-prepared text. Simultaneously, it achieves proactive early detection of cognitive pauses through multimodal perception, eliminating the need for users to manually trigger assistance. This improves the timeliness and naturalness of knowledge completion while ensuring privacy and security, effectively solving the technical problem that existing technologies cannot adapt to different scenarios.

[0072] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S20 includes steps S201 to S202: Step S201: Preprocess the user's physiological behavior data to obtain a multimodal feature vector.

[0073] It should be noted that preprocessing user physiological behavior data includes two core steps: feature extraction and alignment. Due to significant differences in the data types, sampling rates, and physical dimensions of data collected by different sensors, targeted preprocessing is necessary to transform them into feature representations that can be uniformly processed by subsequent models. Then, temporal alignment is performed to ensure that all features correspond to the cognitive state within the same observation time window. User physiological behavior data can be specifically divided into four categories: audio data, gaze data, motion data, and facial data. Preprocessing also performs feature extraction separately for each type of data, corresponding to the observation results output by the four types of sensors. Data from each sensor is written to an independent ring buffer with a capacity of 8192 sampling points. A ring-overlapping mechanism avoids frequent memory allocation. The ring buffer is a fixed-size, contiguous data structure that achieves efficient real-time data stream caching by overwriting the oldest data, supporting zero-copy data exchange, and is suitable for sensor data acquisition and inter-pipeline communication. Because the sampling rates of the four sensors differ significantly (bone conduction 16kHz, eye tracking 120Hz, IMU 200Hz, EMG 500Hz), the number of window sampling points is calculated based on the sampling rate of each channel during reading to ensure time alignment.

[0074] In one feasible implementation, step S201 may include steps B11 to B15: Step B11: Extract the Mel frequency cepstral coefficient features from the audio data to obtain acoustic hesitation features; It should be noted that acoustic hesitation features are a direct indicator of obstruction in the speech production system. When extracting Mel-frequency cepstral coefficients (MFCC) features from audio data, a 50ms sliding window and a 25ms frame shift are typically used to extract 13-dimensional MFCC coefficients and their first and second-order differences. These are then combined with fundamental frequency jitter, energy envelope, and filler word frequency to construct the acoustic hesitation features. MFCC simulates the nonlinear perception characteristics of the human ear at different frequencies, effectively capturing subtle acoustic changes caused by recall difficulties, such as micro-vibrations of the articulatory organs, vowel lengthening, or non-fluent filler words (e.g., "uh," "that"). Compared to simple speech recognition and text analysis, it better reflects the underlying cognitive processing pressure.

[0075] The acoustic hesitation feature consists of 13-dimensional MFCC coefficients, fundamental frequency jitter, energy envelope, and filler word frequency, making it a 16-dimensional feature.

[0076] Step B12: Extract the user's gaze variance and gaze features based on the gaze data to obtain visual wandering features; Visual wandering features are used to characterize users' attention allocation patterns when retrieving memories. When extracting fixation variance based on gaze data, the standard deviations of the fixation point coordinates in the horizontal and vertical directions within a preset time window are calculated separately. When users are in a "speech-on-the-point" state, they often exhibit aimless gaze wandering or staring into space, leading to a significant increase in fixation variance. Simultaneously, by combining gaze features such as pupil diameter change rate and saccade speed, it is possible to distinguish between visual search behavior caused by cognitive load and external attention shifts caused by environmental stimuli, thereby improving the specificity of the features.

[0077] Specifically, five dimensions of user gaze variance in the horizontal direction, gaze variance in the vertical direction, average pupil diameter, maximum saccade speed, and gaze duration can be extracted to form visual wandering features.

[0078] Step B13: Perform smoothness analysis on the motion data to obtain motion pause characteristics; Action pauses reflect disruptions in the coordination between motion planning and execution. Smoothness analysis of motion data typically involves calculating the rate of change or acceleration index of acceleration or angular velocity signals acquired by the inertial measurement unit. During cognitive lapses, users' gestures or head movements often exhibit unnatural stiffness, repetitive micro-movements, or abrupt trajectory changes. These abnormal fluctuations in kinematic characteristics are externalizations of cognitive blockage at the physical level, compensating for detection blind spots in certain silent or gaze-stable scenarios using only voice or visual channels.

[0079] Specifically, four features can be extracted from motion data: acceleration variance, angular velocity rate of change, frequency of micro-pauses, and total energy, which constitute motion pause features.

[0080] Step B14: Perform spectral analysis on the facial data to obtain anxiety micro-expression features; Anxiety micro-expression features capture the instantaneous emotional response accompanying recall failure. When performing spectral analysis on facial data, the focus is on the frequency energy distribution of electromyographic signals in key facial muscle groups such as the corrugator supercilii and zygomaticus major. Cognitive lapses are often accompanied by mild confusion, anxiety, or frustration. These emotions trigger specific changes in facial muscle tension, and their spectral characteristics differ significantly from voluntarily controlled social expressions, providing evidence for cognitive state identification through an emotional dimension independent of overt behavior.

[0081] Specifically, feature extraction and spectral analysis can be performed on facial data to extract RMS energy, spectral centroid, and muscle tension index, totaling three-dimensional features, thereby obtaining micro-expression anxiety features.

[0082] Step B15: Time-synchronize and align the acoustic hesitation feature, the visual wandering feature, the action pause feature, and the anxiety micro-expression feature to construct a multimodal feature vector.

[0083] In practice, the features extracted from each channel can be spliced ​​together to form a 28-dimensional unified feature vector, consisting of 16-dimensional acoustic hesitation features, 5-dimensional visual wandering features, 4-dimensional action pause features, and 3-dimensional anxiety micro-expression features.

[0084] Then, layer normalization is performed on the 28-dimensional unified feature vector to eliminate the differences in feature dimensions of each channel, thereby obtaining a multimodal feature vector.

[0085] Step S202: Detect cognitive lag in users using the multimodal feature vectors.

[0086] In practice, the multimodal feature vectors can be weighted to determine the user's current cognitive lag probability value. The user's cognitive lag probability value can be compared with the preset detection threshold to determine whether the user has cognitive lag.

[0087] In one feasible implementation, step S202 may include steps B21-B23: Step B21: Input the multimodal feature vectors into the pre-trained cross-modal attention fusion model to calculate the correlation weights between each modality feature vector and output the current cognitive lag probability value; It should be noted that the cross-modal attention fusion model employs a multi-head self-attention mechanism, enabling the model to adaptively learn the contribution and interaction patterns of different modal features under different cognitive states. The core idea of ​​the cross-modal attention mechanism is that each attention head is responsible for calculating the attention weights of one sensor feature to the other three features. Specifically, the normalized feature vectors are projected through three linear projection matrices: Q (Query), K (Key), and V (Value), and then the Scaled Dot-Product Attention is calculated: Attention(Q, K, V) = softmax(QK^T / √d_k)·V, where d_k is the dimension of each attention head. The outputs of the four attention heads are concatenated along the feature dimensions to obtain the final fused representation. The cross-modal attention fusion model is deployed on an edge NPU, using INT8 quantization. Model inference consumes no more than 20% of the NPU's computing power, and the single-frame inference latency is controlled within 100ms, meeting real-time requirements. The edge-side NPU is a dedicated neural network processing unit integrated into the chip of mobile / wearable devices. It is hardware-optimized for inference tasks of INT8 quantized models and can complete deep learning inference under low power conditions.

[0088] For example, when a user speaks fluently but their gaze wanders, the model automatically reduces the weight of acoustic features and increases the weight of visual features. When "speech hesitation" and "facial anxiety" occur simultaneously, the model captures this synergistic enhancement effect through an attention mechanism, outputting a higher confidence level for stuttering. Compared to traditional simple splicing or fixed-weighted fusion methods, cross-modal attention fusion can dynamically model the complementary and redundant relationships between modalities, maintaining stable detection performance even when some sensors fail or are subject to noise, thus enhancing the system's adaptability in complex real-world scenarios.

[0089] The fused output is processed through a fully connected layer and a sigmoid activation function, and mapped to a cognitive lapse probability value in the range [0,1]. To avoid false triggers caused by instantaneous fluctuations in the probability values, the system performs a smoothing process on the probability sequence using a moving average of length 10.

[0090] Step B22: If the probability value of cognitive lag is greater than the preset detection threshold, determine that the user has cognitive lag; It should be noted that the preset detection threshold can be set to 0.72, or other values. This embodiment does not limit this. When the smoothed probability value exceeds the detection threshold of 0.72, the system determines that the user is currently in a cognitive stagnation state, triggers the StallEvent event, and passes it to the downstream pipeline.

[0091] Step B23: If the probability value of cognitive lag is less than or equal to the preset detection threshold, determine that the user does not have cognitive lag.

[0092] When the probability value of cognitive lag is less than or equal to 0.72, the user's cognitive processing is considered smooth at the current moment, and there is no recall blockage requiring intervention. The system continues to maintain background detection, waiting for data input in the next observation window. This probability threshold-based judgment mechanism, combined with moving average smoothing, can effectively filter out misjudgments caused by fluctuations in single-frame data while ensuring sensitivity to real cognitive lag events, achieving a reasonable balance between detection accuracy and response latency.

[0093] Understandably, after knowledge completion is triggered, in order to address the interference issue of the auxiliary prompts themselves and ensure that the prompts fade naturally and imperceptibly as the user's cognitive state recovers, rather than mechanically remaining or abruptly disappearing, the method after step S50 includes: if the probability of cognitive lag is less than a preset recovery threshold, obtaining the duration; if the duration is greater than a preset time threshold, controlling the candidate completion information to fade out within a preset time period.

[0094] It should be noted that the preset recovery threshold is a probabilistic boundary for determining whether a user has recovered from a cognitive stagnation state. In this embodiment, the preset recovery threshold is set to 0.30, and the preset time threshold can be set to 2 seconds. By setting a duration window of 2 seconds, the system effectively introduces a confirmation buffer. Only when the low stagnation probability state is stably maintained for more than this duration is the user's cognitive load considered to have been substantially reduced. This time-dimensional verification effectively filters out false recovery signals caused by sensor noise or transient behavioral anomalies, ensuring that the prompt retention strategy matches the user's actual cognitive rhythm.

[0095] Fade-out refers to the process by which the sensory intensity of a cue message (such as visual transparency, audio volume, or vibration amplitude) decreases linearly or non-linearly over time until it reaches zero, which is different from the instantaneous disappearance of a signal that is directly cut off.

[0096] When the probability of cognitive lag is less than the preset recovery threshold and the duration is greater than the preset time threshold, it is determined that the user has resumed fluent expression. Therefore, the prompts for candidate completion information can be controlled to automatically fade out and disappear in 300ms, achieving seamless assistance throughout the process.

[0097] This embodiment preprocesses the user's physiological behavior data to obtain a multimodal feature vector; then, it uses this multimodal feature vector to detect cognitive lapses in the user. Through these steps, the cognitive state information contained in the multi-channel heterogeneous sensing data can be fully exploited, and the complementarity of multimodal features can be used to improve the accuracy and robustness of cognitive lapse detection. Simultaneously, the lightweight edge model design ensures real-time performance and low power consumption during the detection process, meeting the resource constraints of wearable devices and providing a reliable trigger for subsequent accurate and timely knowledge completion.

[0098] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S50 includes steps S501 to S503: Step S501: Obtain the current environment scene mode.

[0099] The system divides the current environment into three scene modes through an environmental soundprint scene classifier: presentation mode, meeting mode, and private conversation mode. Therefore, the scene classifier also uses an INT8 quantization model deployed on the edge. By analyzing the soundprint features (background noise spectrum, reverberation features, and multi-person soundprint overlay mode) collected by the environmental microphone, it outputs scene category labels to obtain the current environmental scene mode.

[0100] Step S502: Determine the preset prompt channel according to the environmental scene mode.

[0101] The preset prompt channel is not pre-fixed to a specific hardware output device, but rather refers to the information transmission path dynamically evaluated by the system as optimal or suboptimal under the constraints of the current scenario. The system maintains an internal scenario-channel mapping strategy, which can be a rule-based decision tree or a dynamic weight model based on real-time sensory load assessment. For example, when the environment is identified as a public speech, since the auditory channel is already occupied by the user's voice output and the visual center of vision needs to focus on the audience or teleprompter, the system automatically identifies the edge area of ​​the visual display channel as the preset prompt channel. When the environment switches to a business meeting, considering that visual attention needs to be focused on the participants or documents, and the environment allows for private listening, the system identifies the bone conduction audio channel as the preset prompt channel. In extremely quiet or private conversation modes where it is inconvenient to make noise or emit light, the haptic feedback channel is prioritized. It should be understood that this determination process is real-time and dynamic; the same wearable device will automatically activate different output channels at different times and in different environments, even seamlessly switching as the user moves from the stage to the audience during a single interaction. In contrast, existing prompting systems often employ fixed configurations (such as always displaying pop-ups or always broadcasting voice prompts), which cannot adapt to complex and ever-changing real-world interactive environments and are prone to interference or information omissions. This embodiment fundamentally solves the problem of the appropriateness of auxiliary information delivery by establishing a dynamic coupling relationship between environmental perception and channel selection.

[0102] In one feasible implementation, the user's current visual and auditory sensory load status can also be assessed. Visual load assessment considers whether the user is currently looking at a screen or performing a visually intensive task; auditory load assessment considers the user's current ambient noise level and speech communication density. The optimal non-intrusive cueing channel is then determined based on the environmental scene pattern and sensory load status.

[0103] Step S503: Based on the environmental scene mode, output the candidate completion information to the user through the preset prompt channel.

[0104] It is understandable that the optimal prompting strategy can be determined through the environmental scenario mode, thereby prompting the user with the corresponding candidate completion information through the preset prompting channel and prompting strategy.

[0105] In one feasible implementation, the environmental scene mode includes a speech mode, and the preset prompt channel includes a visual display channel; step S503 may include steps C11~C12: Step C11: When the environment scene mode is speech mode, configure the preset contrast and semi-transparent visual rendering parameters. It should be noted that the speech mode is a special interactive scenario with extremely high constraints on the user's visual attention allocation. In this mode, the user's central vision usually needs to be continuously focused on the audience, the camera lens, or the physical teleprompter to maintain the continuity of eye contact and body language. Any highlighted or opaque information appearing in the center of the visual field will directly obscure key visual targets, leading to interruptions in the speech or social rudeness. Therefore, this embodiment limits the visual display channel to the user's peripheral visual field. The peripheral visual field refers to the visual perception area that is offset from the user's current primary gaze point by a certain angle.

[0106] The system can also dynamically calculate and adjust the rendering position of prompts based on the gaze coordinates obtained in real time by the eye-tracking sensor, so that it always remains at the relative edge of the current gaze point, thereby adapting to changes in the user's field of vision caused by head rotation or fine-tuning of the gaze during the presentation.

[0107] In practice, the preset contrast can be a low contrast, such as a transparency of about 15%, and semi-transparent text projection can be configured to avoid being noticed by the audience.

[0108] Step C12: Display the candidate completion information in the user's field of vision edge area using the visual rendering parameters through the visual display channel.

[0109] In the fourteenth specific instance, the transparent floating layer of the AR glasses HUD (Head-Up Display) can be used to project low-contrast semi-transparent text prompts. Specifically, the top-1 candidate completion items in the candidate completion information can be mechanically formatted, and then only the core text of the candidate completion items can be selected and rendered through the AR glasses HUD with visual rendering parameters to avoid visual abruptness.

[0110] In one feasible implementation, the environmental scene mode includes a conference mode, and the preset prompt channel includes a bone conduction audio channel; step S503 may include steps C21~C22: Step C21: When the environment scene mode is conference mode, configure a preset volume lower than the ambient background volume; It should be noted that the meeting mode is an interactive scenario that places high demands on both information privacy and environmental awareness. In this scenario, users need to obtain auxiliary information to maintain fluent expression, while also maintaining a complete auditory perception of the speeches, tones, and atmosphere of the surrounding participants to maintain a normal social interaction and collaborative rhythm. Therefore, this embodiment configures a preset volume lower than the ambient background volume as the prompt volume, such as a whisper-level volume, 30dB.

[0111] Step C22: Broadcast the candidate completion information at the preset volume through the bone conduction audio channel.

[0112] While traditional air conduction headphones provide clear voice prompts, they often physically isolate or actively noise-canceling external ambient sounds, causing users to miss interruptions or questions while receiving prompts, resulting in significant communication disruptions and rudeness. External speakers or ordinary open-back headphones pose a significant risk of sound leakage. In contrast, bone conduction audio channels utilize skull vibrations to directly transmit sound waves to the cochlea, allowing users to receive natural ambient sounds without loss while receiving prompts. Therefore, designating bone conduction audio channels as the default prompt channel in conference mode is the optimal technical approach to physically resolve the contradiction between "assisted privacy" and "environmental awareness."

[0113] Specifically, in meeting mode, knowledge category tags, such as [Poetry], [Data], [Terminology], etc., can be added before the prompt text corresponding to the candidate completion information, and the prompt content of the candidate completion information can be broadcast through bone conduction headphones at a whisper-level volume.

[0114] In one feasible implementation, the environmental scene mode includes a private conversation mode, and the preset prompt channel includes a haptic feedback channel; step S503 may include steps C31~C32: Step C31: When the environment scene mode is private conversation mode, configure the vibration parameters of the preset rhythm; Private conversation mode typically corresponds to special interactive environments where users are extremely sensitive to sound and light or where information confidentiality is paramount, such as library reading areas or confidential business negotiations. In these scenarios, close proximity may allow others to perceive sound leakage through bone conduction transducers. Therefore, designating haptic feedback as the preset prompting channel for this mode leverages the privacy and silence advantages of haptic perception. Furthermore, since other prompting methods are easily detected, preset rhythmic vibration parameters can be configured. By combining different vibration frequencies, intervals, and intensities, the number and priority of candidate completion information can be encoded, setting the vibration intensity to a level perceptible only to the user, thus avoiding disturbance to others in the surrounding environment.

[0115] Step C32: Prompt the user with the candidate completion information using the vibration parameters through the haptic feedback channel.

[0116] In specific implementations, for example, if the user does not have AR glasses or headphones, the smartwatch worn by the user can be controlled to emit a specific rhythmic tactile pulse sequence, thereby prompting the user with candidate completion information based on the vibration parameters. Different knowledge categories correspond to different rhythmic patterns to prompt the user to view the candidate completion information displayed on the paired terminal.

[0117] It should be noted that each channel has an availability detection interface, and will automatically degrade to the backup channel when the preferred channel becomes unavailable.

[0118] During the knowledge completion process, the system continuously monitors the inference latency and total power consumption of each pipeline. When the total power consumption exceeds the wearable device's heat dissipation budget, a frequency reduction strategy is implemented: the stutter detection frame rate is adjusted from 50ms / frame to 100ms / frame, or the HNSW retrieval accuracy is reduced (ef_search is reduced from 64 to 32). When the total inference latency on the edge exceeds the 200ms limit, a fast path is activated: skipping the non-critical feature extraction stages of EMG and IMU, retaining only the two core features of bone conduction and eye movement for stutter detection, achieving a balance between accuracy and latency. Dynamic scheduling parameters are passed to the data acquisition stage through a feedback loop to adjust the sensor acquisition frame rate and feature extraction strategy, forming an adaptive closed loop.

[0119] By automatically selecting the most appropriate prompt channel and rendering strategy based on the current environment—using semi-transparent text from AR glasses during presentations, whispering through bone conduction headphones during meetings, and haptic pulses from a smartwatch during private conversations—the gradual fade-in / fade-out rendering strategy avoids abrupt appearances of prompts. The prompts automatically disappear once the user resumes a smooth flow, making the auxiliary behavior completely imperceptible to onlookers.

[0120] This embodiment acquires the current environmental scene mode; determines a preset prompt channel based on the environmental scene mode; and outputs the candidate completion information to the user through the preset prompt channel based on the environmental scene mode. Through the above steps, the prompt output strategy is adaptively adapted to different scene constraints. While ensuring that the prompt information is effectively received by the user, the interference with the user's current interaction process is minimized, truly achieving seamless and private intelligent assistance throughout the process. This solves the technical problems of poor scene adaptability and easy interference with interaction caused by fixed output schemes.

[0121] For example, to help understand the implementation process of the real-time knowledge completion and prompting method obtained by combining this embodiment with the above embodiment one, please refer to... Figure 4 , Figure 4A simplified flowchart of a real-time knowledge completion suggestion method is provided. The process can be divided into four core stages: S1: Multimodal sensor data acquisition and preprocessing. The acquired data may include limb / head movements captured by IMU (Inertial Measurement Unit), micro-expressions or muscle tension captured by EMG (Electromyography Sensor), and speech or eye movement data. Multimodal feature vectors are obtained through preprocessing; S2: Cross-modal attention fusion and cognitive lag detection. The algorithm analyzes the acquired multimodal data to determine whether the user is in a "cognitive lag" state. If the probability of stuttering is greater than 0.72, proceed to step S3: Dialogue Context Extraction. If not, continue collecting data without providing any prompts. After dialogue context extraction, proceed to step S4: End-Side Knowledge Vector Retrieval. Retrieve relevant knowledge based on the dialogue context to obtain candidate completion information. Then proceed to step S5: Environment Scene Classification and Channel Selection. This leads to step S6: Progressive Prompt Rendering and Recovery Detection. If the stuttering is not recovered, continue to provide prompts. If the stuttering probability is less than 0.30 and lasts for 2 seconds, it is determined that the cognitive stuttering has recovered. At this point, continue collecting data, fade out the prompts, and proceed to step S7: Dynamic Scheduling of NPU Heterogeneous Computing Power based on user feedback, thereby feeding the scheduling parameters back to the data collection process.

[0122] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the real-time knowledge completion and prompting method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0123] This application also provides a real-time knowledge completion and prompting device; please refer to [reference needed]. Figure 5 The real-time knowledge completion and prompting device includes: The acquisition module 10 is used to acquire real-time user physiological behavior data collected by multiple heterogeneous sensors on the wearable device.

[0124] The detection module 20 is used to detect cognitive lag based on the user's physiological behavior data.

[0125] The acquisition module 10 is also used to acquire the current dialogue context text when it detects that the user has a cognitive lag.

[0126] The query module 30 is used to query the current dialogue context text to determine the corresponding candidate completion information.

[0127] The output module 40 is used to output the candidate completion information to the user through a preset prompt channel.

[0128] The real-time knowledge completion prompting device provided in this application, employing the real-time knowledge completion prompting method in the above embodiments, can solve the technical problem that current solutions for prompting cognitive lapses cannot adapt to different scenarios. Compared with the prior art, the beneficial effects of the real-time knowledge completion prompting device provided in this application are the same as those of the real-time knowledge completion prompting method provided in the above embodiments, and other technical features in the real-time knowledge completion prompting device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0129] In one embodiment, the detection module 20 is further configured to preprocess the user's physiological behavior data to obtain a multimodal feature vector; The cognitive lag of users is detected by using the multimodal feature vectors.

[0130] In one embodiment, the user's physiological behavior data includes: the user's audio data, gaze data, motion data, and facial data; the detection module 20 is further configured to extract Mel-frequency cepstral coefficient features from the audio data to obtain acoustic hesitation features; Based on the gaze data, the user's gaze variance and gaze features are extracted to obtain visual wandering features; Smoothness analysis is performed on the motion data to obtain motion pause characteristics; Spectral analysis was performed on the facial data to obtain anxiety micro-expression features; The acoustic hesitation feature, the visual wandering feature, the action pause feature, and the anxiety micro-expression feature are time-synchronized and aligned to construct a multimodal feature vector.

[0131] In one embodiment, the detection module 20 is further configured to input the multimodal feature vectors into a pre-trained cross-modal attention fusion model to calculate the correlation weights between each modal feature vector and output the current cognitive lag probability value. If the probability value of cognitive lag is greater than a preset detection threshold, it is determined that the user has cognitive lag. If the probability value of cognitive lag is less than or equal to the preset detection threshold, it is determined that the user does not have cognitive lag.

[0132] In one embodiment, the detection module 20 is further configured to obtain the duration when the probability of cognitive lag is less than a preset recovery threshold; If the duration exceeds a preset time threshold, the candidate completion information is controlled to fade out and disappear within a preset time period.

[0133] In one embodiment, the query module 30 is further configured to encode the current dialogue context text into a semantic query vector; Based on the semantic query vector, a search is performed in the local knowledge base to obtain the corresponding candidate completion information.

[0134] In one embodiment, the query module 30 is further configured to obtain retrieval parameters and search width; Based on the semantic query vector, an approximate nearest neighbor retrieval is performed in the local knowledge base using the retrieval parameters and the search width to obtain multiple candidate completion entries that meet the preset conditions; The corresponding candidate completion information is obtained based on multiple candidate completion entries.

[0135] In one embodiment, the output module 40 is further configured to acquire the current environmental scene mode; Determine the preset prompt channel based on the described environmental scene mode; Based on the aforementioned environmental scenario mode, the candidate completion information is output to the user through the preset prompt channel.

[0136] In one embodiment, the environment scene mode includes a speech mode, and the preset prompt channel includes a visual display channel; the output module 40 is further configured to configure preset contrast and semi-transparent visual rendering parameters when the environment scene mode is a speech mode. The candidate completion information is displayed in the user's field of vision edge area using the visual rendering parameters through the visual display channel.

[0137] In one embodiment, the environmental scene mode includes a conference mode, and the preset prompt channel includes a bone conduction audio channel; the output module 40 is further configured to configure a preset volume lower than the ambient background volume when the environmental scene mode is a conference mode. The candidate completion information is broadcast through the bone conduction audio channel at the preset volume.

[0138] In one embodiment, the environmental scene mode includes a private conversation mode, and the preset prompt channel includes a tactile feedback channel; the output module 40 is further configured to configure vibration parameters of a preset rhythm when the environmental scene mode is a private conversation mode. The haptic feedback channel sends preset rhythmic vibration parameters to the user to prompt for the candidate completion information.

[0139] In one embodiment, the multi-channel heterogeneous sensor includes at least a bone conduction microphone, an eye-tracking sensor, an inertial measurement unit, and a facial electromyography (EMG) sensor; the acquisition module 10 is further configured to acquire audio data of the user collected through the bone conduction microphone, gaze data of the user collected by the eye-tracking sensor, motion data of the user collected by the inertial measurement unit, and facial data of the user collected by the facial EMG sensor. The user's physiological behavior data is obtained by summarizing the audio data, the gaze data, the motion data, and the facial data.

[0140] In one embodiment, the acquisition module 10 is further configured to acquire the user's laryngeal vibration audio signal collected by the bone conduction microphone, and obtain audio data based on the laryngeal vibration audio signal; The coordinates of the user's gaze point and the change in pupil diameter are acquired through the eye-tracking sensor, and gaze data is obtained based on the coordinates of the gaze point and the change in pupil diameter. The user's head posture angular velocity and limb acceleration data are acquired through the inertial measurement unit, and motion data is obtained based on the head posture angular velocity and limb acceleration data. The micro-expression electrical signals of the user's facial muscles are acquired through the facial electromyography sensor, and facial data is obtained based on the micro-expression electrical signals.

[0141] This application provides a real-time knowledge completion and prompting device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the real-time knowledge completion and prompting method in the first embodiment described above.

[0142] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the real-time knowledge completion and prompting device in the embodiments of this application. The real-time knowledge completion and prompting device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), tablets, PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The real-time knowledge completion and prompting device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0143] like Figure 6As shown, the real-time knowledge completion and prompting device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the real-time knowledge completion and prompting device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the real-time knowledge completion prompting device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show real-time knowledge completion prompting devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0144] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0145] The real-time knowledge completion prompting device provided in this application, employing the real-time knowledge completion prompting method in the above embodiments, can solve the technical problem that current solutions for prompting cognitive lapses cannot adapt to different scenarios. Compared with the prior art, the beneficial effects of the real-time knowledge completion prompting device provided in this application are the same as those of the real-time knowledge completion prompting method provided in the above embodiments, and other technical features in this real-time knowledge completion prompting device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0146] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0147] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0148] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the real-time knowledge completion and prompting method in the above embodiments.

[0149] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0150] The aforementioned computer-readable storage medium may be included in the real-time knowledge completion and prompting device; or it may exist independently and not be assembled into the real-time knowledge completion and prompting device.

[0151] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the real-time knowledge completion and prompting device, the real-time knowledge completion and prompting device: acquires real-time user physiological behavior data collected by multiple heterogeneous sensors on the wearable device; performs cognitive lag detection based on the user physiological behavior data; when a cognitive lag is detected, acquires the current dialogue context text; queries the current dialogue context text to determine corresponding candidate completion information; and outputs the candidate completion information to the user through a preset prompting channel.

[0152] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0154] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0155] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described real-time knowledge completion and prompting method. This solves the technical problem that current solutions for prompting cognitive lapses cannot adapt to different scenarios. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the real-time knowledge completion and prompting method provided in the above embodiments, and will not be elaborated upon here.

[0156] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the real-time knowledge completion and prompting method described above.

[0157] The computer program product provided in this application can solve the technical problem that current solutions for prompting cognitive lag cannot adapt to different scenarios. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the real-time knowledge completion prompting method provided in the above embodiments, and will not be repeated here.

[0158] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

[0159] This application discloses A1. A real-time knowledge completion suggestion method, the real-time knowledge completion suggestion method comprising: Acquire real-time user physiological behavior data from multiple heterogeneous sensors on wearable devices; Cognitive lag detection is performed based on the user's physiological behavior data. When a user is detected to have a cognitive lapse, obtain the current conversation context text; The current dialogue context text is queried to determine the corresponding candidate completion information; The candidate completion information is output to the user through a preset prompt channel.

[0160] A2. As described in A1, the step of detecting cognitive lag based on the user's physiological behavior data includes: The user's physiological behavior data is preprocessed to obtain a multimodal feature vector; The cognitive lag of users is detected by using the multimodal feature vectors.

[0161] A3. As described in A2, the user's physiological behavior data includes: the user's audio data, gaze data, motion data, and facial data; The step of preprocessing the user's physiological behavior data to obtain multimodal feature vectors includes: Mel frequency cepstral coefficients feature extraction is performed on the audio data to obtain acoustic hesitation features; Based on the gaze data, the user's gaze variance and gaze features are extracted to obtain visual wandering features; Smoothness analysis is performed on the motion data to obtain motion pause characteristics; Spectral analysis was performed on the facial data to obtain anxiety micro-expression features; The acoustic hesitation feature, the visual wandering feature, the action pause feature, and the anxiety micro-expression feature are time-synchronized and aligned to construct a multimodal feature vector.

[0162] A4. As described in A2, the step of detecting cognitive lapses in users using the multimodal feature vector includes: The multimodal feature vectors are input into a pre-trained cross-modal attention fusion model to calculate the correlation weights between each modality feature vector and output the current cognitive lag probability value. If the probability value of cognitive lag is greater than a preset detection threshold, it is determined that the user has cognitive lag. If the probability value of cognitive lag is less than or equal to the preset detection threshold, it is determined that the user does not have cognitive lag.

[0163] A5. As described in A4, after the step of outputting the candidate completion information to the user through a preset prompt channel, the method further includes: If the probability of cognitive lag is less than a preset recovery threshold, the duration is obtained; If the duration exceeds a preset time threshold, the candidate completion information is controlled to fade out and disappear within a preset time period.

[0164] A6. As described in A1, the step of querying the current dialogue context text to determine the corresponding candidate completion information includes: Encode the current dialogue context text into a semantic query vector; Based on the semantic query vector, a search is performed in the local knowledge base to obtain the corresponding candidate completion information.

[0165] A7. As described in A6, the step of retrieving corresponding candidate completion information from the local knowledge base based on the semantic query vector includes: Get the search parameters and search width; Based on the semantic query vector, an approximate nearest neighbor retrieval is performed in the local knowledge base using the retrieval parameters and the search width to obtain multiple candidate completion entries that meet the preset conditions; The corresponding candidate completion information is obtained based on multiple candidate completion entries.

[0166] A8. As described in A1, the step of outputting the candidate completion information to the user through a preset prompting channel includes: Obtain the current environment scene mode; Determine the preset prompt channel based on the described environmental scene mode; Based on the aforementioned environmental scenario mode, the candidate completion information is output to the user through the preset prompt channel.

[0167] A9. As described in A8, the environmental scene mode includes a speech mode, and the preset prompt channel includes a visual display channel; The step of outputting the candidate completion information to the user through the preset prompt channel based on the environmental scene mode includes: When the environment scene mode is speech mode, configure preset contrast and semi-transparent visual rendering parameters. The candidate completion information is displayed in the user's field of vision edge area using the visual rendering parameters through the visual display channel.

[0168] A10. As described in A8, the environmental scene mode includes a conference mode, and the preset prompt channel includes a bone conduction audio channel; The step of outputting the candidate completion information to the user through the preset prompt channel based on the environmental scene mode includes: When the environment scene mode is conference mode, configure a preset volume that is lower than the ambient background volume; The candidate completion information is broadcast through the bone conduction audio channel at the preset volume.

[0169] A11. As described in A8, the environmental scene mode includes a private conversation mode, and the preset prompt channel includes a tactile feedback channel; The step of outputting the candidate completion information to the user through the preset prompt channel based on the environmental scene mode includes: When the environment scene mode is private conversation mode, and the vibration parameters of the preset rhythm are configured; The haptic feedback channel sends preset rhythmic vibration parameters to the user to prompt for the candidate completion information.

[0170] A12. The method as described in A1, wherein the multi-channel heterogeneous sensor includes at least a bone conduction microphone, an eye-tracking sensor, an inertial measurement unit, and a facial electromyography sensor; The steps for acquiring real-time user physiological behavior data collected by multiple heterogeneous sensors on wearable devices include: The system acquires audio data collected from the user via the bone conduction microphone, gaze data collected from the user via the eye-tracking sensor, motion data collected from the user via the inertial measurement unit, and facial data collected from the user via the facial electromyography sensor. The user's physiological behavior data is obtained by summarizing the audio data, the gaze data, the motion data, and the facial data.

[0171] A13. The method as described in A12, wherein the step of acquiring the user's audio data collected by the bone conduction microphone, the user's gaze data collected by the eye-tracking sensor, the user's motion data collected by the inertial measurement unit, and the user's facial data collected by the facial electromyography sensor includes: Acquire audio signals of the user's throat vibrations through the bone conduction microphone, and obtain audio data based on the audio signals of the throat vibrations; The coordinates of the user's gaze point and the change in pupil diameter are acquired through the eye-tracking sensor, and gaze data is obtained based on the coordinates of the gaze point and the change in pupil diameter. The user's head posture angular velocity and limb acceleration data are acquired through the inertial measurement unit, and motion data is obtained based on the head posture angular velocity and limb acceleration data. The micro-expression electrical signals of the user's facial muscles are acquired through the facial electromyography sensor, and facial data is obtained based on the micro-expression electrical signals.

[0172] This application also discloses B14. A real-time knowledge completion and prompting device, the device comprising: The acquisition module is used to acquire real-time user physiological behavior data collected by multiple heterogeneous sensors on the wearable device; The detection module is used to detect cognitive lag based on the user's physiological behavior data; The acquisition module is also used to acquire the current dialogue context text when it detects that the user has a cognitive lag. The query module is used to query the current dialogue context text to determine the corresponding candidate completion information; The output module is used to output the candidate completion information to the user through a preset prompt channel.

[0173] B15. The apparatus as described in B14, wherein the detection module is further configured to preprocess the user's physiological behavior data to obtain a multimodal feature vector; The cognitive lag of users is detected by using the multimodal feature vectors.

[0174] B16. The device as described in B15, wherein the user's physiological behavior data includes: the user's audio data, gaze data, motion data, and facial data; The detection module is also used to extract Mel frequency cepstral coefficient features from the audio data to obtain acoustic hesitation features; Based on the gaze data, the user's gaze variance and gaze features are extracted to obtain visual wandering features; Smoothness analysis is performed on the motion data to obtain motion pause characteristics; Spectral analysis was performed on the facial data to obtain anxiety micro-expression features; The acoustic hesitation feature, the visual wandering feature, the action pause feature, and the anxiety micro-expression feature are time-synchronized and aligned to construct a multimodal feature vector.

[0175] B17. The apparatus as described in B15, wherein the detection module is further configured to input the multimodal feature vector into a pre-trained cross-modal attention fusion model to calculate the correlation weights between the feature vectors of each modality, and output the current cognitive lag probability value. If the probability value of cognitive lag is greater than a preset detection threshold, it is determined that the user has cognitive lag. If the probability value of cognitive lag is less than or equal to the preset detection threshold, it is determined that the user does not have cognitive lag.

[0176] B18. The apparatus as described in B15, wherein the detection module is further configured to acquire the duration when the probability of cognitive lapse is less than a preset recovery threshold; If the duration exceeds a preset time threshold, the candidate completion information is controlled to fade out and disappear within a preset time period.

[0177] This application also discloses C19. A real-time knowledge completion and prompting device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the real-time knowledge completion and prompting method as described above.

[0178] This application also discloses D20. A storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the real-time knowledge completion and prompting method described above.

Claims

1. A real-time knowledge completion suggestion method, characterized in that, The real-time knowledge completion and prompting method includes: Acquire real-time user physiological behavior data from multiple heterogeneous sensors on wearable devices; Cognitive lag detection is performed based on the user's physiological behavior data. When a user is detected to have a cognitive lapse, obtain the current conversation context text; The current dialogue context text is queried to determine the corresponding candidate completion information; The candidate completion information is output to the user through a preset prompt channel.

2. The method as described in claim 1, characterized in that, The steps for detecting cognitive lag based on the user's physiological behavior data include: The user's physiological behavior data is preprocessed to obtain a multimodal feature vector; The cognitive lag of users is detected by using the multimodal feature vectors.

3. The method as described in claim 2, characterized in that, The user's physiological behavior data includes: the user's audio data, gaze data, movement data, and facial data; The step of preprocessing the user's physiological behavior data to obtain multimodal feature vectors includes: Mel frequency cepstral coefficients feature extraction is performed on the audio data to obtain acoustic hesitation features; Based on the gaze data, the user's gaze variance and gaze features are extracted to obtain visual wandering features; Smoothness analysis is performed on the motion data to obtain motion pause characteristics; Spectral analysis was performed on the facial data to obtain anxiety micro-expression features; The acoustic hesitation feature, the visual wandering feature, the action pause feature, and the anxiety micro-expression feature are time-synchronized and aligned to construct a multimodal feature vector.

4. The method as described in claim 2, characterized in that, The step of detecting cognitive lag in users using the multimodal feature vector includes: The multimodal feature vectors are input into a pre-trained cross-modal attention fusion model to calculate the correlation weights between each modality feature vector and output the current cognitive lag probability value. If the probability value of cognitive lag is greater than a preset detection threshold, it is determined that the user has cognitive lag. If the probability value of cognitive lag is less than or equal to the preset detection threshold, it is determined that the user does not have cognitive lag.

5. The method as described in claim 4, characterized in that, After the step of outputting the candidate completion information to the user through a preset prompt channel, the method further includes: If the probability of cognitive lag is less than a preset recovery threshold, the duration is obtained; If the duration exceeds a preset time threshold, the candidate completion information is controlled to fade out and disappear within a preset time period.

6. The method as described in claim 1, characterized in that, The step of querying the current dialogue context text to determine the corresponding candidate completion information includes: Encode the current dialogue context text into a semantic query vector; Based on the semantic query vector, a search is performed in the local knowledge base to obtain the corresponding candidate completion information.

7. The method as described in claim 6, characterized in that, The step of retrieving corresponding candidate completion information from the local knowledge base based on the semantic query vector includes: Get the search parameters and search width; Based on the semantic query vector, an approximate nearest neighbor retrieval is performed in the local knowledge base using the retrieval parameters and the search width to obtain multiple candidate completion entries that meet the preset conditions; The corresponding candidate completion information is obtained based on multiple candidate completion entries.

8. A real-time knowledge completion and prompting device, characterized in that, The device includes: The acquisition module is used to acquire real-time user physiological behavior data collected by multiple heterogeneous sensors on the wearable device; The detection module is used to detect cognitive lag based on the user's physiological behavior data; The acquisition module is also used to acquire the current dialogue context text when it detects that the user has a cognitive lag. The query module is used to query the current dialogue context text to determine the corresponding candidate completion information; The output module is used to output the candidate completion information to the user through a preset prompt channel.

9. A real-time knowledge completion and prompting device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the real-time knowledge completion suggestion method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the real-time knowledge completion and prompting method as described in any one of claims 1 to 7.

Images