Evaluation system and evaluation method for the real-time fusion of physiological and user interaction data to provide adaptive feedback

Abstract

Evaluation system for obtaining and analyzing user interaction and physiological data to provide adaptive feedback, comprising: a portable multi-sensor device comprising: an electroencephalography sensor designed to measure the user's brainwave signals; and a galvanic skin response sensor designed to measure skin conductivity, which indicates the user's stress level; an interactive assembly device designed to recognize the user's assembly actions and generate corresponding assembly data; a computing device in communication with the wearable multi-sensor device and the interactive montage device, the computing device configured to receive and combine the brainwave signals, the skin conductance, and the montage data. Analyze the combined data in real time to create an estimate of the user's cognitive state and provide adaptive feedback to the user based on the cognitive state estimate via at least one output module. wherein the computing device is further configured to store the brainwave signals, the skin conductance, the compilation data, and the cognitive state estimate in a database for later retrieval and review.

Description

AREA OF DISCLOSURE

[0001] The present disclosure relates generally to interactive systems that monitor the performance and physiological responses of users, and more particularly to a system and method for fusing data from multiple sensors - such as electroencephalography (EEG) and galvanic skin response (GSR) - with task-based user interaction data to provide adaptive, real-time feedback. BACKGROUND OF THE INVENTION

[0002] Various interactive systems have been developed to engage users in physical or virtual tasks while tracking their performance. In recent years, wearable devices and intelligent assembly devices (e.g., programmable blocks or puzzle systems) have become popular for entertainment, educational, or therapeutic purposes. These existing systems often measure basic data, such as the number of errors made when assembling a puzzle, or record limited physiological parameters (e.g., heart rate).

[0003] However, many current solutions fail to fully integrate continuous physiological data from multiple channels (e.g., EEG signals, skin conductance, eye tracking) with real-time performance data, making it impossible to adapt tasks in real time in a highly personalized manner. Furthermore, systems that store such data for later review often lack advanced predictive algorithms that can detect trends or precursors of cognitive decline. Therefore, there is a need for a more robust system and method that comprehensively integrates diverse data streams in real time, providing immediate feedback and long-term analytics for applications in healthcare, education, or other fields. SUMMARY OF THE INVENTION

[0004] The present disclosure provides an assessment system and method for the real-time fusion of physiological and user interaction data, capable of dynamically adapting adaptive feedback to the user's cognitive or emotional state. In certain embodiments, a wearable multi-sensor device captures the user's EEG (electroencephalography) and GSR (galvanic skin response) signals and may additionally capture gaze tracking or pupil dilation data. Simultaneously, an interactive building device, such as a set of smart blocks or a puzzle-based interface, detects the user's actions and captures performance metrics, including building accuracy and completion time.

[0005] A computing device receives these data streams and processes them using one or more predictive models. By correlating physiological signals with user interaction performance data, the assessment system can infer the user's cognitive load, emotional state, or stress level in real time. Based on these inferences, an adaptive feedback mechanism adjusts task parameters—for example, by increasing or decreasing the difficulty level, providing mid-task hints, or modifying the instructional content—to improve user engagement and outcomes. The assessment system stores all collected data in a database, allowing for immediate review and longitudinal analysis across multiple tasks.Clinicians, caregivers, or educators can remotely monitor the user's progress and compare current data with previous baseline values ​​or peer group benchmarks. Thus, the present disclosure provides an improved and robust platform for applications such as training, therapy, or entertainment that require precise monitoring and adaptive interventions in real time.

[0006] Through these and other aspects disclosed herein, the present disclosure overcomes the disadvantages of known systems through comprehensive sensor data acquisition, advanced predictive analytics, real-time feedback loops, and flexible long-term data storage and retrieval. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows a schematic diagram of an embodiment of the evaluation system of the present disclosure. Fig. 2 shows a schematic diagram of an embodiment of the portable multi-sensor device of the present disclosure. Fig. 3 shows a schematic diagram of another embodiment of the portable multi-sensor device of the present disclosure. Fig. 4 shows a schematic diagram of an embodiment of the interactive assembly device of the present disclosure. Fig. 5 shows a schematic diagram of an embodiment of the computing device of the present disclosure. Fig. 6 shows a flowchart of an embodiment of the evaluation method of the present disclosure. DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] With reference to Fig. 1 shows Fig. 1 is a schematic diagram of one embodiment of the assessment system of the present disclosure. In this embodiment, the assessment system 10 includes a wearable multi-sensor device 100, an interactive assembly device 200, a computing device 300, and a database 400. These components work together to provide adaptive, real-time feedback based on aggregated physiological and user interaction data. This configuration supports a variety of applications, such as cognitive assessments, therapeutic interventions, and educational training.

[0008] With reference to the Fig. 2 shows a schematic diagram of one embodiment of the wearable multi-sensor device of the present disclosure. The wearable multi-sensor device 100 is configured to collect user physiological data in real time. In one embodiment, the wearable device 100 includes an electroencephalography (EEG) sensor 110 positioned on or near the user's scalp to measure EEG signals and a galvanic skin response (GSR) sensor 120 that detects changes in the user's skin conductance indicative of stress or emotional states. In some embodiments, the wearable multi-sensor device 100 may also be equipped with an eye-tracking sensor 130, which uses an optical sensor or camera to track the user's gaze direction and pupil changes.In certain embodiments, the portable device 100 includes an integrated edge processor 140 that performs initial filtering of the raw EEG and GSR signals before transmitting them to the computing device 300. This approach improves data accuracy through noise reduction and signal stabilization.

[0009] With reference to Fig. 3 shows a schematic diagram of another embodiment of the wearable multi-sensor device of the present disclosure. In certain embodiments of the present disclosure, the wearable multi-sensor device 100 may include additional biometric sensors beyond the EEG sensor 110 and the GSR sensor 120. For example, the wearable device 100 may include a heart rate sensor 150 that non-invasively measures a user's pulse using photoplethysmography, and an accelerometer 160 that detects the user's body movements or head movements. In one example configuration, the heart rate sensor 150 is positioned near the user's temple or wrist, while the accelerometer 160 is integrated into the main body of the wearable device 100.These additional signals can be captured simultaneously with EEG and GSR data, allowing the evaluation system 10 to create a more holistic profile of the user's physiological and behavioral states. For example, by analyzing heart rate variability along with EEG frequency shifts, the evaluation system 10 can better distinguish between heightened stress responses and mere physical exertion.

[0010] To process the various data streams, the wearable device 100 or the computing device 300 may implement one or more signal processing modules (in this embodiment, the signal processing module 170 is configured in the computing device 300) that clean, filter, and align these streams in real time. For example, EEG signals may be passed through a bandpass filter 172, e.g., 0.5 to 50 Hz, to remove low-frequency fluctuations and high-frequency noise. In some embodiments, the EEG sensor 110 may be further equipped with artifact detection algorithms 112 that identify and suppress signal disturbances caused by muscle activity, eye blinking, or sudden movements of the user.The GSR sensor 120 can also undergo baseline correction to ensure that gradual changes in skin conductance are reliably distinguished from brief spikes due to transient stimuli. Meanwhile, each accelerometer signal can be passed through a motion artifact reduction algorithm, which flags excessive user movement that might otherwise contaminate the EEG or GSR data. Once these filtering or artifact suppression processes are complete, the wearable device 100 or computing device 300 can apply data fusion techniques to precisely synchronize the resulting data streams.For example, each signal can be timestamped and placed on a common timeline, allowing the evaluation system 10 to correlate changes in EEG amplitude or alpha / beta wave ratio with concurrent shifts in GSR, heart rate, or user movement. In some embodiments, a Kalman filter 174 or an extended Kalman filter could be employed to merge data from multiple sensors and reduce inherent measurement noise. By integrating these advanced fusion strategies, the evaluation system 10 obtains a refined view of the user's physiological states, allowing the predictive AI module 320 to more reliably infer nuanced cognitive or emotional states.

[0011] In addition to real-time processing, the data fusion module 310 (as shown in Fig. 5) or an integrated edge processor 140 maintain calibration routines that adapt to user-specific signal characteristics during repeated interactive tasks. For example, individuals with naturally higher baseline GSR values ​​or faster resting heart rates could benefit from personalized filter thresholds or artifact detection algorithms. The evaluation system 10 can store these user-specific calibration parameters in the database 400, ensuring that each subsequent interactive task begins with optimal signal processing settings. In certain embodiments, the machine learning model 322 within the predictive AI module 320 (as shown in Fig. 5) can analyze archived data to improve artifact classification accuracy over time by effectively "learning" from previously encountered noise or drift patterns. This continuous refinement further increases the reliability of the real-time feedback generated by the adaptive feedback engine 330.

[0012] By combining advanced sensors and robust signal processing techniques, the present disclosure provides the flexibility to capture a rich array of physiological metrics, such as EEG signals, GSR readings, heart rate, and movement data, all of which contribute to a more comprehensive assessment of a user's mental and physical state. By minimizing noise and ensuring synchronized data streams, the assessment system 10 can detect subtle correlations, such as whether a sudden increase in stress correlates with a user's hand tremor detected by the accelerometer or a concurrent increase in beta-band EEG activity. These correlations enable more precise adaptive interventions, allowing the invention to promptly respond with task adjustments, calming prompts, or other feedback tailored to the user's current cognitive load and physical state.

[0013] With reference to Fig. 4 shows a schematic diagram of one embodiment of the interactive assembly apparatus of the present disclosure. The interactive assembly apparatus 200 is designed to engage the user in physical or virtual tasks. For example, the interactive assembly apparatus 200 may include a plurality of smart blocks 210, each equipped with force sensors 212 and / or orientation sensors 214 to detect how the smart blocks 210 are placed or rotated. These smart blocks 210 communicate with a base unit 220 via wired or wireless connections, and the base unit 220 collects user performance data such as assembly time, block orientation, accuracy, and number of defects. The interactive assembly apparatus 200 may also be implemented as a touchscreen puzzle or as a 3D-printed kit that records touch inputs and placement data.Regardless of the form, it generates “assembly data” that relates to how the user interacts with the task.

[0014] With reference to Fig. 5 shows a schematic diagram of an embodiment of the computing device of the present disclosure. The computing device 300, as shown in Fig. 5, receives and processes physiological data from wearable device 100 and assembly data from assembly device 200. In one embodiment, it merges the various data streams in a data fusion module 310 and ensures that EEG signals, GSR measurements, and block assembly data are aligned to common temporal references. The predictive AI module 320 analyzes these combined signals to infer the user's cognitive load or emotional state. This predictive AI module 320 includes a trained machine learning model 322 that uses features extracted from the EEG, such as the alpha / beta wave ratio, fluctuations in GSR over time, and user interaction patterns, such as speed and accuracy. Based on this analysis, the adaptive feedback engine 330 adjusts difficulty levels, modifies instructions, or issues calming prompts in real time.This feedback is output via one or more output modules 340, such as a screen, an audio device, or a haptic actuator.

[0015] In certain embodiments of the present disclosure, the predictive AI module 320 within the computing device 300 goes beyond simple pattern recognition by implementing advanced machine learning architectures to analyze time-synchronized data from the wearable multi-sensor device 100 and the interactive assembly device 200. This module may use a combination of neural networks, statistical learning algorithms, or hybrid approaches to extract complex patterns from streams of physiological signals such as EEG and GSR, along with user performance metrics such as processing times, error rates, and reaction speeds.An illustrative example is a deep neural network that takes a concatenated time window of EEG data—possibly filtered into relevant frequency bands such as alpha, beta, and gamma—and correlates these features with concurrent GSR fluctuations and accelerometer readings (if available). By using multiple convolutional layers, the deep neural network can isolate spatiotemporal features indicative of increased stress, sustained concentration, or emerging fatigue, and then incorporate them into the system's real-time assessment of the user's cognitive state.

[0016] In an alternative embodiment, the predictive AI module 320 uses recurrent neural networks (RNNs), such as LSTM (Long Short-Term Memory) or Gated Recurrent Unit (GRU) architectures, to capture dependencies over longer time horizons. These recurrent models track how the user's physiological and performance indicators evolve during an interactive task, enabling more accurate detection of slowly building stress or progressive learning curves. For example, if the user's EEG alpha wave ratio gradually decreases while the GSR remains persistently elevated, the model can infer chronic anxiety rather than a brief startle response. This improved temporal sensitivity allows the adaptive feedback engine 330 to proactively adjust the complexity of puzzles or block arrangement challenges before the user becomes overwhelmed.

[0017] Feature extraction and labeling can be further refined using unsupervised learning techniques, especially when large amounts of unlabeled physiological data are collected in the database. 400 clustering algorithms can classify user states into categories such as "low engagement," "moderate engagement," and "high stress," even without explicit ground-truth labels. Once clusters are identified, they can be labeled or refined over time using supervised methods based on clinical or educational feedback from nurses, therapists, or instructors. The system can also incorporate a semi-supervised learning approach, where user-labeled samples (e.g., moments when the user presses a "help" button) trigger the labeling process, while unlabeled samples are automatically classified by the model as it gains confidence.

[0018] In some embodiments, the predictive AI module 320 implements adaptive threshold algorithms that adjust the stress or cognitive load thresholds on a per-user basis. By analyzing previous interactive tasks, the module identifies each individual's typical EEG and GSR baselines and adjusts the thresholds accordingly. This personalization helps reduce false alarms that may arise from innate physiological differences between users, such as a naturally high GSR in warmer climates or a consistently lower resting alpha-wave ratio in advanced age groups. Such adjustment can be recalibrated periodically if the user or a caregiver detects discrepancies between the system's alerts and the user's subjective experience.

[0019] Model training and updates can occur either locally on the computing device 300 or remotely in a cloud-based environment, depending on computing resources and privacy requirements. In one scenario, after each interactive task, the device 300 uploads raw or partially processed sensor data to the database 400, where a server-side training pipeline incrementally retrains or fine-tunes the neural networks using new data from multiple users. Once the improved models are validated, they can be deployed back to each user's local device. This iterative feedback cycle ensures that the AI ​​module continuously evolves and maintains accuracy even as new sensor technologies are introduced or user populations change.

[0020] By leveraging these sophisticated machine learning strategies, the invention dynamically infers the user's state with increased accuracy, detecting subtle cognitive or emotional signals that may elude simpler rule-based systems. The real-time outputs of the predictive AI module 320 inform the adaptive feedback engine 330, enabling more targeted and context-sensitive interventions—be it in the form of soothing audible cues, step-by-step instructions, or deliberately challenging tasks for users who appear under-challenged. Overall, this fusion of advanced analytics with synchronized multi-sensor data not only strengthens the system's short-term responsiveness but also provides valuable longitudinal data for long-term studies of cognitive performance and well-being.

[0021] Coming back to the Fig. 1, the acquired data is stored in database 400, which can be located locally or on a cloud server. Database 400 contains both raw sensor signals and derived metrics, allowing authorized users to later retrieve and analyze the information. This database 400 facilitates longitudinal studies and comparisons across multiple interactive tasks or multiple users. By managing secure, access-controlled files, the system supports healthcare, research, or educational environments where historical performance trends are relevant.

[0022] In certain embodiments of the present disclosure, the assessment system 10 implements advanced privacy and security measures to protect user data during transmission, storage, and processing. In one embodiment, all physiological signals (e.g., EEG, GSR, heart rate) and user interaction data (e.g., assembly actions, performance metrics) transmitted between the wearable multi-sensor device 100 and the computing device 300 are encrypted using the Advanced Encryption Standard (AES) with a 256-bit key, thereby mitigating the risk of unauthorized access or interception. Data stored in the database 400 may also be encrypted at rest using AES-256 encryption to ensure that sensitive information remains protected even if the storage hardware is compromised.

[0023] To further protect personally identifiable information (PII), the system optionally applies data anonymization techniques before storing or sharing records. For example, the system uses k-anonymity or similar algorithms to obscure individual user records, reducing the likelihood that an unauthorized user can trace certain physiological data back to a specific user. Furthermore, role-based access control (RBAC) layers can regulate the permissions of different user roles—such as administrators, clinicians, or researchers—so that only those with appropriate credentials can access or modify sensitive data. A secure user portal also allows individuals to view, correct, or delete their data, while the system maintains logs of all access events or changes for auditability.

[0024] In addition to privacy and security, performance goals in certain embodiments focus on reducing system latency to enable near-real-time adaptive feedback. For example, the system may guarantee a response time of less than two seconds, from the time EEG or GSR signals are collected and fused with user interaction data to the time an updated task difficulty score is output. By employing advanced signal processing and machine learning algorithms, some embodiments maintain a data accuracy rate of at least 95% in classifying users' stress states or cognitive load levels. This combination of low-latency response times and robust classification accuracy ensures that the assessment system 10 can provide timely and reliable feedback to users in various operating environments.

[0025] With reference to the Fig.6 shows a flowchart of one embodiment of the assessment method of the present disclosure. At the beginning of an interactive task, as shown in step S110, the user logs into the computing device 300, which retrieves relevant profile information from the database 400. Then, as shown in step S120, the user wears the wearable device 100 to ensure that the EEG sensor 110 and the GSR sensor 120 are properly calibrated. If an eye-tracking sensor 130 is present, it must be aligned to collect meaningful data. Next, as shown in step S130, the interactive montage 200 is activated or connected, and any necessary instructions are provided to guide the user. Thereafter, as shown in step S140, the wearable device 100 continuously transmits EEG and GSR data, which is optionally preprocessed to remove noise or artifacts.Meanwhile, as shown in step S150, the interactive assembly device 200 monitors the user's performance in assembling blocks or solving puzzles and generates "assembly data" reflecting the speed, accuracy, time spent, and any notable actions or errors.

[0026] In this embodiment, as shown in step S160, the data fusion module 310 of the computing device 300 merges these physiological data streams with the assembly data and timestamps them to maintain proper synchronization. As shown in step S170, the predictive AI module 320 examines patterns in the EEG waveforms and GSR fluctuations and identifies indicators of cognitive stress or engagement. By correlating these physiological signals with metrics of user interaction—such as the time taken to place a block or the error rate—as shown in step S180, the predictive AI module 320 infers whether the user is struggling, performing well, or showing signs of fatigue.If the derived cognitive load exceeds a predetermined threshold, as illustrated in step S183, the adaptive feedback engine 330 may reduce the difficulty of the assembly task, provide hints, or instruct the user to take a short break. Conversely, if the assessment system 10 determines a relatively low stress level and strong performance, as illustrated in step S187, the assessment system 10 may increase the complexity of the task or present a new challenge.

[0027] During the interactive task, these automatic adjustments continue as new data is received from the wearable device 100 and the assembly device 200. This iterative feedback loop provides a highly personalized experience and ensures that the tasks are neither too easy nor too challenging for the user. The computing device 300 regularly transmits updated sensor data, derived metrics, and user interaction records to the database 400. At the end of the interactive task, the user can receive a summary report detailing average stress levels, cumulative assembly times, error rates, and any relevant trends or anomalies. This summary is also stored in the database 400 for later retrieval and analysis by the user, caregivers, or medical professionals.

[0028] In one embodiment for cognitive training of elderly users, wearable device 100 measures EEG and GSR signals while elderly individuals assemble physical puzzles. The predictive AI module 320 monitors these signals to determine if the user is experiencing unusual stress, confusion, or fatigue. If the detected stress exceeds a threshold, the adaptive feedback engine 330 can provide easier tasks or add supportive instructions. All data generated by the interactive tasks is stored in database 400, allowing caregivers to track progress over time and detect early signs of cognitive decline.

[0029] In another embodiment aimed at children's education, the evaluation system 10 uses smart blocks 210 decorated with different colors and shapes. As children solve color-coded or shape-based puzzles, real-time EEG and GSR measurements provide insight into attention span and emotional engagement. If the evaluation system 10 detects a decline in engagement, it triggers more creative mini-challenges or provides supportive audio prompts. This maintains children's interest, and their continuous physiological responses confirm the effectiveness of the interventions.

[0030] Another embodiment is a remote therapy scenario where healthcare providers access database 400 via a secure portal. Therapists can schedule tasks for patients using wearable device 100 and montage device 200 at home. If assessment system 10 detects increased anxiety or cognitive overload during the task, it can immediately adjust the task. Therapists can review quantitative data—such as EEG waveforms and montage completion rates—and qualitative measurements, including recorded videos of the user's facial expressions or body language, to adjust subsequent therapy tasks.

[0031] Although specific embodiments have been described, those skilled in the art will recognize that variations and modifications may be made without departing from the overall spirit of the invention. For example, wearable device 100 may integrate additional sensors, such as heart rate monitors or accelerometers, while interactive device 200 could be implemented in virtual reality or augmented reality environments instead of physical blocks. Computing device 300 may also employ various machine learning methods, ranging from traditional statistical methods to deep neural networks, depending on system requirements and available computing resources.

[0032] By combining multi-sensor data acquisition, AI-driven analysis, and adaptive feedback mechanisms, this assessment system provides a powerful means for dynamically adapting tasks based on the user's cognitive load or stress level. By integrating real-time assembly data with simultaneous physiological monitoring, the invention not only provides immediate support for the user but also provides extensive data recording for long-term analysis in training, therapy, and research environments. This integrated, iterative approach distinguishes the invention from conventional systems that rely on limited data streams or delayed offline processing.

[0033] The present disclosure provides an evaluation system and method for collecting and analyzing multiple streams of physiological and user interaction data to enable adaptive, real-time feedback. A wearable multi-sensor device measures the user's brainwave signals and skin conductance and optionally collects pupil dilation or eye tracking data. Simultaneously, an interactive montage device generates task-related performance data. A computing device synthesizes these data streams to determine the user's cognitive or emotional state and automatically adjusts the task difficulty or instructional content. The system stores the measured data and associated feedback in a database, enabling trend analysis, personalized profiling, and comparisons across multiple tasks or users.

Claims

[1] Evaluation system for obtaining and analyzing user interaction and physiological data to provide adaptive feedback, comprising: a portable multi-sensor device comprising: an electroencephalography sensor designed to measure the user's brainwave signals; and a galvanic skin response sensor designed to measure skin conductivity, which indicates the user's stress level; an interactive assembly device designed to recognize the user's assembly actions and generate corresponding assembly data; a computing device in communication with the wearable multi-sensor device and the interactive montage device, the computing device configured to receive and combine the brainwave signals, the skin conductance, and the montage data. Analyze the combined data in real time to create an estimate of the user's cognitive state and provide adaptive feedback to the user based on the cognitive state estimate via at least one output module. wherein the computing device is further configured to store the brainwave signals, the skin conductance, the compilation data, and the cognitive state estimate in a database for later retrieval and review. [2] The assessment system of claim 1, wherein the wearable multi-sensor device further comprises a gaze tracking sensor configured to detect gaze direction or pupil dilation, and wherein the computing device fuses the gaze direction or pupil dilation with the brainwave signals, skin conductance, and montage data. [3] The evaluation system of claim 1, wherein the interactive assembly device comprises a plurality of smart blocks, each smart block including at least one force sensor configured to detect the force or pressure applied by the user during assembly of the smart blocks. [4] The assessment system of claim 1, wherein the adaptive feedback comprises automatically adjusting the difficulty level of the interactive assembly task by changing the instructions displayed on a screen. [5] The assessment system of claim 1, further comprising a camera configured to capture a video of the user, wherein the computing device is configured to analyze the user's facial expressions and further refine the cognitive state estimate based on the facial expressions. [6] The assessment system of claim 1, wherein the computing device comprises a predictive model trained to detect the potential onset of cognitive decline by analyzing historical trends in the fused data across multiple user tasks. [7] The assessment system of claim 1, wherein the wearable multi-sensor device further comprises a heart rate sensor configured to measure the user's pulse using photoplethysmography, and wherein the computing device is configured to combine heart rate data with the brainwave signals, skin conductance, and montage data to produce a refined estimate of cognitive state. [8] The assessment system of claim 1, wherein the portable multi-sensor device further comprises an accelerometer configured to detect motion artifacts, and wherein the computing device is configured to reduce noise in the brainwave signals or skin conductance based on motion data from the accelerometer. [9] The assessment system of claim 1, wherein the computing device is further configured to implement one or more filtering or artifact detection algorithms that apply bandpass filtering to the brainwave signals and baseline correction to the skin conductance to improve the accuracy of the cognitive state estimation. [10] The rating system of claim 1, wherein the predictive model is a recurrent neural network trained to detect user stress or cognitive overload by analyzing historical dependencies in the fused data, the predictive model comprising at least one of a long-short-term memory or gated recurrent unit architecture.

Images