A low-cost real-time comprehension level recognition system based on pupil data

Abstract

This disclosure provides a low-cost, real-time comprehension level recognition system based on pupil data. It comprises two parts: a front-end and a back-end. The front-end includes a user interface display and interaction unit that captures facial video stream data from a webcam, collects and generates pupil data in real time, and transmits it to the back-end. The back-end preprocesses the pupil data and performs real-time analysis and prediction through multiple modules, including: a face detection module; a pupil extraction module; a model recognition module: based on a pre-trained deep learning model, using the input pupil sequence data to predict the user's subjective and objective comprehension levels; and an adaptive system: including automated measures based on the user's comprehension level, supplementing additional information explanations to apply the comprehension level and improve the user experience. Ultimately, it achieves automated, real-time assessment of the user's comprehension level in multimodal scenarios.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more specifically, to a low-cost, real-time understanding degree recognition system based on pupil data. Background Technology

[0002] In the current development of human-computer interaction systems, users' ability to understand information (i.e., their level of comprehension) has become a key factor affecting the quality and effectiveness of the interactive experience. Accurate identification and analysis of the level of comprehension not only helps optimize the user experience but also supports the provision of personalized services. However, existing methods for identifying the level of comprehension have several technical limitations, making them difficult to apply widely in the real world.

[0003] First, traditional methods for identifying comprehension levels primarily rely on self-reporting and performance testing, such as listening tests, reading comprehension tests, and multiple-choice tests (technical approaches such as...). Figure 1 (As shown). These methods can provide valuable data and insights in controlled experimental environments, but their results are often not available in real time and lack the ability to reflect the user's immediate state. Furthermore, these methods rely on active user participation and self-reporting, and this subjectivity can lead to biases such as "initial elevation bias" (i.e., users may make inaccurate self-assessments of their understanding in the initial stages). These subjective factors not only affect the reliability of the data but may also disrupt the normal user interaction process in practical applications, degrading the user experience.

[0004] Secondly, in order to overcome these limitations, some understanding-based recognition technologies based on physiological signals and machine learning have been developed in recent years (technical approaches such as...). Figure 2 (As shown). These technologies assess a user's comprehension of information by collecting physiological indicators such as electroencephalogram (EEG) signals and eye-tracking data. While these methods can achieve real-time recognition to some extent and provide a more objective data basis, they still face challenges such as high cost and inconvenience of use. EEG devices require complex hardware support and professional operators, and although eye trackers can capture a user's gaze point and eye trajectory, they are usually expensive and not portable, limiting the application of these technologies in daily life.

[0005] Furthermore, existing comprehension recognition technologies are mostly limited to single-modal scenarios, such as measuring comprehension only for text or speech. This single-mode limitation prevents them from providing effective comprehension assessment in more complex multimodal environments, such as scenarios combining multiple input forms like text, speech, and images. This limitation restricts the applicability of the technology in various application scenarios, such as education, healthcare, and intelligent assistants, which require multimodal interaction.

[0006] Existing technical methods have failed to gain widespread adoption in real-world applications due to their reliance on expensive and complex external devices, lack of real-time feedback capabilities, limitations of single-modality approaches, and negative impacts on user experience. To address these issues, a new technical solution is urgently needed. This solution should possess characteristics such as low cost, real-time performance, user-friendliness, and multimodal support, thereby enabling accurate comprehension assessments for users in a wider range of application scenarios. Summary of the Invention

[0007] The purpose of this disclosure is to provide a low-cost, real-time understanding level recognition system based on pupil data, addressing numerous shortcomings of existing technologies in understanding level recognition, including high cost, single-modal limitations, poor user experience, low automation levels, and lack of real-time feedback. These problems mainly stem from the high dependence of existing methods on expensive external devices and their excessive focus on single-modal data, failing to effectively integrate multiple information sources and interaction modes.

[0008] This technical solution provides a low-cost, real-time understanding degree recognition system based on pupil data, consisting of two parts: front-end and back-end.

[0009] The front-end includes a user interface display and interaction unit. It captures the user's facial video stream data by accessing a webcam, collects and generates the user's pupil data in real time, and transmits the pupil data to the back-end via a Socket connection. Specifically, the front-end uses a facial detection algorithm (MediaPipe) to identify the user's facial region and then locates the eye region. Based on this, a pupil tracking algorithm is used to identify the pupil's position and size. First, the system acquires video stream data through the webcam and extracts the eye region in each frame, accurately calculating the pupil center and diameter using edge detection and image segmentation techniques. Second, the front-end tracks the user's pupil movement in real time, capturing dynamic changes in the pupil, including pupil dilation and constriction (i.e., pupillary response). This process utilizes image processing techniques, including circle detection and contour extraction, to obtain the accurate position of the pupil and update the data. The pupil data (including position, diameter, rate of change, etc.) is transmitted to the back-end in real time via a Socket connection, where the back-end further analyzes this data.

[0010] The backend receives the pupil data transmitted from the frontend and preprocesses it, and performs real-time analysis and prediction through multiple modules, including:

[0011] The face detection module transmits the ratio to the pupil extraction module through the calibration module, and transmits the processed frame data to the pupil extraction module through the face recognition and feature extraction module.

[0012] The pupil extraction module uses a calibration ratio to process frame data, captures and collects pupils, obtains pupil sequence data, and transmits it to the model recognition module.

[0013] The model recognition module takes pupil sequence data as input and employs a pupil extraction deep learning model. It constructs a complete analysis method by embedding convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), bidirectional long short-term memory networks (BiLSTM), and Transformer models. To improve prediction accuracy and stability, a soft voting strategy from ensemble learning is used to integrate the prediction results of different network models. Ensemble learning further enhances the model's adaptability and predictive ability in different contexts. The module analyzes the time-series changes in pupil data to predict the user's subjective and objective understanding. Specifically, the model recognition module first receives pupil data sequences that have been collected and preprocessed by the front end. This data includes pupil size, position, and dynamic changes. Subsequently, it is processed through the following deep learning networks:

[0014] Convolutional Neural Network (CNN): This model employs a two-layer CNN structure, with each layer containing 64 convolutional kernels of 3x3 size. The ReLU activation function is used to enhance non-linear processing capabilities. The first convolutional layer reduces data dimensionality through max pooling with a stride of 2, effectively halving the data dimension. After convolution, the data is fed into a fully connected layer, and finally, a comprehensibility score is generated using the Sigmoid activation function. This structure can quickly extract local features and is suitable for processing high-dimensional time series data.

[0015] Recurrent Neural Networks (RNNs): This model employs a two-layer RNN structure, with each layer containing 256 hidden units. RNNs are highly capable of processing time-series data and are suitable for capturing the dynamic features of time sequences within the data. As data is input, the RNN layers progressively update their internal states, and at the final time point, a fully connected layer generates the user's understanding score.

[0016] Long Short-Term Memory (LSTM) Networks: LSTM uses a two-layer structure with 128 hidden units per layer, effectively handling dependencies in long-term sequences. In pupil data processing, LSTM is particularly suitable for capturing long-term dependencies in pupil size changes over time. LSTM uses a dropout rate of 0.5 to prevent overfitting. The output is mapped to a range of 0 to 1 using a sigmoid function to generate a user comprehension score.

[0017] Bidirectional Long Short-Term Memory (BiLSTM) Network: BiLSTM uses a bidirectional structure (i.e., forward and backward LSTM layers) to enable the model to consider information from both past and future time steps simultaneously, thus providing a comprehensive understanding of pupil data. Each layer contains 128 hidden units and uses a dropout rate of 0.5. The output of the BiLSTM model is converted into a comprehension score using a sigmoid function.

[0018] The Transformer model utilizes self-attention, employing a multi-layer encoder (16 layers) with 256 hidden units per layer and 8 attention heads. This allows for parallel processing of large-scale time-series data, making it suitable for handling long temporal dependencies in pupil data. When generating comprehension scores, the model maps the final encoder output to the scoring space through fully connected layers. The Transformer model's advantage lies in its ability to handle long-term dependencies and its superior performance on large-scale datasets.

[0019] The model predicts user comprehension performance, encompassing both subjective and objective levels of understanding. Subjective comprehension refers to participants' self-reported level of understanding, typically collected through questionnaires, such as participants' perceived understanding of a task or information. Objective comprehension, on the other hand, refers to the proportion of information a user can accurately repeat or reproduce, reflecting their actual grasp of the content. It is usually measured through tests or quizzes, such as the percentage of knowledge points a user can correctly answer or the accuracy with which they repeat given content. These two indicators together provide a comprehensive picture of user comprehension; subjective comprehension reflects cognitive perception, while objective comprehension focuses on actual accuracy. The formula for calculating subjective comprehension can be expressed as:

[0020]

[0021] The comprehension score is the participant's self-reported comprehension score, with the highest possible score being the highest rating, typically 9 points. This metric reflects the user's subjective feelings about the task or information.

[0022] The formula for calculating the degree of objective understanding can be expressed as:

[0023]

[0024] The correctly paraphrased content refers to the portion of the information that the user can accurately understand and paraphrase, while the total content refers to all the content that needs to be paraphrased. This indicator objectively reflects the user's actual grasp of the information.

[0025] Finally, an adaptive system is used to apply the level of understanding to improve the user experience. Specifically, these two metrics are core to the evaluation of understanding performance: subjective understanding focuses on the user's perception, while objective understanding emphasizes accuracy and paraphrasing ability. In the field of artificial intelligence (AI), measuring understanding is becoming increasingly important because users need to understand information such as text, speech, and images to effectively interact with AI systems. By measuring and analyzing users' understanding performance, AI systems can adjust the interactive content according to the user's understanding level, ensuring that information presentation matches the user's comprehension ability, thereby optimizing the interaction effect.

[0026] The preprocessing method includes noise reduction and normalization.

[0027] The technical effects to be achieved by the embodiments of the present invention are as follows:

[0028] This paper proposes a low-cost, real-time, deep learning-based system for recognizing user comprehension levels using pupil data. The system utilizes common webcams to collect user pupil data without requiring any complex external equipment, and can automatically and in real-time assess user comprehension levels in multimodal scenarios. By introducing advanced deep learning algorithms, the system not only significantly reduces hardware costs and operational complexity but also substantially improves recognition accuracy and user experience. This innovative technological approach overcomes many shortcomings of existing technologies, providing a more efficient and practical solution for the field of intelligent human-computer interaction.

[0029] Furthermore, it can achieve the following technical effects:

[0030] 1. Reduce the cost of using the recognition system by collecting users' pupil data through common webcams, eliminating reliance on expensive physiological sensors and significantly reducing equipment procurement and usage costs. Furthermore, using pre-trained deep learning models reduces the computational resources and energy consumption required for model training, thereby lowering the overall system cost.

[0031] 2. Enhance the real-time performance of the comprehension recognition system by collecting and processing the user's pupil data in real time. Utilize various deep learning models (such as CNN, RNN, LSTM, BiLSTM, and Transformer) for predictive analysis to ensure immediate feedback during user interaction. This real-time capability allows the system to dynamically adjust the complexity and presentation of information, thereby improving user comprehension and experience.

[0032] 3. Enhanced support for multimodal scenarios to improve the flexibility of understanding and recognition, enabling it to handle multiple modal inputs including text, voice, images, and combinations thereof, thus broadening its application prospects in various scenarios such as education, healthcare, and entertainment.

[0033] 4. Reduce the burden on users by wearing external devices to improve user experience. Based on the data collection method of webcam, there is no need to wear additional devices. Users can interact in a natural state, which greatly improves the convenience and comfort of use.

[0034] 5. The entire process is systematized to improve the level of automation. The system automatically analyzes the user's pupil data and identifies the level of understanding through a deep learning model, avoiding the cumbersome experimental operations and human intervention in existing methods. This improves the level of automation of the system and enables it to be applied seamlessly in real-world environments.

[0035] 6. Enhance the system's scalability and adaptability with full-stack web system development. The system is designed as a lightweight network plugin that can be easily embedded into existing human-computer interaction systems (such as intelligent education systems and intelligent medical systems). Furthermore, the system requires no external devices, possesses strong adaptability and scalability, and can be applied to various hardware platforms (such as smartphones, tablets, and personal computers). Attached Figure Description

[0036] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.

[0037] Figure 1 This is a schematic diagram illustrating the technology roadmap identified based on the level of understanding of existing conventional methods;

[0038] Figure 2 This is a schematic diagram illustrating a technology roadmap based on existing technologies, physiological instruments, and machine learning.

[0039] Figure 3 This is a schematic diagram illustrating a low-cost, real-time understanding degree recognition system based on pupil data according to an embodiment of the present disclosure;

[0040] Figure 4 This is a schematic diagram illustrating the system architecture of a low-cost, real-time understanding degree recognition system based on pupil data according to an embodiment of the present disclosure. Detailed Implementation

[0041] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0042] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0043] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0044] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0045] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0046] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0047] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0048] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0049] Figure 3 This is a schematic diagram illustrating a low-cost, real-time understanding degree recognition system based on pupil data according to an embodiment of the present disclosure.

[0050] To achieve the aforementioned objectives, the present invention employs the following technical framework: Figure 3 This illustrates a real-time understanding recognition system (AI-Mind-Reader) based on deep learning and pupil data.

[0051] The solution aims to achieve efficient measurement of user understanding in multimodal scenarios without the need for external devices.

[0052] The system's overall design includes a front-end (Vue), a back-end (Flask), and Socket connections for data transmission. The front-end, developed using the Vue framework, is primarily responsible for the user interface display and interaction, including the presentation of text, images, and audio materials, as well as capturing facial video stream data from the user. By accessing the webcam, the front-end collects the user's pupil data in real time and transmits this data to the back-end via Socket connections. The use of Socket connections ensures the real-time and efficient transmission of data between the front-end and back-end, enabling the system to provide immediate feedback during user interaction.

[0053] On the backend, the system is built using the Flask framework and is responsible for receiving pupil data transmitted from the frontend and performing real-time analysis and prediction. The backend system comprises multiple modules, including a face detection module, a pupil extraction module, and a model recognition module. Each module uses a multi-threaded processing mechanism to ensure real-time data processing. The system utilizes tools such as MediaPipe for facial feature detection and pupil data capture and tracking, generating pupil sequence data in real time. First, the system captures the user's facial video stream data by calling a webcam, collects and generates the user's pupil data in real time, and transmits the pupil data to the backend via a Socket connection. Specifically, the frontend uses a face detection algorithm (MediaPipe) to identify the user's facial regions and then locates the eye region. Based on this, a pupil tracking algorithm is used to identify the position and size of the pupil. First, the system acquires video stream data through a webcam and extracts the eye region in each frame, accurately calculating the pupil center and diameter using edge detection and image segmentation techniques. Second, the frontend tracks the movement of the user's pupils in real time, capturing dynamic changes in the pupils, including pupil dilation and constriction (i.e., pupillary response). This process utilizes image processing techniques, including circle detection and contour extraction, to obtain the accurate location of the pupil and update the data. Pupil data (including location, diameter, rate of change, etc.) is transmitted to the backend in real time via a Socket connection. To ensure data accuracy and consistency, the data undergoes preprocessing steps such as denoising and standardization after transmission to the backend, providing high-quality input for subsequent deep learning model analysis.

[0054] Deep learning models are one of the core technologies of this invention. The system embeds five different types of deep learning models, including Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), Bidirectional Long Short-Term Memory Networks (BiLSTM), and Transformer models. A soft voting strategy from ensemble learning is employed to integrate the prediction results of different network models, enhancing the model's adaptability and predictive ability in various scenarios. These models predict the user's level of understanding by analyzing the time-series changes in pupil data. To reduce computational costs and time consumption, these models are pre-trained and optimized for specific tasks. Through multi-model collaboration, the system can provide higher-precision recognition results for understanding levels in various complex multimodal scenarios, thereby improving the user experience and the applicability. Specifically, the model recognition module first receives the pupil data sequence after front-end acquisition and preprocessing, including information on pupil size, position, and dynamic changes. Subsequently, it is processed through the following deep learning networks:

[0055] Convolutional Neural Network (CNN): This model employs a two-layer CNN structure, with each layer containing 64 convolutional kernels of 3x3 size. The ReLU activation function is used to enhance non-linear processing capabilities. The first convolutional layer reduces data dimensionality through max pooling with a stride of 2, effectively halving the data dimension. After convolution, the data is fed into a fully connected layer, and finally, a comprehensibility score is generated using the Sigmoid activation function. This structure can quickly extract local features and is suitable for processing high-dimensional time series data.

[0056] Recurrent Neural Networks (RNNs): This model employs a two-layer RNN structure, with each layer containing 256 hidden units. RNNs are highly capable of processing time-series data and are suitable for capturing the dynamic features of time sequences within the data. As data is input, the RNN layers progressively update their internal states, and at the final time point, a fully connected layer generates the user's understanding score.

[0057] Long Short-Term Memory (LSTM) Networks: LSTM uses a two-layer structure with 128 hidden units per layer, effectively handling dependencies in long-term sequences. In pupil data processing, LSTM is particularly suitable for capturing long-term dependencies in pupil size changes over time. LSTM uses a dropout rate of 0.5 to prevent overfitting. The output is mapped to a range of 0 to 1 using a sigmoid function to generate a user comprehension score.

[0058] Bidirectional Long Short-Term Memory (BiLSTM) Network: BiLSTM uses a bidirectional structure (i.e., forward and backward LSTM layers) to enable the model to consider information from both past and future time steps simultaneously, thus providing a comprehensive understanding of pupil data. Each layer contains 128 hidden units and uses a dropout rate of 0.5. The output of the BiLSTM model is converted into a comprehension score using a sigmoid function.

[0059] The Transformer model utilizes self-attention, employing a multi-layer encoder (16 layers) with 256 hidden units per layer and 8 attention heads. This allows for parallel processing of large-scale time-series data, making it suitable for handling long temporal dependencies in pupil data. When generating comprehension scores, the model maps the final encoder output to the scoring space through fully connected layers. The Transformer model's advantage lies in its ability to handle long-term dependencies and its superior performance on large-scale datasets.

[0060] The model predicts user comprehension performance, encompassing both subjective and objective levels of understanding. Subjective comprehension refers to participants' self-reported level of understanding, typically collected through questionnaires, such as participants' perceived understanding of a task or information. Objective comprehension, on the other hand, refers to the proportion of information a user can accurately repeat or reproduce, reflecting their actual grasp of the content. It is usually measured through tests or quizzes, such as the percentage of knowledge points a user can correctly answer or the accuracy with which they repeat given content. These two indicators together provide a comprehensive picture of user comprehension; subjective comprehension reflects cognitive perception, while objective comprehension focuses on actual accuracy. The formula for calculating subjective comprehension can be expressed as:

[0061]

[0062] The comprehension score is the participant's self-reported comprehension score, with the highest possible score being the highest rating, typically 9 points. This metric reflects the user's subjective feelings about the task or information.

[0063] The formula for calculating the degree of objective understanding can be expressed as:

[0064]

[0065] The correctly paraphrased content refers to the portion of the information that the user can accurately understand and paraphrase, while the total content refers to all the content that needs to be paraphrased. This indicator objectively reflects the user's actual grasp of the information.

[0066] Experimental verification:

[0067] The system generates a user comprehension level assessment result through real-time analysis of pupil data and transmits the result back to the front end for dynamic display via a Socket connection. Users receive real-time feedback during interaction with the system, which can dynamically adjust the complexity and presentation of information based on the user's comprehension level, further optimizing the user experience. In this way, the invention achieves automated, real-time monitoring and feedback of user comprehension levels in multimodal interaction scenarios, providing an innovative solution for multiple application areas such as smart education and smart healthcare.

[0068] Figure 3The presentation details the user interaction flow of the AI-Mind-Reader system and the working mechanism of its adaptive system. The system adjusts its interaction strategies based on user groups (e.g., regular users, elderly users, non-native speakers), and continuously monitors and collects user comprehension data using pupil data and facial feature capture. The adaptive system dynamically adjusts the complexity and presentation format of content to improve user experience. The system is applicable not only to various fields such as conversational AI (e.g., ChatGPT) and smart education (e.g., Khan Academy), but also to autonomous driving and other complex scenarios.

[0069] Figure 4 The document demonstrates the overall structure of the AI-Mind-Reader comprehension level recognition system. The front-end (Vue) is responsible for stimulus presentation (such as text, images, and audio materials) and camera area selection. The captured pupil sequence data is transmitted to the back-end (Flask) via a Socket connection. Multiple modules on the back-end (such as face detection, pupil extraction, and model recognition) work in parallel, processing the input data in real time and generating comprehension level results. Finally, the results are returned to the front-end for display via a Socket connection. The entire system achieves real-time data interaction between the front-end and back-end through Socket connections, effectively supporting comprehension level recognition in multimodal environments.

[0070] This invention realizes a real-time, low-cost, multimodal, pupil data-based level-of-understanding recognition system. Compared with existing technologies, this invention has the following five advantages:

[0071] This invention provides real-time user comprehension level recognition and measurement, offering greater real-time usability. The system captures user facial activities via a webcam, captures pupil sequence data using MediaPipe, and inputs this data into an embedded deep learning model. This invention selects five deep neural network models: CNN, RNN, LSTM, BiLSTM, and Transformer. The model is trained using laboratory pupil sequence data (720 pupil sequences, each with 16381 pupil time points), enabling it to capture complex features from the pupil sequence data and accurately predict the corresponding comprehension level. Based on the input pupil data, the embedded model can calculate the comprehension level in real time and transmit the results to the webpage frontend for real-time updates. Traditional comprehension level measurement methods, such as self-report and comprehension performance tests, rely on periodic assessments, and the user's comprehension process is interrupted by the assessment content, thus lacking good continuity and real-time performance. Compared to existing comprehension level measurement methods, this invention, through real-time capture by a webcam, continuous socket transmission, and real-time calculation by a deep learning model, ensures real-time feedback on comprehension levels. The real-time nature of this invention provides a solid foundation for future applications such as embedding into other interactive systems and dynamically adapting to user comprehension.

[0072] This invention achieves high accuracy in recognizing levels of understanding. The five deep neural networks used in this invention—CNN, RNN, LSTM, BiLSTM, and Transformer—demonstrated good prediction accuracy on the test set (average MSE of 0.0625). Further experimental verification of the prediction accuracy revealed that the best-performing Transformer model achieved a MAPE of 29.5% (meaning a prediction accuracy of 70.5%) for objective levels of understanding and a MAPE of 14.9% (meaning a prediction accuracy of 85.1%) for subjective levels of understanding. The accuracy of this invention in recognizing levels of understanding is significantly higher than that of random guessing (50%) and existing eye-tracking data-based prediction methods (41%).

[0073] This invention reduces the cost of level-of-understanding recognition, offering high cost-effectiveness and ease of use. Developed as a web plugin, it can be easily embedded into existing human-computer interaction systems. All models used in the system are pre-trained, reducing the computational, data, and energy costs for users during training. Furthermore, the system design is lightweight and requires no external devices, necessitating only a computer camera for operation. Compared to existing level-of-understanding recognition methods based on wearable sensors, such as electroencephalography (EEG) and eye trackers, this invention significantly reduces the cost of technology adoption, usage, and maintenance.

[0074] This invention's comprehension level identification incorporates both subjective and objective comprehension levels. Most existing comprehension level identification methods only consider a single dimension of comprehension performance, lacking a comprehensive consideration of both subjective and objective comprehension levels. However, for human-computer interaction systems, both subjective and objective comprehension levels are equally important: objective comprehension reflects the user's actual level of understanding and is the primary objective of the interaction task; the system can adaptively adjust the difficulty of the interaction task and assist in understanding information based on objective comprehension. Subjective comprehension reflects the user's judgment of their own comprehension level and has a significant impact on the quality of the user experience during interaction. Therefore, this invention's comprehensive identification of comprehension levels from both subjective and objective perspectives provides important technical support for the optimization and improvement of human-computer interaction systems.

[0075] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A low-cost, real-time understanding degree recognition system based on pupil data, characterized in that, It includes two parts: front-end and back-end. The front end includes a user interface display and interaction unit, which captures the user's facial video stream data by calling the webcam, collects and generates the user's pupil data in real time, and transmits the pupil data to the back end via a Socket connection; Specifically, the front end uses a facial detection algorithm to identify the user's facial area and then locate the eye area. Based on this, a pupil tracking algorithm is used to identify the position and size of the pupil. First, the system acquires video stream data through a webcam and extracts the eye area in each frame. Then, it accurately calculates the center and diameter of the pupil through edge detection and image segmentation technology. Secondly, the front end uses image processing technology that employs circular detection and contour extraction to obtain the accurate position of the pupil, tracks the movement of the user's pupil in real time, captures the dynamic changes of the pupil, including pupil dilation and constriction, and updates the data accordingly. Pupil data, including position, diameter, and rate of change, is transmitted to the backend in real time via a Socket connection, where the backend performs further analysis on this data. The backend receives the pupil data transmitted from the frontend and preprocesses it, and performs real-time analysis and prediction through multiple modules, including: The face detection module transmits the ratio to the pupil extraction module through the calibration module, and transmits the processed frame data to the pupil extraction module through the face recognition and feature extraction module. The pupil extraction module uses a calibration ratio to process frame data, captures and collects pupils, obtains pupil sequence data, and transmits it to the model recognition module. The model recognition module takes pupil sequence data as input and adopts a pupil extraction deep learning model. By embedding parallel convolutional neural networks, recurrent neural networks, long short-term memory networks, bidirectional long short-term memory networks, and Transformer models, it integrates the prediction results of different network models through a soft voting strategy in ensemble learning to construct a complete analysis method.

2. The low-cost, real-time understanding degree recognition system based on pupil data as described in claim 1, characterized in that, The specific implementation method of the model recognition module is as follows: First, the pupil data sequence, which has been acquired and preprocessed by the front end, is received. The pupil data includes pupil size, position, and dynamic change information. The processing is then performed using a separately designed deep learning network, the structure of which includes: Convolutional Neural Network: A two-layer convolutional neural network structure is adopted, with each layer containing 64 convolutional kernels. The kernel size is 3x3. The ReLU activation function is used to enhance non-linear processing capabilities. The first convolutional layer reduces the data dimension through max pooling with a stride of 2, halving the data dimension. After convolution processing, the data is fed into the fully connected layer, and finally, the comprehension score is generated through the Sigmoid activation function. Recurrent Neural Network: It adopts a two-layer RNN structure, with each layer containing 256 hidden units. At the final time point, a fully connected layer is used to generate the user's understanding score. Long Short-Term Memory Network: It consists of a two-layer structure with 128 hidden units in each layer. In pupil data processing, a Dropout rate of 0.5 is used to prevent overfitting. The output is mapped to between 0 and 1 through the Sigmoid function to generate the user's comprehension score. Bidirectional Long Short-Term Memory Network: Through the bidirectional connection of forward and backward LSTM layers, the model can simultaneously consider past and future time step information. Each layer contains 128 hidden units, uses a Dropout rate of 0.5, and the output is converted into a comprehension score through the Sigmoid function. Transformer model: Utilizing a self-attention mechanism, it processes large-scale time series data in parallel through a 16-layer encoder with 256 hidden units per layer and 8 attention heads. When generating comprehension scores, it maps the final encoder output to the score space through a fully connected layer.

3. The low-cost, real-time understanding degree recognition system based on pupil data as described in claim 2, characterized in that, The separately designed deep learning networks all use the cross-entropy loss function during training, the optimizer adopts the Adam optimization algorithm, the learning setting is 0.00001, the batch size is 25, and the number of training epochs is 100. The performance of each neural network structure is evaluated by mean squared error and prediction accuracy. Regularization and Dropout techniques are used during training to reduce overfitting.

4. The low-cost, real-time understanding degree recognition system based on pupil data as described in claim 3, characterized in that, The predicted user comprehension performance includes subjective comprehension and objective comprehension. Subjective comprehension is defined as the level of comprehension reported by participants and collected through questionnaires. Objective comprehension is defined as the proportion of information that users can accurately repeat or reproduce and is measured through tests or quizzes. Together, these two indicators provide a comprehensive picture of users' understanding of the content. The formula for calculating the degree of subjective understanding is as follows: Among them, the comprehension score is the comprehension score reported by the participants themselves, and the maximum possible score is the highest value of the score; The formula for calculating the degree of objective understanding is as follows: Among them, the content that is correctly paraphrased is the part that the user can accurately understand and paraphrase, and the total content is all the content that needs to be paraphrased. Finally, through an adaptive system, when a low level of comprehension is detected, automated measures are taken to provide additional supplementary information and further explanations of the content to improve the user experience.

5. The low-cost, real-time understanding degree recognition system based on pupil data as described in claim 4, characterized in that, The preprocessing method includes noise reduction and normalization.

Images