A children's literature reading effect evaluation method and system combined with image processing

Abstract

The application discloses a kind of children's literature reading effect evaluation method and system combined with image processing, comprising: by preposed image acquisition device, its face image is collected, and face image sequence is constructed;Face image sequence is preprocessed, and feature area is extracted;Identify the fixation point position of children in reading, pupil change trend, micro-expression category and head posture angle;Text semantic structure and image visual element of children's literature content are acquired, and space-time mapping relationship is established;According to reading progress time stamp, sentence-level semantic unit and main visual object are aligned according to time window, and weight-bearing graph-text association atlas is constructed;Fixation point position and space-time mapping relationship are compared, and the attention focus state of children is determined;According to micro-expression category and emotional label in text semantic structure, degree of matching is analyzed, and emotional input index is generated;Multi-dimensional reading state vector is constructed, and the presentation mode of subsequent literature content or auxiliary guiding strategy is dynamically adjusted.

Description

Technical Field

[0001] This invention relates to a method and system for evaluating reading effectiveness, and more particularly to a method and system for evaluating children's literature reading effectiveness that incorporates image processing. Background Technology

[0002] Assessing the effectiveness of children's literature reading is an important means of measuring children's comprehension, concentration, and emotional engagement during the reading process. It has educational value in promoting early language development, cultivating imagination, and enhancing cognitive abilities, and is receiving widespread attention from families and educational institutions.

[0003] Current assessment methods typically involve teachers or parents judging children's reading performance through observation and questionnaires. This process relies on manual recording and subjective interpretation to form preliminary conclusions about a child's reading status, such as attention span and interest in the story. This information can be used for subsequent teaching adjustments or reading material recommendations. Existing technologies have the following problems: traditional observation and questionnaire methods are easily influenced by the assessor's experience and emotions, limiting their quantification; current e-reading platforms mostly focus on basic interactive data such as page-turning frequency and dwell time, rarely linking them to the semantics of text and images; while some systems incorporate facial recognition technology for attention monitoring, this is primarily geared towards classroom management scenarios and does not deeply integrate visual attention, micro-expression responses, and literary narrative structure; furthermore, most systems lack dynamic feedback mechanisms based on real-time reading status, making it difficult to provide targeted content guidance or emotional support during reading, thus limiting the actual positive impact of assessment results on the reading experience.

[0004] Therefore, it is of great significance to provide a method and system for evaluating the effectiveness of children's literature reading that combines image processing. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to solve the problems existing in the prior art and provide a method and system for evaluating the effectiveness of children's literature reading by combining image processing, so as to achieve objective quantification and real-time intervention of children's reading focus, comprehension and emotional investment, and improve the accuracy and personalization of reading evaluation.

[0006] Technical solution: The present invention provides a method for evaluating the reading effect of children's literature by combining image processing, comprising the following steps:

[0007] S1. Configure a front-facing image acquisition device and a display unit on the children's reading terminal device; continuously acquire facial images of the child during the reading process through the front-facing image acquisition device to construct a facial image sequence;

[0008] S2. Preprocess the facial image sequence to extract feature regions including the eye region, mouth region, and overall facial posture; based on the feature regions, identify the child's gaze point position, pupil change trend, micro-expression category, and head posture angle during reading.

[0009] S3. Obtain the text semantic structure and image visual elements of the currently displayed children's literature content, and establish the spatiotemporal mapping relationship between the text semantic structure and the image visual elements; specifically including: calling a pre-trained language model to perform dependency parsing and sentiment annotation on the text, and extracting sentence-level semantic units; calling a multimodal fusion model to perform instance segmentation and semantic annotation on the illustrations, and identifying the main visual objects and their spatial layout;

[0010] Based on the reading progress timestamp, the sentence-level semantic units are aligned with the main visual objects according to the time window to construct a weighted image-text association graph;

[0011] S4. Compare the fixation point position with the spatiotemporal mapping relationship to determine the current attention focus state of the child;

[0012] S5. Perform a matching degree analysis between the micro-expression categories and the sentiment tags in the text semantic structure to generate an emotional investment index;

[0013] S6. Combine attention focus state, pupil change trend, head posture angle and emotional investment index to construct a multi-dimensional reading state vector; based on the multi-dimensional reading state vector, dynamically adjust the presentation mode of subsequent literary content or trigger auxiliary guidance strategies.

[0014] Furthermore, in step S1, the front-facing image acquisition device is a high frame rate camera, which is fixed to the center of the upper edge of the display unit by a rigid bracket, and the angle between its optical axis and the normal of the display plane is less than 15 degrees.

[0015] Furthermore, in step S2, the facial image sequence is preprocessed, specifically including:

[0016] Perform illumination normalization processing based on Retinex theory on each frame of the image;

[0017] A deep learning-based face detection model was used to locate the bounding boxes of the facial region, and a 68-point facial key point regression network was used to segment the eye region, mouth region and jaw contour.

[0018] The segmented regions are scaled and affine transformed to align them in the standard template coordinate system.

[0019] Furthermore, step S2, identifying the child's gaze point position during reading, specifically includes:

[0020] Based on the spatial geometric relationship between the centers of the irises of both eyes and the feature points of the inner and outer corners of the eyes, the direction vector of the gaze is calculated.

[0021] The gaze direction vector is projected onto the screen coordinate system of the display unit to obtain the pixel coordinates of the gaze point;

[0022] By combining the page's DOM tree structure and content layout information, the pixel coordinates of the gaze point are mapped to the index of the text and image elements on the current page.

[0023] Furthermore, in step S2, the micro-expression categories are classified using a trained three-dimensional convolutional neural network model. The input of the model is a continuous temporal image patch of the mouth region and the eye region, and the output is a category label in a preset micro-expression set, which includes surprise, pleasure, confusion, and boredom.

[0024] Furthermore, in step S4, the fixation point position is compared with the spatiotemporal mapping relationship to determine the child's current attention focus state, specifically:

[0025] If the coordinates of the gaze point fall within the area marked as a narrative key in the text-image association map, it is determined to be focused attention; otherwise, it is determined to be distracted attention.

[0026] Furthermore, in step S5, the emotional investment index The calculation formula is as follows:

[0027]

[0028] in, For the first Micro-expression categories detected within a time window For the sentiment tags of the corresponding text semantic units For predefined matching functions, For time decay weight, This is the length of the sliding window.

[0029] Furthermore, in step S6, the dynamic adjustment of the presentation method of subsequent literary content or the triggering of auxiliary guidance strategies specifically includes:

[0030] When the multi-dimensional reading state vector indicates that attention is not focused and the emotional engagement index is less than 0.3, an interactive question or animated prompt is inserted on the next page.

[0031] When a micro-expression of confusion lasting for more than or equal to 3 seconds is detected and the attention is not focused, the system will automatically revert to the previous key plot and highlight the core sentence.

[0032] When the emotional engagement index is greater than 0.8 and the micro-expression is pleasant, it is recommended to read further materials with similar narrative style or theme.

[0033] Furthermore, step S6 also includes:

[0034] The multi-dimensional reading state vector is input into the strategy decision module, which outputs the optimal intervention action based on the reinforcement learning model.

[0035] The intervention action is applied to the reading rendering engine through the terminal device application layer interface to achieve dynamic adjustment of the content presentation method;

[0036] Record the changes in the state vector before and after each intervention, which are used to update the reward function of the policy model online;

[0037] Determine whether the emotional investment index has increased after the intervention. If so, mark the current intervention strategy as effective and increase its triggering priority in similar scenarios in the future. If not, mark the current intervention strategy as ineffective and reduce its triggering weight, while trying alternative strategies.

[0038] After three consecutive ineffective interventions, a reading difficulty warning was sent to the parent's app, and manual intervention was recommended.

[0039] The present invention provides a children's literature reading effect evaluation system that combines image processing, comprising:

[0040] Data acquisition module: A front-facing image acquisition device and a display unit are configured on the children's reading terminal device; the front-facing image acquisition device continuously acquires facial images of the child during the reading process and constructs a facial image sequence;

[0041] Data feature extraction module: preprocesses the facial image sequence to extract feature regions including the eye region, mouth region, and overall facial posture; based on the feature regions, identifies the child's gaze point position, pupil change trend, micro-expression category, and head posture angle during reading.

[0042] The mapping relationship construction module obtains the text semantic structure and image visual elements of the currently displayed children's literature content, and establishes the spatiotemporal mapping relationship between the text semantic structure and the image visual elements; specifically, it includes: calling a pre-trained language model to perform dependency parsing and sentiment polarity labeling on the text, and extracting sentence-level semantic units; calling a multimodal fusion model to perform instance segmentation and semantic labeling on the illustrations, and identifying the main visual objects and their spatial layout;

[0043] Based on the reading progress timestamp, the sentence-level semantic units are aligned with the main visual objects according to the time window to construct a weighted image-text association graph;

[0044] Focusing state determination module: compares the fixation point position with the spatiotemporal mapping relationship to determine the current child's attention focusing state;

[0045] Emotional Engagement Index Generation Module: Analyzes the matching degree between the micro-expression categories and the emotional tags in the semantic structure of the text to generate an emotional engagement index;

[0046] Dynamic intervention module: Combining attention focus state, pupil change trend, head posture angle, and emotional engagement index, a multi-dimensional reading state vector is constructed; based on the multi-dimensional reading state vector, the presentation mode of subsequent literary content is dynamically adjusted or auxiliary guidance strategies are triggered.

[0047] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0048] (1) By integrating a front-end image acquisition device and a multimodal analysis engine into the reading terminal, the system achieves refined capture of children’s visual attention, micro-expression reactions and head posture; (2) Deeply coupling image features with the semantic structure of literary content breaks through the limitations of traditional systems that rely solely on coarse-grained interactive data such as page-turning time; (3) The constructed multi-dimensional reading state vector can objectively quantify comprehension, focus and emotional investment levels; the dynamic feedback mechanism triggered by this vector can provide personalized guidance in real time during the reading process, realize objective quantification and real-time intervention of children’s reading focus, comprehension and emotional investment, and improve the accuracy and personalization of reading assessment; (4) The entire system does not require human intervention, avoids subjective bias, and provides families and educational institutions with a reproducible and traceable tool for evaluating children’s reading effects. Attached Figure Description

[0049] Figure 1 This is a flowchart of a children's literature reading effect evaluation method combined with image processing provided in an embodiment of the present invention.

[0050] Figure 2 This is a flowchart of the method for triggering intervention strategies according to an embodiment of the present invention.

[0051] Figure 3 This is a flowchart of a method for determining whether to initiate a complete analysis process according to an embodiment of the present invention. Detailed Implementation

[0052] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0053] like Figure 1 As shown, this embodiment provides a method for evaluating children's literature reading effectiveness by combining image processing. Based on computer vision and natural language processing technologies, it achieves a quantitative assessment of children's reading status, including the following steps:

[0054] S101. A front-end image acquisition device and a display unit are configured on the children's reading terminal device.

[0055] The front-facing image acquisition device is an optical sensor capable of continuously capturing sequences of images of a child's face, including a high-frame-rate camera. The front-facing image acquisition device is rigidly fixed to the center of the upper edge of the display unit, with its optical axis forming an angle of less than 15 degrees with the normal to the display plane, ensuring a typical reading distance of 30 cm to 80 cm. The acquired facial images meet the geometric constraints required for subsequent feature extraction.

[0056] S102. Using the aforementioned front-view image acquisition device, facial images are continuously acquired during the child's reading process to construct a facial image sequence.

[0057] The acquired facial images meet the geometric constraints required for subsequent feature extraction.

[0058] S103. Preprocess the facial image sequence to extract feature regions including the eye region, mouth region, and overall facial posture.

[0059] The preprocessing specifically includes: performing illumination normalization processing based on Retinex theory on each frame of image to eliminate the interference of ambient light fluctuations on feature extraction; using a deep learning-based face detection model to locate facial bounding boxes; using a 68-point facial key point regression network to accurately locate the contours of the eyes, mouth, and jaw; and performing scale normalization and affine transformation on the segmented key regions to align them in the standard template coordinate system.

[0060] S104. Based on the feature region, identify the child's gaze point position, pupil change trend, micro-expression category and head posture angle during the reading process.

[0061] Specifically, the identification of the gaze point location includes: calculating the gaze direction vector based on the spatial geometric relationship between the center of the iris of both eyes and the feature points of the inner and outer corners of the eyes; projecting the vector onto the screen coordinate system of the display unit to obtain the pixel coordinates of the gaze point; and mapping the pixel coordinates to the image and text element index of the current page by combining the page DOM tree structure and content layout information.

[0062] The specific calculation process for the pupil change trend is as follows:

[0063] S201, Pupil contour detection and ellipse fitting;

[0064] After preprocessing in step S103, the eye region image is used to accurately extract the edge contour of the pupil in each frame by employing an edge detection algorithm (such as the Canny operator) or a pupil segmentation network based on deep learning. Subsequently, an ellipse fitting algorithm (such as least squares ellipse fitting) is used to model this contour. The fitted ellipse has two parameters: a major axis (major axis, a) and a minor axis (minor axis, b).

[0065] S202, Pupil Diameter Calculation;

[0066] Approximating the pupil as a circle, its diameter D_t at time point t can be calculated using the two axes of a fitted ellipse. The method involves taking the arithmetic mean of the major and minor axes as the estimated pupil diameter for that frame:

[0067] D_t = (a_t + b_t) / 2

[0068] Where a_t and b_t are the lengths of the major and minor axes of the ellipse fitted in the t-th frame image, respectively.

[0069] S203, Time Series Smoothing and Normalization;

[0070] Due to the influence of head micro-movements, instantaneous changes in illumination, and detection noise, the original pupil diameter sequence {D_t} exhibits high-frequency jitter. Therefore, it is necessary to first perform temporal smoothing filtering (such as using a moving average window or a low-pass Butterworth filter) to obtain a smoothed sequence {D'_t}. Next, to eliminate the differences in absolute pupil size between individuals, the sequence can be normalized to the average diameter range of the child during the calm fixation baseline period, resulting in a relative diameter change sequence.

[0071] S204, Calculation of rate of change;

[0072] The rate of change of pupil diameter, P, is defined as an estimate of the first derivative (i.e., the speed of change) of pupil diameter over time. In discrete-time series, the instantaneous rate of change is calculated using the central difference method:

[0073] P_t = (D'{t+Δt} - D'{t-Δt}) / (2 * Δt)

[0074] Here, Δt is the time step (usually the time interval between two consecutive frames, determined by the camera frame rate; for example, at a frame rate of 30fps, Δt ≈ 33.3 ms). The final output is given to the multi-dimensional reading state vector. The P-value is typically the average absolute value or root mean square value of the instantaneous rate of change P_t calculated over an evaluation time window (e.g., 5 seconds) to reflect the average rate of pupil dilation / contraction during that time period, measured in millimeters per second (mm / s). The conversion from pixels per second to millimeters per second requires camera calibration and spatial scaling based on known camera parameters and shooting distance.

[0075] The micro-expression categories are classified using a pre-trained 3D convolutional neural network. The network takes temporal image blocks (e.g., 5 consecutive frames) of the eye and mouth regions as input and outputs category labels from a preset set of micro-expressions. The preset set of micro-expressions includes, but is not limited to, surprise, pleasure, confusion, and boredom.

[0076] The calculation process for the head posture angle is as follows:

[0077] S301, 3D head model corresponds to 2D feature points;

[0078] The system pre-sets a general 3D rigid head model (e.g., a 3D average face model containing dozens of key points, such as the corners of the eyes, the tip of the nose, and the corners of the mouth). In S103, the 2D facial key points (such as eyelids, nose wings, and corners of the mouth) detected by the 68-point facial key point regression network are matched with the corresponding points of this 3D model.

[0079] S302, Attitude Angle Calculation (PnP Algorithm)

[0080] The head pose is determined using the Perspective-n-Point (PnP) algorithm. The input to this algorithm is:

[0081] A set of 3D model points (world coordinate system). The corresponding 2D projection points in the current image frame (image coordinate system). The camera's intrinsic parameter matrix (obtained through pre-calibration, including parameters such as focal length and principal point). The PnP algorithm iteratively optimizes to solve for a rotation vector and a translation vector, minimizing the reprojection error between the 3D points projected onto the image plane after rotation and translation and the detected 2D points.

[0082] S303, From Rotation Vector to Euler Angle

[0083] The obtained rotation vector (usually represented as a 3x1 vector) is transformed into a 3x3 rotation matrix using the Rodrigues transform. Then, the specific Euler angles are decomposed from this rotation matrix:

[0084] Pitch (θ): The angle of rotation around the X-axis. A positive value indicates pitching up and a negative value indicates pitching down.

[0085] Yaw angle (φ): The angle of rotation around the Y-axis. A positive value indicates a right turn, and a negative value indicates a left turn.

[0086] Roll angle (ψ): The angle of rotation around the Z-axis, i.e., the head tilt angle.

[0087] Reading state vectors in multiple dimensions In general, the roll angle is not used directly, or it is treated as a separate dimension.

[0088] S304. Calculate the Euclidean norm of head posture angle changes.

[0089] To quantify the degree to which the head deviates from the front (i.e., the ideal reading posture) using a scalar synthesis, the Euclidean norm (i.e., the length of the vector) of the two-dimensional vector formed by the pitch and yaw angles is calculated:

[0090] H = √(θ² + φ²)

[0091] Here, θ and φ are the absolute values ​​(or offsets relative to the calibration zero point) of the pitch and yaw angles calculated for the current frame (or the average within the current time window). The unit of this value H is degrees (°). A larger H value indicates a greater degree of head tilting, lowering, or turning, potentially suggesting distraction, physical discomfort, or a different reaction to content. In the calculation, the angles are typically filtered and smoothed to eliminate momentary jitter.

[0092] S105. Obtain the text semantic structure and image visual elements of the currently displayed children's literature content, and establish the spatiotemporal mapping relationship between the text semantic structure and the image visual elements.

[0093] Specifically, it includes:

[0094] Pre-trained language models (such as BERT) are invoked to perform dependency parsing and sentiment polarity labeling on the text, extracting sentence-level semantic units;

[0095] Multimodal fusion models (such as Mask R-CNN combined with CLIP) are used to perform instance segmentation and semantic annotation on illustrations, and to identify the main visual objects and their spatial layout.

[0096] Based on the reading progress timestamp, text semantic units and visual objects are aligned according to time windows to construct a weighted image-text association graph.

[0097] The reading progress timestamp is usually provided by the application programming interface of the reading terminal device (such as an e-book application), which records the current time point when the user turns a page, scrolls, or reads aloud.

[0098] The weights represent the importance of the text-image association, for example, numerical values ​​assigned based on the frequency or closeness of mentioning a visual object in the text description, used to distinguish core and secondary elements in the association graph.

[0099] S106. Compare the gaze point position with the spatiotemporal mapping relationship to determine whether the child's visual attention is focused on the current key area of ​​the narrative.

[0100] If the coordinates of the gaze point fall into the area marked as "narrative key" in the image-text association map (such as the image-text block corresponding to the protagonist's image or plot turning point), it is determined that the attention is focused; otherwise, it is determined that the attention is scattered.

[0101] S107. Perform a matching degree analysis between the micro-expression categories and the sentiment tags in the text semantic structure to generate an emotional investment index.

[0102] The sentiment labels are the results obtained in S105 by labeling the text with sentiment polarity using a pre-trained language model (such as BERT), typically categorized as positive, negative, or neutral. The system calculates the sentiment engagement index by matching the child's real-time micro-expression categories with the sentiment labels of the text content.

[0103] The emotional investment index The calculation formula is as follows:

[0104]

[0105] in, For the first Micro-expression categories detected within a time window For the sentiment tags of the corresponding text semantic units This is a predefined matching function (1 for a match, 0 for no match). For time decay weight, This is the length of the sliding window.

[0106] S108. Combining the pupil change trend, head posture angle, and emotional engagement index, construct a multi-dimensional reading state vector.

[0107] The multi-dimensional reading state vector It can be represented as:

[0108]

[0109] in, A represents the focus of attention; a value of 0 indicates distracted attention, and a value of 1 indicates focused attention. The rate of change of pupil diameter (unit: mm / s) The Euclidean norm (in degrees) represents the pitch and yaw angles of the head. Emotional investment index.

[0110] S109. Based on the multi-dimensional reading state vector, dynamically adjust the presentation mode of subsequent literary content or trigger auxiliary guidance strategies.

[0111] The multi-dimensional reading state vector can be a weighted combination of multiple physiological and behavioral indicators. This vector can comprehensively reflect a child's cognitive and emotional state during a reading interaction.

[0112] The specific intervention strategy is as follows:

[0113] When the multi-dimensional reading state vector indicates that attention is not focused and the emotional engagement index is less than 0.3, an interactive question or animated prompt is inserted on the next page.

[0114] When a micro-expression of confusion lasting for more than or equal to 3 seconds is detected and the attention is not focused, the system will automatically revert to the previous key plot and highlight the core sentence.

[0115] When the emotional engagement index is greater than 0.8 and the micro-expression is pleasant, it is recommended to read further materials with similar narrative style or theme.

[0116] like Figure 2 As shown, in step S109, the dynamic adjustment of the presentation method of subsequent literary content or the triggering of auxiliary guidance strategies further includes:

[0117] S401. Input the multi-dimensional reading state vector into the strategy decision module, and the strategy decision module outputs the optimal intervention action based on the reinforcement learning model.

[0118] S402. The intervention action is applied to the reading rendering engine through the application layer interface to achieve dynamic adjustment of the content presentation method;

[0119] S403. Record the changes in the state vector before and after each intervention, which are used to update the reward function of the policy model online;

[0120] S404. Determine whether the emotional investment index has increased after the intervention. If yes, mark the current intervention strategy as effective and increase its triggering priority in similar scenarios in the future; if no, mark the current intervention strategy as ineffective and reduce its triggering weight, while trying alternative strategies.

[0121] S405. After three consecutive ineffective interventions, push a reading difficulty warning to the parent's app and suggest manual intervention.

[0122] The above methods enable a refined assessment of children's comprehension, concentration, and emotional engagement in each reading interaction. Furthermore, the use of a multimodal fusion approach, coupling visual behavioral characteristics with the deep semantics of literary content, ensures the accuracy of assessment results and the targeted nature of intervention strategies.

[0123] The image processing techniques used in this embodiment of the invention can be divided into multiple functional modules. A lightweight convolutional neural network can be used as a micro-expression classifier, and a binocular gaze estimation model can be used as a gaze localization engine.

[0124] The raw image data acquired by the front-end image acquisition device is transmitted to the processor via the device's internal bus. The processor calls the pre-installed algorithm model in the memory and sequentially performs operations such as face detection, key point localization, gaze estimation, and micro-expression classification. Data is transferred between modules through shared memory or message queues to ensure the efficient operation of the processing pipeline.

[0125] Since the front-end image acquisition device and related algorithm modules continuously consume system resources, power consumption can be reduced through an event-driven mechanism. The full analysis process is initiated only when a valid reading action is detected.

[0126] like Figure 3 The flowchart shown illustrates the method for determining whether to initiate the full analysis process, including:

[0127] S501. Detect whether the display unit is in an active reading state. In normal standby mode, the system only runs low-power face detection. Only when the user opens an e-book and enters the text page is it considered to be in a valid reading state.

[0128] S502. When the user is on an active reading page, the front-end image acquisition device and the multimodal analysis engine are activated.

[0129] S503. If no valid face region or gaze point data is detected within a predetermined second time interval (e.g., 10 seconds), the high-load analysis module is paused and the system enters a low-power monitoring mode.

[0130] In this embodiment, the state vector can be sampled multiple times and a moving average can be performed. This can also achieve a smooth estimation of the reading state and avoid misjudgment due to instantaneous noise, thereby preventing inappropriate intervention.

[0131] This embodiment also provides a children's literature reading effect evaluation system that combines image processing, including:

[0132] Data acquisition module: A front-facing image acquisition device and a display unit are configured on the children's reading terminal device; the front-facing image acquisition device continuously acquires facial images of the child during the reading process and constructs a facial image sequence;

[0133] Data feature extraction module: preprocesses the facial image sequence to extract feature regions including the eye region, mouth region, and overall facial posture; based on the feature regions, identifies the child's gaze point position, pupil change trend, micro-expression category, and head posture angle during reading.

[0134] The mapping relationship construction module obtains the text semantic structure and image visual elements of the currently displayed children's literature content, and establishes the spatiotemporal mapping relationship between the text semantic structure and the image visual elements; specifically, it includes: calling a pre-trained language model to perform dependency parsing and sentiment polarity labeling on the text, and extracting sentence-level semantic units; calling a multimodal fusion model to perform instance segmentation and semantic labeling on the illustrations, and identifying the main visual objects and their spatial layout;

[0135] Based on the reading progress timestamp, the sentence-level semantic units are aligned with the main visual objects according to the time window to construct a weighted image-text association graph;

[0136] Focusing state determination module: compares the fixation point position with the spatiotemporal mapping relationship to determine the current child's attention focusing state;

[0137] Emotional Engagement Index Generation Module: Analyzes the matching degree between the micro-expression categories and the emotional tags in the semantic structure of the text to generate an emotional engagement index;

[0138] Dynamic intervention module: Combining attention focus state, pupil change trend, head posture angle, and emotional engagement index, a multi-dimensional reading state vector is constructed; based on the multi-dimensional reading state vector, the presentation mode of subsequent literary content is dynamically adjusted or auxiliary guidance strategies are triggered.

[0139] The system is based on the same inventive concept as the above method, and will not be described in detail here.

[0140] This embodiment also provides a computer device, including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs, when executed by the processors, implement the steps of the children's literature reading effect evaluation method combined with image processing.

[0141] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method for evaluating the effectiveness of children's literature reading combined with image processing.

Claims

1. A method for evaluating the reading effect of children's literature by combining image processing, characterized in that, Includes the following steps: S1. Configure a front-end image acquisition device and display unit on the children's reading terminal device; The front-facing image acquisition device continuously captures facial images of children during reading, constructing a facial image sequence. S2. Preprocess the facial image sequence to extract feature regions including the eye region, mouth region, and overall facial posture; based on the feature regions, identify the child's gaze point position, pupil change trend, micro-expression category, and head posture angle during reading. S3. Obtain the text semantic structure and image visual elements of the currently displayed children's literature content, and establish the spatiotemporal mapping relationship between the text semantic structure and the image visual elements; specifically including: calling a pre-trained language model to perform dependency parsing and sentiment annotation on the text, and extracting sentence-level semantic units; calling a multimodal fusion model to perform instance segmentation and semantic annotation on the illustrations, and identifying the main visual objects and their spatial layout; Based on the reading progress timestamp, the sentence-level semantic units are aligned with the main visual objects according to the time window to construct a weighted image-text association graph; S4. Compare the fixation point position with the spatiotemporal mapping relationship to determine the current attention focus state of the child; S5. Perform a matching degree analysis between the micro-expression categories and the sentiment tags in the text semantic structure to generate an emotional investment index; S6. Combine attention focus state, pupil change trend, head posture angle and emotional investment index to construct a multi-dimensional reading state vector; based on the multi-dimensional reading state vector, dynamically adjust the presentation mode of subsequent literary content or trigger auxiliary guidance strategies.

2. The method for evaluating the reading effect of children's literature combined with image processing according to claim 1, characterized in that, In step S1, the front-facing image acquisition device is a high frame rate camera, which is fixed to the center of the upper edge of the display unit by a rigid bracket, and the angle between its optical axis and the normal of the display plane is less than 15 degrees.

3. The method for evaluating the reading effect of children's literature combined with image processing according to claim 1, characterized in that, In step S2, the facial image sequence is preprocessed, specifically including: Perform illumination normalization processing based on Retinex theory on each frame of the image; A deep learning-based face detection model was used to locate the bounding boxes of the facial region, and a 68-point facial key point regression network was used to segment the eye region, mouth region and jaw contour. The segmented regions are scaled and affine transformed to align them in the standard template coordinate system.

4. The method for evaluating the reading effect of children's literature combined with image processing according to claim 1, characterized in that, Step S2, identifying the child's gaze point position during reading, specifically includes: Based on the spatial geometric relationship between the centers of the irises of both eyes and the feature points of the inner and outer corners of the eyes, the direction vector of the gaze is calculated. The gaze direction vector is projected onto the screen coordinate system of the display unit to obtain the pixel coordinates of the gaze point; By combining the page's DOM tree structure and content layout information, the pixel coordinates of the gaze point are mapped to the index of the text and image elements on the current page.

5. The method for evaluating the reading effect of children's literature combined with image processing according to claim 1, characterized in that, In step S2, the micro-expression categories are classified using a trained three-dimensional convolutional neural network model. The input of the model is a continuous temporal image patch of the mouth region and the eye region, and the output is a category label in a preset micro-expression set, which includes surprise, pleasure, confusion, and boredom.

6. The method for evaluating the reading effect of children's literature combined with image processing according to claim 1, characterized in that, In step S4, the fixation point position is compared with the spatiotemporal mapping relationship to determine the child's current attention focus state, specifically: If the coordinates of the gaze point fall within the area marked as a narrative key in the text-image association map, it is determined to be focused attention; otherwise, it is determined to be distracted attention.

7. The method for evaluating the reading effect of children's literature combined with image processing according to claim 1, characterized in that, In step S5, the emotional investment index The calculation formula is as follows: in, For the first Micro-expression categories detected within a time window For the sentiment tags of the corresponding text semantic units For predefined matching functions, For time decay weight, This is the length of the sliding window.

8. The method for evaluating the reading effect of children's literature combined with image processing according to claim 1, characterized in that, In step S6, the dynamic adjustment of the presentation of subsequent literary content or the triggering of auxiliary guidance strategies specifically includes: When the multi-dimensional reading state vector indicates that attention is not focused and the emotional engagement index is less than 0.3, an interactive question or animated prompt is inserted on the next page. When a micro-expression of confusion lasting for more than or equal to 3 seconds is detected and the attention is not focused, the system will automatically revert to the previous key plot and highlight the core sentence. When the emotional engagement index is greater than 0.8 and the micro-expression is pleasant, it is recommended to read further materials with similar narrative style or theme.

9. The method for evaluating the reading effect of children's literature combined with image processing according to claim 1, characterized in that, Step S6 further includes: The multi-dimensional reading state vector is input into the strategy decision module, which outputs the optimal intervention action based on the reinforcement learning model. The intervention action is applied to the reading rendering engine through the terminal device application layer interface to achieve dynamic adjustment of the content presentation method; Record the changes in the state vector before and after each intervention, which are used to update the reward function of the policy model online; Determine whether the emotional investment index has increased after the intervention. If so, mark the current intervention strategy as effective and increase its triggering priority in similar scenarios in the future. If not, mark the current intervention strategy as ineffective and reduce its triggering weight, while trying alternative strategies. After three consecutive ineffective interventions, a reading difficulty warning was sent to the parent's app, and manual intervention was recommended.

10. A children's literature reading effect evaluation system combining image processing, characterized in that, include: Data acquisition module: A front-facing image acquisition device and display unit are configured on the children's reading terminal device; The front-facing image acquisition device continuously captures facial images of children during reading, constructing a facial image sequence. Data feature extraction module: preprocesses the facial image sequence to extract feature regions including the eye region, mouth region, and overall facial posture; based on the feature regions, identifies the child's gaze point position, pupil change trend, micro-expression category, and head posture angle during reading. The mapping relationship construction module obtains the text semantic structure and image visual elements of the currently displayed children's literature content, and establishes the spatiotemporal mapping relationship between the text semantic structure and the image visual elements; specifically, it includes: calling a pre-trained language model to perform dependency parsing and sentiment polarity labeling on the text, and extracting sentence-level semantic units; calling a multimodal fusion model to perform instance segmentation and semantic labeling on the illustrations, and identifying the main visual objects and their spatial layout; Based on the reading progress timestamp, the sentence-level semantic units are aligned with the main visual objects according to the time window to construct a weighted image-text association graph; Focusing state determination module: compares the fixation point position with the spatiotemporal mapping relationship to determine the current child's attention focusing state; Emotional Engagement Index Generation Module: Analyzes the matching degree between the micro-expression categories and the emotional tags in the semantic structure of the text to generate an emotional engagement index; Dynamic intervention module: Combining attention focus state, pupil change trend, head posture angle, and emotional engagement index, a multi-dimensional reading state vector is constructed; based on the multi-dimensional reading state vector, the presentation mode of subsequent literary content is dynamically adjusted or auxiliary guidance strategies are triggered.

Images