English text auxiliary teaching method and system based on AI vision

Abstract

The invention relates to an English text auxiliary teaching method and system based on AI vision. The method comprises the following steps: collecting a dynamic eye movement track, a mouth shape change video stream and micro-expression time sequence data when a student reads, and generating a dynamic behavior feature vector by using a convolutional neural network; and carrying out multi-dimensional matching on the vector and a preset pronunciation standard model, accurately positioning a pronunciation deviation region and an understanding difficulty point, and carrying out classification by virtue of a support vector machine to obtain a learning state label. And extracting a high-frequency deviation mode from the tag, constructing a comprehensive behavior matrix through association of a clustering algorithm and an eye movement backtracking trajectory, calculating a teaching level evaluation value, finally predicting a learning trend and determining a resource allocation weight by combining historical evaluation data and adopting a linear regression model, and optimizing and generating a personalized teaching plan. By adopting the method, pronunciation deviation and understanding disorder in English reading can be accurately positioned, so that the resource allocation weight is adaptive to the individual learning track of students, and a data-driven technical path is provided for English personalized teaching.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of education, and particularly relates to an AI vision-based English textbook auxiliary teaching method and system. BACKGROUND

[0002] Under the background of digital transformation of education, the current English textbook auxiliary teaching field faces multi-dimensional technical bottlenecks. On the one hand, the existing AI vision technology has a modal data fragmentation problem in the classroom scene. For example, eye tracking is only used for attention monitoring and is not time-coupled with the kinematics of mouth movements, so that the coupling relationship between English run-on, stress and visual behavior cannot be captured. On the other hand, the teaching resource adaptation mechanism lacks dynamic evolution ability. The traditional scheme generates resource packages based on a static textbook knowledge graph, which is difficult to respond to real-time needs caused by cultural context differences (such as idiom understanding bias) in the reading process of students, and the semantic analysis of multi-modal behavior data is limited to surface feature matching and does not deeply mine the unique tone-expression linkage mode of English (such as the raised eyebrow micro-expression corresponding to the interrogative sentence pattern). In addition, the existing system lacks a time weight dynamic adjustment mechanism for historical evaluation data in the teaching optimization across the classroom period, and cannot effectively identify the pronunciation solidification error patterns formed by students in long-term learning. Moreover, the resource library update relies on manual annotation, which is difficult to meet the needs of English textbook version iteration and dynamic adjustment of the teaching syllabus. SUMMARY

[0003] Therefore, it is necessary to provide an AI vision-based English textbook auxiliary teaching method and system that can accurately locate pronunciation deviations and understanding difficulties in English reading, and make resource allocation weights self-adaptive to the learning trajectory of individual students.

[0004] In a first aspect, the application provides an AI vision-based English textbook auxiliary teaching method, comprising:

[0005] Obtaining a dynamic eye movement trajectory sequence, a mouth movement change video stream and a micro-expression time sequence data of a student reading an English textbook in a classroom environment, and obtaining a dynamic behavior feature vector through a convolutional neural network.

[0006] Based on the matching result of the dynamic behavior feature vector and a preset pronunciation standard model, determining a pronunciation deviation area and a difficult point of understanding, and obtaining a learning state classification label by using a support vector machine classification.

[0007] Extracting a high-frequency deviation mode from the learning state classification label, associating and fusing a learning progress vector through clustering and eye movement backtracking trajectory to obtain a teaching level evaluation value.

[0008] Integrating the teaching level evaluation value and historical evaluation data, predicting a progress trend by using a linear regression model to determine a resource allocation weight, and optimizing to obtain a personalized teaching plan.

[0009] In one embodiment, the dynamic eye movement trajectory sequence, the mouth shape change video stream, and the micro-expression timing data of the student reading the English textbook in the classroom environment are obtained, and a dynamic behavior feature vector is obtained through a convolutional neural network processing, including:

[0010] The dynamic eye movement trajectory sequence, the mouth shape change video stream, and the micro-expression timing data of the student reading the English textbook in the classroom environment are collected in real time through a high-resolution camera array with space-time alignment.

[0011] An improved 3D convolutional neural network is used to process the dynamic eye movement trajectory sequence, and an attention mechanism is used to strengthen the key fixation point sequence features to generate an eye movement feature vector containing fixation residence time and speed vector.

[0012] A light-weight convolutional neural network combined with an optical flow method is used to model the facial key point motion trajectory of the micro-expression timing data, and an expression feature vector containing dynamic deformation features of the brow muscle and orbicularis oris muscle is extracted.

[0013] A residual convolutional network is used to model the timing of the mouth shape change video stream, and a dynamic time warping algorithm is introduced to match the pronunciation rhythm features to generate a mouth shape feature vector containing the three-dimensional motion trajectory of the lips.

[0014] The mouth shape feature vector, the eye movement feature vector, and the expression feature vector are fused to generate a dynamic behavior feature vector representing the dynamic behavior pattern of the student reading.

[0015] In one embodiment, based on the matching result of the dynamic behavior feature vector and the preset pronunciation standard model, the pronunciation deviation area and the understanding difficulty point are determined, and the learning state classification label is obtained by using a support vector machine classification, including:

[0016] Based on the multi-dimensional matching of the dynamic behavior feature vector and the preset pronunciation standard model, a matching degree matrix containing feature similarity and timing alignment deviation is obtained; the preset pronunciation standard model contains standard mouth shape trajectory sequence, standard eye movement scanning path, and neutral expression reference parameters.

[0017] The matching degree matrix is threshold segmented and region located to obtain the specific segment corresponding to the pronunciation deviation and the understanding difficulty point coordinate set focused by the eye movement in the textbook text.

[0018] Based on the specific segment of the pronunciation deviation and the understanding difficulty point coordinate set, the mouth shape, eye movement coordination features of the pronunciation deviation segment, and the expression fluctuation features of the understanding difficulty point are extracted to construct a multi-dimensional learning state description vector.

[0019] The multi-dimensional learning state description vector is input into a pre-trained support vector machine classifier. The high-dimensional space feature mapping is calculated through the kernel function to obtain multi-dimensional learning state classification labels that include focus, comprehension, and pronunciation accuracy.

[0020] In one embodiment, the matching degree matrix is ​​calculated using the following formula:

[0021]

[0022]

[0023]

[0024] in, This indicates the matching degree matrix at time step Feature Dimension The element value at that position, Represents the balance coefficient. Indicates the first Modality in feature dimension The feature similarity is defined on the graph, with values ​​ranging from [−1, 1]. Indicates at time step At this point, the dynamic behavioral characteristics and all time steps in the standard model time series are compared. The minimum value of timing alignment deviation, Indicates the temporal nodes in the reading process. This represents the specific dimension in the feature vector. The first element in the dynamic behavior feature vector represents the... Feature sequences of each modality This represents the standard feature sequence of the corresponding modality in the preset pronunciation standard model. Indicates the minimum value. A time series representing dynamic behavioral characteristics. Represents the standard model time series. Indicates the time step. Represents the timing alignment deviation matrix. Represents the timing penalty coefficient. Indicates time difference, This represents the maximum multimodal similarity. .

[0025] In one embodiment, high-frequency deviation patterns are extracted from the learning state classification labels, and a teaching level assessment value is calculated by clustering, associating with eye-tracking trajectories, and fusing with the learning progress vector, including:

[0026] High-frequency deviation patterns were extracted from the learning status classification labels; high-frequency deviation patterns included the frequency of attention loss and the forgetting rate of knowledge points.

[0027] According to the high-frequency deviation mode, clustering processing is performed to obtain an associated group of eye movement backtracking trajectories; and the associated group fuses the clustering result and the eye movement data sequence.

[0028] An integrated behavior association matrix is constructed through the associated group, and a dimension-reduced behavior matrix is obtained through principal component analysis.

[0029] A teaching level evaluation value is generated by fusing the dimension-reduced behavior matrix and a learning progress vector; and the teaching level evaluation value is used for associating the deviation mode and the progress vector.

[0030] If the teaching level evaluation value exceeds a preset threshold, an abnormal sub-matrix is extracted from the integrated behavior association matrix; and the abnormal sub-matrix refines the high-frequency deviation mode through k-means clustering.

[0031] According to the abnormal sub-matrix, the eye movement backtracking trajectory is updated to obtain an optimized trajectory association matrix, and the teaching level evaluation value is recalculated.

[0032] In one embodiment, the teaching level evaluation value is calculated by the following formula:

[0033] exp

[0034] wherein, the teaching level evaluation value is represented by, the dimension-reduced behavior matrix is represented by, the feature dimension after dimension reduction is represented by, the time step is represented by, the learning progress vector is represented by, the normalized progress value at time t is represented by, the first principal component vector is represented by, the column vector of the first principal component vector is represented by, the progress-behavior association vector is represented by, the Frobenius norm is represented by, , the weight coefficient is represented by. In one embodiment, the teaching level evaluation value is integrated with historical evaluation data, a linear regression model is used to predict the progress trend to determine the resource allocation weight, and an individualized teaching plan is optimized, including: The historical evaluation records are obtained, the teaching level evaluation value is merged with the historical evaluation records, a linear regression model with a forgetting factor is used to process the merged data, and a trend slope representing the change in learning state is calculated.

[0035] The historical evaluation records are obtained, the teaching level evaluation value is merged with the historical evaluation records, a linear regression model with a forgetting factor is used to process the merged data, and a trend slope representing the change in learning state is calculated.

[0036] The historical evaluation records are obtained, the teaching level evaluation value is merged with the historical evaluation records, a linear regression model with a forgetting factor is used to process the merged data, and a trend slope representing the change in learning state is calculated. ​​

[0037] Based on the positive and negative of the trend slope and the absolute value size, the associated resource type set is retrieved from the digital resource library combined with the textbook number, and the allocation weight of each type of resource is determined by the entropy weight algorithm.

[0038] According to the resource allocation weight and the current teaching progress, a dynamic adjustment strategy containing resource call priority and push timing is generated.

[0039] From the dynamic adjustment strategy, the student identifier and personalized features are parsed, and the personalized teaching plan containing the weak point reinforcement path and resource adaptation suggestion is constructed by fusing the knowledge graph.

[0040] The knowledge system matching degree of the personalized teaching plan and the textbook number is judged by the semantic similarity algorithm, and if the matching degree is higher than the preset threshold, the digital resource library is updated, the resource type proportion of the corresponding textbook is reorganized according to the resource allocation weight, and the final personalized teaching plan is obtained.

[0041] In the second aspect, the application also provides an English text auxiliary teaching system based on AI vision, which comprises:

[0042] The behavior feature extraction module is used to obtain the dynamic eye movement trajectory sequence, the mouth shape change video stream and the micro-expression time sequence data of the students when reading the English text in the classroom environment, and the dynamic behavior feature vector is obtained by processing through the convolutional neural network.

[0043] The learning state classification module is used to determine the pronunciation deviation area and the understanding difficulty point based on the matching result of the dynamic behavior feature vector and the preset pronunciation standard model, and the learning state classification label is obtained by using the support vector machine classification.

[0044] The teaching level evaluation module is used to extract the high-frequency deviation mode from the learning state classification label, and the teaching level evaluation value is calculated by associating and fusing the learning progress vector through clustering and eye movement backtracking trajectory.

[0045] The teaching plan generation module is used to integrate the teaching level evaluation value and the historical evaluation data, and the resource allocation weight is determined by using the linear regression model to predict the progress trend, and the personalized teaching plan is optimized.

[0046] In the third aspect, the application also provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the previous method when executing the computer program.

[0047] In the fourth aspect, the application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to implement the previous method.

[0048] The AI vision-based English lesson auxiliary teaching method, system, computer device and storage medium, first collect dynamic eye movement trajectory sequences, mouth shape change video streams and micro-expression time sequence data of students reading English lessons in a classroom environment, extract multi-modal features through a convolutional neural network (CNN) and generate a dynamic behavior feature vector; then, the feature vector is matched with a preset pronunciation standard model in multiple dimensions, and based on the matching result, pronunciation deviation areas and understanding difficulties are located, and a support vector machine (SVM) is used to classify to obtain learning state labels including concentration and understanding degree dimensions; subsequently, high-frequency deviation patterns are extracted from the labels, a comprehensive behavior correlation matrix is constructed by clustering algorithm and eye movement backtracking trajectory association, and a teaching level evaluation value is calculated by fusing a learning progress vector; finally, the evaluation value and historical evaluation data are integrated, a linear regression model is used to predict the learning progress trend, and accordingly, the digital resource allocation weight is determined and an individualized teaching plan is optimized. The method realizes the intelligentization of the whole process from behavior feature extraction to teaching strategy generation through multi-modal data fusion and AI algorithm linkage: on the one hand, through the collaborative analysis of eye movement, mouth shape and micro-expression, the precision limitations of traditional single-mode evaluation are broken through, and pronunciation deviations and understanding obstacles in English reading can be accurately located; on the other hand, based on the trend prediction of dynamic evaluation value and historical data, the resource allocation weight can adapt to the learning trajectory of individual students, solving the problem of resource adaptation lag in traditional teaching; in addition, the quantitative calculation of teaching level evaluation and knowledge system matching degree promotes the upgrade of the digital resource library from static storage to dynamic optimization, providing a data-driven technical path for English individualized teaching. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creating laborious work.

[0050] Figure 1 The flowchart of the AI vision-based English lesson auxiliary teaching method provided by the embodiments of the present application;

[0051] Figure 2 The structural block diagram of the AI vision-based English lesson auxiliary teaching system provided by the embodiments of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0053] In one embodiment, as shown in Figure 1 The application provides an AI vision-based English lesson auxiliary teaching method, which comprises the following steps:

[0054] In step S101, the dynamic eye movement trajectory sequence, mouth movement video stream and micro-expression time sequence data of students reading English lessons in the classroom environment are acquired, and a dynamic behavior feature vector is obtained through a convolutional neural network.

[0055] Specifically, through eye tracking devices, high-definition cameras and other sensors, the dynamic eye movement trajectory sequence (including gaze point coordinates, saccade path and other time sequence data), mouth movement video stream (lip movement image with a resolution of no less than 720P) and micro-expression time sequence data (frame-level record of dynamic changes of facial muscle groups) of students reading English lessons in the classroom environment are collected in real time. After the three types of heterogeneous data are normalized, they are input into a pre-trained convolutional neural network (CNN) model, spatial features (such as mouth contour and eye key points) are extracted through multiple convolutional layers, time dimension features are processed through a time pooling layer, and finally a dynamic behavior feature vector with unified dimensions is output by a fully connected layer, realizing the conversion from original perception data to abstract semantic features.

[0056] In step S102, based on the matching result of the dynamic behavior feature vector and the preset pronunciation standard model, the pronunciation deviation area and the understanding difficulty point are determined, and a learning state classification label is obtained by using a support vector machine classification.

[0057] Further, the generated dynamic behavior feature vector is matched with the preset pronunciation standard model (including a standard mouth shape feature library and an eye movement reference trajectory) step by step, and through the calculation of cosine similarity, dynamic time warping (DTW) distance and other indicators, the pronunciation deviation area (such as the mouth shape deformation frame segment corresponding to a specific phoneme) and the understanding difficulty point (such as the abnormal residence node of the eye movement trajectory at a long and difficult sentence) are located. Based on the deviation features, an input matrix is constructed, a support vector machine (SVM) classifier is used for binary classification and multi-classification processing, and a learning state classification label containing “pronunciation error type”, “understanding obstacle level” and “attention state” is output. The label system conforms to the ISO 25010 education software quality model specification.

[0058] In step S103, a high-frequency deviation pattern is extracted from the learning state classification label, and a teaching level evaluation value is calculated by associating and fusing the learning progress vector through clustering and eye movement backtracking trajectory.

[0059] The time series statistics of the learning state classification labels are performed, deviation patterns (such as the occurrence rate of connected reading errors in virtual mood sentence patterns) with a frequency higher than a threshold are extracted, the deviation patterns are associated with the eye movement backtracking trajectory data through a K-means clustering algorithm, and a comprehensive behavior association matrix with a dimension of [deviation type x time step x feature weight] is constructed. A standardized learning progress vector (the element value corresponds to the completion degree of the knowledge points in the teaching outline) is introduced, and the evaluation value of the quantitative teaching level is calculated through matrix point multiplication and weighted summation operation, which represents the matching degree of the current behavior state of the student and the teaching target, and the value range is [0, 1].

[0060] In step S104, the teaching level evaluation value is integrated with the historical evaluation data, the linear regression model is used to predict the progress trend to determine the resource allocation weight, and the personalized teaching plan is optimized.

[0061] Specifically, the current teaching level evaluation value is integrated with the historical evaluation data (the evaluation results of the previous N classes constitute a time series), a linear regression model with an exponential forgetting factor is used for trend fitting, and the slope parameter is used to predict the progress rate of the student in the dimensions of knowledge point mastery and pronunciation accuracy. Based on the prediction result, the allocation weight of each type of resource (video explanation, interactive exercise, and text analysis) in the digital resource library is determined by combining the information entropy weight method, and the personalized teaching plan containing the weak point strengthening path and the resource pushing time sequence is generated according to the resource calling priority algorithm and the teaching progress constraint condition. The plan content needs to meet the corresponding requirements of the CEFR English ability framework level.

[0062] The above-mentioned AI vision-based English lesson auxiliary teaching method first collects dynamic eye movement trajectory sequences, mouth shape change video streams and micro-expression time sequence data of students reading English lessons in the classroom environment, extracts multi-modal features through a convolutional neural network (CNN) and generates a dynamic behavior feature vector; then the feature vector is matched with a preset pronunciation standard model in multiple dimensions, the pronunciation deviation area and understanding difficulties are located based on the matching result, and the learning state label containing dimensions such as concentration and understanding degree is obtained by using a support vector machine (SVM) classification; then the high-frequency deviation mode is extracted from the label, a comprehensive behavior correlation matrix is constructed by associating the eye movement backtracking trajectory through a clustering algorithm, and a teaching level evaluation value is calculated by fusing a learning progress vector; finally, the evaluation value and historical evaluation data are integrated, a linear regression model is used to predict the learning progress trend, and the digital resource allocation weight is determined and the personalized teaching plan is optimized accordingly. This method realizes the intelligentization of the whole process from behavior feature extraction to teaching strategy generation through multi-modal data fusion and AI algorithm linkage: on the one hand, through the cooperative analysis of eye movement, mouth shape and micro-expression, the precision limitations of traditional single-mode evaluation are broken through, and the pronunciation deviation and understanding obstacles in English reading can be accurately located; on the other hand, based on the trend prediction of dynamic evaluation value and historical data, the resource allocation weight can adapt to the learning trajectory of individual students, solving the problem of resource adaptation lag in traditional teaching; in addition, the quantitative calculation of teaching level evaluation and knowledge system matching degree promotes the upgrade of digital resource library from static storage to dynamic optimization, providing a data-driven technical path for English personalized teaching.

[0063] In one of the embodiments, acquiring dynamic eye movement trajectory sequences, mouth shape change video streams and micro-expression time sequence data of students reading English lessons in the classroom environment, and obtaining a dynamic behavior feature vector through a convolutional neural network processing can include the following steps:

[0064] Step S201, through a high-resolution camera array with space-time alignment, dynamic eye movement trajectory sequences, mouth shape change video streams and micro-expression time sequence data of students reading English lessons in the classroom environment are collected in real time.

[0065] Step S202, an improved 3D convolutional neural network is used to process the dynamic eye movement trajectory sequences, and the key fixation point sequence features are strengthened combined with an attention mechanism to generate an eye movement feature vector containing fixation residence time-speed vectors.

[0066] Step S203, a light-weight convolutional neural network combined with an optical flow method is used to model the facial key point motion trajectory of the micro-expression time sequence data, and an expression feature vector containing dynamic deformation features of the brow muscle and orbicularis oris muscle is extracted.

[0067] Step S204, the lip shape change video stream is modeled by a residual convolutional network, a dynamic time warping algorithm is introduced to match pronunciation rhythm characteristics, and a lip shape feature vector containing three-dimensional motion trajectories of lips is generated.

[0068] Step S205, the lip shape feature vector, the eye movement feature vector and the expression feature vector are fused to generate a dynamic behavior feature vector for representing the dynamic behavior pattern of the student during reading.

[0069] Specifically, by spatiotemporal alignment of a high-resolution camera array (spatial resolution is set to 1080P, time frame rate is 30fps, and three different angle cameras are included to cover the whole area of the face), three types of original data are collected in real time in a classroom environment: dynamic eye movement trajectory sequence (two-dimensional coordinates of pupil center, mapping position of fixation point in text area, and interframe transfer path are recorded every frame, sampling interval is 10ms), lip shape change video stream (focus on lips and surrounding area, resolution is 720P, each frame contains upper and lower lip contour coordinates, gingival exposure degree, etc.), and micro-expression time series data (covering the whole face, recording pixel-level dynamic changes of 21 key points such as eyebrow arch, eye, and mouth). Through a hardware-level time stamp synchronization mechanism (error ≤1ms), the alignment of the three types of data on the time axis is ensured, and the input with consistent spatiotemporal reference for subsequent feature extraction is provided.

[0070] Further, for the dynamic eye movement trajectory sequence, an improved 3D convolutional neural network (on the basis of traditional 3D-CNN, the first layer of convolution kernel size is adjusted from 3×3×3 to 5×5×1 to adapt to the spatiotemporal sparsity of eye movement data, and redundant calculation is reduced) is input, and through 4 layers of convolution operation, the spatial distribution features (such as the concentration degree of fixation points in difficult sentence area) and time sequence features (such as the continuity of saccade path) of the trajectory are extracted; the network embeds a channel attention module, and by calculating the information entropy of different fixation point sequences (the information entropy weight of fixation points in the corresponding area of a difficult sentence is increased by 30%), the key fixation point features are strengthened, and finally a 128-dimensional eye movement feature vector is output, which contains two core dimensions of fixation residence time (cumulative residence time of a single point, unit ms) and velocity vector (direction angle and speed of fixation point transfer, unit ° / s).

[0071] Preferably, for micro-expression timing data, a lightweight convolutional neural network (based on MobileNet architecture improvement, set the expansion rate of depth separable convolution to 2 to expand the receptive field) is used for initial feature extraction, and the dense optical flow method is used to calculate the pixel motion vector between adjacent frames, and the three-dimensional motion trajectory (x, y coordinates are pixel positions, z coordinates are depth estimation values) of 21 facial key points (including eyebrow muscle, orbicularis oris muscle, orbicularis muscle, etc. Core area) is modeled; Through first-order derivative (motion speed) and second-order derivative (acceleration) analysis of the trajectory, 16 dynamic deformation features such as eyebrow muscle contraction amplitude (unit: mm) and orbicularis oris muscle opening and closing frequency (unit: Hz) are extracted, and a 128-dimensional expression feature vector is generated.

[0072] For mouth shape change video stream, input residual convolutional network (based on ResNet-18 improvement, add timing convolutional layer in the 3rd and 4th residual blocks to capture dynamic features), extract spatial features of lip contour (such as lip peak distance, upper and lower lip thickness ratio) through 6-layer convolution and residual connection, and capture the continuous change rule of mouth shape through a bidirectional LSTM layer (hidden unit number 64); Introduce dynamic time warping (DTW) algorithm, align the extracted mouth shape timing features with the standard pronunciation rhythm template of international phonetic alphabet (including 48 phonetic alphabet mouth shape change period benchmarks), correct the timing offset (alignment error ≤2 frames) caused by speed difference, generate 128-dimensional mouth shape feature vector, including lip three-dimensional motion trajectory (time sequence of x, y, z coordinates, sampling frequency 30Hz).

[0073] A "feature splicing + dynamic weighting" fusion strategy is used to integrate three types of feature vectors: first, the 128-dimensional mouth shape feature vector, the 128-dimensional eye movement feature vector, and the 128-dimensional expression feature vector are spliced in the feature dimension to form a 384-dimensional initial fusion feature; Then calculate the dynamic weight of each modal feature (assign 0.4-0.5 weight to mouth shape feature in pronunciation stage, assign 0.35-0.45 weight to eye movement feature in understanding stage) through 3-layer MLP (hidden layer dimension 256-128), and adjust the initial fusion feature by weighting, finally generate a 256-dimensional dynamic behavior feature vector, which fully represents the visual attention mode, facial emotional dynamics and pronunciation organ movement state of students when reading English texts.

[0074] The embodiment realizes high-precision extraction and integration of dynamic behavior characteristics through hierarchical optimization technology design: first, the spatiotemporal alignment multi-view camera array and the hardware-level synchronization mechanism ensure the consistency of the original data in the spatiotemporal dimension, solve the feature mispositioning problem caused by the delay of multi-modal data collection, and provide a reliable data foundation for subsequent fusion; second, the network model is customized according to the characteristics of eye movement, micro-expression, and mouth shape data (the improved 3D convolutional neural network adapts to the sparsity of eye movement, the lightweight network adapts to the real-time performance of micro-expression, and the residual network adapts to the complex dynamics of mouth shape), and the key features are strengthened by combining attention mechanism, optical flow method, DTW algorithm, etc., so that the extraction accuracy of single modal features is improved by 15%-20% compared with general models; third, the dynamic weighted multi-modal fusion strategy realizes the feature emphasis of different learning stages (pronunciation, understanding), and the generated dynamic behavior feature vector covers multi-dimensional information of vision, expression, and pronunciation, overcoming the limitations of one-sidedness of single modal data representation, providing comprehensive and accurate feature input for subsequent learning state evaluation (such as pronunciation deviation positioning and understanding difficulty point identification), and supporting the accurate implementation of personalized teaching intervention.

[0075] In one of the embodiments, based on the matching result of the dynamic behavior feature vector and the preset pronunciation standard model, the pronunciation deviation area and the understanding difficulty point are determined, and the learning state classification label is obtained by using the support vector machine classification, which can include the following steps:

[0076] Step S301, based on the dynamic behavior feature vector and the preset pronunciation standard model, multi-dimensional matching is performed to obtain a matching degree matrix containing feature similarity and time sequence alignment deviation; the preset pronunciation standard model contains standard mouth shape trajectory sequence, standard eye movement scanning path and neutral expression reference parameters.

[0077] Step S302, threshold segmentation and region positioning are performed on the matching degree matrix to obtain specific segments corresponding to pronunciation deviation and understanding difficulty point coordinate set focused by eye movement in the text.

[0078] Step S303, based on the specific segments of pronunciation deviation and the understanding difficulty point coordinate set, the mouth shape and eye movement coordination features of the pronunciation deviation segments and the expression fluctuation features of the understanding difficulty points are extracted, and a multi-dimensional learning state description vector is constructed.

[0079] Step S304, the multi-dimensional learning state description vector is input into the pre-trained support vector machine classifier, the high-dimensional space feature mapping is calculated through the kernel function, and the multi-dimensional learning state classification label containing attention degree, understanding degree and pronunciation accuracy is obtained.

[0080] Specifically, the dynamic behavior feature vector is matched with a preset pronunciation standard model in multiple dimensions: the preset pronunciation standard model includes a standard mouth shape trajectory sequence (a three-dimensional motion benchmark of the lips corresponding to the International Phonetic Alphabet), a standard eye movement scanning path (an optimal fixation point transfer mode for reading a textbook), and a neutral expression benchmark parameter (a static coordinate range of facial key points); by calculating the differences between the dynamic behavior feature vector and the standard model in spatial features (such as lip contour similarity) and time series (such as pronunciation rhythm alignment), a matching degree matrix is generated, which includes feature similarity (value range [0, 1]) and time sequence alignment deviation (unit frame). The adaptive threshold segmentation (threshold dynamically adjusted according to the difficulty of the textbook, range [0.3, 0.6]) is used on the matching degree matrix, and the area with a feature similarity lower than the threshold is marked as a deviation area, which is located in the textbook text combined with the time sequence alignment deviation, to obtain specific segments (accurate to the word or phrase level) corresponding to pronunciation deviation and eye movement focus difficulty point coordinate set (two-dimensional coordinates of the text page). Based on the above pronunciation deviation segments and understanding difficulty point coordinate set, two types of core features are extracted: for the pronunciation deviation segments, the time correlation of the mouth shape feature vector and the eye movement feature vector (such as the time difference between the peak value of the lip movement and the fixation point residence) is calculated to form the mouth shape-eye movement coordination feature; for the understanding difficulty points, the expression feature vector fluctuation amplitude (such as the peak value of the brow muscle contraction acceleration) of the corresponding position is analyzed to obtain the expression fluctuation feature; the two types of features are integrated into a 128-dimensional multi-dimensional learning state description vector. The description vector is input into a pre-trained support vector machine (SVM) classifier (kernel function uses radial basis function RBF, optimized by 100,000 + samples training), which maps low-dimensional features to high-dimensional space through kernel function, realizes nonlinear classification, and outputs multi-dimensional learning state classification labels including concentration (three levels: high / medium / low), understanding degree (four levels: complete understanding / partial understanding / fuzzy / not understanding), and pronunciation accuracy (five levels: standard / slight deviation / obvious deviation / serious deviation / error).

[0081] This embodiment achieves precise quantification of learning states through hierarchical processing: First, a pre-set pronunciation standard model covers multiple dimensions of benchmarks, including lip shape, eye movement, and facial expression. Combined with a matrix generated by multi-dimensional matching, it overcomes the limitations of traditional single-feature matching and provides a comprehensive basis for deviation localization. Second, the combination of adaptive threshold segmentation and text region localization achieves precise mapping between pronunciation deviation segments and points of difficulty in understanding (localization error ≤ 2 words), solving the problem of "deviation being disconnected from the text" in traditional assessment. Third, the extraction of lip shape-eye movement collaborative features and facial expression fluctuation features strengthens the correlation representation of behavioral data, making the learning state description vector closer to the real learning process. Fourth, the pre-trained SVM classifier improves classification accuracy through high-dimensional mapping, and the output multi-dimensional labels provide a directly applicable quantitative basis for subsequent personalized teaching interventions, supporting the precise adjustment of teaching strategies.

[0082] In one embodiment, the matching degree matrix can be calculated using the following formula:

[0083]

[0084]

[0085]

[0086] in, This indicates the matching degree matrix at time step Feature Dimension The element value at that position, Represents the balance coefficient. Indicates the first Modality in feature dimension The feature similarity is defined on the graph, with values ​​ranging from [−1, 1]. Indicates at time step At this point, the dynamic behavioral characteristics and all time steps in the standard model time series are compared. The minimum value of timing alignment deviation, Indicates the temporal nodes in the reading process. This represents the specific dimension in the feature vector. The first element in the dynamic behavior feature vector represents the... Feature sequences of each modality This represents the standard feature sequence of the corresponding modality in the preset pronunciation standard model. Indicates the minimum value. A time series representing dynamic behavioral characteristics. Represents the standard model time series. Indicates the time step. Represents the timing alignment deviation matrix. Represents the timing penalty coefficient. representing a time difference, representing a multimodal similarity maximum, .

[0087] The embodiment realizes precise quantitative matching of dynamic behavior features and standard models by jointly modeling multimodal feature similarity and time alignment deviation: the balance coefficient a is used to adjust the weight proportion of different modalities (mouth shape, eye movement, expression) in the matching degree calculation, ensuring the collaborative consideration of multi-dimensional features; feature similarity quantifies the spatial difference between each modality feature and the standard model, and the time alignment deviation takes the minimum deviation value through the dynamic time warping idea, effectively solving the time sequence misplacement problem caused by speed difference; the time penalty coefficient β is introduced to weight the time difference, and the multimodal similarity maximum Norm is normalized to make the matching degree matrix element value uniformly mapped to the [0,1] interval, which not only retains the spatial-temporal double difference information of the original features, but also ensures the comparability of cross-modal and cross-time step data, providing a high-precision quantitative basis for subsequent deviation area positioning and learning state classification.

[0088] In one of the embodiments, high-frequency deviation patterns are extracted from the learning state classification labels, and teaching level evaluation values are calculated by clustering and associating eye movement backtracking trajectories and fusing learning progress vectors, which can include the following steps:

[0089] Step S401, extracting high-frequency deviation patterns from the learning state classification labels; the high-frequency deviation patterns include attention dispersion frequency and knowledge point forgetting rate.

[0090] Step S402, clustering according to the high-frequency deviation patterns to obtain an associated group of eye movement backtracking trajectories; the associated group fuses the clustering results and eye movement data sequences.

[0091] Step S403, constructing a comprehensive behavior association matrix through the associated group, and obtaining a reduced dimension behavior matrix through principal component analysis.

[0092] Step S404, fusing the reduced dimension behavior matrix and the learning progress vector to generate a teaching level evaluation value; the teaching level evaluation value is used to associate the deviation pattern and the progress vector.

[0093] Step S405, if the teaching level evaluation value exceeds a preset threshold, extracting an abnormal submatrix from the comprehensive behavior association matrix; the abnormal submatrix refines the high-frequency deviation pattern through k-means clustering.

[0094] Step S406, updating the eye movement backtracking trajectory according to the abnormal submatrix to obtain an optimized trajectory association matrix, and recalculating the teaching level evaluation value.

[0095] Specifically, high-frequency deviation patterns are extracted from the learning state classification labels, including attention distraction frequency (the number of times the fixation point deviates from the text area per unit time) and knowledge point forgetting rate (the frequency of the appearance of understanding difficulty points of the learned knowledge points). The high-frequency deviation patterns are clustered (using the DBSCAN density clustering algorithm, with a neighborhood radius of 0.5 and a minimum sample number of 5), samples with similar deviation characteristics are divided into the same group, and are associated with eye movement backtracking trajectory data (including fixation point coordinates, dwell time, and other time series information) to form an associated group that integrates the clustering results and eye movement data sequences.

[0096] Further, an integrated behavior association matrix with dimensions [deviation pattern x time step x feature weight] is constructed by associating the group, and principal component analysis (PCA) is used for dimension reduction processing (retaining 95% of the variance contribution rate) to obtain a reduced behavior matrix. The reduced behavior matrix and the learning progress vector (element values correspond to the mastery degree of the teaching outline knowledge points, with a value range of [0, 1]) are weighted and fused (the weight of the reduced matrix is 0.6, and the weight of the progress vector is 0.4) to generate a teaching level evaluation value (with a value range of [0, 1]), which is used to quantify the association degree of deviation patterns and learning progress. If the evaluation value exceeds the preset threshold (default 0.7), an abnormal sub-matrix (corresponding to the top 30% of the matrix elements in terms of evaluation value contribution) is extracted from the integrated behavior association matrix, the high-frequency deviation patterns are refined through K-means clustering (cluster number K=3), the association rules of eye movement backtracking trajectories are updated, and an optimized trajectory association matrix is obtained. The teaching level evaluation value is recalculated.

[0097] The present embodiment realizes dynamic evaluation and optimization of teaching level through multi-level data processing. The quantitative extraction of high-frequency deviation patterns (attention distraction frequency, knowledge point forgetting rate) provides measurable indicators for learning state analysis, solving the problem of qualitative description in traditional evaluation. The association and fusion of clustering processing and eye movement data realize the spatio-temporal mapping of deviation patterns and visual behavior, making behavior analysis more traceable. Principal component analysis dimension reduction reduces computational complexity while retaining key information, improving evaluation efficiency. The fusion calculation of evaluation value and learning progress establishes a quantitative association between deviation patterns and teaching goals, supporting precise adaptation of teaching resources. The abnormal sub-matrix extraction and clustering refinement mechanism based on the evaluation threshold makes the eye movement trajectory association rules and deviation pattern analysis dynamically evolve with the learning process, providing more accurate quantitative basis for personalized teaching intervention.

[0098] In one embodiment, the teaching level evaluation value can be calculated by the following formula:

[0099] exp

[0100] wherein, denotes a teaching level evaluation value, denotes a dimension reduction behavior matrix, denotes a feature dimension after dimension reduction, denotes a time step, denotes a learning progress vector, denotes a moment normalized progress value, denotes a first principal component vector, denotes a first column vector of , denotes a progress-behavior correlation vector, fitted by , denotes a Frobenius norm, , denotes a weight coefficient.

[0101] The embodiment realizes quantitative representation and dynamic correlation of teaching state by fusion modeling of the dimension reduction behavior matrix and the learning progress vector: the dimension reduction behavior matrix obtained by principal component analysis retains more than 95% of the key information of behavior characteristics, and the similarity between the dimension reduction behavior matrix and the first principal component vector is calculated by Frobenius norm to capture the overall distribution characteristics of high-frequency deviation patterns; the learning progress vector adopts the moment progress value after standardization, and the progress-behavior correlation vector is linearly fitted to strengthen the mapping relationship between the knowledge point mastery degree and the behavior characteristics; the weight coefficient and (the default values are 0.6 and 0.4, respectively) dynamically adjust the contribution ratio of behavior characteristics and learning progress in evaluation, so that the evaluation value can reflect not only the current behavior deviation pattern of students, but also the completion progress of the teaching goal; the exponential function mapping normalizes the comprehensive calculation result to the interval [0,1], ensuring the interpretability and cross-scene comparability of the evaluation value, and improving the teaching state representation accuracy by 18%-22% compared with the traditional single-dimensional evaluation method, providing a scientific and quantitative decision basis for personalized teaching resource allocation and strategy adjustment.

[0102] In one of the embodiments, the teaching level evaluation value is integrated with historical evaluation data, a linear regression model is used to predict the progress trend to determine the resource allocation weight, and an optimized personalized teaching plan can include the following steps:

[0103] Step S501, obtaining historical evaluation records, merging the teaching level evaluation value with the historical evaluation records, processing the merged data by using a linear regression model with a forgetting factor, and calculating a trend slope representing the change of learning state.

[0104] Step S502, based on the positive and negative of the trend slope and the absolute value size, the teaching material number is combined to retrieve the associated resource type set from the digital resource library, and the allocation weight of each type of resource is determined by the entropy weight algorithm.

[0105] Step S503, according to the resource allocation weight and the current teaching progress, a dynamic adjustment strategy containing resource call priority and push timing is generated.

[0106] Step S504, the student identifier and personalized features are parsed from the dynamic adjustment strategy, and the personalized teaching plan containing the weak point reinforcement path and resource adaptation suggestion is constructed by fusing the knowledge graph.

[0107] Step S505, the semantic similarity algorithm is used to judge the knowledge system matching degree of the personalized teaching plan and the teaching material number, if the matching degree is higher than the preset threshold, the digital resource library is updated, the resource type proportion of the corresponding teaching material is reorganized according to the resource allocation weight, and the final personalized teaching plan is obtained.

[0108] Specifically, first, the historical evaluation records are obtained, the current teaching level evaluation value is combined with the historical data to form a time series data set, a linear regression model with forgetting factor (the forgetting factor λ is set to 0.85) is used to process the data set, and the trend slope representing the change of learning state is calculated by least square fitting. The slope reflects the progress rate of students in knowledge point mastery, pronunciation accuracy and other dimensions.

[0109] Based on the positive and negative of the trend slope and the absolute value size (positive value indicates progress, negative value indicates regression, and absolute value reflects the change amplitude), the associated resource type set (such as video explanation, interactive exercise, text analysis, etc.) is retrieved from the digital resource library according to the teaching material number, the information entropy and redundancy of each type of resource are calculated by the entropy weight algorithm, and the resource allocation weight (the sum of the weights is 1) is determined. According to the weight and the current teaching progress (knowledge point completion degree), a dynamic adjustment strategy containing resource call priority (high / medium / low three levels), push timing (specific time node) is generated.

[0110] Further, the student identifier and personalized features (such as high-frequency deviation mode, learning preference) are parsed from the dynamic adjustment strategy, and the personalized teaching plan is constructed by fusing the knowledge graph (containing knowledge point association relationship, difficulty level), including weak point reinforcement path (such as special training for connected reading error) and resource adaptation suggestion (such as recommending specific difficulty listening materials). The matching degree of the teaching plan and the teaching material knowledge system is judged by the semantic similarity algorithm (cosine similarity is used), if the matching degree is higher than the preset threshold (default 0.7), the digital resource library is updated, the resource type proportion of the corresponding teaching material is reorganized according to the resource allocation weight, and the final dynamic optimized personalized teaching plan is formed.

[0111] The linear regression model with a forgetting factor in this embodiment assigns time decay weights to historical data, more accurately capturing the recent learning state change trend of students, and improving the trend prediction accuracy by 15-20% compared to traditional linear regression; the entropy weight algorithm automatically determines the resource weight based on data characteristics, avoiding artificial experience bias, and making the resource allocation more in line with the actual needs of students; the dynamic adjustment strategy couples the resource priority with the teaching progress, ensuring the timeliness and pertinence of resource pushing; the fusion of knowledge graph and personalized features enables the teaching plan to have the closed-loop ability of "weak point positioning - path planning - resource adaptation", improving the accuracy of teaching intervention; fifth, the resource library updating mechanism based on semantic similarity promotes the upgrade of digital resources from static storage to dynamic evolution

[0112] In one of the embodiments, as shown in Figure 2 The present application also provides an AI vision-based English lesson auxiliary teaching system, which can include the following steps:

[0113] The behavior feature extraction module 601 is used to obtain the dynamic eye movement trajectory sequence, mouth movement video stream and micro-expression time sequence data of students reading English lessons in the classroom environment, and obtain a dynamic behavior feature vector through a convolutional neural network.

[0114] The learning state classification module 602 is used to determine the pronunciation deviation area and understanding difficulty point based on the matching result of the dynamic behavior feature vector and the preset pronunciation standard model, and obtain a learning state classification label by using a support vector machine classification.

[0115] The teaching level evaluation module 603 is used to extract a high-frequency deviation pattern from the learning state classification label, associate and fuse a learning progress vector by clustering and eye movement backtracking trajectory to calculate a teaching level evaluation value.

[0116] The teaching plan generation module 604 is used to integrate the teaching level evaluation value and historical evaluation data, predict the progress trend by using a linear regression model to determine the resource allocation weight, and optimize to obtain a personalized teaching plan.

[0117] The above-mentioned AI vision-based English text auxiliary teaching system, the behavior feature extraction module collects the dynamic eye movement trajectory, the mouth shape change video and the micro-expression time sequence data of the student reading the English text in real time through the space-time aligned camera array, generates the dynamic behavior feature vector after the convolutional neural network processing, and realizes the feature abstraction of the multi-modal behavior data. The learning state classification module matches the feature vector with the preset pronunciation standard model, locates the pronunciation deviation area and the understanding difficulty point, uses the support vector machine classification to obtain the learning state label containing the concentration, pronunciation accuracy and other dimensions, and completes the semantic mapping from the feature to the state. The teaching level evaluation module extracts the high-frequency deviation mode from the classification label, constructs a comprehensive behavior matrix by clustering and associating with the eye movement backtracking trajectory, fuses the learning progress vector to calculate the teaching level evaluation value, and quantizes the matching degree of the learning state and the teaching target. The teaching plan generation module integrates the current and historical evaluation data, predicts the progress trend by using the linear regression model to determine the resource allocation weight, generates the personalized teaching plan containing the weak point reinforcement path by combining the knowledge graph, and finally forms the dynamically adapted teaching scheme through semantic matching optimization.

[0118] The multi-modal feature extraction of the embodiment breaks through the limitation of single-modal evaluation, and improves the behavior representation accuracy by 15%-20%; the state classification and evaluation module establishes the quantitative association between the learning state and the teaching target through deviation mode clustering and progress fusion; the teaching plan generation module solves the problem of resource adaptation lag in traditional teaching based on historical trend prediction and dynamic allocation of resource weight. The modules cooperate to provide a traceable and optimized technical framework for English personalized teaching, and improve the intervention accuracy.

[0119] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0120] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the AI vision-based English text auxiliary teaching method and system as described above when executing the computer program.

[0121] In one embodiment, a computer readable storage medium is provided, having stored thereon a computer program, which when executed by a processor implements the steps of any of the above method embodiments.

[0122] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts are described in the part of the method embodiments. The above described device embodiments are only illustrative, wherein the components described as separate components can or can not be physically separated, the components displayed as units can or can not be physical units, i.e. can be located in one place or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the present disclosure according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0123] The above described embodiments only express several implementation manners of the present application, the description is more specific and detailed, but it should not be understood as a limitation to the patent scope of the application. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1. An AI vision-based English lesson auxiliary teaching method, characterized in that, The method comprises: obtaining a dynamic eye movement trajectory sequence, a mouth shape change video stream and a micro-expression timing data of a student reading an English textbook in a classroom environment, and obtaining a dynamic behavior feature vector through a convolutional neural network processing; based on the matching result of the dynamic behavior feature vector and the preset pronunciation standard model, determining the pronunciation deviation area and the understanding difficulty point, and obtaining the learning state classification label by using the support vector machine classification; extracting a high-frequency deviation mode from the learning state classification label, and calculating a teaching level evaluation value by associating and fusing a learning progress vector through clustering and eye movement backtracking trajectory; integrating the teaching level evaluation value and historical evaluation data, predicting the progress trend by using a linear regression model to determine the resource allocation weight, and optimizing to obtain a personalized teaching plan.

2. The method of claim 1, wherein, The method comprises: through a high-resolution camera array with space-time alignment, real-time collection of dynamic eye movement trajectory sequence, mouth shape change video stream and micro-expression timing data of a student reading an English textbook in a classroom environment; using an improved 3D convolutional neural network to process the dynamic eye movement trajectory sequence, combining an attention mechanism to strengthen the key fixation point sequence features, and generating an eye movement feature vector containing fixation residence time and speed vector; using a lightweight convolutional neural network combined with an optical flow method to model the facial key point motion trajectory of the micro-expression timing data, and extracting an expression feature vector containing eyebrow muscle and orbicularis oris dynamic deformation features; using a residual convolutional network to model the timing of the mouth shape change video stream, introducing a dynamic time warping algorithm to match the pronunciation rhythm features, and generating a mouth shape feature vector containing the three-dimensional motion trajectory of the lips; fusing the mouth shape feature vector, eye movement feature vector and expression feature vector to generate a dynamic behavior feature vector representing the dynamic behavior mode of the student reading.

3. The method of claim 1, wherein, The method comprises: based on the dynamic behavior feature vector and the preset pronunciation standard model, a multi-dimensional matching is performed to obtain a matching degree matrix containing feature similarity and timing alignment deviation; the preset pronunciation standard model contains standard mouth shape trajectory sequence, standard eye movement scanning path and neutral expression reference parameters; threshold segmentation and region positioning are performed on the matching degree matrix to obtain specific segments corresponding to pronunciation deviation and eye movement focused understanding difficulty point coordinate set in the textbook text; based on the specific segments of pronunciation deviation and the understanding difficulty point coordinate set, the mouth shape, eye movement coordination features of the pronunciation deviation segments and the expression fluctuation features of the understanding difficulty points are extracted to construct a multi-dimensional learning state description vector; the multi-dimensional learning state description vector is input into a pre-trained support vector machine classifier, the high-dimensional space feature mapping is calculated through a kernel function, and a multi-dimensional learning state classification label containing concentration, understanding degree and pronunciation accuracy is obtained.

4. The method of claim 3, wherein, The matching degree matrix is calculated by the following formula: wherein, denotes the element value of the matching degree matrix at time step , feature dimension , denotes the balance coefficient, denotes the feature similarity of the modal at feature dimension , with a value range of [−1, 1], denotes the minimum value of the time series alignment deviation of the dynamic behavior feature and all time steps in the standard model time series sequence at time step , denotes the time series node in the reading process, denotes the specific dimension in the feature vector, denotes the feature sequence of the modal in the dynamic behavior feature vector, denotes the standard feature sequence of the corresponding modal in the preset pronunciation standard model, denotes the minimum value, denotes the time series sequence of the dynamic behavior feature, denotes the standard model time series sequence, denotes the time step length, denotes the time series alignment deviation matrix, denotes the time series penalty coefficient, denotes the time difference, denotes the maximum value of the multi-modal similarity, .

5. The method of claim 1, wherein, The high-frequency deviation mode is extracted from the learning state classification label, and a teaching level evaluation value is calculated by clustering and associating the eye movement backtracking trajectory and fusing a learning progress vector, including: extracting a high-frequency deviation mode from the learning state classification label; the high-frequency deviation mode includes attention dispersion frequency and knowledge point forgetting rate; performing clustering processing according to the high-frequency deviation mode to obtain an associated group of eye movement backtracking trajectories; the associated group fuses clustering results and eye movement data sequences; constructing a comprehensive behavior association matrix through the associated group, and obtaining a reduced dimension behavior matrix through principal component analysis dimension reduction; fusing the reduced dimension behavior matrix and the learning progress vector to generate a teaching level evaluation value; the teaching level evaluation value is used to associate the deviation mode and the progress vector; if the teaching level evaluation value exceeds a preset threshold, an abnormal sub-matrix is extracted from the comprehensive behavior association matrix; the abnormal sub-matrix refines the high-frequency deviation mode through k-means clustering; updating the eye movement backtracking trajectory according to the abnormal sub-matrix to obtain an optimized trajectory association matrix, and recalculating a teaching level evaluation value.

6. The method of claim 5, wherein, The teaching level evaluation value is calculated by the following formula: exp wherein, denotes a teaching level evaluation value, denotes a dimension reduction behavior matrix, denotes a feature dimension after dimension reduction, denotes a time step, denotes a learning progress vector, denotes a moment normalized progress value, denotes a first principal component vector, denotes a first column vector of the , denotes a progress-behavior correlation vector, by fitting, denotes a Frobenius norm, , denotes a weight coefficient.

7. The method of claim 1, wherein, The teaching level evaluation value is integrated with historical evaluation data, and a linear regression model is used to predict the progress trend to determine the resource allocation weight, and an individualized teaching plan is optimized, including: obtain historical evaluation records, merge the teaching level evaluation value with the historical evaluation records, process the merged data using a linear regression model with a forgetting factor, and calculate a trend slope representing the change in learning state; based on the positive and negative of the trend slope and the absolute value size, combined with the textbook number, retrieve the associated resource type set from the digital resource library, and determine the allocation weight of each type of resource through an entropy weight algorithm; generate a dynamic adjustment strategy including resource call priority and push timing according to the resource allocation weight and the current teaching progress; parse the student identifier and individualized features from the dynamic adjustment strategy, and fuse the knowledge graph to build an individualized teaching plan including weak point reinforcement paths and resource adaptation suggestions; determine the knowledge system matching degree of the individualized teaching plan and the textbook number through a semantic similarity algorithm, if the matching degree is higher than a preset threshold, trigger the digital resource library to update, reorganize the resource type proportion of the corresponding textbook according to the resource allocation weight, and obtain the final individualized teaching plan.

8. An AI vision-based English lesson auxiliary teaching system, characterized in that, The system comprises: a behavior feature extraction module for obtaining a dynamic eye movement trajectory sequence, a mouth movement change video stream and a micro-expression timing data of a student reading an English textbook in a classroom environment, and obtaining a dynamic behavior feature vector through a convolutional neural network; a learning state classification module for determining a pronunciation deviation area and a understanding difficulty point based on the matching result of the dynamic behavior feature vector and a preset pronunciation standard model, and obtaining a learning state classification label by using a support vector machine classification; a teaching level evaluation module for extracting a high-frequency deviation mode from the learning state classification label, and calculating a teaching level evaluation value by clustering and associating the eye movement backtracking trajectory and fusing a learning progress vector. A teaching plan generation module is configured to integrate the teaching level evaluation value with historical evaluation data, determine a resource allocation weight by using a linear regression model to predict a progress trend, and optimize to obtain a personalized teaching plan. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 7.

Images