English reading material difficulty self-adaptive pushing method and system based on reinforcement learning

Abstract

The application discloses an English reading material difficulty self-adaptive pushing method and system based on reinforcement learning, and relates to the technical field of reinforcement learning. The method comprises the following steps: obtaining a touch-drag rate sequence, a paragraph backtracking number sequence and a word long-press query duration sequence generated by a target user in the process of reading current English materials; calculating reading fluency deviation and instant understanding confidence; taking the difficulty level of the current material, the reading fluency deviation and the instant understanding confidence as a three-element state vector, inputting the three-element state vector into a deep Q network policy model to output a difficulty adjustment action; calculating a reward correction value according to the number of correct reading comprehension questions and the reading fluency deviation, combining a benchmark reward value to obtain an actual reward value; constructing an experience tuple and updating the parameters of the deep Q network policy model; and pushing new materials to a user terminal according to the difficulty adjustment action. The application realizes dynamic matching of the difficulty of reading materials and the ability of the user.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, specifically to a method and system for adaptively pushing English reading materials based on reinforcement learning difficulty. Background Technology

[0002] In leveled reading exercises on online English learning platforms, the system needs to recommend reading materials of appropriate difficulty based on the user's English level. Traditional methods mainly rely on the user's self-reported grade level or the results of a one-time vocabulary test, manually matching and recommending reading materials of fixed difficulty levels.

[0003] This traditional method has significant drawbacks: First, there is a discrepancy between the user's self-reported grade level and their actual English proficiency. Some users are in higher grades but have weak English foundations, while others are in lower grades but have outstanding reading abilities; grade information fails to reflect actual reading ability. Second, a one-time vocabulary test only reflects the user's static level at the time of the test and cannot capture the dynamic changes in ability during continuous reading. Third, the fixed-level recommendation model leads to a long-term mismatch between the difficulty of the materials and the user's current level: when the materials are too difficult, users frequently experience reading difficulties and low comprehension; when the materials are too easy, users lack challenge and progress is slow.

[0004] Therefore, there is an urgent need for an adaptive push method that can dynamically adjust the difficulty of materials based on the user's real-time reading behavior, so that the difficulty of materials always matches the user's current ability, thereby improving learning efficiency and participation. Summary of the Invention

[0005] This invention addresses the technical problems in existing technologies, such as the reliance on user-reported grade level or one-time test results for matching the difficulty of materials, which leads to a long-term mismatch between the recommended level and the user's actual level, the inability to capture dynamic changes in user ability, and the underutilization of user reading behavior feedback. It provides a method and system for adaptive recommendation of English reading materials based on reinforcement learning.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, this invention provides a method for adaptively pushing English reading materials based on the difficulty level of reinforcement learning, including: Obtain the sequence of touch drag rate, paragraph back number, and word long press duration generated by the target user while reading the current English material; Based on the touch drag rate sequence and the standard page turning rate, the reading fluency deviation is calculated, and based on the duration sequence of long-press query of words and the average reading time of paragraphs, the instant comprehension confidence is calculated. The three-element state vector is constructed by the difficulty level of the current material, the reading fluency deviation, and the instant understanding confidence. It is input into a pre-trained deep Q-network policy model and outputs a difficulty adjustment action, which includes increasing the difficulty level, decreasing the difficulty level, or maintaining the difficulty level. After the target user completes the subsequent reading comprehension questions of the current material, a reward correction value is calculated based on the number of correct answers and the reading fluency deviation, and combined with the baseline reward value, the actual reward value is obtained. The three-element state vector, the difficulty adjustment action, the actual reward value, and the next state vector are combined into an experience tuple, which is stored in a circular experience queue. Every ten experience tuples, a priority sampling of a small batch of samples is triggered to update the parameters of the deep Q-network policy model. According to the aforementioned difficulty adjustment action, new materials of corresponding difficulty level are selected from the graded reading corpus and pushed to the target user's terminal.

[0007] Secondly, this invention provides an adaptive recommendation system for English reading materials based on reinforcement learning, comprising: The behavior collection module is used to acquire the touch drag rate sequence, paragraph back number sequence, and word long press query duration sequence generated by the target user during the reading of the current English material; The deviation calculation module is used to calculate the reading fluency deviation based on the touch drag rate sequence and the material standard page turning rate, and to calculate the instant comprehension confidence based on the duration sequence of the long press query of the word and the average reading time of the paragraph. The action output module is used to input a three-element state vector consisting of the difficulty level of the current material, the reading fluency deviation, and the instant understanding confidence into a pre-trained deep Q-network policy model, and output a difficulty adjustment action, wherein the difficulty adjustment action includes increasing the difficulty level, decreasing the difficulty level, or maintaining the difficulty. The reward calculation module is used to calculate a reward correction value based on the number of correct answers and the reading fluency deviation after the target user completes several subsequent reading comprehension questions of the current material, and to obtain the actual reward value by combining it with the baseline reward value. The model update module is used to form an experience tuple from the three-element state vector, the difficulty adjustment action, the actual reward value and the next state vector, store it in the circular experience queue, and trigger a priority sampling of small batch samples every ten experience tuples to update the parameters of the deep Q network policy model. The material push module is used to select new materials of corresponding difficulty level from the graded reading corpus according to the difficulty adjustment action and push them to the target user terminal.

[0008] The beneficial effects of this invention are: Compared to existing technologies, this invention first calculates reading fluency deviation using a touch-drag rate sequence and calculates instant comprehension confidence using a word long-press query duration sequence. Combined with the material difficulty level, this forms a three-element state vector, quantifying the implicit representation of user reading behavior into a computable strategy input. Secondly, a deep Q-network strategy model is used to implement difficulty adjustment decisions. After the user completes reading comprehension questions, the number of correct answers and fluency deviation are combined to calculate a reward correction value, ensuring the reward signal reflects both objective answer results and subjective reading process quality. Thirdly, a cyclical experience queue combined with a priority sampling mechanism is designed, retaining only the highest-priority experience tuples. Every ten new samples trigger a network parameter update, effectively controlling computational resource consumption while ensuring learning efficiency. Finally, the system selects materials of corresponding levels from the graded corpus based on the difficulty adjustment actions output by the model, forming a closed-loop adaptive system. This invention achieves real-time dynamic matching of reading material difficulty with user ability, improving the intelligence level of the graded reading system. Attached Figure Description

[0009] Figure 1 A flowchart illustrating the method for adaptively pushing English reading materials based on reinforcement learning provided by this invention; Figure 2 A logical diagram illustrating the reinforcement learning-based method for adaptively pushing English reading materials based on difficulty, provided by this invention. Figure 3 This is a schematic diagram of the structure of the reinforcement learning-based English reading material difficulty adaptive push system provided by the present invention.

[0010] In the attached diagram, the components represented by each number are as follows: Behavior acquisition module 11, deviation calculation module 12, action output module 13, reward calculation module 14, model update module 15, and material push module 16. Detailed Implementation

[0011] Example 1, as Figure 1 , Figure 2 As shown, this embodiment of the invention provides a method for adaptively pushing English reading materials based on reinforcement learning, including: S10: Obtain the sequence of touch drag rate, paragraph back number, and word long press query duration generated by the target user during the reading of the current English material; In the graded reading scenarios of online English learning platforms, users engage in reading via the touchscreen of their devices. Their interactive behaviors while reading materials implicitly reveal rich information about their text comprehension level. For example, excessively fast page-turning may indicate that the material is too simple and the user is skimming through it; excessively slow page-turning and frequent backtracking to previous paragraphs may indicate that the material is too difficult and the user needs to read it repeatedly; frequent long-presses to look up words may indicate that there are too many unfamiliar words and the user's vocabulary is insufficient to support the current level of difficulty. Traditional methods cannot capture these behavioral signals in real time, causing the system's judgment of the user's comprehension status to lag behind the actual reading process.

[0012] Therefore, this step collects three types of behavioral data in real time while the user is reading the current English material, generating corresponding sequences for each. Specifically, the touch-drag rate sequence represents the smoothness of the user's page turning, the paragraph back-click frequency sequence represents the frequency of the user's rereading, and the long-press lookup duration sequence represents the difficulty the user faces in understanding new words. By collecting these sequences, the user's implicit understanding can be transformed into calculable quantitative features, providing a data foundation for subsequent difficulty adjustment decisions.

[0013] Specifically, the system acquires sequences of touch dragging rates, paragraph back navigation times, and word long-press lookup durations generated by the target user while reading the current English material, including: Listen to the target user's touch movement events on the reading interface, and record the start and end coordinates of each touch movement, as well as the corresponding start and end times; Calculate the Euclidean distance between the starting coordinates and the ending coordinates, combine it with the time difference between the starting time and the ending time to obtain the instantaneous drag rate, arrange all instantaneous drag rates in chronological order, and generate the touch drag rate sequence. Monitor the target user's scrolling operation on the reading page. When the scrolling direction is from the bottom of the page to the top and the number of text paragraphs scrolled across is at least one, accumulate one paragraph back event. Arrange the paragraph rollback events accumulated during the current reading period in chronological order of occurrence to generate the paragraph rollback count sequence; Monitor the target user's long press operation on the text area, record the start time and end time of the long press, and calculate the duration of a single word long press query; Arrange the durations of all single word long-press queries within the current reading period in the order in which the queries occurred to generate a sequence of the durations of the word long-press queries. The sequence lengths of the touch drag rate sequence, the paragraph back number sequence, and the word long press query duration sequence are calculated respectively. When the length of any sequence is less than three times, extend the acquisition time window until the end of the current material reading; If, after the reading end time, there are still sequences with a length less than three, the sequences with insufficient length will be padded to three. The padding value is the corresponding average value of the target user in historical readings. If there is no historical data, it will be padded to zero.

[0014] First, the system listens for touch events on the reading interface, recording the start and end coordinates, as well as the start and end times, of each touch. When a user performs a page-turning or swiping operation on the touchscreen, the terminal operating system triggers a touch event. In the callback function of this touch event, the system obtains the start coordinates when the finger touches the screen and the end coordinates before it leaves the screen, while also recording the timestamp of the event trigger.

[0015] Secondly, the Euclidean distance between the starting and ending coordinates is calculated. Combined with the time difference between the starting and ending times, the instantaneous drag rate is obtained. All instantaneous drag rates are then arranged in chronological order to generate a touch drag rate sequence. Specifically, the Euclidean distance is calculated as follows: the square root of the sum of the squares of the differences in the x-coordinates and y-coordinates of the starting and ending coordinates. The time difference equals the ending time minus the starting time. The instantaneous drag rate equals the Euclidean distance divided by the time difference, in pixels per second.

[0016] Each touch and swipe generates an instantaneous drag rate. Arranging all instantaneous drag rates in the order in which the user performs the swipe operation forms a touch drag rate sequence. This sequence reflects the variation in the speed at which the user turns pages or swipes during reading. The faster the speed, the smoother the reading experience; the slower the speed or the greater the fluctuation, the more likely the user is encountering difficulties in understanding.

[0017] Simultaneously, the system monitors the target user's scrolling actions on the reading page. When the scrolling direction is from the bottom of the page upwards and the scrolling crosses at least one text paragraph, a paragraph back event is accumulated. The scrolling direction is determined by comparing the change in the position of the content at the top of the page before and after scrolling. If the content displayed at the top of the page after scrolling is ahead of the content displayed at the top of the page before scrolling (i.e., the user swiped upwards to move the page content downwards), it is determined to be a movement from the bottom of the page upwards. The number of text paragraphs crossed is estimated by dividing the scrolling distance by the paragraph height. When the user scrolls backwards and crosses at least one complete paragraph, it indicates that the user is returning to reread the previous content, and a paragraph back event is accumulated. The accumulated paragraph back events within the current reading period are arranged chronologically to generate a paragraph back event sequence. Each item in this sequence represents the number of back events that occurred at the corresponding time point. The density of events in the sequence reflects the frequency of user rereading; a higher frequency indicates greater difficulty in understanding the material.

[0018] Simultaneously, the system monitors long-press actions by the target user on the text area, recording the start and end times of the long-press, and calculating the duration of a single long-press query. When a user long-presses on a word, the system pops up a dictionary lookup window. The system monitors long-press events, recording the moment the finger is pressed as the start time and the moment the finger is released as the end time. The duration of a single long-press query is equal to the end time minus the start time, in seconds. The longer the duration, the higher the user's level of confusion regarding the word, and the more time may be needed to understand the meaning or example sentences. The durations of all single long-press queries within the current reading period are arranged in the order of occurrence to generate a sequence of long-press query durations.

[0019] Next, the sequence lengths of the touch drag rate sequence, paragraph back count sequence, and word long press query duration sequence were calculated. The sequence length refers to the number of elements in the sequence, reflecting the total number of times the user performed the corresponding operation during reading.

[0020] If any sequence length is less than three times, the data collection time window is extended until the current material is finished being read. Because deep learning strategy models require a sufficient number of samples to stably calculate statistical features, if a certain type of behavior occurs too infrequently, the statistical results are easily affected by random factors. Therefore, the data collection time window is extended from the preset sampling period to the moment the user finishes reading the entire current material, during which time behavioral data continues to be collected. The preset sampling period refers to the default time for the system to collect behavioral data after the user begins reading. This sampling period represents the standard collection window when the system does not detect insufficient data. It is set based on the average reading time of graded reading materials and the user's attention span, for example, it can be set to 5 minutes. Within this period, the system collects three types of behavioral data generated by the user: touch dragging, paragraph back navigation, and long-press word lookup. If, at the end of 5 minutes, a certain sequence length is still less than three times, the data collection window is extended until the current material is finished being read.

[0021] Furthermore, if a sequence length is still less than three elements after the reading end time, the insufficiently long sequences are padded to three elements. The padding value is the average value from the target user's historical reading data. For example, if the target user's average touch-drag speed for each of the five materials they have read in the past is 150 pixels per second, this average value is used to pad the missing elements in the sequence. If the target user is using the system for the first time and no historical data is available, the padding value is set to zero. This padding mechanism ensures that each sequence has at least three valid elements when input into subsequent calculation modules, avoiding situations where the state vector cannot be calculated or the calculation results are unreliable due to insufficient data.

[0022] In summary, by collecting the aforementioned sequences of touch drag rate, paragraph back-paste frequency, and word long-press query duration, the implicit understanding state of users can be transformed into calculable quantitative features, providing a data foundation for subsequent difficulty adjustment decisions. Each sequence is arranged chronologically according to the user's actions during reading, preserving complete temporal information and enabling the system to capture dynamic changes in users at different reading stages. The length of the sequence depends on the number of actions performed by the user during the reading process; the longer the sequence, the richer the available behavioral features and the more accurate the characterization of the user's understanding state.

[0023] S20: Calculate the reading fluency deviation based on the touch drag rate sequence and the material standard page turning rate, and calculate the instant comprehension confidence based on the duration sequence of the long press query of the word and the average reading time of the paragraph; Secondly, after obtaining the touch drag speed sequence, it is necessary to quantify the fluency of user reading. English materials of different difficulty levels have different standard page-turning speed ranges. Materials of lower difficulty have shorter texts and simpler vocabulary, resulting in faster user reading speeds and higher upper and lower limits for standard page-turning speeds. Materials of higher difficulty have more unfamiliar words and more complex sentence structures, leading to slower user reading speeds and lower upper and lower limits for standard page-turning speeds. The user's actual average drag speed is compared to the standard page-turning speed range corresponding to the current material's difficulty level: if the actual speed is below the lower limit, it indicates that the user's page-turning speed is too slow and there is reading difficulty; the fluency deviation is negative, with a larger deviation indicating greater difficulty. If the actual speed is above the upper limit, it indicates that the user's page-turning speed is too fast and the material is too simple; the fluency deviation is positive, with a larger deviation indicating easier material. If the actual speed is between the upper and lower limits, it indicates normal reading fluency, with zero fluency deviation.

[0024] Simultaneously, after obtaining the duration sequence of long-press queries for words, it is necessary to quantify the user's confidence in comprehension during the reading process. The average reading time per paragraph is calculated based on the total number of words in the paragraph and the user's baseline reading speed. If the user's sustained press time when performing a word query is significantly longer than the time required to normally read a paragraph, it indicates that the user encountered a significant comprehension obstacle at that word, and the confidence in immediate comprehension is low. If the query duration is close to the average reading time per paragraph, it indicates that the user can understand the word meaning with only a short pause, and the confidence in immediate comprehension is high. This confidence in immediate comprehension represents the user's grasp of vocabulary and sentence comprehension in the current reading material, and is used to characterize the cognitive load when facing unfamiliar words and the speed at which the user recovers reading fluency from the query operation.

[0025] Through the two calculation steps described above, the original behavioral sequence data can be transformed into two standardized quantitative indicators: reading fluency deviation, which reflects the user's adaptation to the difficulty of the current material in the reading speed dimension, and instant comprehension confidence, which reflects the user's mastery of the vocabulary and syntax of the current material in the comprehension depth dimension. This enables the subsequent reinforcement learning agent to perceive the user's current real reading state and provide an accurate basis for difficulty adjustment.

[0026] Specifically, based on the touch drag rate sequence and the standard page-turning rate, a reading fluency deviation is calculated, and based on the duration sequence of long-press queries for words and the average reading time of paragraphs, an instant comprehension confidence score is calculated, including: Calculate the average drag rate based on the touch drag rate sequence; Read the upper and lower limits of the standard page-turning rate corresponding to the current difficulty level from the metadata of the current material; The metadata refers to the attribute description information attached to each English material in the graded reading corpus. The attribute description information includes at least the material difficulty level, the upper limit of the standard page-turning rate, the lower limit of the standard page-turning rate, and the baseline reward value. The average dragging rate is compared with the lower limit of the standard page-turning rate and the upper limit of the standard page-turning rate. If the average drag rate is lower than the lower limit of the standard page-turning rate, then the reading smoothness deviation is determined based on the degree of deviation between the lower limit of the standard page-turning rate and the average drag rate; If the average drag rate is higher than the upper limit of the standard page-turning rate, then the reading smoothness deviation is determined based on the degree of deviation between the average drag rate and the upper limit of the standard page-turning rate; If the average dragging speed is between the lower limit of the standard page-turning speed and the upper limit of the standard page-turning speed, then the reading smoothness deviation is set to zero; Calculate the average query time based on the duration sequence of the long-press queries for the aforementioned words; Obtain the total number of words in the current paragraph, and calculate the average reading time of the paragraph based on the target user's baseline reading speed; Based on the average query time and the average reading time of the paragraph, the time ratio is calculated, the time ratio is subjected to arctangent transformation, and the instant understanding confidence is calculated based on the transformation result.

[0027] First, based on the previously obtained touch drag rate sequence, the average drag rate is calculated. The average drag rate is equal to the arithmetic mean of all instantaneous drag rates in the touch drag rate sequence. For example, suppose the touch drag rate sequence contains N instantaneous drag rate values; the average drag rate is the sum of these values ​​divided by N, expressed in pixels per second. This average drag rate reflects the average speed at which the user turns pages or swipes while reading the current material.

[0028] Secondly, the upper and lower limits of the standard page-turning speed, corresponding to the current difficulty level, are retrieved from the metadata of the current material. The metadata consists of attribute descriptions accompanying each English material in the graded reading corpus, including at least the material's difficulty level, the upper and lower limits of the standard page-turning speed, and a baseline reward value. The upper and lower limits of the standard page-turning speed are obtained through statistical analysis of the dragging speed distribution of historical users on materials at this difficulty level. For example, the 70th percentile of the average dragging speed of historical users is used as the lower limit, and the 130th percentile of the average dragging speed is used as the upper limit. Specifically, the lower limit of the standard page-turning speed represents the minimum acceptable page-turning speed for normal reading of materials at this difficulty level among historical users; a speed below this indicates that the current user has reading difficulties. The upper limit of the standard page-turning speed represents the maximum acceptable page-turning speed for normal reading of materials at this difficulty level among historical users; a speed above this indicates that the current user is reading too easily, and the material may be too simple. These two thresholds together constitute a reasonable range for the user's page-turning speed, used to determine the current user's reading fluency relative to the difficulty level of the material.

[0029] Then, the average dragging speed is compared with the lower and upper limits of the standard page-turning speed. If the average dragging speed is lower than the lower limit, it indicates that the user's page-turning speed is below normal, resulting in reading difficulties. In this case, the reading fluency deviation is equal to the difference between the lower limit and the average dragging speed, divided by the lower limit. In other words, the reading fluency deviation is the difference between the lower limit and the average dragging speed divided by the lower limit. This reading fluency deviation value is positive, with a maximum of 1; the larger the value, the more severe the reading difficulties.

[0030] If the average dragging speed exceeds the upper limit of the standard page-turning speed, it indicates that the user's page-turning speed is higher than normal, and the material is too simple. In this case, the reading fluency deviation is equal to the difference between the average dragging speed and the upper limit of the standard page-turning speed, divided by the upper limit. This reading fluency deviation is a positive number; the larger the value, the simpler the material. Furthermore, if the average dragging speed is between the lower and upper limits of the standard page-turning speed, it indicates that the user's reading fluency is normal, and the reading fluency deviation is set to zero.

[0031] The resulting reading fluency deviation represents the degree of deviation between the user's current page-turning speed and the expected speed of the material. When the average dragging speed is lower than the lower limit of the standard page-turning speed, the reading fluency deviation is positive, and the larger the value, the more difficult the reading is for the user and the more difficult the material is. When the average dragging speed is higher than the upper limit of the standard page-turning speed, the reading fluency deviation is positive, and the larger the value, the easier the reading is for the user and the less difficult the material is. This reading fluency deviation is used in the state vector to reflect the user's adaptation to the current material in the reading speed dimension.

[0032] Furthermore, based on the duration sequence of long-press queries for words, the average query duration is calculated. This average query duration is equal to the arithmetic mean of the durations of all individual queries in the sequence. For example, suppose the duration sequence of long-press queries for words contains M duration values; the average query duration is the sum of these values ​​divided by M, in seconds. This average query duration reflects the average length of time a user pauses to query due to unfamiliar words during reading.

[0033] Next, the total number of words in the current paragraph is obtained, and combined with the target user's baseline reading speed, the average reading time of the paragraph is calculated. The baseline reading speed is taken from the target user's historical statistics, representing the number of words the user can read per unit of time. Assuming the baseline reading speed is in words per second and the total number of words in the paragraph is W, then the average reading time of the paragraph is equal to W divided by the baseline reading speed, in seconds. In another embodiment, if the baseline reading speed is in words per minute, the baseline reading speed is first divided by 60 to convert it to words per second, and then the division operation is performed to ensure that the calculated result is consistent with the unit of the average query time.

[0034] Furthermore, based on the average query time and the average paragraph reading time, the time ratio is calculated, an arctangent transformation is applied to the time ratio, and the instant understanding confidence is calculated based on the transformation result. Specifically, the time ratio equals the average query time divided by the average paragraph reading time, and its value ranges from 0 to positive infinity. When the average query time is much less than the average paragraph reading time, the time ratio approaches 0; when the average query time is much greater than the average paragraph reading time, the time ratio approaches positive infinity. The arctangent transformation maps the time ratio to the interval between 0 and π / 2. Specifically, the instant understanding confidence is calculated as follows: confidence equals 1 minus 2 / π multiplied by the arctangent transformation result. That is, confidence = 1 - (2 / π) × (arctan time ratio).

[0035] When the duration ratio is 0, the arctangent result is 0, and the confidence level is 1. When the duration ratio approaches positive infinity, the arctangent result approaches π / 2, and the confidence level approaches 0. Through this transformation, the larger the duration ratio, the lower the confidence level, and the two show a monotonically decreasing relationship, with the confidence level always remaining within the open interval of 0 to 1. The instant understanding confidence level reflects the user's implicit understanding behind their word search behavior; a higher confidence level indicates a better grasp of vocabulary, while a lower confidence level indicates a more severe vocabulary barrier.

[0036] S30: The three-element state vector is constructed by the difficulty level of the current material, the reading fluency deviation, and the instant understanding confidence. It is input into the pre-trained deep Q-network policy model and outputs a difficulty adjustment action, wherein the difficulty adjustment action includes increasing the difficulty level, decreasing the difficulty level, or maintaining the difficulty level. Furthermore, after calculating the reading fluency deviation and instant comprehension confidence, these two indicators need to be integrated with the difficulty level of the current material to form a complete vector describing the user's current reading state. The difficulty level is a discrete value, for example, ranging from 1 to 10, with higher values ​​indicating more difficult material. The reading fluency deviation is a continuous value; its sign and magnitude reflect the direction and degree of deviation of the user's page-turning speed from the expected material. The instant comprehension confidence is a continuous value between 0 and 1, reflecting the user's understanding and grasp of vocabulary and sentences. These three values ​​are concatenated in a fixed order to form a three-element state vector, such as [Difficulty Level, Reading Fluency Deviation, Instant Comprehension Confidence]. This three-element state vector simultaneously includes the objective difficulty of the material, the user's performance in the speed dimension, and their performance in the comprehension dimension, comprehensively characterizing the user's reading state on the current material.

[0037] Next, the three-element state vector is input into a pre-trained deep Q-network policy model. The deep Q-network policy model is a deep neural network with three input layer nodes, corresponding to the three dimensions of the three-element state vector; and three output layer nodes, corresponding to the Q-values ​​of the three actions: difficulty increase, difficulty decrease, and difficulty maintenance. The Q-value represents the expected cumulative reward after performing an action in the current state. During training, the model has learned the mapping relationship from the state vector to the Q-values ​​of each action by studying a large amount of historical user reading behavior data. During the model's forward computation, the input state vector undergoes a nonlinear transformation through multiple hidden layers, outputting the Q-values ​​of the three actions.

[0038] Then, the action with the highest Q-value is selected as the difficulty adjustment action. If the Q-value of the difficulty increase action is the highest, it means that the model judges that the current state is suitable for increasing the difficulty of the materials, and the system will push materials of a higher difficulty level to the user; if the Q-value of the difficulty decrease action is the highest, it means that the model judges that the current materials are too difficult and needs to be reduced to maintain the user's learning motivation; if the Q-value of the difficulty maintenance action is the highest, it means that the model judges that the current difficulty matches the user's ability, and continues to push materials of the same level.

[0039] Specifically, the core of this step lies in introducing reinforcement learning into the problem of adaptive material difficulty adjustment, enabling the system to automatically select the optimal difficulty adjustment strategy based on the current state. The deep Q-network policy model, through learning from a large amount of historical data, can capture the complex nonlinear relationship between reading fluency deviations, instant understanding confidence, and difficulty adjustment actions, outputting actions that better match the user's actual needs, thus achieving a dynamic and precise match between material difficulty and the user's ability.

[0040] The training steps of the deep Q-network policy model include: Collect reading behavior data generated by multiple historical users during graded reading practice, and extract a three-element state vector at consecutive time points from each reading session. The three-element state vector consists of the difficulty level of the current material, the reading fluency deviation, and the instant comprehension confidence. Record the actual difficulty adjustment actions performed under each state vector, and calculate the corresponding actual reward value after the user completes the subsequent reading comprehension questions; Capture the next state vector after the action is executed, and combine the state vector, the difficulty adjustment action, the actual reward value, and the next state vector into a quadruple sample in chronological order; Collect all quadruple samples to construct a training sample set; Construct a deep Q-network, wherein the input layer of the deep Q-network has three nodes, which correspond to the three dimensions of the three-element state vector, and the output layer has three nodes, which correspond to the Q values ​​of the three actions of difficulty increase, difficulty decrease, and difficulty maintenance, respectively. The hidden layer adopts a fully connected structure and uses a non-linear activation function. A small batch of quadruplet samples are randomly selected from the training sample set. For each quadruplet sample, the state vector is input into the current network to obtain the current Q value, and the next state vector is input into the target network to obtain the target Q value. The target value is calculated based on the Bellman equation, and the mean square error between the current Q value and the target value is used as the loss function. The network weights of the deep Q-network are updated using the backpropagation algorithm, and the weights of the deep Q-network are synchronized to the target network every fixed number of update rounds. Repeat the process of extraction, loss calculation, and weight update until the average value of the loss function on the validation sample set no longer decreases for multiple consecutive rounds. The trained deep Q network is then used as the deep Q network policy model.

[0041] First, reading behavior data from multiple historical users during graded reading exercises was collected, and a three-element state vector was extracted from each reading session at consecutive time points. The three-element state vector consists of the difficulty level of the current material, reading fluency deviation, and immediate comprehension confidence. Each time point corresponds to a difficulty adjustment decision moment, such as after a user finishes reading a material or before the next push notification.

[0042] Simultaneously, the actual difficulty adjustment actions performed under each state vector are recorded, and the corresponding actual reward value is calculated after the user completes the subsequent reading comprehension questions. Specifically, the difficulty adjustment actions include three types: difficulty increase, difficulty decrease, and difficulty maintenance. The calculation method for the actual reward value will be explained in detail in subsequent steps; this actual reward value is used to reflect the quality of the difficulty adjustment action.

[0043] Next, the next state vector after the action is executed is captured. The next state vector refers to the three-element state vector recalculated after the user finishes reading the newly pushed material, based on the difficulty level of the new material, the user's reading fluency deviation, and immediate comprehension confidence. The state vector, difficulty adjustment action, actual reward value, and next state vector are combined into a four-tuple sample in chronological order. The four elements are: state vector S, action A, reward value R, and next state vector S'. This four-tuple records the complete process of the user's state transition and is the basic unit for training the deep Q-network. All four-tuple samples are collected to construct a training sample set. Specifically, a large number of four-tuple samples are extracted from multiple reading sessions of multiple historical users to form a sufficiently large training dataset.

[0044] Furthermore, a deep Q-network is constructed. The input layer of the deep Q-network has three nodes, corresponding to the three dimensions of the three-element state vector: difficulty level, reading fluency bias, and immediate comprehension confidence. The output layer also has three nodes, corresponding to the Q-values ​​of the three actions: difficulty increase, difficulty decrease, and difficulty maintenance. The hidden layers employ a fully connected structure; for example, two hidden layers can be used: the first layer contains 128 neurons, and the second layer contains 64 neurons. Each neuron uses the ReLU function as a non-linear activation function, introducing non-linear transformation capabilities. The deep Q-network constructs two identical network replicas: the current network and the target network. The current network is used to select actions and calculate the current Q-value, while the target network is used to calculate the target Q-value.

[0045] Next, a small batch of quadruplet samples is randomly selected from the previously constructed training sample set. For each quadruplet sample, the state vector is input into the current network, which outputs the Q-values ​​of the three actions. The Q-value corresponding to the actual action is extracted from these samples and used as the current Q-value. The next state vector is input into the target network, which outputs the Q-values ​​of the three actions in the next state. The maximum value is then selected. The target value is calculated based on the Bellman equation. The Bellman equation is a core recursive relation in reinforcement learning, expressed as: the target value equals the actual reward value plus a discount factor multiplied by the maximum Q-value of the next state. The discount factor ranges from 0 to 1 and is used to balance the importance of immediate rewards and future rewards. The mean squared error between the current Q-value and the target value is used as the loss function. The mean squared error is calculated as: the square of the difference between the current Q-value and the target value, and then the average value is taken.

[0046] For example, in a feasible embodiment, the hyperparameter settings during the training process of the deep Q-network are as follows: the discount factor γ is set to 0.95 to balance the weights of immediate and future rewards; the learning rate is set to 0.001 to control the step size of each gradient update; the batch size is set to 32, that is, 32 quadruplet samples are randomly selected for each mini-batch update; the experience replay pool capacity is set to 10000 to store historical quadruplet samples; the target network synchronization frequency is set to 100, that is, the weights of the current network are copied to the target network after every 100 updates of the current network; the maximum number of training epochs is set to 500; and the early stopping patience value is set to 20, that is, training is terminated early if the loss on the validation set does not decrease after 20 consecutive epochs. With the above architecture and parameter configuration, the deep Q-network can stably converge to a better strategy.

[0047] The network weights of the deep Q-network are updated using the backpropagation algorithm. Backpropagation calculates the gradient of the loss function with respect to the weights of each layer, working from the output layer to the input layer, and adjusts the weight values ​​along the gradient descent direction to reduce the loss function. Every fixed number of update iterations, such as every 100 updates, the weights of the current network are completely copied to the target network, keeping the parameters of the target network synchronized with the current network. This synchronization mechanism stabilizes the training process and prevents the target value from oscillating drastically due to frequent changes in network parameters.

[0048] Finally, the process of extracting, calculating the loss, and updating the weights is repeated until the average value of the loss function on the validation set no longer decreases for multiple consecutive rounds. Specifically, the validation set is an independent subset partitioned from the training set; it does not participate in network weight updates and is only used to evaluate the model's generalization performance. For example, when the average loss on the validation set does not decrease for 20 consecutive training rounds, the model is considered to have converged, and training is stopped. At this point, the trained Deep Q-Network is used as the Deep Q-Network policy model for subsequent actual push decisions.

[0049] Through the training process described above, the deep Q-network learns the mapping relationship from the three-element state vector to the adjustment actions of various difficulties, and can select the action with the highest expected cumulative reward in a given state.

[0050] In practical use, a three-element state vector is constructed using the current material's difficulty level, reading fluency deviation, and instant comprehension confidence. This vector is then input into a trained deep Q-network policy model to obtain difficulty adjustment actions.

[0051] S40: After the target user completes the subsequent reading comprehension questions of the current material, calculate the reward correction value based on the number of correct questions and the reading fluency deviation, and combine it with the baseline reward value to obtain the actual reward value; Furthermore, after the target user completes reading the current material, the system typically presents several reading comprehension questions to assess the user's understanding of the content. The accuracy rate of these questions directly reflects the quality of the user's understanding and is an important basis for evaluating the effectiveness of difficulty adjustment. However, calculating rewards solely based on accuracy has limitations: users may barely answer the questions correctly after numerous queries and rollbacks, but the reading experience is poor, and the actual difficulty of the material remains too high. Therefore, this step combines the number of correct reading comprehension questions with reading fluency deviation to jointly calculate the reward correction value, ensuring that the reward signal simultaneously reflects both objective comprehension effectiveness and subjective reading experience.

[0052] Specifically, after the target user completes several subsequent reading comprehension questions of the current material, a reward correction value is calculated based on the number of correct answers and the reading fluency deviation. This correction value is then combined with a baseline reward value to obtain the actual reward value, including: The number of correct answers is calculated by counting all reading comprehension questions answered correctly by the target user after completing the current material. The basic accuracy rate is determined based on the proportion of correct answers to the total number of reading comprehension questions. The reading fluency deviation is obtained, the fluency correction factor is calculated, and the reward correction value is obtained by combining it with the basic accuracy rate; The baseline reward value is read from the metadata of the current material, wherein the baseline reward value is the average level of the reward adjustment values ​​obtained by all users in the historical push of the current material; The actual reward value is calculated based on the reward correction value and the baseline reward value.

[0053] First, the system counts the number of correct answers among all reading comprehension questions answered correctly by the target user after completing the current material. For example, the reading comprehension section typically consists of 5 to 10 questions, including multiple choice, true / false, or fill-in-the-blank questions. An automated grading system then determines the correctness of each question and accumulates the correct answer count.

[0054] Secondly, the basic accuracy rate is determined based on the proportion of correct answers to the total number of reading comprehension questions. For example, if the total number of reading comprehension questions is T and the number of correct answers is C, then the basic accuracy rate equals C divided by T, with a value ranging from 0 to 1. The basic accuracy rate reflects the user's level of understanding of the material; a higher accuracy rate indicates a more thorough understanding of the material.

[0055] Next, the reading fluency deviation is obtained, a fluency correction factor is calculated, and a reward correction value is obtained by combining it with the baseline accuracy. The reading fluency deviation, calculated in the previous steps, reflects the degree of deviation between the user's page-turning speed and the expected speed of the material. A large reading fluency deviation indicates significant unfluency during reading, suggesting that the material difficulty may not be suitable for the user, even if the baseline accuracy is high. Therefore, a fluency correction factor is needed to attenuate the reward. Specifically, the fluency correction factor is calculated as follows: the fluency correction factor equals 1 divided by the sum of the reading fluency deviation and a positive decimal. The positive decimal is usually taken as 0.1 to avoid a denominator of zero. When the reading fluency deviation is large, the fluency correction factor is small, resulting in a stronger attenuation effect. The reward correction value equals the baseline accuracy multiplied by the fluency correction factor, and its value is truncated to be between 0 and 1. This reward correction value considers both the user's accuracy in understanding the content and the user's smooth reading experience; neither is dispensable.

[0056] Next, the baseline reward value is read from the metadata of the current material. The baseline reward value is the average level of the reward correction values ​​obtained by all users in the historical pushes of the current material, and is stored in the material's metadata. This baseline reward value represents the typical performance of the material among ordinary users and can be used as a reference for comparing the performance of the current user.

[0057] Finally, the actual reward value is calculated based on the reward adjustment value and the baseline reward value. The actual reward value equals the reward adjustment value multiplied by the baseline reward value. The physical meaning of this product is: the baseline reward value, as the average performance score of this material among historical users, is multiplied by the current user's reward adjustment value to obtain the current user's weighted performance on this material. This calculation method keeps the reward value within a range close to the reward adjustment value, facilitating stable updates of the deep Q-network across different materials and users.

[0058] S50: The three-element state vector, the difficulty adjustment action, the actual reward value, and the next state vector are combined into an experience tuple, stored in a circular experience queue, and every ten experience tuples a priority sampling of small batches of samples is triggered to update the parameters of the deep Q network strategy model. Furthermore, after obtaining the actual reward value and determining the next state vector, the entire process of this difficulty adjustment decision needs to be recorded as an experience sample for subsequent model parameter updates.

[0059] Specifically, the three-element state vector, the difficulty adjustment action, the actual reward value, and the next state vector are combined into an experience tuple, stored in a circular experience queue, and every ten experience tuples a priority sampling of a small batch of samples is triggered to update the parameters of the deep Q-network policy model, including: The three-element state vector, the difficulty adjustment action, the actual reward value, and the next state vector are combined into an experience tuple in chronological order. Calculate a priority value for each empirical tuple; The experience tuples are stored in a fixed-capacity circular experience queue. When the queue is full, the newly stored experience tuples overwrite the existing experience tuples with the lowest priority value in the queue. Whenever the cumulative number of newly stored experience tuples reaches ten, a priority sampling process is triggered, and a small batch of samples is drawn. For each empirical tuple in the mini-batch samples, the deviation between the Q value output by the current network and the target value is calculated. All deviations are summed to obtain the total loss. The network weights of the deep Q-network policy model are updated through the backpropagation algorithm to reduce the total loss. After completing the network weight update, the accumulated count of newly stored experience tuples will be reset to zero, and sampling and updating will be triggered again when the count reaches ten again.

[0060] First, the three-element state vector, difficulty adjustment action, actual reward value, and next state vector are combined into an experience tuple in chronological order. This experience tuple is a four-tuple that records the complete decision-making process of the user starting from the current state, performing a difficulty adjustment action, receiving a reward, and entering a new state.

[0061] Subsequently, a priority value is calculated for each empirical tuple. The priority value is used to measure the importance of the empirical tuple in subsequent sampling; the higher the value, the higher the learning value of the empirical tuple.

[0062] Specifically, a priority value is calculated for each empirical tuple, including: Obtain the actual reward value from the experience tuple and calculate the absolute value of the reward; Obtain the temporal difference error corresponding to the empirical tuple, wherein the temporal difference error is the difference between the predicted Q value and the target value of the current network for the difficulty adjustment action under the state vector in the empirical tuple; Based on the time-series differential error, the absolute value of the error is calculated, and the absolute value of the reward is combined with the absolute value of the error to obtain the basic priority value. The priority value is obtained by adding a positive decimal to the base priority value.

[0063] First, the actual reward value in the experience tuple is obtained, and the absolute reward value is calculated. The actual reward value is the value calculated in step S40, and the absolute reward value is equal to the absolute value of the actual reward value. The larger the absolute reward value, the greater the reward intensity generated by the difficulty adjustment action, and the higher the learning value of the experience tuple.

[0064] Secondly, the temporal difference error corresponding to the empirical tuple is obtained. Temporal difference error is a metric used in deep Q-networks to measure the accuracy of the current network's predictions. The specific calculation method is as follows: The state vector from the empirical tuple is input into the current network, which outputs the Q-values ​​of three actions. The Q-value corresponding to the actual difficulty adjustment action executed in the empirical tuple is extracted, called the current Q-value. The next state vector is input into the target network, which outputs the Q-values ​​of the three actions in the next state. The maximum value is taken. The target value is calculated according to the Bellman equation, which equals the actual reward value plus a discount factor multiplied by the maximum Q-value of the next state. The temporal difference error equals the current Q-value minus the target value. The absolute value of the temporal difference error reflects the gap between the current network's prediction and the true target. The larger the absolute value of the error, the richer the correction information contained in the empirical tuple, and the more worthy it is to be learned first.

[0065] Furthermore, based on the temporal difference error, the absolute value of the error is calculated, and the absolute value of the reward is combined with the absolute value of the error to obtain the basic priority value. Specifically, the absolute value of the error is equal to the absolute value of the temporal difference error. The basic priority value can be calculated by multiplying the absolute value of the reward by the absolute value of the error. This product takes into account two factors: a large absolute value of the reward indicates that the event result corresponding to the empirical tuple is significant and worth focusing on; a large absolute value of the error indicates that the current network's prediction of the empirical tuple is very inaccurate, and the sample needs to be used first to correct the network. Multiplying the two gives empirical tuples with both high absolute values ​​of reward and high absolute values ​​of error a higher basic priority.

[0066] Finally, a positive decimal is added to the base priority value to obtain the priority value. This positive decimal is typically 0.01 or 0.001 to ensure that the priority value of each empirical tuple is greater than zero. Without this positive decimal, when the absolute value of the reward or error is zero, the base priority value might be zero, resulting in the empirical tuple having zero priority and never being selected during sampling, thus being wasted. The final priority value is used to calculate the sampling probability distribution in subsequent priority sampling processes; a larger value indicates a higher probability that the empirical tuple will be selected. Specifically, through this priority value calculation mechanism, the experience replay queue can automatically retain and prioritize the use of empirical tuples with high learning value, accelerating the convergence speed of the deep Q-network policy model.

[0067] Furthermore, experience tuples are stored in a fixed-capacity circular experience queue. A circular experience queue is a first-in, first-out (FIFO) storage structure with a fixed maximum length, for example, a capacity of 2000 experience tuples. When the queue is not full, newly added experience tuples are appended to the tail of the queue. When the queue is full, newly added experience tuples overwrite the existing experience tuple with the lowest priority value in the queue. This eviction policy ensures that higher-priority experience tuples are always retained in the queue, while lower-priority historical experience tuples are gradually evicted.

[0068] Specifically, a priority sampling process is triggered whenever the cumulative number of newly stored empirical tuples reaches ten, and a mini-batch of samples is drawn. The reason for setting an update every ten empirical tuples is as follows: if an update is triggered for each newly stored sample, the gradient estimation variance is large, and the update direction is unstable; if too many samples are accumulated before an update, the model cannot respond promptly to the latest changes in user capabilities. Ten samples strike a balance between reducing gradient variance and maintaining timely response. Meanwhile, the batch size of the mini-batch is set to 32 for the following reasons: if the batch size is too small, the gradient variance remains large; if the batch size is too large, computational overhead increases and the model may get trapped in local optima. 32 is a commonly used batch size in deep reinforcement learning, striking a balance between stability and efficiency.

[0069] Whenever the cumulative number of newly stored experience tuples reaches ten, a priority sampling process is triggered, and a small batch of samples is drawn, including: Traverse all experience tuples in the circular experience queue, read the priority value corresponding to each experience tuple, and sort all experience tuples in descending order according to the priority value to obtain a sorted list of experience tuples. Calculate the sum of the priority values ​​of the first N empirical tuples in the sorted empirical tuple list, where N is equal to the pre-set batch size, and determine whether the summation result exceeds half of the total sum of the priority values ​​of all empirical tuples. If more than half are selected, the first N empirical tuples in the sorted empirical tuple list are selected as mini-batch samples. If the number of empirical tuples does not exceed half, then an empirical tuple is selected from the sorted list of empirical tuples at fixed intervals. The fixed interval is determined based on the ratio of the total number of empirical tuples in the sorted empirical tuple list to the batch size, and the starting position is the first empirical tuple in the sorted empirical tuple list, until the batch size of empirical tuples is selected to form a small batch sample.

[0070] First, iterate through all experience tuples in the circular experience queue, read the priority value corresponding to each experience tuple, and sort all experience tuples in descending order of priority value, obtaining a sorted list of experience tuples. The first experience tuple in this list has the highest priority, and the last experience tuple has the lowest priority. Next, calculate the sum of the priority values ​​of the first N experience tuples in the sorted list, where N is equal to the pre-set batch size. Specifically, the batch size N is usually set to 32. This value strikes a balance between computational efficiency and gradient stability, avoiding excessive gradient noise caused by too small a batch and preventing update delays caused by too large a batch. It is a widely adopted default configuration in the field of deep reinforcement learning. Finally, determine whether the summation result exceeds half of the total sum of the priority values ​​of all experience tuples.

[0071] If the sum of the priority values ​​of the top N empirical tuples exceeds half the sum of the priority values ​​of all empirical tuples, it indicates that the top N empirical tuples with the highest priority account for most of the total priority weight, and the learning value of these empirical tuples is far higher than that of the other empirical tuples. In this case, the top N empirical tuples in the sorted list are directly selected as mini-batch samples.

[0072] Furthermore, if the sum of the priority values ​​of the first N empirical tuples does not exceed half the sum of the priority values ​​of all empirical tuples, it indicates that the top N empirical tuples with the highest priority have not yet gained an absolute advantage, and simply selecting the top N may lead to insufficient sample diversity. In this case, empirical tuples are selected from the sorted empirical tuple list at fixed intervals. The fixed interval is calculated by dividing the total number of empirical tuples in the sorted empirical tuple list by the batch size and then rounding down. The starting position for selection is the first empirical tuple in the sorted empirical tuple list, i.e., the empirical tuple with the highest priority. Then, subsequent empirical tuples are selected sequentially at fixed intervals until the batch size of empirical tuples is reached. For example, if the total number of empirical tuples is 200 and the batch size is 32, then the fixed interval is 200 divided by 32 and rounded down, which is 6. Empirical tuples with indices 1, 7, 13, 19... are selected until 32 are selected. This method of sampling at fixed intervals can prioritize the selection of high-priority samples while also taking into account medium-priority and low-priority samples, maintaining sampling diversity and avoiding model overfitting to a few extremely high-priority samples.

[0073] Specifically, by combining the two sampling strategies mentioned above, the priority sampling process can ensure that high-learning-value samples are fully utilized while maintaining the diversity of sample distribution, thereby improving the training effect and generalization ability of the deep Q-network strategy model.

[0074] Further, for each empirical tuple in the mini-batch, the deviation between the current network output Q-value and the target value is calculated. Specifically, the state vector in the empirical tuple is input into the current network, which outputs the Q-values ​​for three actions: difficulty increase, difficulty decrease, and difficulty maintenance. The Q-value corresponding to the actual executed action is extracted and recorded as the current Q-value. The next state vector is input into the target network, which outputs the Q-values ​​for the three actions in the next state. The maximum value among them is recorded as the maximum Q-value of the next state. The target value is calculated according to the Bellman equation, which equals the actual reward value plus a discount factor multiplied by the maximum Q-value of the next state. The difference between the current Q-value and the target value is calculated, and the square of this difference is taken as the deviation of the empirical tuple. The deviations of all empirical tuples in the mini-batch are summed and divided by the number of mini-batch samples to obtain the mean squared error loss. The gradient of the loss function with respect to the weights of each layer of the deep Q-network policy model is calculated using the backpropagation algorithm. The network weights are updated along the gradient descent direction to reduce the mean squared error loss. The above process is repeated until the model converges.

[0075] Finally, after updating the network weights, the accumulated count of newly stored experience tuples is reset to zero, waiting for the next accumulation of ten to trigger sampling and updating again. Through this mechanism, the system performs a batch update every ten new experience tuples, ensuring that the model can learn from new experiences in a timely manner while avoiding the high variance and instability problems caused by single-sample updates. Simultaneously, the priority sampling mechanism allows the model to focus more on experience tuples with high learning value, accelerating the training convergence process. The fixed-capacity design of the circular experience queue limits memory usage and avoids the computational resource expansion caused by the infinite accumulation of historical samples.

[0076] In summary, through this online learning mechanism, the deep Q-network policy model can continuously optimize its difficulty adjustment decision-making ability in the continuous push service, and adaptively adapt to changes in user capabilities.

[0077] S60: Based on the aforementioned difficulty adjustment action, select new materials of the corresponding difficulty level from the graded reading corpus and push them to the target user's terminal.

[0078] Finally, after the deep Q-network strategy model outputs the difficulty adjustment action, the system needs to determine the new material difficulty level based on the action, and then select materials of the corresponding difficulty level from the graded reading corpus and push them to the user.

[0079] A graded reading corpus is a pre-organized database of English reading materials according to difficulty levels, with each difficulty level corresponding to a batch of materials. Difficulty levels are typically set to multiple levels, such as 1 to 10, with level 1 being the easiest and level 10 the most difficult. Each piece of material is labeled with its difficulty level upon being added to the database, along with metadata information, including the upper limit of the standard page-turning rate, the lower limit of the standard page-turning rate, and a baseline reward value.

[0080] The difficulty adjustment action includes three scenarios. Specifically, if the model outputs a difficulty increase action, the system increments the difficulty level of the current material by one, resulting in a new difficulty level; if the model outputs a difficulty decrease action, the system decrements the difficulty level of the current material by one, resulting in a new difficulty level; if the model outputs a difficulty hold action, the system maintains the current material's difficulty level but replaces it with another new material of the same level to avoid the user repeatedly reading the same material and developing a memory effect. After determining the new difficulty level, the system randomly selects a material of that level from the graded reading corpus, or prioritizes recommending materials that the user has not yet read based on their reading history.

[0081] This push notification is completed through the push interface of the terminal application. Specifically, the system sends the selected new material, accompanying reading comprehension questions, and the material's metadata to the target user's terminal. Once the user receives the material and begins reading, the system simultaneously initiates behavioral data collection and monitoring, recording the user's touch and drag events, paragraph back events, and long-press word lookup events on the new material, preparing for the next round of difficulty adjustment decisions.

[0082] In addition, after selecting new materials of corresponding difficulty levels from the graded reading corpus according to the aforementioned difficulty adjustment action and pushing them to the target user's terminal, the process also includes: Record the sequence of touch drag rate, paragraph back number, and word long press query duration generated by the target user during the reading of new materials; The baseline reading speed of the target user is updated based on the difficulty level of the new material and the target user's reading fluency deviation and instant comprehension confidence level on the new material. The benchmark reading speed is updated in the following way: The average reading time of the target users on all historical materials is weighted and averaged with the average reading time on the new materials. Obtain the real-time average level of the reward correction value obtained by all users after the actual push of the new material, and perform a weighted average of the real-time average level and the benchmark reward value stored in the new material's metadata to obtain the updated benchmark reward value, and write it back to the new material's metadata; When the deep Q-network strategy model outputs a difficulty-maintaining action in multiple consecutive pushes, and the actual reward value after each push is higher than the current material benchmark reward value, the exploration rate of the deep Q-network strategy model will be reduced by a preset ratio.

[0083] First, the system records the sequence of touch dragging rate, paragraph backtracking frequency, and word long-press query duration generated by the target user during the reading of new materials. The system then restarts behavioral data collection and monitoring while the user is reading new materials, recording these three behavioral sequences in the same manner as in step S10. These sequences are used to calculate the user's reading fluency deviation and immediate comprehension confidence level on the new materials.

[0084] Secondly, the baseline reading speed for the target users is updated based on the difficulty level of the new material and their reading fluency deviation and immediate comprehension confidence. The baseline reading speed represents the number of words a user can read per unit of time and is a crucial parameter for calculating the average reading time of a paragraph. As a user's reading ability improves, the baseline reading speed should gradually increase; conversely, if a user consistently performs poorly on low-difficulty materials, the baseline reading speed should be appropriately lowered. Specifically, the update method involves a weighted average of the target user's average reading time across all historical materials and the average reading time on the current new material. The average reading time of historical materials reflects the user's long-term reading ability, while the average reading time of the current new material reflects the user's current performance. This weighted averaging maintains the stability of the baseline reading speed while allowing it to gradually adjust to changes in user ability. The weighting coefficients can be dynamically set based on the number of materials; for example, a higher weight can be assigned when there is a large number of historical materials.

[0085] Next, the real-time average level of the reward correction values ​​obtained by all users after the actual push of the new material is obtained. This real-time average level is then weighted and averaged with the baseline reward value stored in the new material's metadata to obtain the updated baseline reward value, which is then written back to the new material's metadata. The updated baseline reward value equals the update weight multiplied by the real-time average level, plus one minus the update weight multiplied by the original baseline reward value. The update weight typically ranges from 0 to 1, for example, 0.3, to control the impact of new observations on the update result. The real-time average level refers to the arithmetic mean of the reward correction values ​​obtained by all users after the actual push of the current material; the original baseline reward value is stored in the material's metadata. Through this weighted averaging, the baseline reward value of the new material can be continuously corrected based on user feedback, making it increasingly accurate in reflecting the true difficulty level of the material and providing a more reliable reference benchmark for subsequent user reward calculations.

[0086] Furthermore, when the Deep Q-Network strategy model consistently outputs actions with maintained difficulty in multiple consecutive pushes, and the actual reward value after each push is higher than the current baseline reward value, the exploration rate of the Deep Q-Network strategy model is reduced by a preset percentage. For example, the initial exploration rate of the Deep Q-Network strategy model is set to 0.2, meaning that there is a 20% probability of randomly selecting an action instead of choosing the network's output action during decision-making, used to explore the unknown action space in the early stages of training. When the model maintains the same difficulty in multiple consecutive pushes and the actual reward value is higher than the baseline value, it indicates that the model has stably found a suitable difficulty level for the current user. Reducing the exploration rate at this point reduces random attempts, allowing the model to focus more on utilizing the learned strategy and improving the stability of pushes. The reduction method is, for example, by halving the preset percentage, i.e., updating the exploration rate to the current exploration rate multiplied by 0.5. However, the exploration rate has a lower limit, for example, a minimum reduction to 0.01, to prevent the exploration from completely disappearing and causing the model to lose its ability to adapt to new situations.

[0087] In summary, through the aforementioned multi-layered update mechanism, the system can update the baseline reading speed at the individual user level, update the baseline reward value at the material level, and dynamically adjust the exploration rate at the model level, forming a complete adaptive closed loop that enables the graded reading recommendation system to be continuously optimized and operate stably in the long term.

[0088] In summary, the embodiments of this application have at least the following technical effects: This invention first collects multi-dimensional behavioral data, such as touch dragging rate, paragraph backtracking frequency, and word long-press query duration, to quantitatively characterize the user's comprehension state during reading from three dimensions: page-turning smoothness, rereading frequency, and vocabulary query, solving the problem that traditional methods cannot perceive real-time comprehension status. Second, it calculates a state vector based on reading fluency deviation and instant comprehension confidence, enabling the system to accurately determine the difficulty level of the current material relative to the user's ability, providing a scientific basis for difficulty adjustment. Third, it employs a deep Q-network strategy model, using a three-element state vector composed of difficulty level, fluency deviation, and confidence, outputting actions to increase, decrease, or maintain difficulty, achieving dynamic matching between material difficulty and user ability. Furthermore, it calculates a reward correction value based on reading comprehension question accuracy and reading fluency deviation, combining this with a baseline reward value to obtain the actual reward value, ensuring the reward function reflects both objective comprehension performance and subjective reading experience. Finally, it uses a circular experience queue combined with a priority sampling mechanism to update model parameters, prioritizing the retention of high-value experiences, triggering a batch update every ten experience tuples, ensuring model convergence speed while reducing computational overhead. This invention solves the long-standing problems of mismatch between material difficulty and users' actual skill level, inability to capture dynamic changes in ability, and underutilization of reading behavior feedback, thereby improving learning efficiency and engagement.

[0089] Example 2, as Figure 3As shown, based on the same inventive concept as the reinforcement learning-based English reading material difficulty adaptive recommendation method provided in Embodiment 1, this embodiment of the invention also provides a reinforcement learning-based English reading material difficulty adaptive recommendation system, including: The behavior collection module 11 is used to acquire the touch drag rate sequence, paragraph back number sequence, and word long press query duration sequence generated by the target user during the reading of the current English material; The deviation calculation module 12 is used to calculate the reading fluency deviation based on the touch drag rate sequence and the material standard page turning rate, and to calculate the instant comprehension confidence based on the duration sequence of the long press query of the word and the average reading time of the paragraph. The action output module 13 is used to input a three-element state vector consisting of the difficulty level of the current material, the reading fluency deviation, and the instant understanding confidence into a pre-trained deep Q-network strategy model, and output a difficulty adjustment action, wherein the difficulty adjustment action includes increasing the difficulty level, decreasing the difficulty level, or maintaining the difficulty level. The reward calculation module 14 is used to calculate a reward correction value based on the number of correct answers and the reading fluency deviation after the target user completes the subsequent reading comprehension questions of the current material, and to obtain the actual reward value by combining the base reward value. The model update module 15 is used to form an experience tuple from the three-element state vector, the difficulty adjustment action, the actual reward value and the next state vector, store it in the circular experience queue, and trigger a priority sampling of small batch samples every ten experience tuples to update the parameters of the deep Q network strategy model. The material push module 16 is used to select new materials of corresponding difficulty level from the graded reading corpus according to the difficulty adjustment action and push them to the target user terminal.

[0090] The behavior acquisition module 11 is specifically used for: Specifically, the system acquires sequences of touch dragging rates, paragraph back navigation times, and word long-press lookup durations generated by the target user while reading the current English material, including: Listen to the target user's touch movement events on the reading interface, and record the start and end coordinates of each touch movement, as well as the corresponding start and end times; Calculate the Euclidean distance between the starting coordinates and the ending coordinates, combine it with the time difference between the starting time and the ending time to obtain the instantaneous drag rate, arrange all instantaneous drag rates in chronological order, and generate the touch drag rate sequence. Monitor the target user's scrolling operation on the reading page. When the scrolling direction is from the bottom of the page to the top and the number of text paragraphs scrolled across is at least one, accumulate one paragraph back event. Arrange the paragraph rollback events accumulated during the current reading period in chronological order of occurrence to generate the paragraph rollback count sequence; Monitor the target user's long press operation on the text area, record the start time and end time of the long press, and calculate the duration of a single word long press query; Arrange the durations of all single word long-press queries within the current reading period in the order in which the queries occurred to generate a sequence of the durations of the word long-press queries. The sequence lengths of the touch drag rate sequence, the paragraph back number sequence, and the word long press query duration sequence are calculated respectively. When the length of any sequence is less than three times, extend the acquisition time window until the end of the current material reading; If, after the reading end time, there are still sequences with a length less than three, the sequences with insufficient length will be padded to three. The padding value is the corresponding average value of the target user in historical readings. If there is no historical data, it will be padded to zero.

[0091] Specifically, the deviation calculation module 12 is used for: Specifically, based on the touch drag rate sequence and the standard page-turning rate, a reading fluency deviation is calculated, and based on the duration sequence of long-press queries for words and the average reading time of paragraphs, an instant comprehension confidence score is calculated, including: Calculate the average drag rate based on the touch drag rate sequence; Read the upper and lower limits of the standard page-turning rate corresponding to the current difficulty level from the metadata of the current material; The metadata refers to the attribute description information attached to each English material in the graded reading corpus. The attribute description information includes at least the material difficulty level, the upper limit of the standard page-turning rate, the lower limit of the standard page-turning rate, and the baseline reward value. The average dragging rate is compared with the lower limit of the standard page-turning rate and the upper limit of the standard page-turning rate. If the average drag rate is lower than the lower limit of the standard page-turning rate, then the reading smoothness deviation is determined based on the degree of deviation between the lower limit of the standard page-turning rate and the average drag rate; If the average drag rate is higher than the upper limit of the standard page-turning rate, then the reading smoothness deviation is determined based on the degree of deviation between the average drag rate and the upper limit of the standard page-turning rate; If the average dragging speed is between the lower limit of the standard page-turning speed and the upper limit of the standard page-turning speed, then the reading smoothness deviation is set to zero; Calculate the average query time based on the duration sequence of the long-press queries for the aforementioned words; Obtain the total number of words in the current paragraph, and calculate the average reading time of the paragraph based on the target user's baseline reading speed; Based on the average query time and the average reading time of the paragraph, the time ratio is calculated, the time ratio is subjected to arctangent transformation, and the instant understanding confidence is calculated based on the transformation result.

[0092] The action output module 13 is specifically used for: The training steps of the deep Q-network policy model include: Collect reading behavior data generated by multiple historical users during graded reading practice, and extract a three-element state vector at consecutive time points from each reading session. The three-element state vector consists of the difficulty level of the current material, the reading fluency deviation, and the instant comprehension confidence. Record the actual difficulty adjustment actions performed under each state vector, and calculate the corresponding actual reward value after the user completes the subsequent reading comprehension questions; Capture the next state vector after the action is executed, and combine the state vector, the difficulty adjustment action, the actual reward value, and the next state vector into a quadruple sample in chronological order; Collect all quadruple samples to construct a training sample set; Construct a deep Q-network, wherein the input layer of the deep Q-network has three nodes, which correspond to the three dimensions of the three-element state vector, and the output layer has three nodes, which correspond to the Q values ​​of the three actions of difficulty increase, difficulty decrease, and difficulty maintenance, respectively. The hidden layer adopts a fully connected structure and uses a non-linear activation function. A small batch of quadruplet samples are randomly selected from the training sample set. For each quadruplet sample, the state vector is input into the current network to obtain the current Q value, and the next state vector is input into the target network to obtain the target Q value. The target value is calculated based on the Bellman equation, and the mean square error between the current Q value and the target value is used as the loss function. The network weights of the deep Q-network are updated using the backpropagation algorithm, and the weights of the deep Q-network are synchronized to the target network every fixed number of update rounds. Repeat the process of extraction, loss calculation, and weight update until the average value of the loss function on the validation sample set no longer decreases for multiple consecutive rounds. The trained deep Q network is then used as the deep Q network policy model.

[0093] The reward calculation module 14 is specifically used for: Specifically, after the target user completes several subsequent reading comprehension questions of the current material, a reward correction value is calculated based on the number of correct answers and the reading fluency deviation. This correction value is then combined with a baseline reward value to obtain the actual reward value, including: The number of correct answers is calculated by counting all reading comprehension questions answered correctly by the target user after completing the current material. The basic accuracy rate is determined based on the proportion of correct answers to the total number of reading comprehension questions. The reading fluency deviation is obtained, the fluency correction factor is calculated, and the reward correction value is obtained by combining it with the basic accuracy rate; The baseline reward value is read from the metadata of the current material, wherein the baseline reward value is the average level of the reward adjustment values ​​obtained by all users in the historical push of the current material; The actual reward value is calculated based on the reward correction value and the baseline reward value.

[0094] Specifically, the model update module 15 is used for: Specifically, the three-element state vector, the difficulty adjustment action, the actual reward value, and the next state vector are combined into an experience tuple, stored in a circular experience queue, and every ten experience tuples a priority sampling of a small batch of samples is triggered to update the parameters of the deep Q-network policy model, including: The three-element state vector, the difficulty adjustment action, the actual reward value, and the next state vector are combined into an experience tuple in chronological order. Calculate a priority value for each empirical tuple; The experience tuples are stored in a fixed-capacity circular experience queue. When the queue is full, the newly stored experience tuples overwrite the existing experience tuples with the lowest priority value in the queue. Whenever the cumulative number of newly stored experience tuples reaches ten, a priority sampling process is triggered, and a small batch of samples is drawn. For each empirical tuple in the mini-batch samples, the deviation between the Q value output by the current network and the target value is calculated. All deviations are summed to obtain the total loss. The network weights of the deep Q-network policy model are updated through the backpropagation algorithm to reduce the total loss. After completing the network weight update, the accumulated count of newly stored experience tuples will be reset to zero, and sampling and updating will be triggered again when the count reaches ten again.

[0095] Specifically, a priority value is calculated for each empirical tuple, including: Obtain the actual reward value from the experience tuple and calculate the absolute value of the reward; Obtain the temporal difference error corresponding to the empirical tuple, wherein the temporal difference error is the difference between the predicted Q value and the target value of the current network for the difficulty adjustment action under the state vector in the empirical tuple; Based on the time-series differential error, the absolute value of the error is calculated, and the absolute value of the reward is combined with the absolute value of the error to obtain the basic priority value. The priority value is obtained by adding a positive decimal to the base priority value.

[0096] Specifically, whenever the cumulative number of newly stored experience tuples reaches ten, a priority sampling process is triggered, and a small batch of samples is drawn, including: Traverse all experience tuples in the circular experience queue, read the priority value corresponding to each experience tuple, and sort all experience tuples in descending order according to the priority value to obtain a sorted list of experience tuples. Calculate the sum of the priority values ​​of the first N empirical tuples in the sorted empirical tuple list, where N is equal to the pre-set batch size, and determine whether the summation result exceeds half of the total sum of the priority values ​​of all empirical tuples. If more than half are selected, the first N empirical tuples in the sorted empirical tuple list are selected as mini-batch samples. If the number of empirical tuples does not exceed half, then an empirical tuple is selected from the sorted list of empirical tuples at fixed intervals. The fixed interval is determined based on the ratio of the total number of empirical tuples in the sorted empirical tuple list to the batch size, and the starting position is the first empirical tuple in the sorted empirical tuple list, until the batch size of empirical tuples is selected to form a small batch sample.

[0097] The material delivery module 16 is specifically used for: After selecting new materials of corresponding difficulty levels from the graded reading corpus according to the aforementioned difficulty adjustment action and pushing them to the target user's terminal, the process also includes: Record the sequence of touch drag rate, paragraph back number, and word long press query duration generated by the target user during the reading of new materials; The baseline reading speed of the target user is updated based on the difficulty level of the new material and the target user's reading fluency deviation and instant comprehension confidence level on the new material. The benchmark reading speed is updated in the following way: The average reading time of the target users on all historical materials is weighted and averaged with the average reading time on the new materials. Obtain the real-time average level of the reward correction value obtained by all users after the actual push of the new material, and perform a weighted average of the real-time average level and the benchmark reward value stored in the new material's metadata to obtain the updated benchmark reward value, and write it back to the new material's metadata; When the deep Q-network strategy model outputs a difficulty-maintaining action in multiple consecutive pushes, and the actual reward value after each push is higher than the current material benchmark reward value, the exploration rate of the deep Q-network strategy model will be reduced by a preset ratio.

Claims

1. A method for adaptively recommending English reading materials based on reinforcement learning, characterized in that, The method includes: Obtain the sequence of touch drag rate, paragraph back number, and word long press duration generated by the target user while reading the current English material; Based on the touch drag rate sequence and the standard page turning rate, the reading fluency deviation is calculated, and based on the duration sequence of long-press query of words and the average reading time of paragraphs, the instant comprehension confidence is calculated. The three-element state vector is constructed by the difficulty level of the current material, the reading fluency deviation, and the instant understanding confidence. It is input into a pre-trained deep Q-network policy model and outputs a difficulty adjustment action, which includes increasing the difficulty level, decreasing the difficulty level, or maintaining the difficulty level. After the target user completes the subsequent reading comprehension questions of the current material, a reward correction value is calculated based on the number of correct answers and the reading fluency deviation, and combined with the baseline reward value, the actual reward value is obtained. The three-element state vector, the difficulty adjustment action, the actual reward value, and the next state vector are combined into an experience tuple, which is stored in a circular experience queue. Every ten experience tuples, a priority sampling of a small batch of samples is triggered to update the parameters of the deep Q-network policy model. According to the aforementioned difficulty adjustment action, new materials of corresponding difficulty level are selected from the graded reading corpus and pushed to the target user's terminal.

2. The method for adaptively pushing English reading materials based on reinforcement learning according to claim 1, characterized in that, Obtain the following sequences of touch drag rate, paragraph back navigation count, and word long-press query duration generated by the target user while reading the current English material: Listen to the target user's touch movement events on the reading interface, and record the start and end coordinates of each touch movement, as well as the corresponding start and end times; Calculate the Euclidean distance between the starting coordinates and the ending coordinates, combine it with the time difference between the starting time and the ending time to obtain the instantaneous drag rate, arrange all instantaneous drag rates in chronological order, and generate the touch drag rate sequence. Monitor the target user's scrolling operation on the reading page. When the scrolling direction is from the bottom of the page to the top and the number of text paragraphs scrolled across is at least one, accumulate one paragraph back event. Arrange the paragraph rollback events accumulated during the current reading period in chronological order of occurrence to generate the paragraph rollback count sequence; Monitor the target user's long press operation on the text area, record the start time and end time of the long press, and calculate the duration of a single word long press query; Arrange the durations of all single word long-press queries within the current reading period in the order in which the queries occurred to generate a sequence of the durations of the word long-press queries. The sequence lengths of the touch drag rate sequence, the paragraph back number sequence, and the word long press query duration sequence are calculated respectively. When the length of any sequence is less than three times, extend the acquisition time window until the end of the current material reading; If, after the reading end time, there are still sequences with a length less than three, the sequences with insufficient length will be padded to three. The padding value is the corresponding average value of the target user in historical readings. If there is no historical data, it will be padded to zero.

3. The method for adaptively pushing English reading materials based on reinforcement learning according to claim 1, characterized in that, Based on the touch drag rate sequence and the standard page-turning rate, a reading fluency deviation is calculated, and based on the duration sequence of long-press queries for words and the average reading time of paragraphs, an instant comprehension confidence score is calculated, including: Calculate the average drag rate based on the touch drag rate sequence; Read the upper and lower limits of the standard page-turning rate corresponding to the current difficulty level from the metadata of the current material; The metadata refers to the attribute description information attached to each English material in the graded reading corpus. The attribute description information includes at least the material difficulty level, the upper limit of the standard page-turning rate, the lower limit of the standard page-turning rate, and the baseline reward value. The average dragging rate is compared with the lower limit of the standard page-turning rate and the upper limit of the standard page-turning rate. If the average drag rate is lower than the lower limit of the standard page-turning rate, then the reading smoothness deviation is determined based on the degree of deviation between the lower limit of the standard page-turning rate and the average drag rate. If the average drag rate is higher than the upper limit of the standard page-turning rate, then the reading smoothness deviation is determined based on the degree of deviation between the average drag rate and the upper limit of the standard page-turning rate; If the average dragging speed is between the lower limit of the standard page-turning speed and the upper limit of the standard page-turning speed, then the reading smoothness deviation is set to zero. Calculate the average query time based on the duration sequence of the long-press queries of the words; Obtain the total number of words in the current paragraph, and calculate the average reading time of the paragraph based on the target user's baseline reading speed; Based on the average query time and the average reading time of the paragraph, the time ratio is calculated, the time ratio is subjected to arctangent transformation, and the instant understanding confidence is calculated based on the transformation result.

4. The method for adaptively pushing English reading materials based on reinforcement learning according to claim 1, characterized in that, The training steps for the deep Q-network policy model include: Collect reading behavior data generated by multiple historical users during graded reading practice, and extract a three-element state vector at consecutive time points from each reading session. The three-element state vector consists of the difficulty level of the current material, the reading fluency deviation, and the instant comprehension confidence. Record the actual difficulty adjustment actions performed under each state vector, and calculate the corresponding actual reward value after the user completes the subsequent reading comprehension questions; Capture the next state vector after the action is executed, and combine the state vector, the difficulty adjustment action, the actual reward value, and the next state vector into a quadruple sample in chronological order; Collect all quadruple samples to construct a training sample set; Construct a deep Q-network, wherein the input layer of the deep Q-network has three nodes, which correspond to the three dimensions of the three-element state vector, and the output layer has three nodes, which correspond to the Q values ​​of the three actions of difficulty increase, difficulty decrease, and difficulty maintenance, respectively. The hidden layer adopts a fully connected structure and uses a non-linear activation function. A small batch of quadruplet samples are randomly selected from the training sample set. For each quadruplet sample, the state vector is input into the current network to obtain the current Q value, and the next state vector is input into the target network to obtain the target Q value. The target value is calculated based on the Bellman equation, and the mean square error between the current Q value and the target value is used as the loss function. The network weights of the deep Q-network are updated using the backpropagation algorithm, and the weights of the deep Q-network are synchronized to the target network every fixed number of update rounds. Repeat the process of extraction, loss calculation, and weight update until the average value of the loss function on the validation sample set no longer decreases for multiple consecutive rounds. The trained deep Q network is then used as the deep Q network policy model.

5. The method for adaptively pushing English reading materials based on reinforcement learning according to claim 1, characterized in that, After the target user completes several subsequent reading comprehension questions of the current material, a reward correction value is calculated based on the number of correct answers and the reading fluency deviation. This correction value is then combined with the baseline reward value to obtain the actual reward value, including: The number of correct answers is calculated by counting all reading comprehension questions answered correctly by the target user after completing the current material. The basic accuracy rate is determined based on the proportion of correct answers to the total number of reading comprehension questions. The reading fluency deviation is obtained, the fluency correction factor is calculated, and the reward correction value is obtained by combining it with the basic accuracy rate; The baseline reward value is read from the metadata of the current material, wherein the baseline reward value is the average level of the reward adjustment values ​​obtained by all users in the historical push of the current material; The actual reward value is calculated based on the reward correction value and the baseline reward value.

6. The method for adaptively pushing English reading materials based on reinforcement learning according to claim 1, characterized in that, The three-element state vector, the difficulty adjustment action, the actual reward value, and the next state vector are combined into an experience tuple, which is stored in a circular experience queue. Every ten experience tuples, a priority sampling of a small batch of samples is triggered to update the parameters of the deep Q-network policy model, including: The three-element state vector, the difficulty adjustment action, the actual reward value, and the next state vector are combined into an experience tuple in chronological order. Calculate a priority value for each empirical tuple; The experience tuples are stored in a fixed-capacity circular experience queue. When the queue is full, the newly stored experience tuples overwrite the existing experience tuples with the lowest priority value in the queue. Whenever the cumulative number of newly stored experience tuples reaches ten, a priority sampling process is triggered, and a small batch of samples is drawn. For each empirical tuple in the mini-batch samples, the deviation between the Q value output by the current network and the target value is calculated. All deviations are summed to obtain the total loss. The network weights of the deep Q-network policy model are updated through the backpropagation algorithm to reduce the total loss. After completing the network weight update, the accumulated count of newly stored experience tuples will be reset to zero, and sampling and updating will be triggered again when the count reaches ten again.

7. The method for adaptively pushing English reading materials based on reinforcement learning according to claim 6, characterized in that, Calculate a priority value for each empirical tuple, including: Obtain the actual reward value from the experience tuple and calculate the absolute value of the reward; Obtain the temporal difference error corresponding to the empirical tuple, wherein the temporal difference error is the difference between the predicted Q value and the target value of the current network for the difficulty adjustment action under the state vector in the empirical tuple; Based on the time-series differential error, the absolute value of the error is calculated, and the absolute value of the reward is combined with the absolute value of the error to obtain the basic priority value. The priority value is obtained by adding a positive decimal to the base priority value.

8. The method for adaptively pushing English reading materials based on reinforcement learning according to claim 6, characterized in that, Whenever the cumulative number of newly stored experience tuples reaches ten, a priority sampling process is triggered, and a small batch of samples is drawn, including: Traverse all experience tuples in the circular experience queue, read the priority value corresponding to each experience tuple, and sort all experience tuples in descending order according to the priority value to obtain a sorted list of experience tuples. Calculate the sum of the priority values ​​of the first N empirical tuples in the sorted empirical tuple list, where N is equal to the pre-set batch size, and determine whether the summation result exceeds half of the total sum of the priority values ​​of all empirical tuples. If more than half are selected, the first N empirical tuples in the sorted empirical tuple list are selected as mini-batch samples. If the number of empirical tuples does not exceed half, then an empirical tuple is selected from the sorted list of empirical tuples at fixed intervals. The fixed interval is determined based on the ratio of the total number of empirical tuples in the sorted empirical tuple list to the batch size, and the starting position is the first empirical tuple in the sorted empirical tuple list, until the batch size of empirical tuples is selected to form a small batch sample.

9. The method for adaptively pushing English reading materials based on reinforcement learning according to claim 1, characterized in that, After selecting new materials of corresponding difficulty levels from the graded reading corpus according to the aforementioned difficulty adjustment action and pushing them to the target user's terminal, the process also includes: Record the sequence of touch drag rate, paragraph back number, and word long press query duration generated by the target user during the reading of new materials; The baseline reading speed of the target user is updated based on the difficulty level of the new material and the target user's reading fluency deviation and instant comprehension confidence level on the new material. The benchmark reading speed is updated in the following way: The average reading time of the target users on all historical materials is weighted and averaged with the average reading time on the new materials. Obtain the real-time average level of the reward correction value obtained by all users after the actual push of the new material, and perform a weighted average of the real-time average level and the benchmark reward value stored in the new material's metadata to obtain the updated benchmark reward value, and write it back to the new material's metadata; When the deep Q-network strategy model outputs a difficulty-maintaining action in multiple consecutive pushes, and the actual reward value after each push is higher than the current material benchmark reward value, the exploration rate of the deep Q-network strategy model will be reduced by a preset ratio.

10. A reinforcement learning-based adaptive difficulty recommendation system for English reading materials, characterized in that, The method for adaptively pushing English reading materials based on reinforcement learning as described in any one of claims 1-9 includes: The behavior collection module is used to acquire the touch drag rate sequence, paragraph back number sequence, and word long press query duration sequence generated by the target user during the reading of the current English material; The deviation calculation module is used to calculate the reading fluency deviation based on the touch drag rate sequence and the material standard page turning rate, and to calculate the instant comprehension confidence based on the duration sequence of the long press query of the word and the average reading time of the paragraph. The action output module is used to input a three-element state vector consisting of the difficulty level of the current material, the reading fluency deviation, and the instant understanding confidence into a pre-trained deep Q-network policy model, and output a difficulty adjustment action, wherein the difficulty adjustment action includes increasing the difficulty level, decreasing the difficulty level, or maintaining the difficulty. The reward calculation module is used to calculate a reward correction value based on the number of correct answers and the reading fluency deviation after the target user completes several subsequent reading comprehension questions of the current material, and to obtain the actual reward value by combining it with the baseline reward value. The model update module is used to form an experience tuple from the three-element state vector, the difficulty adjustment action, the actual reward value and the next state vector, store it in the circular experience queue, and trigger a priority sampling of small batch samples every ten experience tuples to update the parameters of the deep Q network policy model. The material push module is used to select new materials of corresponding difficulty level from the graded reading corpus according to the difficulty adjustment action and push them to the target user's terminal.

Images