A sight-based auxiliary memory method, device, system, equipment and medium

Abstract

The embodiments of the present application provide a sight-based auxiliary memory method, apparatus, system, device, and medium. The electronic device can obtain a sight image of a user while reading an article; for each sentence unit, obtain a target image of the sentence unit from the sight image; determine a sight trajectory based on the sight information in the target image; obtain the user's correct answer based on the probability of correct answer; train an auxiliary memory model; determine a memory time interval based on the correct answer and the number of answer rounds; after the memory time interval, return to the step of obtaining a sight image of the user's sight while reading the article until the correct answer meets a preset condition. Since the user will reread the article and re-answer the questions selected by the electronic device at each memory time interval until the user's correct answer meets the preset condition, the electronic device can assist the user in memorizing the article.

Description

Technical Field

[0001] The present application relates to the technical field of line of sight analysis, and in particular to a line of sight-based auxiliary memory method, device, system, equipment and medium. Background Art

[0002] When a user reads an article, the user's visual line information such as the number of blinks, the number of glances back, the reading frequency, and the gaze time can be collected, and then the reading memory ability assessment model, the reading comprehension ability assessment model, and the reading reasoning ability assessment model are used respectively to determine the user's reading ability and attention based on the above-collected visual line information.

[0003] In the above evaluation method, the user's reading ability and attention can only be evaluated through the visual information collected during the reading process, and it cannot help the user remember the read article. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a sight-based auxiliary memory method, device, system, equipment, and medium to assist users in memorizing articles. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a sight-based auxiliary memory method, the method comprising:

[0006] Obtaining a sight line image of the user while reading the article;

[0007] For each sentence unit in the article, obtaining a target image corresponding to the sentence unit from the sight line image, wherein the sentence unit is at least one continuous sentence, and the target image is the sight line image corresponding to the paragraph including the context of the sentence unit;

[0008] Determining a sight line trajectory corresponding to the target image according to the sight line information of the user in the target image;

[0009] Obtaining a title corresponding to the article based on a probability of correct answer, and obtaining the correctness of the user's answer to the title, wherein the probability of correct answer is predicted by an auxiliary memory model;

[0010] The auxiliary memory model is trained based on the reading content segment, the eye tracking track, and the correctness of the answer, wherein the reading content segment is the paragraph including the context of the sentence unit, and the auxiliary memory model is used to predict the probability of the user answering the question correctly based on the text features of the input reading content segment and the eye tracking temporal features of the input eye tracking track;

[0011] Determining a memory time interval based on the correctness of the answers and the rounds of the user answering the questions;

[0012] After the duration of the memory time interval has passed, return to the step of obtaining the sight image of the user's sight during reading the article until the correct answer meets the preset conditions, wherein the preset conditions indicate that the user's memory of the article has achieved the expected memory effect.

[0013] Optionally, the auxiliary memory model includes a first branch network, a second branch network and a fully connected network;

[0014] The step of training the auxiliary memory model based on the reading content segment, the eye tracking track and the correct answer status includes:

[0015] Inputting the reading content segment into the first branch network and the gaze trajectory into the second branch network, so that the first branch network extracts text features of the reading content segment and inputs the features into the fully connected network; the second branch network extracts sight timing features of the sight trajectory and inputs the features into the fully connected network, so that the fully connected network fuses the text features and the sight timing features and outputs a correct prediction based on the fused features;

[0016] Determining a loss function value of the auxiliary memory model according to a difference between the correct prediction and the correct answer;

[0017] Based on the loss function value, the parameters of the auxiliary memory model are adjusted through back propagation.

[0018] Optionally, before the step of training the auxiliary memory model based on the reading content segment, the eye tracking track, and the correctness of the answer, the method further includes:

[0019] Extracting word features for each sentence unit in the article;

[0020] Clustering the sentence units according to the extracted features to determine the category to which each sentence unit belongs;

[0021] Determining sentence units in the reading content segment that belong to the same category as the sentence unit and the sentence unit as associated segments;

[0022] Determining sentence units in the reading content segment that belong to a category different from the category to which the sentence unit belongs, as well as the sentence unit, as unrelated segments;

[0023] The step of inputting the reading content segment into the first branch network includes:

[0024] The associated segment and the unassociated segment are input into the first branch network, so that the first branch network extracts text features of the associated segment and the unassociated segment respectively.

[0025] Optionally, the step of obtaining the title corresponding to the article based on the probability of correct answer includes:

[0026] Get the preset number of questions corresponding to each sentence unit;

[0027] From a preset number of questions, multiple questions are selected based on the probability of correct answers as the questions corresponding to the article.

[0028] Optionally, the step of extracting multiple questions from a preset number of questions based on the probability of correct answers as the questions corresponding to the article includes:

[0029] For each question in the questions corresponding to each sentence unit, the reading content segment corresponding to the sentence unit corresponding to the question and the eye tracking trajectory corresponding to the reading content segment are input into the auxiliary memory model to obtain the prediction correct probability corresponding to the question;

[0030] The reciprocal of the correct prediction probability is used as the score corresponding to the question;

[0031] The ratio between the score corresponding to the question and the total score of the questions corresponding to each sentence unit is used as the extraction ratio corresponding to the question;

[0032] According to the extraction ratio of the topics corresponding to each sentence unit, at least one topic is extracted from the preset number of topics corresponding to each sentence unit.

[0033] Optionally, the step of obtaining a preset number of questions corresponding to each sentence unit includes:

[0034] For each sentence unit, extract the target keyword of the sentence unit, match the target keyword with the keyword corresponding to each question in the pre-established question bank, obtain the matching degree between the sentence unit and each question, and obtain a preset number of questions with the highest corresponding matching degree; or,

[0035] According to the pre-established correspondence between sentence units and questions, a preset number of questions corresponding to each sentence unit is determined.

[0036] Optionally, the step of determining the memory time interval based on the correctness of the answers and the rounds of answering performed by the user includes:

[0037] The ratio between the number of questions answered correctly and the number of questions corresponding to the article is used as the answer accuracy rate;

[0038] Based on the answer accuracy rate, the number of rounds the user takes to answer the question, and the Ebbinghaus curve, the memory time interval Δ is calculated according to the following formula:

[0039] Δ=f(η)·g(k)

[0040] Wherein, η is the correct answer rate, g(k) is the Ebbinghaus curve, k is the answer round, and f(η) is the Sigmoid function after the vertical axis value is magnified.

[0041] Optionally, the preset condition is that the user's correct answer rate for the questions for a preset number of consecutive times is greater than a preset correct rate, wherein the correct answer rate is the ratio between the number of questions answered correctly and the number of questions corresponding to the article.

[0042] In a second aspect, an embodiment of the present application provides a sight-based auxiliary memory device, the device comprising:

[0043] An image acquisition module is used to acquire an image of the user's sight while reading an article;

[0044] an image determination module, configured to obtain, for each sentence unit in the article, a target image corresponding to the sentence unit from the sight line image, wherein the sentence unit is at least one continuous sentence, and the target image is the sight line image corresponding to the paragraph including the context of the sentence unit;

[0045] a trajectory determination module, configured to determine a sight line trajectory corresponding to the target image based on the sight line information of the user in the target image;

[0046] An answer status acquisition module, configured to acquire the title corresponding to the article based on a probability of correct answer, and acquire the correctness of the user's answer to the title, wherein the probability of correct answer is predicted by an auxiliary memory model;

[0047] a model training module, configured to train the auxiliary memory model based on a reading content segment, the eye tracking track, and the correctness of the answer, wherein the reading content segment is the paragraph including the context of the sentence unit, and the auxiliary memory model is configured to predict the probability of the user answering the question correctly based on the text features of the input reading content segment and the eye tracking temporal features of the input eye tracking track;

[0048] An interval determination module, configured to determine a memory time interval based on the correctness of the answer and the number of rounds of answering performed by the user;

[0049] The return execution module is used to return to the step of obtaining the sight image of the user's sight during reading the article after the length of the memory time interval has passed, until the correct answer meets the preset conditions, wherein the preset conditions represent that the user's memory of the article has achieved the expected memory effect.

[0050] Optionally, the auxiliary memory model includes a first branch network, a second branch network and a fully connected network;

[0051] The model training module includes:

[0052] a parameter input submodule, configured to input the reading content segment into the first branch network and the sight track into the second branch network, so that the first branch network extracts text features of the reading content segment and inputs the features into the fully connected network; the second branch network extracts sight timing features of the sight track and inputs the features into the fully connected network, so that the fully connected network fuses the text features and the sight timing features and outputs a correct prediction based on the fused features;

[0053] a function value determination submodule, configured to determine a loss function value of the auxiliary memory model according to a difference between the correct prediction and the correct answer;

[0054] A parameter adjustment submodule, configured to adjust the parameters of the auxiliary memory model by back propagation based on the loss function value;

[0055] The device further comprises:

[0056] A word feature extraction module, configured to extract word features from each sentence unit in the article;

[0057] a category determination module, configured to cluster the sentence units according to the extracted features and determine the category to which each sentence unit belongs;

[0058] an associated segment determining module, configured to determine sentence units in the reading content segment that belong to the same category as the sentence unit and the sentence unit as associated segments;

[0059] an unrelated segment determination module, configured to determine a sentence unit in the reading content segment that belongs to a category different from the category to which the sentence unit belongs, and the sentence unit, as an unrelated segment;

[0060] The parameter input submodule includes:

[0061] a parameter input unit, configured to input the associated segment and the unassociated segment into the first branch network, so that the first branch network extracts text features of the associated segment and the unassociated segment respectively;

[0062] The answer status acquisition module includes:

[0063] The topic acquisition submodule is used to obtain a preset number of topics corresponding to each sentence unit;

[0064] A topic extraction submodule is used to extract multiple topics from a preset number of topics based on the probability of correct answers as topics corresponding to the article;

[0065] The topic extraction submodule includes:

[0066] a question extraction unit, configured to input, for each question corresponding to each sentence unit, a reading content segment corresponding to the sentence unit and a sight track corresponding to the reading content segment into the auxiliary memory model to obtain a predicted correct probability corresponding to the question;

[0067] a score determination unit, configured to use the inverse of the predicted correct probability as the score corresponding to the question;

[0068] an extraction ratio determination unit, configured to use the ratio between the score corresponding to the question and the total score of the question corresponding to each sentence unit as the extraction ratio corresponding to the question;

[0069] a topic extraction unit, configured to extract at least one topic from a preset number of topics corresponding to each sentence unit according to an extraction ratio of topics corresponding to each sentence unit;

[0070] The topic acquisition submodule includes:

[0071] A first question acquisition unit is configured to extract target keywords from each sentence unit, match the target keywords with keywords corresponding to each question in a pre-established question bank, obtain a matching degree between the sentence unit and each question, and obtain a preset number of questions with the highest matching degree;

[0072] A second topic acquisition unit is used to determine a preset number of topics corresponding to each sentence unit based on a pre-established correspondence between the sentence units and the topics;

[0073] The interval determination module includes:

[0074] a correctness determination submodule, configured to use the ratio of the number of questions answered correctly to the number of questions corresponding to the article as the correctness of the answer;

[0075] The time interval determination submodule is configured to calculate the memory time interval Δ based on the answer accuracy rate, the number of answer rounds performed by the user, and the Ebbinghaus curve according to the following formula:

[0076] Δ=f(η)·g(k)

[0077] Wherein, η is the correct answer rate, g(k) is the Ebbinghaus curve, k is the answer round, and f(η) is the Sigmoid function after the vertical axis value is magnified;

[0078] The preset condition is that the user's correct answer rate for the questions for a preset number of consecutive times is greater than a preset correct rate, wherein the correct answer rate is the ratio between the number of questions with correct answers and the number of questions corresponding to the article.

[0079] In a third aspect, an embodiment of the present application provides a sight-based auxiliary memory system, the system comprising a server and a student-end device, the student-end device comprising a display and a camera, wherein:

[0080] The display is used to display the article that the student wants to read;

[0081] The camera is used to capture an image of the student's sight while reading the article;

[0082] The server is configured to obtain the sight line image; for each sentence unit in the article, obtain a target image corresponding to the sentence unit from the sight line image; determine a sight line trajectory corresponding to the target image based on the sight line information of the user in the target image; obtain the question corresponding to the article based on the probability of correct answer, and obtain the correctness of the user's answer to the question; train the auxiliary memory model based on the reading content segment, the sight line trajectory and the correctness of the answer; determine the memory time interval based on the correctness of the answer and the answer rounds of the user; after the length of the memory time interval has passed, trigger the display to display the article, trigger the camera to capture the sight line image of the student's sight line during the reading process, and return to the step of obtaining the sight line image of the user's sight line during the reading process until the correctness of the answer meets the preset condition;

[0083] Among them, the preset condition represents that the user's memory of the article has achieved the expected memory effect, the sentence unit is at least one continuous sentence, the target image is the sight image corresponding to the paragraph including the context of the sentence unit, the reading content segment is the paragraph including the context of the sentence unit, and the auxiliary memory model is used to predict the probability of the user's correct answer to the question based on the text features of the input reading content segment and the sight timing features of the input sight trajectory, and the correct answer probability is predicted by the auxiliary memory model.

[0084] Optionally, the system further includes a teacher terminal device;

[0085] The teacher-side device is used to obtain the preset number of times and the preset accuracy rate input by the teacher, and to receive the correct answer status sent by the server and display the correct answer status;

[0086] The server is further configured to determine, based on the preset number of times and the preset accuracy rate, that the preset condition is that the user's accuracy rate in answering questions for a preset number of consecutive times is greater than the preset accuracy rate.

[0087] Optionally, the student-end device further includes a wristband;

[0088] The server is further configured to send a reminder instruction to the wristband after the memory time interval has elapsed;

[0089] The wristband is used to output a prompt message when receiving the reminder instruction to prompt the student to read the article.

[0090] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0091] Memory for storing computer programs;

[0092] The processor is configured to implement any of the methods described in the first aspect above when executing a program stored in the memory.

[0093] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any of the methods described in the first aspect above.

[0094] Beneficial effects of the embodiments of the present application:

[0095] In the solution provided by the embodiment of the present application, the electronic device can obtain a sight image of the user's sight situation during the reading process of the article; for each sentence unit in the article, obtain a target image corresponding to the sentence unit from the sight image, wherein the sentence unit is at least one continuous sentence, and the target image is a sight image corresponding to the paragraph including the context of the sentence unit; determine the sight track corresponding to the target image according to the sight information of the user in the target image; obtain the title corresponding to the article based on the probability of correct answer, and obtain the correct answer of the user to the title, wherein the probability of correct answer is predicted by the auxiliary memory model; based on the reading content fragment, the sight track and the The auxiliary memory model is trained based on the accuracy of the user's answers to the questions, wherein the reading content segment is a paragraph that includes the context of the sentence unit. The auxiliary memory model is used to predict the probability of the user answering the question correctly based on the text features of the input reading content segment and the eye gaze timing features of the input eye gaze trajectory. The memory interval is determined based on the accuracy of the answers and the number of rounds in which the user answers the questions. After the memory interval, the process returns to the step of obtaining a sight image of the user's sight during the reading process until the accuracy of the answers meets a preset condition, wherein the preset condition indicates that the user's memory of the corresponding article has achieved the expected memory effect. Because the auxiliary memory model can predict the accuracy of the user's answers to the questions based on the eye gaze timing features of the eye gaze trajectory and the text features of the reading content segment, the output of the auxiliary memory model can reflect the user's memory quality. Since the user rereads the article and re-answers the questions selected by the electronic device at each memory interval until the accuracy of the user's answers meets the preset condition, the electronic device can assist the user in memorizing the article.

[0096] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0098] Figure 1 A flowchart of a sight-based auxiliary memory method provided in an embodiment of the present application;

[0099] Figure 2 for Figure 1 A specific flow chart of step S105 in the embodiment shown;

[0100] Figure 3 Based on Figure 1 A flow chart of determining associated segments and unassociated segments according to the embodiment;

[0101] Figure 4 Based on Figure 1 A schematic structural diagram of the auxiliary memory model of the illustrated embodiment;

[0102] Figure 5 for Figure 1 A specific flow chart of step S104 in the embodiment shown;

[0103] Figure 6 for Figure 5 A specific flow chart of step S502 in the embodiment shown;

[0104] Figure 7 for Figure 1 A specific flow chart of step S106 in the embodiment shown;

[0105] Figure 8 Based on Figure 1 A schematic diagram of the Ebbinghaus curve of the illustrated embodiment;

[0106] Figure 9 Based on Figure 1 A schematic diagram of a process of assisting a user in memory according to the embodiment shown;

[0107] Figure 10 A schematic structural diagram of a sight-based auxiliary memory device provided in an embodiment of the present application;

[0108] Figure 11 A schematic structural diagram of a sight-based auxiliary memory system provided in an embodiment of the present application;

[0109] Figure 12 Based on Figure 11 Another structural diagram of the auxiliary memory system of the illustrated embodiment;

[0110] Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0111] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0112] To assist users in memorizing articles, embodiments of the present application provide a sight-based memorization method, apparatus, system, electronic device, computer-readable storage medium, and computer program product. The following first introduces a sight-based memorization method provided by embodiments of the present application.

[0113] The sight-based memory aid method provided in the embodiments of the present application can be applied to any electronic device that needs to assist the user in memorizing, for example, a tablet computer, a desktop computer, etc., without specific limitation. For the sake of clarity, it will be referred to as an electronic device in the following text.

[0114] like Figure 1 As shown, a sight-based auxiliary memory method, the method comprising:

[0115] S101, obtaining a sight line image of a user's sight line during reading an article;

[0116] S102, for each sentence unit in the article, obtaining a target image corresponding to the sentence unit from the sight line image;

[0117] The sentence unit is at least one continuous sentence, and the target image is a sight line image corresponding to a paragraph including the context of the sentence unit.

[0118] S103, determining a sight line trajectory corresponding to the target image according to the sight line information of the user in the target image;

[0119] S104, obtaining a title corresponding to the article based on a probability of correct answer, and obtaining the correctness of the user's answer to the title, wherein the probability of correct answer is predicted by an auxiliary memory model;

[0120] S105, training the auxiliary memory model based on the reading content segment, the eye tracking trajectory, and the correctness of the answer;

[0121] The reading content segment is the paragraph including the context of the sentence unit, and the auxiliary memory model is used to predict the probability of the user's correct answer to the question based on the text features of the input reading content segment and the sight timing features of the input sight trajectory.

[0122] S106, determining a memory time interval according to the correctness of the answers and the rounds of the user answering the questions.

[0123] It can be seen that in an embodiment of the present application, the electronic device can obtain a sight image of the user's sight during the reading process of the article; for each sentence unit in the article, obtain a target image corresponding to the sentence unit from the sight image, wherein the sentence unit is at least one continuous sentence, and the target image is the sight image corresponding to the paragraph including the context of the sentence unit; determine the sight trajectory corresponding to the target image based on the user's sight information in the target image; obtain the title corresponding to the article, and obtain the correctness of the user's answer to the title; based on the reading content segment, the sight trajectory and the correctness of the answer, train the auxiliary memory model, wherein the reading content segment is the paragraph including the context of the sentence unit, and the auxiliary memory model is used to predict the correctness of the user's answer to the question based on the text features of the input reading content segment and the sight timing features of the input sight trajectory; determine the memory time interval based on the correctness of the answer and the number of rounds in which the user answers the question; after the length of the memory time interval, return to the step of obtaining the sight image of the user's sight during the reading process until the correctness of the answer meets the preset condition, wherein the preset condition indicates that the user's memory of the article has achieved the expected memory effect. Because the memory support model predicts the accuracy of the user's answers to questions based on the temporal characteristics of the gaze trajectory and the textual features of the reading content, the output of the memory support model can reflect the user's memory quality. Since the user rereads the article and re-answers the questions selected by the electronic device at each memorization interval until the user's answer accuracy meets the preset conditions, the electronic device can assist the user in memorizing the article.

[0124] In order to determine the user's sight line during reading, the camera can capture the user's sight line during reading to obtain a sight line image. In step S101, the electronic device can obtain the sight line image captured by the camera.

[0125] Since the topics expressed in the context paragraphs are usually similar, the context segment can be used as a whole to predict and examine the user's memory effect. In order to determine the user's eye trajectory when reading the context segment, and then predict the user's memory effect based on the eye trajectory's eye timing characteristics, the electronic device can obtain the eye image corresponding to the paragraph corresponding to the sentence unit and including the context of the sentence unit from the eye image for each sentence unit in the article, that is, execute step S102. Among them, the sentence unit can be at least one continuous sentence. For example, the sentence unit can be a sentence or two continuous sentences.

[0126] In one embodiment, the context paragraph corresponding to a sentence unit includes the sentence unit, a first number of sentences preceding the sentence unit, and a second number of sentences following the sentence unit. For example, if an article includes 50 sentences, namely sentences 1 to 50, and the currently targeted sentence unit is sentences 20 to 23, and the first and second numbers are 2 and 3 respectively, then the sentence unit includes sentences 18 to 26. Furthermore, the electronic device can obtain the image corresponding to sentences 18 to 26, i.e., the target image, from the user's line of sight image while reading sentences 1 to 50.

[0127] Since the user's sight line trajectory during reading can reflect the user's reading habits, the user's memory effect can be predicted based on the reading habits. Therefore, the electronic device can determine the sight line trajectory corresponding to the target image based on the user's sight line information in the target image, that is, execute step S103.

[0128] Specifically, the electronic device can determine the coordinates of the user's sight point during the reading process based on the user's eye direction in the target image, and determine the user's sight trajectory in combination with the timing information between the target images.

[0129] In one embodiment, the electronic device can capture a gaze video of a user's gaze while reading an article; for each sentence unit in the article, a target video segment corresponding to the sentence unit is captured from the gaze video segment; and based on the user's gaze information in the target video segment, the gaze trajectory corresponding to the target video segment is determined. The target video segment is the gaze video segment corresponding to the paragraph containing the context of the sentence unit.

[0130] To determine the user's memory of the article, the electronic device can obtain the corresponding article title based on the probability of correct answers. If the probability of correct answers for a title is low, it indicates that the user may have poor memory of the reading content corresponding to the title. Therefore, the electronic device can select a title with a low probability of correct answers and encourage the user to answer the title. In this way, the electronic device can obtain the correctness of the user's answer to the question, that is, execute step S104. The correctness of the answer can be used to indicate whether the user answered the question correctly or incorrectly.

[0131] If the user's answer to the question is correct, it means that the user has a good memory effect on the corresponding sentence unit. If the user's answer to the question is incorrect, it means that the user has a poor memory effect on the corresponding sentence unit.

[0132] To predict the user's memory effect of a reading content segment, the electronic device can create an algorithmic model of the effect of the user's gaze trajectory on memory based on the user's gaze trajectory. Specifically, the electronic device can train the auxiliary memory model based on the reading content segment, gaze trajectory, and correct answer results, that is, execute step S105. The reading content segment is the paragraph that includes the context of the sentence unit.

[0133] The auxiliary memory model extracts text features from the input reading content fragments and extracts gaze temporal features from the input gaze trajectory. The auxiliary memory model can fuse the text features with the gaze temporal features. The fused features reflect the mapping relationship between the user's gaze temporal features and their memory status. Based on the fused features, the auxiliary memory model can predict the probability of the user answering the question correctly, thereby reflecting the predicted user's memory status.

[0134] Among them, the auxiliary memory model can be any model that can construct a mapping relationship between the temporal characteristics of the line of sight and the memory status, such as a long short-term memory (LSTM) network, a logistic regression model, a support vector machine (SVM), a transformer model, etc.

[0135] In one embodiment, the electronic device can train the auxiliary memory model for each question answered by the user based on the reading content segment, eye tracking, and correctness of the answer corresponding to the question. In this case, the parameter corresponding to the correctness of the answer is one-dimensional.

[0136] In another embodiment, the electronic device can train the auxiliary memory model based on the reading content segment, eye tracking, and correct answer status corresponding to each question. In this case, the dimension of the parameter corresponding to the correct answer status can be equal to the number of questions.

[0137] The memory interval can be the time interval between two consecutive readings of the article by the user. If the user's correct answers account for a high proportion of the questions answered correctly, it indicates that the user has a high degree of memory for the article, and the memory interval can be longer. The user can answer questions after reading the article. If the user answers questions many times, it means that the user has read the article many times, and thus the user's memory of the article may be high. The electronic device can determine the memory interval based on the correct answer percentage and the number of rounds the user answered questions, that is, execute step S106.

[0138] Since the user's memory retention rate of the article content may decrease after a period of time, the electronic device can remind the user to reread the article after the memory time interval has passed and return to step S101. In other words, a cyclical approach is used to consolidate the user's memory of the article.

[0139] If the user's correct answer to the question meets the preset conditions, it means that the user's memory effect of the article has reached the expected memory effect, and the electronic device can stop the loop, that is, not return to execute step S101.

[0140] In one embodiment, the preset condition may be that the user's correct answer rate for each question is greater than a preset correct rate for a preset number of consecutive times. The correct answer rate is the ratio of the number of questions answered correctly to the number of questions corresponding to the article.

[0141] For example, if the application scenario is to assist students in memorizing the text, the teacher can set the preset number of times and the preset accuracy rate according to the degree of mastery of the text required by the students. If the student needs to have a higher degree of mastery of Text A, the teacher can set the preset number of times and the preset accuracy rate to 3 times and 90% respectively. In this way, if the student answers the questions for 3 consecutive times with an accuracy rate higher than 90%, the electronic device will stop looping. If the student needs to have a lower degree of mastery of Text B, the teacher can set the preset number of times and the preset accuracy rate to 2 times and 60% respectively. In this way, if the student answers the questions for 2 consecutive times with an accuracy rate higher than 60%, the electronic device will stop looping.

[0142] Since the auxiliary memory model is trained based on a user's eye track, after the user's memory effect for an article reaches the expected memory effect, the auxiliary memory model can also assist the user in remembering other articles.

[0143] As can be seen, in the embodiments of the present application, since the auxiliary memory model can predict the correctness of the user's answer to the question based on the temporal characteristics of the gaze trajectory and the textual characteristics of the reading content segment, the output of the auxiliary memory model can reflect the user's memory quality. Since the user will reread the article and re-answer the questions selected by the electronic device at each memory interval until the user's correct answer meets the preset conditions, the electronic device can assist the user in memorizing the article.

[0144] As an implementation method of the embodiment of the present application, the auxiliary memory model includes a first branch network, a second branch network and a fully connected network. Figure 2As shown, the step of training the auxiliary memory model based on the reading content segment, the eye tracking and the correct answer situation may include:

[0145] S201: Input the reading content segment into the first branch network, and input the gaze trajectory into the second branch network, so that the first branch network extracts text features of the reading content segment and inputs them into the fully connected network; the second branch network extracts gaze temporal features of the gaze trajectory and inputs them into the fully connected network, so that the fully connected network fuses the text features and the gaze temporal features, and outputs a correct prediction based on the fused features;

[0146] If the auxiliary memory model includes a first branch network, a second branch network and a fully connected network, then the electronic device can input the reading content segment and the sight trajectory into the first branch network and the second branch network respectively. The first branch network can extract text features of the reading content segment and input the text features into the fully connected network. The second branch network can extract the sight timing features of the sight trajectory and input the sight timing features into the fully connected network.

[0147] After receiving the text features and gaze timing features, the fully connected network can fuse the text features and gaze timing features. Since the fused features reflect the mapping relationship between the user's gaze timing features and their memory, the fully connected network can use these fused features to predict the user's correct answer to the question.

[0148] In one embodiment, the auxiliary memory model is a dual LSTM network, and the first branch network and the second branch network are respectively a branch of the dual LSTM network.

[0149] S202, determining a loss function value of the auxiliary memory model according to the difference between the correct prediction and the correct answer;

[0150] Since there may be a difference between the predicted correctness output by the auxiliary memory model and the correctness of the answer actually given by the user, the electronic device can use the correctness of the answer as a label and determine the loss function value of the auxiliary memory model based on the difference between the predicted correctness and the correctness of the answer.

[0151] S203: Based on the loss function value, adjust the parameters of the auxiliary memory model through back propagation.

[0152] In order to improve the accuracy of the auxiliary memory model in predicting correct situations, the electronic device can adjust the parameters of the auxiliary memory model through back propagation based on the loss function value, so that the difference between the predicted correct situation output by the auxiliary memory model based on the adjusted parameters and the correct situation of the user's actual answer is reduced.

[0153] It can be seen that in the embodiment of the present application, the electronic device can input the reading content segment into the first branch network and the sight track into the second branch network, so that the first branch network extracts the text features of the reading content segment and inputs them into the fully connected network, and the second branch network extracts the sight time sequence features of the sight track and inputs them into the fully connected network, so that the fully connected network fuses the text features and the sight time sequence features, and outputs the predicted correctness based on the fused features; according to the difference between the predicted correctness and the correct answer, the loss function value of the auxiliary memory model is determined; based on the loss function value, the parameters of the auxiliary memory model are adjusted by back propagation. Since there may be a difference between the predicted correctness output by the auxiliary memory model and the correct answer of the user's actual answer, the electronic device can calculate the loss function value based on the above difference, and adjust the parameters of the auxiliary memory model by back propagation based on the loss function value, so that the difference between the predicted correctness output by the auxiliary memory model based on the adjusted parameters and the correct answer of the user's actual answer is reduced. In this way, the accuracy of the predicted correctness output by the auxiliary memory model can be improved.

[0154] As an implementation method of the present application, Figure 3 As shown, before the step of training the auxiliary memory model based on the reading content segment, the eye tracking track and the correct answer situation, the method may further include:

[0155] S301, extracting word features for each sentence unit in the article;

[0156] In order to analyze the text features of an article, the electronic device can, for each sentence unit corresponding to the reading content segment, treat the sentence units in the reading content segment that are similar to the word features of the sentence unit and the sentence unit as associated segments; treat the sentence units in the reading content segment that are not similar to the word features of the sentence unit and the sentence unit as unrelated segments, and then determine the text features of the reading content segment based on the associated content and the unrelated content.

[0157] In order to determine the word features of the sentence units, the electronic device may extract word features for each sentence unit in the article, and the extracted word features may be vectors.

[0158] In one embodiment, the electronic device may perform word embedding on each sentence unit to extract word features corresponding to each sentence unit. The word embedding algorithms used by the electronic device to perform word embedding on the sentence unit include the Word to Vector (Word2Vec) algorithm, the Global Vectors for Word Representation (GloVe) algorithm, and the Bidirectional Encoder Representations from Transformers (BERT) algorithm.

[0159] S302, clustering the sentence units according to the extracted features to determine the category to which each sentence unit belongs;

[0160] To determine the degree of similarity between sentence units in the reading content segment and the word features of the sentence unit, the electronic device can cluster the sentence units based on the extracted features and determine the category to which each sentence unit belongs. Sentence units belonging to the same category as the sentence unit are sentence units with similar word features to the sentence unit. Sentence units belonging to a different category than the sentence unit are sentence units with dissimilar word features to the sentence unit.

[0161] In one embodiment, the electronic device may calculate the geometric distance between word features corresponding to sentence units, and determine the category to which a sentence unit belongs if the geometric distance between the corresponding word feature and the word feature corresponding to the cluster center is less than a preset distance. Specifically, the geometric distance may be a cosine distance, a Euclidean distance, or a Manhattan distance.

[0162] S303, determining the sentence units in the reading content segment that belong to the same category as the sentence unit and the sentence unit as associated segments;

[0163] After determining the category to which each sentence unit belongs, the electronic device can determine, based on the category to which each sentence unit belongs, sentence units in the reading content segment that belong to the same category as the sentence unit and the sentence unit as associated segments.

[0164] For example, if the reading content segment corresponding to sentence unit 1 includes sentence unit 1-sentence unit 5, sentence unit 1-sentence unit 3 belong to category A, and sentence unit 3 and sentence unit 4 belong to category B, then the electronic device can determine sentence unit 1-sentence unit 3 as associated segments.

[0165] S304 : Determine the sentence units in the reading content segment that belong to a category different from the category to which the sentence unit belongs, as well as the sentence unit, as unrelated segments.

[0166] After determining the category to which each sentence unit belongs, the electronic device can determine, based on the category to which each sentence unit belongs, sentence units in the reading content segment that belong to a category different from that of the sentence unit and the sentence unit itself as unrelated segments.

[0167] Continuing with the previous example, the electronic device may determine sentence unit 1, sentence unit 4, and sentence unit 5 as unrelated segments.

[0168] In one embodiment, the step of inputting the reading content segment into the first branch network may include:

[0169] The associated segment and the unassociated segment are input into the first branch network, so that the first branch network extracts text features of the associated segment and the unassociated segment respectively.

[0170] In order to improve the effect of the auxiliary memory model on extracting text features of the article, after determining the associated fragments and the unrelated fragments, the electronic device can input the associated fragments and the unrelated fragments into the first branch network, so that the first branch network extracts the text features of the associated fragments and the text features of the unrelated fragments respectively.

[0171] For example, when the auxiliary memory model is a dual LSTM network, the structural diagram of the auxiliary memory model can be as follows: Figure 4 As shown, the electronic device can obtain the gaze trajectory 402 based on the coordinates of the user's gaze point and the timing information in the target image 401, and determine the associated segment 403 and the unassociated segment 404 corresponding to the reading content segment according to the category to which the sentence unit included in the reading content segment corresponding to the target image 401 belongs.

[0172] The electronic device can input the gaze trajectory 402 into the first branch LSTM network 405 so that the first branch LSTM network 405 outputs the gaze timing feature 406; the electronic device can input the associated segment 403 and the unassociated segment 404 into the second branch LSTM network 407 so that the second branch LSTM network 407 outputs the text feature 408.

[0173] The first branch LSTM network 405 can input the extracted sight timing feature 406 into the fully connected network, and the second branch LSTM network 407 can input the extracted text feature 408 into the fully connected network, so that the fully connected network can fuse the sight timing feature 406 and the text feature 408 to obtain the fused feature 409, and output the correct prediction 410 based on the fused feature 409.

[0174] It can be seen that in the embodiment of the present application, the electronic device can extract word features for each sentence unit in the article; cluster the sentence units according to the extracted features to determine the category to which each sentence unit belongs; determine the sentence units in the reading content segment that belong to the same category as the sentence unit and the sentence unit as associated segments; input the associated segments and the unassociated segments into the first branch network, so that the first branch network extracts the text features of the associated segments and the text features of the unassociated segments respectively. In order to analyze the text features of the article, the electronic device can determine the associated segments and the unassociated segments for the reading content segment corresponding to each sentence unit, and then determine the text features of the reading content segment based on the associated content and the unassociated content. In this way, the extracted text features can more accurately reflect the features of the reading content segment.

[0175] As an implementation method of the present application, Figure 5 As shown, the step of obtaining the title corresponding to the article based on the probability of correct answer may include:

[0176] S501, obtaining a preset number of questions corresponding to each sentence unit;

[0177] In order to test the user's memory of each sentence unit in the article, the electronic device can obtain a preset number of questions corresponding to each sentence unit. In this way, the number of questions corresponding to each sentence unit obtained by the electronic device is equal, and the user's memory of each sentence unit can be tested evenly.

[0178] For example, if the article includes sentence unit 1 to sentence unit 20, and the preset number is 3, then the electronic device can obtain 3 topics corresponding to sentence unit 1 to sentence unit 20 respectively.

[0179] S502: Extract multiple questions from a preset number of questions based on the probability of correct answers as the questions corresponding to the article.

[0180] In order to improve the efficiency of testing users and conduct targeted testing on questions that users have poor memory for, the electronic device can extract multiple questions from a preset number of questions based on the probability of correct answers as the corresponding questions for the article.

[0181] In a first embodiment, the electronic device can extract a first number of questions from a preset number of questions corresponding to each sentence unit, and use the question corresponding to each sentence unit as the question corresponding to the article. Continuing with the example in step S501, if the first number is 1, the electronic device can extract one question from the three questions corresponding to sentence units 1-20, obtaining one question corresponding to each of sentence units 1-20. In this way, since the number of questions extracted for each sentence unit is the same, the user's memory for each sentence unit can be examined evenly while improving the efficiency of examining the user's memory.

[0182] In a second embodiment, the electronic device may randomly select multiple topics from a preset number of topics as the topics corresponding to the article. Continuing with the example in step S501, the electronic device may randomly select 20 topics from the 60 topics obtained and use the 20 selected topics as the topics corresponding to the article.

[0183] As can be seen, in the embodiment of the present application, the electronic device can obtain a preset number of questions corresponding to each sentence unit; from the preset number of questions, multiple questions are extracted based on the probability of correct answers as the corresponding questions of the article. In order to improve the efficiency of user testing, the electronic device can extract multiple questions from the preset number of questions as the corresponding questions of the article. In this way, the efficiency of user testing can be improved while testing the user's memory of the article.

[0184] As an implementation of an embodiment of the present application, when the user is in the first round of answering questions, the step of extracting multiple questions from a preset number of questions based on the probability of correct answers as the questions corresponding to the article may include:

[0185] Randomly select a target number of questions from the preset number of questions; or, select at least one question in the same proportion from the preset number of questions corresponding to each sentence unit.

[0186] In the first embodiment, since the user's answering round is the first round, the electronic device cannot judge the user's memory of each sentence unit based on the correctness of the user's answers, and when the user's answering round is the first round, the auxiliary memory model has not yet started training, and the correct situation cannot be predicted through the auxiliary memory model output. Therefore, the electronic device can randomly select a target number of questions from a preset number of questions.

[0187] For example, if the preset number is 5, and the target numbers corresponding to sentence unit 1-sentence unit 3 are 2, 1, and 3 respectively, then the electronic device can randomly select 2 questions from the 5 questions corresponding to sentence unit 1; randomly select 1 question from the 5 questions corresponding to sentence unit 2; and randomly select 3 questions from the 5 questions corresponding to sentence unit 3.

[0188] In a second embodiment, in order to determine the user's memory of each sentence unit, the electronic device may extract at least one question in the same proportion from a preset number of questions corresponding to each sentence unit.

[0189] For example, if the preset number is 3, the electronic device may extract 2 topics from the 3 topics corresponding to sentence units 1 to 3 respectively.

[0190] As an implementation method of the present application, Figure 6 As shown, when the round in which the user answers the questions is not the first round, the step of extracting multiple questions from the preset number of questions as the questions corresponding to the article may include:

[0191] S601, for each question corresponding to each sentence unit, inputting the reading content segment corresponding to the sentence unit corresponding to the question and the eye tracking trajectory corresponding to the reading content segment into the auxiliary memory model to obtain a prediction correct probability corresponding to the question;

[0192] Before extracting the questions, in order to increase the extraction ratio of questions corresponding to sentences that the user has poor memory for, the electronic device can, for each question corresponding to each sentence unit, input the reading content segment corresponding to the sentence unit corresponding to the question and the eye track corresponding to the reading content segment into the auxiliary memory model to obtain the prediction correct probability corresponding to the question.

[0193] For example, if the topics corresponding to sentence unit 1 are topics 1-3, the topics corresponding to sentence unit 2 are topics 4-6, and the topics corresponding to sentence unit 3 are topics 7-9, then the electronic device can input the reading content segment 1 corresponding to sentence unit 1 and the sight track 1 corresponding to the reading content segment 1 into the auxiliary memory model to obtain a 50% prediction correct probability for topics 1-3; the electronic device can input the reading content segment 2 corresponding to sentence unit 2 and the sight track 2 corresponding to the reading content segment 2 into the auxiliary memory model to obtain a 25% prediction correct probability for topics 4-6; the electronic device can input the reading content segment 3 corresponding to sentence unit 3 and the sight track 3 corresponding to the reading content segment 3 into the auxiliary memory model to obtain a 20% prediction correct probability for topics 7-9.

[0194] S602, taking the inverse of the correct prediction probability as the score corresponding to the question;

[0195] If the probability of correct prediction for a question is low, it indicates that the user's memory of the sentence unit corresponding to the question is poor, so the extraction rate of the question corresponding to the sentence unit can be higher. To increase the extraction rate for questions with a low probability of correct prediction, the electronic device can use the inverse of the probability of correct prediction as the score corresponding to the question.

[0196] Continuing with the example in step S601, the scores corresponding to questions 1 to 3 are the inverse of the predicted correct probability of 50%, that is, 2; the scores corresponding to questions 4 to 6 are the inverse of the predicted correct probability of 25%, that is, 4; and the scores corresponding to questions 7 to 9 are the inverse of the predicted correct probability of 20%, that is, 5.

[0197] S603, taking the ratio of the score corresponding to the question to the total score of the questions corresponding to each sentence unit as the extraction ratio corresponding to the question;

[0198] Since the score corresponding to a question can represent the weight of the question being extracted, the electronic device can use the ratio between the score corresponding to the question and the total score of the question corresponding to each sentence unit as the extraction ratio corresponding to the question.

[0199] Continuing with the example in step S602, the total score of questions 1 to 9 is 33, the corresponding extraction ratios of questions 1 to 3 are 2 / 33, the corresponding extraction ratios of questions 4 to 6 are 4 / 33, and the corresponding extraction ratios of questions 7 to 9 are 5 / 33.

[0200] S604: Extract at least one topic from a preset number of topics corresponding to each sentence unit according to the extraction ratio of the topics corresponding to each sentence unit.

[0201] After determining the extraction ratio corresponding to each topic, the electronic device may extract at least one topic from a preset number of topics corresponding to each sentence unit according to the extraction ratio of the topic corresponding to each sentence unit.

[0202] Continuing with the example in step S603, if the number of questions to be extracted by the electronic device is 5, then the electronic device can extract 1 question from Question 1 to Question 3 corresponding to sentence unit 1, extract 2 questions from Question 4 to Question 6 corresponding to sentence unit 2, and extract 2 questions from Question 7 to Question 9 corresponding to sentence unit 3 according to the extraction ratio of the questions corresponding to each sentence unit.

[0203] It can be seen that in the embodiment of the present application, the electronic device can randomly extract a target number of questions from a preset number of questions when the user is in the first round of answering questions; or extract at least one question in the same proportion from the preset number of questions corresponding to each sentence unit; when the user is not in the first round of answering questions, for each question corresponding to each sentence unit, the reading content segment corresponding to the sentence unit corresponding to the question and the eye track corresponding to the reading content segment are input into the auxiliary memory model to obtain the predicted correct probability corresponding to the question; the inverse of the predicted correct probability is used as the score corresponding to the question; the ratio between the score corresponding to the question and the total score of the question corresponding to each sentence unit is used as the extraction ratio corresponding to the question; according to the extraction ratio of the question corresponding to each sentence unit, at least one question is extracted from the preset number of questions corresponding to each sentence unit. In order to make the extraction ratio corresponding to the question with a lower predicted correct probability higher, the electronic device can use the inverse of the predicted correct probability as the score corresponding to the question. Because the score corresponding to a question can represent the weight of the question in extraction, the electronic device can use the ratio between the score corresponding to the question and the total score of the questions corresponding to each sentence unit as the extraction ratio corresponding to the question. After determining the extraction ratio corresponding to each question, the electronic device can extract questions from a preset number of questions according to the extraction ratio. In this way, for questions that the user has a poor memory, the electronic device can extract a higher ratio, which can improve the effectiveness of assisting the user's memory.

[0204] As an implementation of an embodiment of the present application, the step of obtaining a preset number of questions corresponding to each sentence unit may include:

[0205] For each sentence unit, extract the target keyword of the sentence unit, match the target keyword with the keyword corresponding to each question in the pre-established question bank, obtain the matching degree between the sentence unit and each question, and obtain a preset number of questions with the highest corresponding matching degree; or,

[0206] According to the pre-established correspondence between sentence units and questions, a preset number of questions corresponding to each sentence unit is determined.

[0207] In the first embodiment, in order to obtain the topic corresponding to each sentence unit, the electronic device can extract the target keyword of each sentence unit, match the target keyword with the keyword corresponding to each topic in a pre-established question bank, obtain the matching degree between the sentence unit and each topic, and obtain a preset number of topics with the highest matching degree.

[0208] For example, the target keyword corresponding to sentence unit 1 is keyword A, and the keywords corresponding to topics 1-3 are keyword B, keyword C, and keyword D, respectively. The electronic device can match keyword A with keyword B, keyword C, and keyword D, and obtain matching degrees of 80%, 50%, and 10% for sentence unit 1 and topics 1-3, respectively. If the preset number is 2, the electronic device can obtain topics 1 and 2.

[0209] As an implementation method, the electronic device can map the target keywords of each sentence unit and the keywords corresponding to each question into vectors, and for each sentence unit, calculate the geometric distance between the vector corresponding to the target keyword corresponding to the sentence unit and the vector corresponding to the keyword corresponding to each question, and obtain a preset number of questions with the shortest geometric distance.

[0210] In a second embodiment, in order to determine the topic corresponding to each sentence unit and to assess the user's memory performance based on the corresponding topic, a correspondence between the sentence unit and the topic can be established before obtaining the topic corresponding to each sentence unit. In this way, the electronic device can determine a preset number of topics corresponding to each sentence unit based on the pre-established correspondence.

[0211] For example, if in the above correspondence, the topics corresponding to sentence unit 1-sentence unit 3 are topic 1, topic 2 and topic 3 respectively, and the preset number is 1, then the electronic device can determine topic 1 corresponding to sentence unit 1, topic 2 corresponding to sentence unit 2, and topic 3 corresponding to sentence unit 3 based on the correspondence.

[0212] In a third embodiment, if the question bank includes a sub-question bank, and the questions included in the sub-question bank belong to the same category, then the electronic device can extract the target keyword of each sentence unit, match the target keyword with the keyword corresponding to each sub-question bank in the pre-established question bank, obtain the matching degree between the sentence unit and each sub-question bank, and obtain a preset number of questions in the corresponding sub-question bank with the highest matching degree.

[0213] For example, the target keyword corresponding to sentence unit 1 is keyword A, and the keywords corresponding to sub-question banks 1-3 are keyword B, keyword C, and keyword D, respectively. The electronic device can match keyword A with keyword B, keyword C, and keyword D, and obtain matching degrees of 60%, 90%, and 10% for sentence unit 1 and sub-question banks 1-3, respectively. If the preset number is 2, the electronic device can obtain two questions from sub-question bank 2.

[0214] It can be seen that in the embodiment of the present application, the electronic device can extract the target keyword of each sentence unit, match the target keyword with the keyword corresponding to each question in the pre-established question bank, obtain the matching degree between the sentence unit and each question, and obtain a preset number of questions with the highest matching degree; or, based on the pre-established correspondence between the sentence unit and the question, determine the preset number of questions corresponding to each sentence unit. Since the electronic device can select the question corresponding to the sentence unit based on the matching degree between the target keyword corresponding to the sentence unit and the keyword corresponding to the question, or determine the question corresponding to the sentence unit based on the pre-established correspondence, the electronic device can quickly and accurately determine the question of the sentence unit.

[0215] As an implementation method of the present application, Figure 7 As shown, the step of determining the memory time interval based on the correctness of the answer and the answer round of the user may include:

[0216] S701, taking the ratio of the number of questions answered correctly to the number of questions corresponding to the article as the answer accuracy rate;

[0217] Since the answer accuracy rate is the ratio between the number of correctly answered questions and the total number of questions, the electronic device may use the ratio between the number of correctly answered questions and the number of questions corresponding to the article as the answer accuracy rate.

[0218] For example, if the number of questions corresponding to the article is 20 and the number of correct answers to the questions is 15, the electronic device may use the ratio of 15 to 20, that is, 75%, as the answer accuracy rate.

[0219] S702 , based on the answer accuracy rate, the number of answering rounds of the user and the Ebbinghaus curve, calculate the memory time interval Δ according to the following formula: Δ=f(η)·g(k).

[0220] Wherein, η is the correct answer rate, g(k) is the Ebbinghaus curve, k is the answer round, and f(η) is the Sigmoid function after the vertical axis value is magnified.

[0221] To ensure that the determined memory interval is neither too long nor too short, the vertical axis of the Sigmoid function can be amplified, with the accuracy rate of the answer serving as the independent variable. This way, the accuracy rate of the answer is positively correlated with the Sigmoid function value. A higher accuracy rate indicates better memory of the article, and the memory interval can be longer. Here, f(η) is the amplified vertical axis Sigmoid function, and η is the accuracy rate of the answer.

[0222] The Ebbinghaus curve can reflect the relationship between memory retention rate and time. The schematic diagram of the Ebbinghaus curve can be shown as follows: Figure 8 As shown, the horizontal axis represents time, and the vertical axis represents the user's memory retention rate for the article. The first curve 801 shows the relationship between memory retention rate and time when the user does not reread the article. The second curve 802 shows the relationship between memory retention rate and time when the user rereads the article one day, one week, one month, and six months after the first reading. The function values ​​g(k) of the Ebbinghaus curves corresponding to answer rounds k of 2, 3, 4, and 5 are 1 day, one week, one month, and six months, respectively.

[0223] It can be seen that in the embodiment of the present application, the electronic device can use the ratio between the number of questions with correct answers and the number of questions corresponding to the article as the answer accuracy rate; based on the answer accuracy rate, the number of answering rounds in which the user answers the questions, and the Ebbinghaus curve, the memory interval Δ is calculated according to the following formula: Δ = f(η) g(k); wherein η is the answer accuracy rate, g(k) is the Ebbinghaus curve, k is the answering round, and f(η) is the Sigmoid function after the vertical axis value is magnified. Since the electronic device can determine the memory time interval based on the user's answer accuracy rate, combined with the user's answering rounds and the Ebbinghaus curve, the electronic device can quickly and accurately determine the memory time interval.

[0224] As an implementation method of the embodiment of the present application, the flowchart of assisting the user in memorizing can be as follows: Figure 9 As shown, the following steps may be specifically included:

[0225] S901, the article shows;

[0226] In order to enable the user to read the article, the data collection module 921 can display the article.

[0227] S902, capturing eye movement data;

[0228] In order to determine the temporal characteristics of the user's sight while reading the article, the data acquisition module 921 can capture eye movement data, wherein the eye movement data can be an image or video of the user's sight while reading the article.

[0229] S903, keyword extraction and matching;

[0230] In order to determine the related segments and unrelated segments in the reading content segments corresponding to the eye movement data, the sight line analysis module 931 can extract the word features corresponding to each sentence unit, cluster the sentence units according to the extracted features, and determine the category to which each sentence unit belongs.

[0231] S904, segmentation of associated segments and unassociated segments;

[0232] The sight line analysis module 931 can determine the sentence units in the reading content segment that belong to the same category as the sentence unit and the sentence unit as associated segments; and determine the sentence units in the reading content segment that belong to a category different from the sentence unit and the sentence unit as unrelated segments.

[0233] S905, obtaining sight timing information;

[0234] The sight line analysis module 931 can determine the user's sight point location based on the eye movement data. Combined with the timing information included in the eye movement data, the sight line analysis module 931 can determine the user's sight line timing information while reading the article. The sight line timing information can be the user's sight line trajectory.

[0235] S906, determining the predicted answer status;

[0236] The content update module 941 can input the associated fragments, unrelated fragments and line of sight timing information into the auxiliary memory model, so that the auxiliary memory model can extract the text features corresponding to the associated fragments and the unrelated fragments, extract the line of sight timing features corresponding to the line of sight timing information, and perform feature fusion on the text features and the line of sight timing features, and determine the user's predicted answer based on the fused features.

[0237] S907, update the question extraction ratio;

[0238] After determining the predicted answer situation, in order to increase the extraction ratio corresponding to questions with a lower correct answer rate in the predicted answer situation, the content update module 941 can update the question extraction ratio according to the predicted answer situation.

[0239] S908, topic extraction;

[0240] After determining the extraction ratio corresponding to each question, the content update module 941 can extract the questions so that the user can answer the questions.

[0241] S909, statistics of correct answers;

[0242] After the user completes the answer, the content assessment module 951 can count the correctness of the user's answer, wherein the correctness of the answer includes whether the answer result of the question is correct or incorrect.

[0243] S910, generating training data;

[0244] After determining whether the user's answer is correct, the content assessment module 951 can generate training data, which includes related segments, unrelated segments, sight timing information, and answer correctness.

[0245] S911, determine whether the correct rate of the most recent preset number of answers is greater than the preset correct rate; if yes, end; if not, execute step S912;

[0246] If the user's most recent correct answer rates for the preset number of times are all greater than the preset correct rate, it indicates that the user's memory of the article has reached the expected memory effect, and the loop can be stopped. If at least one of the user's most recent correct answer rates for the preset number of times is not greater than the preset correct rate, it indicates that the user's memory of the article has not reached the expected memory effect, and step S912 can be executed to continue the loop.

[0247] S912, update memory time interval;

[0248] When at least one of the correct rates of the user's most recent preset answers is not greater than the preset correct rate, the parameter updating module 961 may update the memory time interval according to the correctness of the user's answers and the rounds of answering the questions performed by the user.

[0249] S913, Update the mapping network of gaze behavior and memory

[0250] In order to retrain the auxiliary memory model, the parameter updating module 961 may update the parameters in the auxiliary memory model according to the training data determined in step S910. After waiting for the memory time interval, the loop is continued, that is, step S901 is returned to be executed.

[0251] It should be noted that in the technical solution of this application, the operations involved in obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0252] Corresponding to the above-mentioned sight-based auxiliary memory method, an embodiment of the present application also provides a sight-based auxiliary memory device. The sight-based auxiliary memory device provided in the embodiment of the present application is introduced below.

[0253] like Figure 10 As shown, a sight-based auxiliary memory device, the device comprising:

[0254] The image acquisition module 1001 is used to acquire an image of the user's sight line when reading an article;

[0255] An image determination module 1002 is configured to obtain, for each sentence unit in the article, a target image corresponding to the sentence unit from the sight line image, wherein the sentence unit is at least one continuous sentence, and the target image is the sight line image corresponding to the paragraph including the context of the sentence unit;

[0256] A trajectory determination module 1003 is configured to determine a sight line trajectory corresponding to the target image based on the sight line information of the user in the target image;

[0257] An answer status acquisition module 1004 is configured to acquire the question corresponding to each sentence unit based on a probability of correct answer, and acquire the correctness of the user's answer to the question, wherein the probability of correct answer is predicted by an auxiliary memory model;

[0258] The model training module 1005 is configured to train the auxiliary memory model based on the reading content segment, the eye tracking track, and the correctness of the answer, wherein the reading content segment is the paragraph including the context of the sentence unit, and the auxiliary memory model is configured to predict the probability of the user's correct answer to the question based on the text features of the input reading content segment and the eye tracking temporal features of the input eye tracking track;

[0259] An interval determination module 1006 is configured to determine a memory time interval based on the correctness of the answer and the rounds of answering performed by the user;

[0260] Return to execution module 1007, which is used to return to the step of obtaining the line of sight image of the user during reading the article after the length of the memory time interval has passed, until the correct answer meets the preset conditions, wherein the preset conditions represent that the user's memory of the article has achieved the expected memory effect.

[0261] It can be seen that in the embodiment of the present application, the electronic device can obtain a sight image of the user's sight situation during the reading process of the article; for each sentence unit in the article, obtain the target image corresponding to the sentence unit from the sight image, wherein the sentence unit is at least one continuous sentence, and the target image is the sight image corresponding to the paragraph including the context of the sentence unit; determine the sight track corresponding to the target image according to the sight information of the user in the target image; obtain the title corresponding to the article based on the probability of correct answer, and obtain the correct answer of the user to the title, wherein the probability of correct answer is predicted by the auxiliary memory model; based on the reading content fragment, sight track and The auxiliary memory model is trained based on the correctness of the answer, wherein the reading content segment is a paragraph that includes the context of the sentence unit. The auxiliary memory model is used to predict the probability of the user's correct answer to the question based on the input text features of the reading content segment and the input gaze timing features of the gaze trajectory. The memory interval is determined based on the correctness of the answer and the number of rounds in which the user answers the question. After the duration of the memory interval, the process returns to the step of obtaining a gaze image of the user's gaze during the reading process until the correctness of the answer meets a preset condition, wherein the preset condition indicates that the user's memory of the corresponding article has achieved the expected memory effect. Because the auxiliary memory model can predict the correctness of the user's answer to the question based on the gaze timing features of the gaze trajectory and the text features of the reading content segment, the output of the auxiliary memory model can reflect the user's memory quality. Since the user rereads the article and re-answers the question selected by the electronic device at each memory interval until the correctness of the answer meets the preset condition, the electronic device can assist the user in memorizing the article.

[0262] As an implementation of an embodiment of the present application, the auxiliary memory model includes a first branch network, a second branch network and a fully connected network;

[0263] The model training module 1005 may include:

[0264] a parameter input submodule, configured to input the reading content segment into the first branch network and the sight track into the second branch network, so that the first branch network extracts text features of the reading content segment and inputs the features into the fully connected network; the second branch network extracts sight timing features of the sight track and inputs the features into the fully connected network, so that the fully connected network fuses the text features and the sight timing features and outputs a correct prediction based on the fused features;

[0265] a function value determination submodule, configured to determine a loss function value of the auxiliary memory model according to a difference between the correct prediction and the correct answer;

[0266] The parameter adjustment submodule is used to adjust the parameters of the auxiliary memory model through back propagation based on the loss function value.

[0267] As an implementation of the embodiment of the present application, the above-mentioned device may further include:

[0268] A word feature extraction module, configured to extract word features from each sentence unit in the article;

[0269] a category determination module, configured to cluster the sentence units according to the extracted features and determine the category to which each sentence unit belongs;

[0270] an associated segment determining module, configured to determine sentence units in the reading content segment that belong to the same category as the sentence unit and the sentence unit as associated segments;

[0271] an unrelated segment determination module, configured to determine a sentence unit in the reading content segment that belongs to a category different from the category to which the sentence unit belongs, and the sentence unit, as an unrelated segment;

[0272] The parameter input submodule includes:

[0273] The parameter input unit is used to input the associated segment and the unassociated segment into the first branch network, so that the first branch network extracts text features of the associated segment and the unassociated segment respectively.

[0274] As an implementation of the embodiment of the present application, the answer status acquisition module 1004 may include:

[0275] The topic acquisition submodule is used to obtain a preset number of topics corresponding to each sentence unit;

[0276] The topic extraction submodule is used to extract multiple topics from a preset number of topics based on the probability of correct answers as topics corresponding to the article.

[0277] As an implementation method of the embodiment of the present application, the above-mentioned question extraction submodule may include:

[0278] a question extraction unit, configured to input, for each question corresponding to each sentence unit, a reading content segment corresponding to the sentence unit and a sight track corresponding to the reading content segment into the auxiliary memory model to obtain a predicted correct probability corresponding to the question;

[0279] a score determination unit, configured to use the inverse of the predicted correct probability as the score corresponding to the question;

[0280] an extraction ratio determination unit, configured to use the ratio between the score corresponding to the question and the total score of the question corresponding to each sentence unit as the extraction ratio corresponding to the question;

[0281] The topic extraction unit is used to extract at least one topic from a preset number of topics corresponding to each sentence unit according to the extraction ratio of the topics corresponding to each sentence unit.

[0282] As an implementation of an embodiment of the present application, the above-mentioned topic acquisition submodule may include:

[0283] A first question acquisition unit is configured to extract target keywords from each sentence unit, match the target keywords with keywords corresponding to each question in a pre-established question bank, obtain a matching degree between the sentence unit and each question, and obtain a preset number of questions with the highest matching degree;

[0284] The second question obtaining unit is used to determine a preset number of questions corresponding to each sentence unit according to a pre-established correspondence between the sentence units and the questions.

[0285] As an implementation of the embodiment of the present application, the interval determination module 1006 may include:

[0286] a correctness determination submodule, configured to use the ratio of the number of questions answered correctly to the number of questions corresponding to the article as the correctness of the answer;

[0287] The time interval determination submodule is configured to calculate the memory time interval Δ based on the answer accuracy rate, the number of answer rounds performed by the user, and the Ebbinghaus curve according to the following formula:

[0288] Δ=f(η)·g(k)

[0289] Wherein, η is the correct answer rate, g(k) is the Ebbinghaus curve, k is the answer round, and f(η) is the Sigmoid function after the vertical axis value is magnified.

[0290] As an implementation method of an embodiment of the present application, the above-mentioned preset condition is that the user's correct answer rate for the questions for a preset number of consecutive times is greater than the preset correct rate, wherein the correct answer rate is the ratio between the number of questions with correct answers and the number of questions corresponding to the article.

[0291] Corresponding to the above-mentioned sight-based auxiliary memory method, the embodiment of the present application also provides a sight-based auxiliary memory system. The sight-based auxiliary memory system provided by the embodiment of the present application is introduced below.

[0292] like Figure 11 As shown, a sight-based auxiliary memory system includes a server 1101 and a student terminal device 1102, wherein the student terminal device 1102 includes a display 1103 and a camera 1104, wherein:

[0293] The display 1103 is used to display the article that the student wants to read;

[0294] The camera 1104 is used to capture the sight line image of the student while reading the article;

[0295] The server 1101 is configured to obtain the sight line image; obtain a target image corresponding to each sentence unit in the article from the sight line image; determine a sight line trajectory corresponding to the target image based on the sight line information of the user in the target image; obtain the question corresponding to the article based on the probability of correct answer, and obtain the correctness of the user's answer to the question; train the auxiliary memory model based on the reading content segment, the sight line trajectory, and the correctness of the answer; determine a memory time interval based on the correctness of the answer and the rounds of answering the question by the user; after the length of the memory time interval has elapsed, trigger the display 1103 to display the article, trigger the camera 1104 to capture the sight line image of the student's sight line during the reading process, and return to the step of obtaining the sight line image of the user's sight line during the reading process until the correctness of the answer meets the preset condition;

[0296] Among them, the preset condition represents that the user's memory of the article has achieved the expected memory effect, the sentence unit is at least one continuous sentence, the target image is the sight image corresponding to the paragraph including the context of the sentence unit, the reading content segment is the paragraph including the context of the sentence unit, and the auxiliary memory model is used to predict the probability of the user's correct answer to the question based on the text features of the input reading content segment and the sight timing features of the input sight trajectory, and the correct answer probability is predicted by the auxiliary memory model.

[0297] It can be seen that in the embodiment of the present application, the electronic device can obtain a sight image of the user's sight situation during the reading process of the article; for each sentence unit in the article, obtain the target image corresponding to the sentence unit from the sight image, wherein the sentence unit is at least one continuous sentence, and the target image is the sight image corresponding to the paragraph including the context of the sentence unit; determine the sight track corresponding to the target image according to the sight information of the user in the target image; obtain the title corresponding to the article based on the probability of correct answer, and obtain the correct answer of the user to the title, wherein the probability of correct answer is predicted by the auxiliary memory model; based on the reading content fragment, sight track and The auxiliary memory model is trained based on the correctness of the answer, wherein the reading content segment is a paragraph that includes the context of the sentence unit. The auxiliary memory model is used to predict the probability of the user's correct answer to the question based on the input text features of the reading content segment and the input gaze timing features of the gaze trajectory. The memory interval is determined based on the correctness of the answer and the number of rounds in which the user answers the question. After the duration of the memory interval, the process returns to the step of obtaining a gaze image of the user's gaze during the reading process until the correctness of the answer meets a preset condition, wherein the preset condition indicates that the user's memory of the corresponding article has achieved the expected memory effect. Because the auxiliary memory model can predict the correctness of the user's answer to the question based on the gaze timing features of the gaze trajectory and the text features of the reading content segment, the output of the auxiliary memory model can reflect the user's memory quality. Since the user rereads the article and re-answers the question selected by the electronic device at each memory interval until the correctness of the answer meets the preset condition, the electronic device can assist the user in memorizing the article.

[0298] As an implementation method of the embodiment of the present application, the above system may further include a teacher terminal device;

[0299] The teacher-side device is used to obtain the preset number of times and the preset accuracy rate input by the teacher, and is used to receive the correct answer status sent by the server 1101 and display the correct answer status;

[0300] The server 1101 is further configured to determine, based on the preset number of times and the preset accuracy rate, that the preset condition is that the user's accuracy rate in answering questions for a preset number of consecutive times is greater than the preset accuracy rate.

[0301] As an implementation method of the embodiment of the present application, the student terminal device 1102 may also include a wristband;

[0302] The server 1101 is further configured to send a reminder instruction to the wristband after the memory time interval has elapsed;

[0303] The wristband is used to output a prompt message when receiving the reminder instruction to prompt the student to read the article.

[0304] As an implementation of the embodiment of the present application, another structural diagram of the sight-based auxiliary memory system can be as follows: Figure 12 As shown, it can specifically include a server end 1201, a student end device 1202 and a teacher end device 1203.

[0305] Server 1204 in server 1201 can send the article a student needs to memorize to teaching tablet 1206, which then displays the article. As the student reads the article, a wide-angle camera on top of teaching tablet 1206 can record the student's line of sight. After the student finishes reading, teaching tablet 1206 can send student learning data 1205 to server 1201, including the line of sight image.

[0306] The server 1204 can obtain the student learning data 1205 sent by the teaching tablet 1206, and for each sentence unit in the article, obtain the target image corresponding to the sentence unit from the line of sight image in the student learning data 1205, and determine the line of sight trajectory corresponding to the target image based on the line of sight information of the user in the target image.

[0307] Server 1204 can obtain the corresponding question for each sentence unit and send the obtained question to teaching tablet 1206, so that teaching tablet 1206 can display the received question. After the student completes the answer to the question on the teaching tablet, server 1204 can obtain the reading data sent by teaching tablet 1206. The student learning data 1205 also includes the correctness of the student's answer to the question. The reading data includes student learning data 1205.

[0308] The server 1204 can train the auxiliary memory model based on the reading content fragments, eye trajectory, and correct answer situation. The memory time interval is determined based on the correct answer situation and the round of answering in which the user answers the question. After the length of the memory time interval has passed, the server 1204 can trigger the teaching tablet 1206 to display the article; send a clock reminder signal to the teaching bracelet 1207 to make the teaching bracelet 1207 remind the student to read the article displayed on the teaching tablet 1206; trigger the wide-angle camera on the top of the teaching tablet 1206 to capture the sight image of the student's sight situation during the reading process. And the server 1204 can return to the step of obtaining the student learning data 1205 sent by the teaching tablet 1206 until the student's correct answer situation meets the preset conditions.

[0309] The server 1204 can send the student's answer accuracy to the teacher's terminal device 1203, so that the learning situation statistics device 1208 can display the student's answer accuracy. Among them, the student's answer accuracy can reflect the student's mastery of the article, and the learning situation statistics device 1208 can be a computer. The teacher can use the learning situation statistics device 1208 to collect statistics on the student's mastery of the article and adjust the memory plan based on the student's mastery level. Specifically, the teacher can send an adjustment parameter instruction to adjust the parameters of the preset conditions through the learning situation statistics device 1208. The learning situation statistics device 1208 can provide the teacher with a recommended teaching plan based on the received student's mastery level.

[0310] The present application also provides an electronic device, such as Figure 13 Shown, including:

[0311] Memory 1301, used for storing computer programs;

[0312] The processor 1302 is configured to implement the sight-based auxiliary memory method described in any of the above embodiments when executing the program stored in the memory 1301.

[0313] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1302, the communication interface, and the memory 1301 communicate with each other via the communication bus.

[0314] It can be seen that in the embodiment of the present application, the electronic device can obtain a sight image of the user's sight situation during the reading process of the article; for each sentence unit in the article, obtain the target image corresponding to the sentence unit from the sight image, wherein the sentence unit is at least one continuous sentence, and the target image is the sight image corresponding to the paragraph including the context of the sentence unit; determine the sight track corresponding to the target image according to the sight information of the user in the target image; obtain the title corresponding to the article based on the probability of correct answer, and obtain the correct answer of the user to the title, wherein the probability of correct answer is predicted by the auxiliary memory model; based on the reading content fragment, sight track and The auxiliary memory model is trained based on the correctness of the answer, wherein the reading content segment is a paragraph that includes the context of the sentence unit. The auxiliary memory model is used to predict the probability of the user's correct answer to the question based on the input text features of the reading content segment and the input gaze timing features of the gaze trajectory. The memory interval is determined based on the correctness of the answer and the number of rounds in which the user answers the question. After the duration of the memory interval, the process returns to the step of obtaining a gaze image of the user's gaze during the reading process until the correctness of the answer meets a preset condition, wherein the preset condition indicates that the user's memory of the corresponding article has achieved the expected memory effect. Because the auxiliary memory model can predict the correctness of the user's answer to the question based on the gaze timing features of the gaze trajectory and the text features of the reading content segment, the output of the auxiliary memory model can reflect the user's memory quality. Since the user rereads the article and re-answers the question selected by the electronic device at each memory interval until the correctness of the answer meets the preset condition, the electronic device can assist the user in memorizing the article.

[0315] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0316] The communication interface is used for communication between the above electronic device and other devices.

[0317] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0318] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0319] In another embodiment provided in the present application, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned sight-based auxiliary memory methods are implemented.

[0320] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any sight-based auxiliary memory method in the above embodiments.

[0321] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), etc.

[0322] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0323] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device, system computer-readable storage medium, and computer program product embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0324] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A sight-based auxiliary memory method, characterized in that: The method comprises: Obtaining a sight line image of the user while reading the article; For each sentence unit in the article, obtaining a target image corresponding to the sentence unit from the sight line image, wherein the sentence unit is at least one continuous sentence, and the target image is the sight line image corresponding to the paragraph including the context of the sentence unit; Determining a sight line trajectory corresponding to the target image according to the sight line information of the user in the target image; Obtaining a title corresponding to the article based on a probability of correct answer, and obtaining the correctness of the user's answer to the title, wherein the probability of correct answer is predicted by an auxiliary memory model; The auxiliary memory model is trained based on the reading content segment, the eye tracking track, and the correctness of the answer, wherein the reading content segment is the paragraph including the context of the sentence unit, and the auxiliary memory model is used to predict the probability of the user answering the question correctly based on the text features of the input reading content segment and the eye tracking temporal features of the input eye tracking track; Determining a memory time interval based on the correctness of the answers and the rounds of the user answering the questions; After the duration of the memory time interval has passed, return to the step of obtaining the sight image of the user's sight during reading the article until the correct answer meets the preset conditions, wherein the preset conditions indicate that the user's memory of the article has achieved the expected memory effect.

2. The method according to claim 1, characterized in that The auxiliary memory model includes a first branch network, a second branch network and a fully connected network; The step of training the auxiliary memory model based on the reading content segment, the eye tracking track and the correct answer status includes: Inputting the reading content segment into the first branch network and the gaze trajectory into the second branch network, so that the first branch network extracts text features of the reading content segment and inputs the features into the fully connected network; the second branch network extracts sight timing features of the sight trajectory and inputs the features into the fully connected network, so that the fully connected network fuses the text features and the sight timing features and outputs a correct prediction based on the fused features; Determining a loss function value of the auxiliary memory model according to a difference between the correct prediction and the correct answer; Based on the loss function value, the parameters of the auxiliary memory model are adjusted through back propagation.

3. The method according to claim 2, characterized in that Before the step of training the auxiliary memory model based on the reading content segment, the eye tracking track and the correct answer status, the method further includes: Extracting word features for each sentence unit in the article; Clustering the sentence units according to the extracted features to determine the category to which each sentence unit belongs; Determining sentence units in the reading content segment that belong to the same category as the sentence unit and the sentence unit as associated segments; Determining sentence units in the reading content segment that belong to a category different from the category to which the sentence unit belongs, as well as the sentence unit, as unrelated segments; The step of inputting the reading content segment into the first branch network includes: The associated segment and the unassociated segment are input into the first branch network, so that the first branch network extracts text features of the associated segment and the unassociated segment respectively.

4. The method according to claim 1, wherein The step of obtaining the title corresponding to the article based on the probability of correct answer includes: Get the preset number of questions corresponding to each sentence unit; From a preset number of questions, multiple questions are selected based on the probability of correct answers as the questions corresponding to the article.

5. The method according to claim 4, characterized in that The step of extracting a plurality of questions from a preset number of questions based on the probability of correct answers as the questions corresponding to the article includes: For each question in the questions corresponding to each sentence unit, the reading content segment corresponding to the sentence unit corresponding to the question and the eye tracking trajectory corresponding to the reading content segment are input into the auxiliary memory model to obtain the prediction correct probability corresponding to the question; The reciprocal of the correct prediction probability is used as the score corresponding to the question; The ratio between the score corresponding to the question and the total score of the questions corresponding to each sentence unit is used as the extraction ratio corresponding to the question; According to the extraction ratio of the topics corresponding to each sentence unit, at least one topic is extracted from the preset number of topics corresponding to each sentence unit.

6. The method according to claim 4 or 5, characterized in that The step of obtaining a preset number of questions corresponding to each sentence unit includes: For each sentence unit, extract the target keyword of the sentence unit, match the target keyword with the keyword corresponding to each question in the pre-established question bank, obtain the matching degree between the sentence unit and each question, and obtain a preset number of questions with the highest corresponding matching degree; or, According to the pre-established correspondence between sentence units and questions, a preset number of questions corresponding to each sentence unit is determined.

7. The method according to any one of claims 1 to 5, characterized in that The step of determining the memory time interval according to the correctness of the answer and the answering round of the user includes: The ratio between the number of questions answered correctly and the number of questions corresponding to the article is used as the answer accuracy rate; Based on the answer accuracy rate, the number of rounds the user takes to answer the question, and the Ebbinghaus curve, the memory time interval Δ is calculated according to the following formula: Δ=f(η)·g(k) Wherein, η is the correct answer rate, g(k) is the Ebbinghaus curve, k is the answer round, and f(η) is the Sigmoid function after the vertical axis value is magnified.

8. The method according to any one of claims 1 to 5, characterized in that The preset condition is that the user's correct answer rate for the questions for a preset number of consecutive times is greater than a preset correct rate, wherein the correct answer rate is the ratio between the number of questions with correct answers and the number of questions corresponding to the article.

9. A sight-based auxiliary memory device, characterized in that: The device comprises: An image acquisition module is used to acquire an image of the user's sight while reading an article; an image determination module, configured to obtain, for each sentence unit in the article, a target image corresponding to the sentence unit from the sight line image, wherein the sentence unit is at least one continuous sentence, and the target image is the sight line image corresponding to the paragraph including the context of the sentence unit; a trajectory determination module, configured to determine a sight line trajectory corresponding to the target image based on the sight line information of the user in the target image; An answer status acquisition module, configured to acquire the title corresponding to the article based on a probability of correct answer, and acquire the correctness of the user's answer to the title, wherein the probability of correct answer is predicted by an auxiliary memory model; a model training module, configured to train the auxiliary memory model based on a reading content segment, the eye tracking track, and the correctness of the answer, wherein the reading content segment is the paragraph including the context of the sentence unit, and the auxiliary memory model is configured to predict the probability of the user answering the question correctly based on the text features of the input reading content segment and the eye tracking temporal features of the input eye tracking track; An interval determination module, configured to determine a memory time interval based on the correctness of the answer and the number of rounds of answering performed by the user; The return execution module is used to return to the step of obtaining the sight image of the user's sight during reading the article after the length of the memory time interval has passed, until the correct answer meets the preset conditions, wherein the preset conditions represent that the user's memory of the article has achieved the expected memory effect.

10. The device according to claim 9, characterized in that The auxiliary memory model includes a first branch network, a second branch network and a fully connected network; The model training module includes: a parameter input submodule, configured to input the reading content segment into the first branch network and the sight track into the second branch network, so that the first branch network extracts text features of the reading content segment and inputs the features into the fully connected network; the second branch network extracts sight timing features of the sight track and inputs the features into the fully connected network, so that the fully connected network fuses the text features and the sight timing features and outputs a correct prediction based on the fused features; a function value determination submodule, configured to determine a loss function value of the auxiliary memory model according to a difference between the correct prediction and the correct answer; A parameter adjustment submodule, configured to adjust the parameters of the auxiliary memory model by back propagation based on the loss function value; The device further comprises: A word feature extraction module, configured to extract word features from each sentence unit in the article; a category determination module, configured to cluster the sentence units according to the extracted features and determine the category to which each sentence unit belongs; an associated segment determining module, configured to determine sentence units in the reading content segment that belong to the same category as the sentence unit and the sentence unit as associated segments; an unrelated segment determination module, configured to determine a sentence unit in the reading content segment that belongs to a category different from the category to which the sentence unit belongs, and the sentence unit, as an unrelated segment; The parameter input submodule includes: a parameter input unit, configured to input the associated segment and the unassociated segment into the first branch network, so that the first branch network extracts text features of the associated segment and the unassociated segment respectively; The answer status acquisition module includes: The topic acquisition submodule is used to obtain a preset number of topics corresponding to each sentence unit; A topic extraction submodule is used to extract multiple topics from a preset number of topics based on the probability of correct answers as topics corresponding to the article; The topic extraction submodule includes: a question extraction unit, configured to input, for each question corresponding to each sentence unit, a reading content segment corresponding to the sentence unit and a sight track corresponding to the reading content segment into the auxiliary memory model to obtain a predicted correct probability corresponding to the question; a score determination unit, configured to use the inverse of the predicted correct probability as the score corresponding to the question; an extraction ratio determination unit, configured to use the ratio between the score corresponding to the question and the total score of the question corresponding to each sentence unit as the extraction ratio corresponding to the question; a topic extraction unit, configured to extract at least one topic from a preset number of topics corresponding to each sentence unit according to an extraction ratio of topics corresponding to each sentence unit; The topic acquisition submodule includes: A first question acquisition unit is configured to extract target keywords from each sentence unit, match the target keywords with keywords corresponding to each question in a pre-established question bank, obtain a matching degree between the sentence unit and each question, and obtain a preset number of questions with the highest matching degree; A second topic acquisition unit is used to determine a preset number of topics corresponding to each sentence unit based on a pre-established correspondence between the sentence units and the topics; The interval determination module includes: a correctness determination submodule, configured to use the ratio of the number of questions answered correctly to the number of questions corresponding to the article as the correctness of the answer; The time interval determination submodule is configured to calculate the memory time interval Δ based on the answer accuracy rate, the number of answer rounds performed by the user, and the Ebbinghaus curve according to the following formula: Δ=f(η)·g(k) Wherein, η is the correct answer rate, g(k) is the Ebbinghaus curve, k is the answer round, and f(η) is the Sigmoid function after the vertical axis value is magnified; The preset condition is that the user's correct answer rate for the questions for a preset number of consecutive times is greater than a preset correct rate, wherein the correct answer rate is the ratio between the number of questions with correct answers and the number of questions corresponding to the article.

11. A sight-based auxiliary memory system, characterized in that: The system includes a server and a student-end device, wherein the student-end device includes a display and a camera, wherein: The display is used to display the article that the student wants to read; The camera is used to capture an image of the student's sight while reading the article; The server is configured to obtain the sight line image; for each sentence unit in the article, obtain a target image corresponding to the sentence unit from the sight line image; determine a sight line trajectory corresponding to the target image based on the sight line information of the user in the target image; obtain the question corresponding to the article based on the probability of correct answer, and obtain the correctness of the user's answer to the question; train the auxiliary memory model based on the reading content segment, the sight line trajectory and the correctness of the answer; determine the memory time interval based on the correctness of the answer and the answer rounds of the user; after the length of the memory time interval has passed, trigger the display to display the article, trigger the camera to capture the sight line image of the student's sight line during the reading process, and return to the step of obtaining the sight line image of the user's sight line during the reading process until the correctness of the answer meets the preset condition; Among them, the preset condition represents that the user's memory of the article has achieved the expected memory effect, the sentence unit is at least one continuous sentence, the target image is the sight image corresponding to the paragraph including the context of the sentence unit, the reading content segment is the paragraph including the context of the sentence unit, and the auxiliary memory model is used to predict the probability of the user's correct answer to the question based on the text features of the input reading content segment and the sight timing features of the input sight trajectory, and the correct answer probability is predicted by the auxiliary memory model.

12. The system according to claim 11, wherein: The system also includes a teacher terminal device; The teacher-side device is used to obtain the preset number of times and the preset accuracy rate input by the teacher, and to receive the correct answer status sent by the server and display the correct answer status; The server is further configured to determine, based on the preset number of times and the preset accuracy rate, that the preset condition is that the user's accuracy rate in answering questions for a preset number of consecutive times is greater than the preset accuracy rate.

13. The system according to claim 11 or 12, characterized in that The student-end device also includes a wristband; The server is further configured to send a reminder instruction to the wristband after the memory time interval has elapsed; The wristband is used to output a prompt message when receiving the reminder instruction to prompt the student to read the article.

14. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 8 when executing a program stored in a memory.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images