A personalized text summary generation method and system integrating eye movement data

Abstract

This invention discloses a method and system for generating personalized text summaries that integrates eye movement data. The system collects eye movement data from users reading text on a screen and captures a text image interface. The system then analyzes the eye movement data and identifies the coordinates of individual characters in the text image. A gaze-to-word mapping method is then constructed. Word scores are then calculated based on eye movement fixations, saccades, and reflexes, and keywords are extracted based on the scores. After keyword extraction, the keywords are grouped into eye movement key information, encoded using a bidirectional long-short-term memory network, and an internal attention model is used to address duplicate words generated during the summary generation process. The system then inputs the keywords into a pointer generator network to generate a personalized text summary. The text summary generated by this method achieves significant personalization, scoring high in relevant evaluation metrics. It integrates user personalized information while reducing the number of manual steps required by the user.

Description

Technical Field

[0001] The present application belongs to the technical field of automatic text summarization, and specifically relates to a method and system for generating personalized text summaries by integrating eye movement data. Background Art

[0002] Automatic text summarization is an important research area in artificial intelligence, information retrieval, and natural language processing. Summaries can help users efficiently find the information they need, concisely convey the main points of an article, and quickly review previously read articles. Automatic text summarization can be categorized into extractive summarization, generative summarization, and a fusion of extractive and generative summarization. Extractive summarization directly selects a few sentences from the source text as summaries based on various metrics. However, such summaries fail to summarize the entire text. Generative summaries generate words that are not present in the source text but convey similar information to the user. However, such summaries may replace specific nouns in the source text, such as places or names, thus altering the source text's content. Methods that fuse extractive and generative summarization introduce a generation probability mechanism to determine whether to copy words from the source text or select words with similar meanings from a vocabulary. Other methods first select a few sentences from the source text and then combine these sentences into a new text as input to the summary generation module, generating a fusion of extractive and generative summaries. These methods only generate the same summary for the same text, lacking differentiation and personalization.

[0003] In recent years, there has been an endless stream of research on generating personalized text summaries. There are methods that use weighting based on keyword vectors, where users must manually insert keywords and weights into the model; methods that extract important sentences from documents based on personalized data, where users need to manually add data to the text, and the generated sentences can only come from the source text; methods that generate text summaries based on user profile information, where users need to fill in their personal information and update it promptly; and methods that use eye movement heat maps to extract highly significant sentences and combine them for text compression to generate personalized summaries. However, this method of extracting sentences cannot accurately express the user's interests, and the sentences in the summary may be incoherent. Therefore, compared to previous methods of generating personalized text summaries, generating personalized text summaries faces the challenge of how to automatically and accurately capture user preferences.

[0004] Reading is not only a crucial psychological activity for humans, but also a crucial means for acquiring information from the outside world. During reading, the eyes continuously move between different locations within a text to extract and integrate relevant information, ultimately achieving comprehension of the entire text. Different users exhibit distinct reading patterns due to their diverse backgrounds and preferences. Analyzing eye movement data and the visual attention behaviors it represents while reading, and generating personalized text summaries based on this data, is of great value in advancing natural language processing, information retrieval, and other fields. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for generating personalized text summaries by integrating eye movement data, so as to solve the technical problems in the prior art of lack of generation of personalized text summaries and low quality of generated personalized text summaries.

[0006] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0007] A personalized text summary generation method integrating eye movement data, comprising:

[0008] Collect eye movement data of users reading text on the screen and capture text image interface;

[0009] Recognize the text in the captured text image interface and determine the coordinate range of each word in the image. Combined with the coordinates of the user's eye gaze point when reading the text, a gaze point-word mapping method is constructed.

[0010] Analyze the eye movement data collected while users read text on the screen, extract eye movement features during reading, remove stop words, calculate the fixation time ratio and return time ratio of each word based on different eye movement features, and extract the top k words with the highest ratio as text keywords;

[0011] The extracted text keywords are combined into eye movement key information, which is encoded using a bidirectional long short-term memory network. Combined with the internal attention model, the context vector is calculated. The encoded eye movement key information and context vector are input into the pointer generator network to calculate the final probability distribution of the words, forming a text summary model and generating personalized text summaries.

[0012] Furthermore, the gaze point-word mapping method includes:

[0013] The fixation point is mapped to a single character, and the fixation point sequence is converted into a character sequence by calculating and matching the fixation point coordinates and the recognized single character coordinate range;

[0014] A single character is mapped to the corresponding word, and the text is divided into words using the word segmentation method. The character sequence and the word sequence are matched in string order, and the character sequence is converted into the corresponding word sequence.

[0015] Furthermore, the eye movement features include eye fixations, eye saccades, and eye returns. Eye movement features are analyzed using a word sequence converted from fixation points. Consecutive identical words in the word sequence represent consecutive fixations, and the number of consecutive word appearances represents the number of consecutive fixations. When a word in the word sequence appears at intervals, the coordinates of the fixation points corresponding to the preceding and following words are compared. When the coordinate difference between the preceding and following words in the sequence is within a certain threshold, it is considered that the words are at the same position in the text, and a return behavior is considered to have occurred at this time.

[0016] Furthermore, the calculation function used to calculate the fixation time ratio and the return time ratio of each word is as follows:

[0017]

[0018] Among them, f w represents the fixation time of word w, r w represents the look-back time of word w, n represents the word number in the text, PF(w) represents the fixation time ratio of word w, f i represents the fixation time of the i-th word in the text, PR(w) represents the return time ratio of word w, r i Indicates the lookback time of the i-th word in the text.

[0019] Furthermore, the text summarization model integrates eye movement key information and attention mechanism, and the calculation function adopted is as follows:

[0020]

[0021] Among them, h i Represents the hidden layer state of the input sequence, s t Denotes the decoder hidden state, h k Indicates the hidden state of key information of eye movement, v, W h ,W s , W k is the training parameter, e i t represents the attention weight distribution.

[0022] The present application also proposes a personalized text summary generation system that integrates eye movement data, including a processor and a memory storing a plurality of computer instructions. When the computer instructions are executed by the processor, the steps of the personalized text summary generation method that integrates eye movement data are implemented.

[0023] This application proposes a personalized text summary generation method and system that integrates eye movement data. This method analyzes the user's eye movement data while reading text and identifies the text content on the screen. It extracts relevant eye movement features as indicators for extracting text keywords. It then uses a bidirectional long-short-term memory network to encode the key eye movement information that constitutes the keywords, thereby achieving personalized text summary generation by combining an internal attention model and a pointer generator network. This application combines eye movement tracking with text image recognition, extracting text keywords based on eye movement data during reading to guide personalized text summary generation, thereby improving the quality and personalization of text summaries.

[0024] The beneficial effects of this application are mainly reflected in: (1) it solves the problem that the personalization effect of the existing text summary model is not obvious, integrates eye movement data to extract keywords to guide text summary generation, and improves the personalization effect of the model and the quality of text summary; (2) it solves the problem that the previous personalized text automatic summary system requires users to manually fill in personal information, and is a non-contact real-time text summary generation method; (3) it can realize the recording and tracking of users' eye movement data and visual attention to the corresponding text. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of a personalized text summary generation method that integrates eye movement data according to the present invention;

[0026] Figure 2 Schematic diagram of the text keyword extraction process of the present invention;

[0027] Figure 3 Schematic diagram of the personalized text summary generation process of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0029] The overall idea of this application is that the eye movement data of reading texts contains a lot of usable information, among which the user's gaze, scan and look back characteristics show the cognitive processing process during reading. Compared with traditional methods, this application can naturally generate personalized text summaries based on user characteristics, by extracting eye movement characteristics during reading, combining with text content, analyzing the words that the user is interested in, and extracting personalized keywords. In addition, the eye movement key information is composed of keywords to guide the generation of personalized text summaries. After the eye movement key information is encoded by a bidirectional long short-term memory network, it is combined with an internal attention model and input into the pointer generator network to generate a personalized text summary. The internal attention model can eliminate repeated words in the summary generation process, and the pointer generator network can solve the problem of unregistered words. Combining text keyword extraction and text summary generation, this application can construct a text summary generation model that integrates eye movement data to achieve personalized text summary generation.

[0030] In one embodiment, Figure 1 As shown in FIG, a personalized text summary generation method integrating eye movement data is provided, including:

[0031] Step S1: collecting eye movement data of a user reading text on a screen and capturing a text image interface.

[0032] This application uses an eye tracking device to capture the user's eye movement data when reading text, and at the same time captures the text image on the screen in the background.

[0033] Step S2: Recognize the characters in the captured text image interface, determine the coordinate range of each character in the image, and construct a gaze point-word mapping method based on the coordinates of the user's eye gaze point when reading the text.

[0034] Specifically, the mapping from gaze points to individual characters is calculated and matched based on the gaze point coordinates and the coordinate range of the recognized individual characters. If the gaze point falls within a certain character coordinate range, the gaze point is mapped to that character, completing the conversion from gaze point sequence to character sequence. The mapping of individual characters to their corresponding words uses a word segmentation method to segment the recognized text into individual words. The character sequence is then matched with the word sequence in string order, and the character sequence is converted into the corresponding word sequence, completing the construction of a complete gaze point-to-word mapping process.

[0035] Step S3: Analyze the eye movement data collected when the user reads the text on the screen, extract the eye movement features during reading, remove stop words, calculate the fixation time ratio and return time ratio of each word based on different eye movement features, and extract the top k words with the highest ratio as text keywords.

[0036] Eye movement features include eye gaze, eye scan and eye return features. Different features reflect different cognitive processing situations of users. The score of each word in the text is calculated based on the features.

[0037] Specifically, such as Figure 2 As shown in the figure, the analysis of eye movement features uses a word sequence converted from the fixation point. Consecutive identical words in the word sequence represent continuous fixations, and the number of consecutive word appearances is the number of consecutive fixations. When a word in the word sequence appears at intervals, the coordinates of the fixation points corresponding to the preceding and following words are compared. When the coordinate difference between the preceding and following words in the sequence is within a certain threshold, it is considered that these are words at the same position in the text, and a return gaze behavior is considered to have occurred at this time. Then, the proportion of fixation time and return gaze time of the word are calculated, and the calculation function used is as follows:

[0038]

[0039] Among them, f w represents the fixation time of word w, r w represents the look-back time of word w, n represents the word number in the text, PF(w) represents the fixation time ratio of word w, f i represents the fixation time of the i-th word in the text, PR(w) represents the return time ratio of word w, r i Indicates the lookback time of the i-th word in the text.

[0040] It should be noted that stop words have a certain impact on the calculation results of words, and Chinese and English stop words should be removed for calculation.

[0041] Step S4: The extracted text keywords are combined into eye movement key information, which is encoded using a bidirectional long short-term memory network. Combined with the internal attention model, the context vector is calculated, and the encoded eye movement key information and context vector are input into the pointer generator network to calculate the final probability distribution of the words, forming a text summary model and generating a personalized text summary.

[0042] like Figure 3 As shown in the figure, this step combines the extracted text keywords into eye movement key information. After being encoded by the bidirectional long short-term memory network, the hidden layer state is represented as Then it is added to the network structure and combined with the attention mechanism to guide text summary generation. The calculation function used is as follows:

[0043]

[0044] Among them, h i Represents the hidden layer state of the input sequence, s t Denotes the decoder hidden state, h kIndicates the hidden state of key information of eye movement, v, W h ,W s , W k are training parameters, represents the attention weight distribution.

[0045] Use the inner attention model to convert the attention weights Normalize and define a new weight distribution e′ ti , calculate the new context vector and The calculated results are input into the pointer generator network to Calculate the probability distribution of the output of the words in the vocabulary. The probability P generated at time step t gen The calculation function of ∈[0,1] is as follows:

[0046]

[0047] Among them, sigmoid represents the activation function, s t is the decoder state, h k is the hidden layer state of the key information of eye movement, w c e , w c d , w s , w k Represents training parameters.

[0048] Next, P gen Used as a soft switch to pass from P vocab Sampling is used to generate words from the vocabulary, or to copy the words of the source text by sampling from the attention distribution. The following probability distribution is obtained through the vocabulary and the source text vocabulary. The calculation function is as follows:

[0049]

[0050] Among them, P input (w) represents the probability distribution of words copied from the source text through the attention distribution. If w is an out-of-vocabulary word, then P vocab (w) is 0, then the model selects words from the source text as output; if w does not exist in the source text, then P input If (w) is 0, the model selects words from the vocabulary as output.

[0051] The final generation probability distribution of words is calculated through the above formula. When the text summary model generates words, the words with the highest probability are output, thereby realizing personalized text summary generation by integrating eye movement data.

[0052] When in use, the user reads the text on the screen, and a heat map and personalized text summary are generated based on the eye movement data collected during reading. Compared with other text summarization methods, the text summary generated by the method of the present invention has a personalized effect and is close to the text summary written by the user himself.

[0053] In one embodiment, the present application also provides a personalized text summary generation system that integrates eye movement data, including a processor and a memory storing a plurality of computer instructions. When the computer instructions are executed by the processor, the steps of the personalized text summary generation method that integrates eye movement data are implemented.

[0054] The specific limitations of the personalized text summary generation system that integrates eye movement data can be found in the limitations of the personalized text summary generation method that integrates eye movement data above, and will not be repeated here. The personalized text summary generation system that integrates eye movement data can be implemented in whole or in part through software, hardware, or a combination thereof. It can be embedded in or independent of the processor in the computer device in hardware form, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the above corresponding operations.

[0055] The memory and processor are electrically connected, directly or indirectly, to enable data transmission or interaction. For example, these components may be electrically connected via one or more communication buses or signal lines. The memory stores a computer program executable on the processor, and the processor executes the computer program stored in the memory to implement the network topology layout method according to the embodiments of the present invention.

[0056] The memory may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store a program, and the processor executes the program after receiving an execution instruction.

[0057] The processor may be an integrated circuit chip with data processing capabilities. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor.

[0058] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A personalized text summary generation method integrating eye movement data, characterized in that: The following steps are involved: 1) Collect eye movement data of users reading text on the screen and capture the text image interface; 2) Recognize the characters in the captured text image interface and determine the coordinate range of each character in the image. Combined with the coordinates of the user's eye gaze point when reading the text, a gaze point-word mapping method is constructed; 3) Analyze the eye movement data collected while users read text on the screen, extract eye movement features during reading, remove stop words, calculate the fixation time ratio and return time ratio for each word based on different eye movement features, and extract the top k words with the highest ratio as text keywords; The calculation function used to calculate the fixation time ratio and return time ratio of each word is as follows: Among them, f w represents the fixation time of word w, r w represents the look-back time of word w, n represents the word number in the text, PF(w) represents the fixation time ratio of word w, f i represents the fixation time of the i-th word in the text, PR(w) represents the return time ratio of word w, r i represents the lookback time of the i-th word in the text; The text summarization model described above combines eye movement key information and attention mechanism, and the calculation function used is as follows: Among them, h i represents the hidden layer state of the input sequence, s t represents the decoder hidden state, h k Indicates the hidden state of key information of eye movement, v,W h ,W s , W k is the training parameter, e i t represents the attention weight; 4) The extracted text keywords are combined into eye movement key information, which is encoded using a bidirectional long short-term memory network. Combined with the internal attention model, the context vector is calculated. The encoded eye movement key information and context vector are input into the pointer generator network to calculate the final probability distribution of the words, forming a text summary model and generating a personalized text summary.

2. The personalized text summary generation method based on eye movement data according to claim 1 is characterized in that: In step 2), the fixation-word mapping method includes: 2.1) Mapping gaze points to single characters. Matching the gaze point coordinates with the recognized single character coordinate range to convert the gaze point sequence into a character sequence. 2.2) A single character is mapped to its corresponding word. The text is segmented into words using a word segmentation method. The character sequence is matched with the word sequence in string order, and the character sequence is converted into the corresponding word sequence.

3. The personalized text summary generation method based on eye movement data fusion according to claim 1 is characterized in that: In step 3), the eye movement features include eye gaze, eye saccade and eye return.

4. A personalized text summary generation system integrating eye movement data, comprising a processor and a memory storing a plurality of computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images