A system that uses eye tracking to assist paralyzed aphasics in language output

Abstract

A system that assists paralyzed aphasics in language output through eye tracking. The system learns and analyzes the objects that the user has gazed at, the language and non-language sounds in the external environment, and the language text output by the user at the moment, and stores them in the system's dictionary model. When the user gazes at a series of objects in a certain order, or collects external environmental sounds, the system will display the candidate texts in the user's dictionary model corresponding to the objects and external environmental sounds through a display device for the user to select and convert into sounds to be emitted through a speaker. The system also provides preset emergency texts that can be used in emergency situations. The present invention enables users to more efficiently convert the surrounding environment into language output that is closer to their habits by learning the environmental objects and environmental sound inputs around the user themselves, as well as the user's own language history habits, so that users such as paralyzed aphasics can conduct basic and important language communication with the outside world.

Description

Technical Field

[0001] The present invention relates to an image recognition and language input technology, belonging to the field of language information technology, and specifically to a system for assisting paralyzed aphasic people in language output through eye tracking. Background Art

[0002] In reality, there are cases where patients lose their language output. For example, patients who cannot speak with their mouths and whose hands and feet are paralyzed, and patients whose vocal organs are intubated and whose hands and feet are inconvenient to move. These patients can neither speak nor write, and can only communicate inefficiently through primitive methods such as asking others and blinking.

[0003] The eye-tracking keyboard input method of China's iFlytek and the United States' Microsoft has a general principle: by having the user wear a glasses-like screen, a virtual keyboard is displayed on the screen, and then the user's gaze is tracked through an eye tracker to obtain the keys on the keyboard that the user is looking at to realize the virtual keyboard typing function.

[0004] Problems:

[0005] (1) There are users who cannot use keyboard input or are inefficient in using keyboard input, such as people who do not know how to use Pinyin or other keyboard input methods, people with poor eyesight who cannot easily distinguish closely arranged letter keys, etc. Therefore, such users are not suitable for the above virtual keyboard.

[0006] (2) The above-mentioned existing virtual keyboard input fails to consider the interaction between the user and the surrounding environment, such as the language content spoken by the interlocutor, the objects or texts seen in the surrounding environment, the non-language sounds heard in the surrounding environment, etc.

[0007] (3) The above-mentioned existing virtual keyboard input fails to directly provide users with personalized interactive information. For example: the patient's most urgent pain, breathing difficulties, toileting, eating and drinking information; the names of the patient's relatives, common items, and frequently visited places; the patient's emotional information such as joy, anger, sorrow, and happiness. Summary of the invention

[0008] The present invention aims to solve the problem that some users do not know how to use keyboard input method or the keyboard input method is inefficient due to vision problems, and thus cannot use glasses-type screens and eye trackers to perform language input.

[0009] The present invention aims to solve the problem that a virtual keyboard input method using a glasses-like screen and eye tracker tracking cannot provide users with rapid language input related to the surrounding interactive environment and the user's personalized needs.

[0010] A system for assisting paralyzed aphasic people in language output through eye tracking, comprising a hardware part and a software part;

[0011] The hardware part includes: External recognition camera: used to shoot the external environment;

[0012] Head-mounted lens glasses display: translucent design, users can see the surrounding environment through the display;

[0013] Video output interface: output the content displayed on the head-mounted lens glasses display to the external video playback device in real time;

[0014] Eye tracker: tracks the focus of the user's glasses to complete interactive operations;

[0015] and microphones and speakers;

[0016] The software part includes: dictionary association sentence module: display alternative words or sentences;

[0017] Sound-to-text conversion module: converts language and non-language sounds into text;

[0018] Image recognition module: recognize images as text;

[0019] Dictionary storage module: stores the "item group + text group" records that the user has looked at and output in chronological order;

[0020] Eye movement word editing module: displays an editing function menu for the words that the user is gazing at.

[0021] Furthermore, the system includes a foreground learning mode: used for the user to actively input external environment-related language to the system;

[0022] Background learning mode: used for the system to recognize and learn the user's voice and the objects they have looked at in the background, without the user having to confirm the operation;

[0023] Dictionary training analysis mode: used for the system to train models of images and texts in the dictionary storage module. The final model is: when an image or word is input into the model, the model outputs the most relevant text corresponding to it;

[0024] Normal use mode: The system automatically generates optional text based on the item the user's line of sight is on. After the user selects and confirms, the text is converted into speech through the voice-to-text conversion module and played through the speaker;

[0025] Emergency Use Mode: The system has a special preset text dictionary for users to use in emergency situations.

[0026] Furthermore, the specific process of the foreground learning mode is as follows:

[0027] (1) The external recognition camera is turned on and transmits the video signal to the image recognition module, which recognizes the object in the field of view;

[0028] (2) The eye tracker tracks the user's gaze and determines the objects where the user's gaze rests;

[0029] (3) The user speaks and voice output is performed;

[0030] (4) The system starts timing;

[0031] (5) The microphone collects the user's voice and transmits the voice data to the sound-to-text conversion module;

[0032] (6) a voice-to-text conversion module converts the above voice into alternative text options;

[0033] (7) The head-mounted lens glasses display highlights the item that the user's gaze is focused on, and the image recognition module converts the item into an alternative text option and displays it next to the item;

[0034] (8) The eye tracker records the moment when the user switches his / her gaze to an object;

[0035] (9) User voice output ends;

[0036] (10) The system stops timing;

[0037] (11) The head-mounted lens glasses display displays the above-mentioned alternative text options and a screenshot of the object where the user's gaze rests as determined by the eye tracker, the text options are separated by the eye movement time, and arranged in the order of the user's voice output and the time of the gaze rest; during the timing period, the text at the nth time and the object where the user's gaze rests as determined by the eye tracker at the nth time are highlighted together, the text is divided into words, and the part of speech is marked;

[0038] (12) The user edits the words displayed separately through the eye movement word editing module;

[0039] (13) The head-mounted lens glasses display prompts the user to confirm;

[0040] (14) The user confirms by looking at the prompt through the eye tracker;

[0041] (15) After the user confirms, the above text option and a group of item screenshots are stored in the dictionary storage module as a "item group + text group" record in the above time sequence;

[0042] (16) Go back and repeat steps (2)-(15) to continue inputting more "item group + text group" records into the system's dictionary storage module;

[0043] (17) End the front desk learning mode.

[0044] Furthermore, the specific process of the background learning mode is as follows:

[0045] (1) The external recognition camera is turned on and transmits the video signal to the image recognition module, which recognizes the object in the field of view;

[0046] (2) The eye tracker tracks the user's gaze and determines the objects where the user's gaze rests;

[0047] (3) The user speaks and voice output is performed;

[0048] (4) The system starts timing;

[0049] (5) The microphone collects the user's voice clips of duration T and transmits the voice data to the sound-to-text conversion module;

[0050] (6) a voice-to-text conversion module converts the voice clips into text;

[0051] (7) The image recognition module converts the object into text;

[0052] (7) The eye tracker records the moment when the user switches his / her gaze to an object;

[0053] (8) User voice output ends;

[0054] (9) The system stops timing;

[0055] (10) The system backend retrieves the text within the timing period and the screenshots of the objects where the user's gaze rested as determined by the eye tracker, and segments the text according to the eye movement time, and arranges the text in the order of the user's voice output and the time of gaze rest; within the timing period, the text at the nth moment and the object where the user's gaze rested as determined by the eye tracker at the nth moment are aligned; the text is segmented into words and the parts of speech are marked;

[0056] (11) The above text options and a group of item screenshots are stored in the dictionary storage module as a "item group + text group" record in the above time sequence;

[0057] (12) Go back and repeat steps (3)-(11) to continue inputting more "item group + text group" records into the dictionary storage module;

[0058] (13) End the background learning mode.

[0059] Furthermore, the specific process of the dictionary training analysis mode is as follows:

[0060] (1) Extracting the “item group + text group” stored in the dictionary storage module according to the eye movement moment;

[0061] (2) Extract the words in each text segment and mark the part of speech of each word;

[0062] (3) Inputting the “item group + text group” stored in the dictionary into the training system for learning and training;

[0063] (4) Dictionary model training: object images are used as input parameters, and words with parts of speech are used as output parameters;

[0064] (5) Parameter weight setting: The weight of the words corresponding to the object images in the foreground learning mode is greater than the weight of the words corresponding to the object images in the background learning mode;

[0065] (6) Dictionary model training is completed;

[0066] (7) When one or more object images or words are input to the model, the output is a text list containing words, which are classified by part of speech. The words in each part of speech classification are sorted from high to low according to their relevance.

[0067] Furthermore, the specific process of the normal use mode is as follows:

[0068] (1) The external recognition camera is turned on and transmits the video signal to the image recognition module, which recognizes the object in the field of view;

[0069] (2) The eye tracker tracks the user's gaze and determines the objects where the user's gaze rests;

[0070] (3) A head-mounted lens eyewear display that highlights the item the user's gaze is focused on;

[0071] (4) The object image is input into the dictionary model, and the head-mounted lens glasses display displays a text list output by the dictionary model, where the text list contains candidate texts sorted by part of speech classification and relevance;

[0072] (5) Users select words using an eye tracker;

[0073] (6) The system combines the words selected by the user into an original sentence in the order selected by the user;

[0074] (7) a dictionary association sentence formation module, which calls the user's dictionary model, reorders the candidate words according to the rules in the dictionary model, and generates multiple association sentences;

[0075] (8) The system collects language sounds in the surrounding environment through a microphone;

[0076] (9) The system converts the above-mentioned language-like sounds into text through the sound-to-text conversion module, and divides them into words, which are displayed on the head-mounted lens glasses display as candidate words in step (4) for the user to select;

[0077] (10) The system monitors whether the text converted from the language-like sound in the surrounding environment is a question sentence;

[0078] (11) If the system detects the above question, a quick response option is displayed on the head-mounted lens glasses display, and the user selects the answer through the eye tracker;

[0079] (12) The system collects non-verbal sounds in the surrounding environment through a microphone;

[0080] (13) The system converts the non-language sound into text through a sound-to-text conversion module, and divides it into words, which are displayed on the head-mounted lens glasses display as candidate words in step (4) for the user to select;

[0081] (14) Repeat steps (4) to (13);

[0082] (15) The original sentence and the associated sentences form a list of candidate sentences, and the user selects the candidate sentences through an eye tracker;

[0083] (16) determining the selected sentence and displaying it in the area to be uttered on the head-mounted lens glasses display;

[0084] (17) The system has one or more areas to be sounded in the head-mounted lens glasses display;

[0085] (18) Display of word separation in the sentence in the area to be pronounced;

[0086] (19) The user edits the words displayed separately through the eye movement word editing module;

[0087] (20) A quick edit button is set next to the sentence in the area to be pronounced. After the user selects the quick edit button through the eye tracker, the system pops up a sentence editing option;

[0088] (21) In the head-mounted lens glasses display, a sound button is provided next to the area to be sounded;

[0089] (22) The user selects the sound using an eye tracker;

[0090] (23) converting the sentences in the to-be-voiced area into speech through a voice-to-text conversion module, and playing the speech through a speaker;

[0091] (24) All sentences spoken by the user in normal mode are also stored in the dictionary storage module as an "item group + text group" record.

[0092] Furthermore, the specific process of the emergency use mode is as follows:

[0093] (1) In the head-mounted lens glasses display, an emergency mode activation area is set in a prominent position;

[0094] (2) The user activates the emergency use mode using an eye tracker;

[0095] (3) The system displays the preset emergency text words on the head-mounted lens glasses display;

[0096] (4) The user selects words using an eye tracker;

[0097] (5) A special preset text dictionary generates a list of associated words based on the words selected by the user;

[0098] (6) The user selects words from the above-mentioned associative word list through the eye tracker, and the system generates associative sentences for all the selected words according to the sentence generation rules in the dictionary through a special preset text dictionary;

[0099] (7) Return to step (5) until the user generates the final sentence and displays it in the area to be uttered in the head-mounted lens glasses display;

[0100] (8) converting the sentences in the to-be-voiced area into speech through a voice-to-text conversion module, and playing the speech through a speaker;

[0101] (9) Once the user activates the emergency mode, the system speaker will sound an alarm to attract other personnel to come to assist;

[0102] (10) The video output interface automatically outputs the video displayed in the head-mounted lens glasses display so that other personnel can guide the user's thoughts in real time and quickly understand the user's intentions.

[0103] Beneficial effects: The present invention learns the environmental objects and environmental sound inputs around the user, as well as the user's own language history habits, so that the user can more efficiently convert the surrounding environment into a language output that is closer to his or her own habits, and can enable users such as paralyzed and aphasic people to conduct basic and important language communication with the outside world. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 It is the object recognition in the foreground learning mode of the present invention;

[0105] Figure 2 The speech output by the user in the foreground learning mode of the present invention is converted into text;

[0106] Figure 3 are two objects that the user has successively gazed at in the background learning mode of the present invention;

[0107] Figure 4The background analysis of the objects that the user has gazed at and the user's output language by the background of the system of the present invention;

[0108] Figure 5 It is the recognition text of the object by the system in the normal use mode of the present invention and the text selected by the user in sequence by looking at it;

[0109] Figure 6 The original sentences and associated sentences generated in the normal use mode of the present invention and the editing of sentences by the user;

[0110] Figure 7 It is that in the normal use mode of the present invention, the system converts the sound of the external language question into text and provides a quick answer option;

[0111] Figure 8 It is the user selecting and making sentences from the text converted from the external language in the normal use mode of the present invention;

[0112] Fig. 9 It is that in the normal use mode of the present invention, the system converts external non-language sounds into text and provides candidate sentences;

[0113] Fig.10 It is an emergency candidate word activated by a user in the emergency use mode of the present invention;

[0114] Fig.11 In the emergency use mode of the present invention, after the user selects a candidate word, the system provides an associative candidate sentence. DETAILED DESCRIPTION

[0115] The present invention discloses a system for assisting paralyzed aphasic people in language output through eye tracking, including a hardware part and a software part;

[0116] The hardware part includes: External recognition camera: used to shoot the external environment;

[0117] Head-mounted lens glasses display: translucent design, users can see the surrounding environment through the display;

[0118] Video output interface: output the content displayed on the head-mounted lens glasses display to the external video playback device in real time;

[0119] Eye tracker: tracks the focus of the user's glasses to complete interactive operations;

[0120] and microphones and speakers;

[0121] The software part includes: dictionary association sentence module, which is used to display alternative words or sentences;

[0122] The sound-to-text conversion module is used to convert language and non-language sounds into text;

[0123] An image recognition module, used to recognize images as text;

[0124] A dictionary storage module is used to store the records of "item group + text group" that the user has looked at and output in chronological order;

[0125] The eye movement word editing module is used to display an editing function menu for the word that the user is gazing at.

[0126] The system contains five modes:

[0127] The front-end learning mode is used for users to actively input external environment-related languages ​​into the system;

[0128] Background learning mode, in which the system recognizes and learns the user's voice and the objects they have looked at in the background, without the user having to confirm the operation;

[0129] The dictionary training analysis mode is used for the system to train the model of images and texts in the dictionary storage module. The final model is: when an image or word is input to the model, the model outputs the most relevant text corresponding to it;

[0130] In normal use mode, the system automatically generates optional text based on the item the user's line of sight is on. After the user selects and confirms, the text is converted into speech through the voice-to-text conversion module and played through the speaker.

[0131] In the emergency use mode, the system has a special preset text dictionary for users to use in important emergency situations such as life and health.

[0132] Embodiment 1:

[0133] Foreground learning mode example:

[0134] (1) Figure 1 As shown in the figure, the user's eyes look at the objects: the cup and the tea box in sequence. The eye tracker records that the user looks at the cup in the first second and the tea box in the second second.

[0135] (2) Alternative texts recognized by the image recognition module are displayed next to the object. For example, the cup is recognized as “cup, jar, water cup”, and the tea box is recognized as “tea, tea leaves, box, can”.

[0136] (3) Figure 2 As shown, the user's voice output is "Take a cup and make a cup of tea", which is separated by eye movement time. "Take a cup" is said in the first second, and "Make a cup of tea" is said in the second second.

[0137] (4) The system converts the user's speech into text, divides it into words, and marks the parts of speech.

[0138] (5) The head-mounted lens glasses display arranges and displays the objects that the user has looked at in the order of eye movement time: the text converted from the user's voice is also arranged in the order of eye movement time.

[0139] (6) The user gazes at the separated word "cup", activating the word editing module, and the words recognized by the image recognition module or synonyms of "cup" such as "jar, water cup", etc. are displayed below "cup".

[0140] (7) The user can look at the corresponding word and replace it with "cup" in the sentence.

[0141] (8) The user looks at the "Confirm" button and records this group of "object group + text group", namely {first second: [water cup image], [take] (verb), [cup] (noun), [jar] (noun), [water cup] (noun); second second: [tea box image], [brew] (verb), [cup] (quantifier), [tea] (noun), [tea leaves] (noun), [box] (noun), [jar] (noun)}, and stores it in the dictionary storage module.

[0142] Embodiment 2:

[0143] Example of background learning mode:

[0144] (1) Figure 3 As shown in the figure, the user's eyes look at the objects: the cup and the coffee box. The eye tracker records that the user looks at the cup in the first second and the coffee box in the second second.

[0145] (2) Figure 4 As shown, the system background image recognition module identifies the alternative texts corresponding to the image. For example, the cup is identified as "cup, jar, water cup", and the coffee box is identified as "coffee, box, can".

[0146] (3) The user’s voice output is “Take a cup and make some coffee”, separated by eye movement time. In the first second, “Take a cup” is said, and in the second second, “Take a cup and make some coffee” is said.

[0147] (4) The system background converts the user's voice into text, divides it into words, and marks the parts of speech.

[0148] (5) The system background records this group of “object group + text group”, i.e., {first second: [water cup image], [take] (verb), [cup] (noun), [jar] (noun), [water cup] (noun); second second: [coffee box image], [brew] (verb), [cup] (quantifier), [coffee] (noun), [box] (noun), [jar] (noun), [drink] (verb), [bar] (particle)}, and stores it in the dictionary storage module.

[0149] Embodiment 3:

[0150] Examples of normal usage mode:

[0151] (1) As shown in Figure 5 , the image recognition module recognizes the face and coffee box in the field of view. The face recognition identifies two options, "Xiaowang" and "Xiaozhang". The coffee boxes are respectively recognized as the verb "brew", "drink", the noun "coffee" and "jar".

[0152] (2) The user gazed at "Xiaowang", "brew", "coffee", and "drink" in sequence. As shown in Figure 6 , the system composes the original sentence "Xiaowang brews coffee to drink" in this order.

[0153] (3) The dictionary association sentence composition module retrieves the user's dictionary model and searches and analyzes the above-mentioned words that the user has gazed at in the dictionary model. In this example, as shown in Figure 6 , the diachronic sentences containing the candidate words "Xiaowang", "drink", and "coffee" are found in the user's dictionary model: "Xiaowang invites to drink coffee".

[0154] (4) As shown in Figure 6 , when the user gazes at the noun "coffee" in the associative sentence "Xiaowang invites to drink coffee", the eye movement word editing module is activated, and the system pops up a word menu. In this example, the synonymous word "tea" that can be replaced appears below "coffee".

[0155] (5) When the user finally replaces "coffee" with "tea", the finally confirmed sentence is "Xiaowang invites to drink tea", which is displayed in the area to be pronounced.

[0156] (6) When the user selects "pronounce", the system converts the sentence in the area to be pronounced into speech through the speech-text conversion module, and the speech is played through the speaker.

[0157] (7) The system collects the language sounds in the surrounding environment through the microphone. As shown in Figure 7 , in this example, Xiaowang replies, "Thank you. Shall I brew a cup of tea for you too?"

[0158] (8) As shown in Figure 7 , the system converts Xiaowang's reply into text through the speech-text conversion module and splits it into words, which are displayed on the head-mounted lens glasses display.

[0159] (9) As shown in Figure 7 , the system detects that Xiaowang's reply is a question, and quick response options, "Okay, thank you" and "You're welcome", are displayed on the head-mounted lens glasses display. The user can select an answer through the eye tracker.

[0160] (10) As shown in Figure 8As shown, here the user chooses to form a new original sentence by looking at the segmented words in Xiao Wang's reply.

[0161] (11) The user selects "I", "brew" in sequence, looks at "tea" to activate the menu, and then selects "coffee" from the same category.

[0162] (12) Figure 8 As shown, the final sentence is "I make coffee", which is displayed in the area to be uttered.

[0163] (13) When the user selects "speak", the system converts the sentences in the area to be spoken into speech through the sound-to-text conversion module, and the speech is played through the speaker.

[0164] (14) Fig. 9 As shown in FIG. 1 , the system collects non-language sounds in the surrounding environment through a microphone. In this example, the system collects the sound of knocking in the ambient sound.

[0165] (15) The system converts the knocking sound into the text “knock on the door” through the sound-to-text conversion module and divides it into words.

[0166] (16) Fig. 9 As shown, the user dictionary model performs association and historical sentence search and provides three candidate sentences: "Someone knocks on the door", "Go open the door", and "Your dad is back".

[0167] (17) The user can select the corresponding sentence to speak.

[0168] Embodiment 4:

[0169] Examples of emergency use mode:

[0170] (1) Fig.10 As shown, in the head-mounted lens glasses display, an emergency mode activation area is set in a prominent position in the upper right corner.

[0171] (2) The user activates the emergency mode using an eye tracker.

[0172] (3) Fig.10 As shown, the system displays preset emergency text words: "pain, toilet, breathe" in the head-mounted lens glasses display.

[0173] (4) Fig.11 As shown, the user selected the word "pain" using an eye tracker.

[0174] (5) A special preset text dictionary generates an associative text list based on the words selected by the user. Fig.11 As shown, "heartache", "headache", "stomachache".

[0175] (6) When the user selects "speak", the voice-to-text conversion module converts the sentences in the area to be spoken into voice, and the voice is played through the speaker.

[0176] (7) Once the user activates the emergency mode, the system speaker will sound an alarm to attract other people to come and assist.

[0177] (8) Once the user activates the emergency use mode, the video output interface automatically outputs the video displayed in the head-mounted lens glasses display so that other personnel can guide the user's thoughts in real time and quickly understand the user's intentions.

[0178] Embodiment 5:

[0179] Example of dictionary training analysis mode:

[0180] (1) Extracting the “item group + text group” stored in the dictionary storage module in the order of eye movement moments;

[0181] (2) Split all texts into words and mark the part of speech of each word, including: names, nouns, verbs, quantifiers, prepositions, modal particles, adjectives, negative words, etc.

[0182] (3) Given an object image, search the text in the dictionary storage module in the following order: ① the text of the object recognized by the image recognition module, ② the most frequently appearing noun in the text group corresponding to the object at this eye movement moment, ③ verbs, ④ adjectives, ⑤ quantifiers, and so on.

[0183] (4) Given a text word, search for the most frequently appearing noun, verb, adjective, quantifier, etc. in the n words before and after the word in all text groups. n can be an integer such as 1, 2, or 3.

[0184] (5) Parameter weight setting: the frequency weight of the text corresponding to the object image or word in the foreground learning mode is greater than the frequency weight of the text corresponding to the object image or word in the background learning mode;

[0185] (6) When one or more object images or text words are input into the model, the output is a text list containing words, which are classified by part of speech. The words in each part of speech classification are sorted from high to low according to the above frequency.

Claims

1. A system for assisting paralyzed aphasic people in language output through eye tracking, characterized in that: Including hardware part and software part; The hardware part includes: External recognition camera: used to shoot the external environment; Head-mounted lens glasses display: translucent design, users can see the surrounding environment through the display; Video output interface: output the content displayed on the head-mounted lens glasses display to an external video playback device in real time; Eye tracker: tracks the focus of the user's glasses to complete interactive operations; and microphones and speakers; The software part includes: dictionary association sentence module: call the user's dictionary model, re-order the candidate words according to the rules in the dictionary model, and generate multiple association sentences; Sound-to-text conversion module: converts language and non-language sounds into text; Image recognition module: converts items into multiple sets of alternative text options and displays them next to the items; Dictionary storage module: stores the "item group + text group" records that the user has looked at and output in chronological order; Eye movement word editing module: displays an editing function menu for the words that the user is looking at; The system includes a foreground learning mode: used for the user to actively input external environment-related language to the system; Background learning mode: used for the system to recognize and learn the user's voice and the objects they have looked at in the background, without the user having to confirm the operation; Dictionary training analysis mode: used for the system to train models of images and texts in the dictionary storage module. The final model is: when an image or word is input into the model, the model outputs the most relevant text corresponding to it; Normal use mode: The system automatically generates optional text based on the item the user's line of sight is on. After the user selects and confirms, the text is converted into speech through the voice-to-text conversion module and played through the speaker; Emergency Use Mode: The system has a special preset text dictionary for users to use in emergency situations; The specific process of the foreground learning mode is as follows: (1) The external recognition camera is turned on and transmits the video signal to the image recognition module, which recognizes the object in the field of view; (2) The eye tracker tracks the user's gaze and determines the objects where the user's gaze rests; (3) The user speaks and voice output is performed; (4) The system starts timing; (5) The microphone collects the user's voice and transmits the voice data to the sound-to-text conversion module; (6) a voice-to-text conversion module converts the above voice into alternative text options; (7) The head-mounted lens glasses display highlights the item that the user's gaze is focused on, and the image recognition module converts the item into an alternative text option and displays it next to the item; (8) The eye tracker records the moment when the user switches his / her gaze to an object; (9) User voice output ends; (10) The system stops timing; (11) The head-mounted lens glasses display displays the above-mentioned alternative text options and a screenshot of the object where the user's gaze rests as determined by the eye tracker, the text options are separated by the eye movement time, and arranged in the order of the user's voice output and the time of the gaze rest; during the timing period, the text at the nth time and the object where the user's gaze rests as determined by the eye tracker at the nth time are highlighted together, the text is divided into words, and the part of speech is marked; (12) The user edits the words displayed separately through the eye movement word editing module; (13) The head-mounted lens glasses display prompts the user to confirm; (14) The user confirms by looking at the prompt through the eye tracker; (15) After the user confirms, the above text option and a group of item screenshots are stored in the dictionary storage module as a "item group + text group" record in the above time sequence; (16) Go back and repeat steps (2)-(15) to continue inputting more "item group + text group" records into the system's dictionary storage module; (17) End the front desk learning mode; The specific process of the normal use mode is as follows: (1) The external recognition camera is turned on and transmits the video signal to the image recognition module, which recognizes the object in the field of view; (2) The eye tracker tracks the user's gaze and determines the objects where the user's gaze rests; (3) A head-mounted lens eyewear display that highlights the item the user's gaze is focused on; (4) The object image is input into the dictionary model, and the head-mounted lens glasses display displays a text list output by the dictionary model, where the text list contains candidate texts sorted by part of speech classification and relevance; (5) Users select words using an eye tracker; (6) The system combines the words selected by the user into an original sentence in the order selected by the user; (7) a dictionary association sentence formation module, which calls the user's dictionary model, reorders the candidate words according to the rules in the dictionary model, and generates multiple association sentences; (8) The system collects language sounds in the surrounding environment through a microphone; (9) The system converts the above-mentioned language sounds into text through the sound-to-text conversion module, and divides them into words, which are displayed on the head-mounted lens glasses display as candidate words in step (4) for the user to select; (10) The system monitors whether the text converted from the language-like sound in the surrounding environment is a question sentence; (11) If the system detects the above question, a quick response option is displayed on the head-mounted lens glasses display, and the user selects the answer through the eye tracker; (12) The system collects non-verbal sounds in the surrounding environment through a microphone; (13) The system converts the non-language sound into text through a sound-to-text conversion module, and divides it into words, which are displayed on the head-mounted lens glasses display as candidate words in step (4) for the user to select; (14) Repeat steps (4) to (13); (15) The original sentence and the associated sentences form a list of candidate sentences, and the user selects the candidate sentences through an eye tracker; (16) determining the selected sentence and displaying it in the area to be uttered on the head-mounted lens glasses display; (17) The system has one or more areas to be sounded in the head-mounted lens glasses display; (18) Display of word separation in the sentence in the area to be pronounced; (19) The user edits the words displayed separately through the eye movement word editing module; (20) A quick edit button is set next to the sentence in the area to be pronounced. After the user selects the quick edit button through the eye tracker, the system pops up a sentence editing option; (21) In the head-mounted lens glasses display, a sound button is provided next to the area to be sounded; (22) The user selects the sound using an eye tracker; (23) converting the sentences in the to-be-voiced area into speech through a voice-to-text conversion module, and playing the speech through a speaker; (24) All sentences spoken by the user in normal mode are also stored in the dictionary storage module as an "item group + text group" record.

2. A system for assisting paralyzed aphasic people in language output through eye tracking according to claim 1, characterized in that: The specific process of the background learning mode is as follows: (1) The external recognition camera is turned on and transmits the video signal to the image recognition module, which recognizes the object in the field of view; (2) The eye tracker tracks the user's gaze and determines the objects where the user's gaze rests; (3) The user speaks and voice output is performed; (4) The system starts timing; (5) The microphone collects the user's voice clips of duration T and transmits the voice data to the sound-to-text conversion module; (6) a voice-to-text conversion module converts the voice clips into text; (7) The image recognition module converts the object into text; (7) The eye tracker records the moment when the user switches his / her gaze to an object; (8) User voice output ends; (9) The system stops timing; (10) The system backend retrieves the text within the timing period and the screenshots of the objects where the user's gaze rested as determined by the eye tracker, and segments the text according to the eye movement time, and arranges the text in the order of the user's voice output and the time of gaze rest; within the timing period, the text at the nth moment and the object where the user's gaze rested as determined by the eye tracker at the nth moment are aligned; the text is segmented into words and the parts of speech are marked; (11) The above text options and a group of item screenshots are stored in the dictionary storage module as a "item group + text group" record in the above time sequence; (12) Go back and repeat steps (3)-(11) to continue inputting more "item group + text group" records into the dictionary storage module; (13) End the background learning mode.

3. The system for assisting paralyzed aphasic people in language output through eye tracking according to claim 1, characterized in that: The specific process of the dictionary training analysis mode is as follows: (1) Extracting the "item group + text group" stored in the dictionary storage module according to the eye movement time; (2) Extract the words in each text segment and mark the part of speech of each word; (3) Inputting the "item group + text group" stored in the dictionary into the training system for learning and training; (4) Dictionary model training: object images are used as input parameters, and words with parts of speech are used as output parameters; (5) Parameter weight setting: The weight of the words corresponding to the object images in the foreground learning mode is greater than the weight of the words corresponding to the object images in the background learning mode; (6) Dictionary model training is completed; (7) When one or more object images or words are input to the model, the output is a text list containing words, which are classified by part of speech. The words in each part of speech classification are sorted from high to low according to their relevance.

4. The system for assisting paralyzed aphasic people in language output through eye tracking according to claim 1, characterized in that: The specific process of the emergency use mode is as follows: (1) In the head-mounted lens glasses display, an emergency mode activation area is set in a prominent position; (2) The user activates the emergency use mode using an eye tracker; (3) The system displays the preset emergency text words on the head-mounted lens glasses display; (4) The user selects words using an eye tracker; (5) A special preset text dictionary generates a list of associated words based on the words selected by the user; (6) The user selects words from the above-mentioned associative word list through the eye tracker, and the system generates associative sentences for all the selected words according to the sentence generation rules in the dictionary through a special preset text dictionary; (7) Return to step (5) until the user generates the final sentence and displays it in the area to be uttered in the head-mounted lens glasses display; (8) converting the sentences in the to-be-voiced area into speech through a voice-to-text conversion module, and playing the speech through a speaker; (9) Once the user activates the emergency mode, the system speaker will sound an alarm to attract other personnel to come to assist; (10) The video output interface automatically outputs the video displayed in the head-mounted lens glasses display so that other personnel can guide the user's thoughts in real time and quickly understand the user's intentions.

Images