Life scene auxiliary method based on electroencephalogram-eye movement information, storage medium and equipment

Abstract

The invention provides a life scene assisting method based on electroencephalogram-eye movement information, a storage medium and equipment. The life scene assisting method comprises the following steps: acquiring first electroencephalogram data; if it is judged that the user is in the Alzheimer's disease morbidity state according to the first electroencephalogram data, the electroencephalogram data are obtained again according to a preset credible mode and recorded as reliable electroencephalogram data; if it is judged that the user is in an attention concentration state according to the reliable electroencephalogram data and the preorder electroencephalogram data, real-time eye movement data are obtained according to a preset reliable mode; inputting the real-time eye movement data into the eye movement data extraction model to obtain reliable eye movement data; if it is determined that the user intends to control the living scene object according to the reliable electroencephalogram data, the preorder electroencephalogram data and the reliable eye movement data, determining a target control object of the user according to the reliable eye movement data and a map of the living scene where the user is located; collecting limb parameters of the user according to a second preset frequency; and if the limb parameters indicate that the user does not act within the preset time threshold, triggering object operation reminding.

Description

Technical Field

[0001] The present invention relates to the field of smart home appliance control, and in particular to a life scene assistance method, storage medium and device based on electroencephalogram (EEG)-eye movement information. Background Art

[0002] Alzheimer's disease (AD), which accounts for 70% of all dementia cases, is a progressive, irreversible neurodegenerative disease characterized by memory loss and cognitive decline. As these symptoms progress, patients experience a decline in their ability to manipulate objects in everyday situations, such as forgetting how to operate the remote control when they want to watch TV or which button to press when using a rice cooker.

[0003] Numerous existing methods exist for interacting with smart home appliances. These methods, for example, utilize the user's EEG and eye gaze to transmit control commands to the appliance system, or perform model training based on the user's EEG information to match it with smart appliance behavioral commands, thereby controlling smart home appliances based on the user's EEG information. However, for Alzheimer's patients, EEG information can decay as the disease progresses, or become chaotic in certain areas. Eye movement data can also show issues such as inattention or frequent gaze movements. These methods of controlling smart home appliances using EEG and eye movement data are prone to command errors or failure to recognize specific commands when used by Alzheimer's patients. Summary of the Invention

[0004] Based on this, the present invention provides a life scene assistance method, storage medium and device based on EEG-eye movement information, which combines EEG data and eye movement data to jointly determine whether the user is in the state of Alzheimer's disease and whether there is a desire to operate objects in life scenes, and combines the user's body movement monitoring to give the user corresponding object operation prompt information, thereby improving the autonomy of the sick user in object operation in life scenes.

[0005] In a first aspect, the present invention provides a life scene assistance method based on EEG-eye movement information, comprising:

[0006] Acquiring first EEG data;

[0007] If it is determined based on the first EEG data that the user is in a state of Alzheimer's disease, reacquire the EEG data in a preset reliable manner and record it as reliable EEG data;

[0008] If the user is judged to be in a state of concentration based on reliable EEG data and previous EEG data, real-time eye movement data is obtained according to a preset reliable mode;

[0009] Input real-time eye movement data into the eye movement data extraction model to obtain reliable eye movement data;

[0010] If it is determined based on the reliable EEG data, the preceding EEG data, and the reliable eye movement data that the user intends to manipulate an object in a life scene, the user's target manipulation object is determined based on the reliable eye movement data and the user's life scene map;

[0011] collecting the user's limb parameters at a second preset frequency;

[0012] If the limb parameter indicates that the user has not made any action within a preset time threshold, an object operation reminder is triggered.

[0013] Furthermore, the process of determining whether the user is in an Alzheimer's disease state based on the first EEG data is as follows:

[0014] The first EEG data is input into a trained deep learning model to determine whether the user is in a state of Alzheimer's disease.

[0015] Furthermore, the EEG data obtained again in a preset reliable manner is recorded as reliable EEG data, specifically:

[0016] Collect EEG data in real time at the third frequency;

[0017] The average or median of the collected EEG data was recorded as the reliable EEG data;

[0018] The third frequency is higher than the first frequency of the first EEG data acquisition.

[0019] Furthermore, the EEG data is re-acquired according to a preset reliable method and recorded as reliable EEG data, specifically:

[0020] EEG data of the frontal lobe area are collected at the fourth frequency, and EEG data of other areas are collected at the third frequency;

[0021] The average or median of the EEG data of the frontal lobe region collected at the fourth frequency is taken, and then combined with the average or median of the EEG data of other regions collected at the third frequency to form reliable EEG data;

[0022] The fourth frequency is higher than the third frequency, and the third frequency is higher than the first frequency of the first EEG data acquisition.

[0023] Furthermore, reliable EEG data and previous EEG data are used to collaboratively determine whether the user is in a state of concentration, specifically:

[0024] If the reliable EEG data The peak value of wave energy and the previous EEG data The increase in the peak value of the wave energy exceeds the first threshold, and the reliable EEG data Wave and The ratio of the wave to the previous EEG data Wave and If the ratio of the two waves decreases by more than a second threshold, it is determined that the user is in a state of concentration.

[0025] Furthermore, the user's intention to manipulate objects in life scenes is determined based on reliable EEG data, previous brain data, and reliable eye movement data. Specifically:

[0026] If the reliable EEG data The peak value of wave energy and the previous EEG data The increase in the peak energy of the wave exceeds the first threshold, and the reliable EEG data Wave and The ratio of the wave to the previous EEG data Wave and If the ratio of the two waves decreases by more than the second threshold, and the average gaze time in the reliable eye movement data exceeds the gaze time threshold and the number of gaze point shifts is less than the attention distraction threshold, it is determined that the user has the intention to manipulate objects in life scenes.

[0027] Furthermore, the user's target manipulation object is determined based on the reliable eye movement data and the user's life scene map, specifically:

[0028] determining the user's gaze focus based on the reliable eye movement data;

[0029] The user's visual focus is projected onto the user's life scene map to determine the user's target manipulation object.

[0030] Furthermore, the life scene assistance method based on EEG-eye movement information also includes:

[0031] Acquiring first EEG data and first eye movement data;

[0032] If it is determined based on the first EEG data and the first eye movement data that the user is in an Alzheimer's disease state, the EEG data is reacquired according to a preset trustworthy mode and recorded as reliable EEG data.

[0033] In a second aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the life scene assistance methods based on EEG-eye movement information in the first aspect.

[0034] In a third aspect, the present invention further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it executes any one of the life scene assistance methods based on EEG-eye movement information in the first aspect.

[0035] The beneficial effects of adopting the above technical solution are as follows: this embodiment monitors the user's static EEG data and eye movement data to provide a more reliable EEG data and eye movement data collection method for users in a state of Alzheimer's disease, such as increasing the collection frequency of specific cerebral cortical areas, or increasing the collection frequency of eye movement data, or collecting EEG data and eye movement data according to preset weights, etc. The specific EEG data and eye movement data collection method in the disease state is combined with a deep learning network to extract key data features that affect the patient's attention, and then provide corresponding auxiliary prompts for home appliance operations, so that EEG data and eye movement data with high credibility can be extracted when the user with Alzheimer's disease is in a disease state, and accurate auxiliary prompts for home appliance operation can be obtained. In the case of decreased cognitive and memory abilities, the patient's self-care ability, quality of life and sense of happiness in life scenarios can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0037] Figure 1 This is a schematic diagram of an EEG-eye movement information acquisition device in one embodiment of the present application;

[0038] Figure 2 This is a schematic diagram of a life scene assistance method based on EEG-eye movement information in one embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In order to explain the present invention in more detail, the life scene auxiliary method, storage medium and device based on EEG-eye movement information provided by the present invention are specifically described below in conjunction with the drawings.

[0040] Unless otherwise defined, the technical or scientific terms used in this application should have the usual meanings understood by people with ordinary skills in the field to which the invention belongs. The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "an" or "the" do not indicate a quantity limitation, but rather indicate the presence of at least one. Words such as "include" or "comprise" mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0041] Alzheimer's disease (AD), which accounts for 70% of all dementia cases, is a progressive, irreversible neurodegenerative disease characterized by memory loss and cognitive decline. As these symptoms progress, patients experience a decline in their ability to manipulate objects in everyday situations, such as forgetting how to operate the remote control when they want to watch TV or which button to press when using a rice cooker.

[0042] As the proportion of the aging population increases, the number of Alzheimer's patients is also growing. How to ensure and improve the quality of life of Alzheimer's patients, who are more common among the elderly, and improve the quality of life of the elderly have become issues of social concern.

[0043] Based on this, combined with the Figure 1 The EEG-eye movement information acquisition device and the attached Figure 2 The diagram shows a life scene assistance method based on EEG-eye movement information. The present invention provides a life scene assistance method based on EEG-eye movement information. By fusing EEG information and eye movement information, the method accurately judges the action intentions of Alzheimer's patients in life scenes, and gives specific life scene prompt information in combination with the patient's life scene. This is beneficial for Alzheimer's patients to accurately trigger specific prompt information of a device or an object in the life scene when they are in the state of illness, and improve the patient's self-care ability, quality of life and sense of happiness in life scenes when their cognitive and memory abilities decline.

[0044] It should be noted that the life scene assistance method based on EEG-eye movement information in this embodiment is based on the attached Figure 1The EEG-eye movement information acquisition device shown is implemented to respectively collect EEG information and eye movement information of the user through the EEG-eye movement information acquisition device.

[0045] In the EEG-eye movement information acquisition device, the EEG acquisition part is designed with an integrated cap body made of lightweight, breathable and elastic flexible materials to ensure comfort when worn for a long time; the inside of the cap fits the curve of the head, evenly distributes pressure, and avoids local oppression.

[0046] Among them, the cap body has a built-in electrode array, which is set in the key EEG signal collection areas of the brain, such as the frontal lobe, temporal lobe and other positions. In order to better fit the user's EEG signal collection area, the electrode is made of a special conductive material that can be embedded in the cap body and adaptively fit the scalp. It can reduce skin irritation and improve biocompatibility while ensuring good conductivity. The location of the electrode array in the cap body corresponds to the frontal lobe area, temporal lobe area and parietal lobe area of ​​the user's cerebral cortex; the frontal lobe area is specifically the central area of ​​the user's forehead and the areas on both sides of the forehead near the temples. The frontal lobe area is closely related to cognition, decision-making, attention and other functions. When the user is in a state of focused thinking, the signals collected in this area are The proportion of wave energy is increased by 30%-40% compared to the relaxed state; the temporal lobe area is specifically located on both sides of the user's ears. This area is mainly responsible for auditory processing, memory, and emotion. When the user hears a familiar voice or recalls the past, the EEG signal in this area is wave and There will be obvious fluctuations in the wave frequency band. The proportion of wave energy increases by about 20%-30% compared to the normal state. The proportion of wave energy increases by 15%-25% compared with the normal state; the parietal lobe area is specifically the area at the back of the head near the occipital lobe, which is mainly related to functions such as body sensation and spatial perception. When the user imagines simple limb movements, the area The proportion of wave energy increases by about 10%-20% compared to the normal state. The electrode array of the above-mentioned EEG acquisition part sends EEG information to the central processing module via a wireless network.

[0047] It should be noted that EEG signals can be divided into Wave (0.5~ 4 Hz), Wave (4~8 Hz), Wave (8~12 Hz), Waves (12~30 Hz) and Waves (30~100 Hz) and other fluctuations in different frequency ranges. In the frequency domain analysis of EEG signals, researchers focus on exploring the energy distribution patterns of different frequency components and their functional coupling mechanisms. Among them, Power Spectral Density (PSD) analysis can effectively reveal the dynamic changes of brain activity by quantifying the energy intensity of each frequency band; Spectral Coherence is used to evaluate the degree of synchronization of different brain regions in specific frequency bands, reflecting the functional connectivity between neuronal clusters. Taking the study of Alzheimer's Disease (AD) as an example, the analysis based on PSD and spectral coherence shows that the EEG signals of AD patients show a significant attenuation of dynamic characteristics, which is manifested as a decrease in energy throughout the brain, especially in the brain. Waves (8-12Hz) and The 12-30 Hz frequency band is more obvious. At the same time, the power spectrum distribution undergoes characteristic changes, with the energy of high-frequency components shifting to low-frequency components, which is closely related to the degeneration of neuronal synaptic function and the disintegration of neural networks.

[0048] The eye movement acquisition component of the EEG-eye movement information acquisition device is designed as a pair of glasses, including a fundus camera and a near-infrared light array. The fundus camera is used to collect fundus data, eye position (such as pupil position and shape), and eye movements (such as eye rotation and gaze direction). The near-infrared light array illuminates the eye without affecting the user's visual experience, allowing the fundus camera to clearly and accurately record specific eye movement data. The fundus camera can be connected to the central processing module via a wireless network or USB interface.

[0049] Based on the above-mentioned EEG-eye movement information acquisition device, the embodiment of the present application also provides an application scenario of the life scene assistance method based on EEG-eye movement information, and the application scenario includes the terminal device provided in the embodiment, and the terminal device includes but is not limited to a smart phone and a computer device, wherein the computer device can be at least one of a desktop computer, a portable computer, a laptop computer, a mainframe computer, a tablet computer, etc. The terminal device receives the EEG information and eye movement information sent by the EEG-eye movement information acquisition device, and provides specific reminder information of objects in the life scene in combination with the life scene map described by the user, and combines the attached Figure 2 The diagram shows a life scene assistance method based on EEG-eye movement information. For the specific process, please refer to the embodiment of the life scene assistance method based on EEG-eye movement information.

[0050] Step S100: Acquire first EEG data.

[0051] Specifically, the EEG acquisition portion of the EEG-eye movement information acquisition device acquires first EEG data in real time at a first frequency. The first EEG data acquired at the first frequency is primarily used to monitor the user's daily state and determine whether the user is experiencing Alzheimer's disease.

[0052] In step S200, if it is determined based on the first EEG data that the user is in an Alzheimer's disease state, the EEG data is reacquired according to a preset trustworthy mode and recorded as reliable EEG data.

[0053] Specifically, the first EEG data, as resting-state EEG data, can record the user's EEG data when the user is not performing a specific cognitive task or stimulation. The process of determining whether the user is in the onset state of Alzheimer's disease based on the first EEG data can be as follows:

[0054] The first EEG data is input into a trained deep learning model to determine whether the user is in a state of Alzheimer's disease.

[0055] It should be noted that in the resting state EEG data, the first EEG data can be wave and The power ratio of the wave frequency band serves as the primary marker for determining the onset of Alzheimer's disease using EEG data. Furthermore, the aforementioned deep learning models include, but are not limited to, artificial neural networks (ANNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs). These deep learning models are trained using EEG datasets from Alzheimer's patients and healthy individuals. Parameter adjustment and regularization are performed on the deep learning models, enabling them to quickly and accurately determine whether a user is experiencing Alzheimer's disease based on their collected EEG data.

[0056] Furthermore, given that task-state EEG data differs in amplitude, waveform, peak value, and other aspects between a user in an Alzheimer's-like state and a normal state, to more accurately identify the user's operational intent, after determining that the user is in an Alzheimer's-like state, EEG data needs to be re-collected to obtain reliable EEG data. This reliable EEG data acquisition process is performed according to a preset trust model, which collects EEG data in real time at a third frequency, which is higher than the first frequency.

[0057] Furthermore, due to the frontal lobe Waves are associated with attentional activity and are more pronounced in patients with Alzheimer's disease during set tasks, such as looking at specific content. In the case where the increase in the proportion of wave energy is insufficient, the above-mentioned preset trustworthy mode can also be to collect EEG data of the frontal lobe area at a fourth frequency and EEG data of other areas at a third frequency. The fourth frequency is higher than the third frequency. The average or median of the EEG data of the frontal lobe area collected at the fourth frequency is taken, and then combined with the average or median of the EEG data of other areas collected at the third frequency to form reliable EEG data. By collecting EEG data from different areas of the cerebral cortex at different frequencies and increasing the data collection frequency for areas with significantly weakened EEG data, the reliability of EEG data can be greatly improved while controlling the amount of EEG data processing.

[0058] Step S300: If it is determined based on the reliable EEG data and the previous EEG data that the user is in a state of concentration, real-time eye movement data is obtained according to a preset reliable mode.

[0059] In this embodiment, in order to improve the accuracy of judging the change of user attention, this embodiment collaboratively judges whether the user is in a state of concentration based on reliable EEG data and previous EEG data, specifically:

[0060] If the reliable EEG data The peak value of wave energy and the previous EEG data The increase in the peak value of the wave energy exceeds the first threshold, and the reliable EEG data Wave and The ratio of the wave to the previous EEG data Wave and If the ratio of the waves decreases by more than the second threshold, it is determined that the user is in a state of concentration. The reliable EEG data expression for the user being in a state of concentration is:

[0061] ,

[0062] in, For reliable EEG data Peak wave energy, For the previous EEG data Peak wave energy, is the first threshold, which can be set to 0.20 in this embodiment. For the previous EEG data Amplitude value, For the previous EEG data Amplitude value, For reliable EEG data Amplitude value, For reliable EEG data Amplitude value, is the second threshold, which can be set to 0.15 in this embodiment.

[0063] It should be noted that, in this embodiment, the preceding EEG data refers to the EEG data collected in the previous time period of the reliable EEG data. This EEG data can be used to observe the changes in specific parameters in the reliable EEG data, which is more conducive to accurately capturing changes in the user's attention. In this embodiment, the setting of the preceding EEG data is mainly used to determine whether the energy of the specific frequency band of the EEG data has changed significantly. The preceding EEG data can be EEG data in an onset state or in a normal state. The value ranges of the first threshold and the second threshold are adjusted according to the different states corresponding to the preceding EEG data to achieve accurate capture of changes in the user's attention.

[0064] Furthermore, considering that users with Alzheimer's disease often experience visual distraction and decreased gaze control, and that eye movement data plays an important role in determining a user's intention to manipulate objects in real life, this embodiment, after determining that the user is in a state of focused attention, acquires real-time eye movement data in a preset reliable mode. Compared to the ordinary eye movement data acquisition mode, this preset reliable mode has more eye movement data acquisition points and a longer acquisition period. The real-time eye movement data includes pupil center position, corneal reflection point, pupil diameter, eye rotation position, etc.

[0065] Step S400: inputting the real-time eye movement data into an eye movement data extraction model to obtain reliable eye movement data.

[0066] Because eye movement data from users with Alzheimer's disease is subject to increased noise, real-time eye movement data must be fed into an eye movement data extraction model to filter out noise and perform feature extraction to obtain reliable eye movement data. This reliable eye movement data includes average gaze duration, number of gaze shifts, and more.

[0067] Furthermore, during the training process, the eye movement data extraction model adjusts the parameters or structure of the neural network, combines the adaptive algorithm, and dynamically optimizes according to the real-time eye movement data to improve the accuracy of extracting reliable eye movement data under pathological conditions.

[0068] Step S500: If it is determined based on reliable EEG data, previous EEG data and reliable eye movement data that the user intends to manipulate objects in life scenes, the user's target manipulation object is determined based on the reliable eye movement data and the user's life scene map.

[0069] Among them, the user's intention to manipulate objects in life scenes is determined based on reliable EEG data, previous brain data, and reliable eye movement data. Specifically:

[0070] If the reliable EEG data The peak value of wave energy and the previous EEG data The increase in the peak energy of the wave exceeds the first threshold, and the reliable EEG data Wave and The ratio of the wave to the previous EEG data Wave and If the ratio of the two waves decreases by more than the second threshold, and the average gaze time in the reliable eye movement data exceeds the gaze time threshold and the number of gaze point shifts is less than the attention distraction threshold, it is determined that the user has the intention to manipulate objects in life scenes.

[0071] The aforementioned fixation time threshold can be a preset value or the average fixation time in historical eye movement data recorded during the Alzheimer's patient's past object manipulation processes; the aforementioned distraction threshold can be a preset value or the average number of gaze shifts in historical eye movement data recorded during the Alzheimer's patient's past object manipulation processes. It should be noted that as the user's disease progression progresses, the aforementioned fixation time threshold and distraction threshold will also adjust accordingly. For example, the fixation time threshold may decrease with progressive disease progression, while the distraction threshold may increase.

[0072] In addition, the user's target manipulation object is determined based on the reliable eye movement data and the user's life scene map, specifically:

[0073] determining the user's gaze focus based on the reliable eye movement data;

[0074] The user's visual focus is projected onto the user's life scene map to determine the user's target manipulation object.

[0075] The user's life scene map consists of two types: known life scenes and unknown life scenes. For known life scenes (such as home, hospital ward, nursing home, etc.), the location of each object in the pre-built life scene map is marked, and the projection point of the user's gaze focus is matched one by one with the coordinates of the location of each object in the life scene map to determine the user's target object. For unknown life scenes (such as hospitals, parks, etc.), the front-facing visible light camera of the eye movement collection part of the EEG-eye movement information collection device is used to monitor and identify objects in the real-time video stream through computer vision and biometric recognition algorithms (such as the BioCV algorithm), and the user's gaze focus is combined to determine the user's target object. The BioCV algorithm abandons the traditional CNN architecture and instead uses a Transformer architecture combined with a self-attention mechanism to extract features from collected images of unknown life scenes, capturing global visual associations. The extracted visual features are then fed into a large multimodal model based on the ALBEF / BLIP framework, comprising a visual encoder, a text encoder, and a cross-modal decoder, which collaborate to understand the unknown life scene images. Masked language modeling and image-text contrastive learning are used to improve generalization capabilities. Cross-modal contrastive learning is used to maximize the similarity between image and text pairs in a joint vector space. Self-supervised tasks (such as mask image reconstruction and text generation) are then used to further learn, mapping images and text into the same vector space, thus providing feedback to the device. Furthermore, the aforementioned object detection and recognition in real-time video streams of unknown life scenes can also be achieved using other existing methods for object detection and recognition in video streams, such as combining trained deep learning models (such as background subtraction, meanshift, and Camshift) to detect and recognize objects in video streams.

[0076] Step S600: collecting user's limb parameters at a second preset frequency.

[0077] Specifically, based on the present embodiment, the front visible light camera in the EEG-eye movement information acquisition device can acquire the user's limb parameters at a second preset frequency.

[0078] Step S700: If the limb parameter indicates that the user has not made any action within a preset time threshold, an object operation reminder is triggered.

[0079] Specifically, the collected user limb parameters are combined with the BioCV algorithm to identify user actions. If the user is detected to have made no action within a preset time threshold, but it is determined based on reliable EEG data, previous EEG data and reliable eye movement data that the user has the intention to manipulate objects in life scenes, it can be inferred that the user is in an ill state and wants to manipulate objects in life scenes but is not sure how to operate the object. At this time, an object operation reminder is triggered to remind the user how to operate the specific object. The BioCV algorithm abandons the traditional CNN architecture and instead uses a Transformer architecture combined with a self-attention mechanism to extract features from collected images of unknown life scenes, capturing global visual associations. The extracted visual features are then fed into a large multimodal model based on the ALBEF / BLIP framework, comprising a visual encoder, a text encoder, and a cross-modal decoder, which collaborate to understand the unknown life scene images. Masked language modeling and image-text contrastive learning are used to improve generalization. Cross-modal contrastive learning maximizes the similarity between image and text pairs in a joint vector space. Self-supervised tasks (such as masked image reconstruction and text generation) are then used to further learn, mapping images and text into the same vector space, thus providing feedback to the device. Furthermore, parametric time series analysis can be used to identify collected user body parameters. These parameters are fed into a specific model, such as Hidden Markov Models (HMMS) and Linear Dynamical Systems (LDSs), to identify whether the user has made an action or what type of action.

[0080] It should be noted that in the embodiment of the present application, the object operation reminder to the user can be sent to the electronic device carried by the user for display through the object operation reminder, including an AR display device provided in the eye movement collection part of the EEG-eye movement information collection device carried by the user, and the operation method of the object is displayed to the user in the form of an AR display. A speaker can also be provided in the EEG-eye movement information collection device carried by the user to inform the user of the operation method of the object through voice broadcast.

[0081] Furthermore, in another technical solution of this embodiment, in order to improve the accuracy of determining whether the user is in a state of Alzheimer's disease, eye movement data may be introduced in steps S100 and S200 to determine:

[0082] Step S100 can be replaced by step S101:

[0083] Step S101: Acquire first EEG data and first eye movement data.

[0084] The EEG-eye movement information acquisition device uses an EEG acquisition component to acquire first EEG data in real time at a first frequency, while an eye movement acquisition component to acquire first eye movement data in real time at a second frequency. The first EEG data acquired at the first frequency and the first eye movement data acquired at the second frequency are primarily used to monitor the user's daily status and determine whether the user is experiencing Alzheimer's disease.

[0085] Step S200 can be replaced by step S201:

[0086] In step S201 , if it is determined that the user is in an Alzheimer's disease state based on the first EEG data and the first eye movement data, the EEG data is reacquired according to a preset trustworthy mode and recorded as reliable EEG data.

[0087] The first EEG data and the first eye movement data can be input into a trained deep learning model according to preset weights to determine whether the user is in a state of Alzheimer's disease. The dual confirmation of EEG data and eye movement data improves the accuracy of the judgment of the user's disease state.

[0088] It should be understood that although the Figure 2 The steps in the flowchart are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 2 At least part of the steps may include multiple sub-steps or sub-stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0089] In one embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned life scene assistance method based on EEG-eye movement information.

[0090] The computer-readable storage medium may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), a hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code for executing any of the steps of the above-described method. This program code can be read from or written to one or more computer program products, and the program code may be compressed in a suitable form.

[0091] In one embodiment, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the above-mentioned life scenario assistance method based on EEG-eye movement information when executing the computer program.

[0092] The computer device includes a memory, a processor, and one or more computer programs, wherein the one or more computer programs can be stored in the memory and configured to be executed by one or more processors, and the one or more application programs are configured to execute the above-mentioned life scene assistance method based on EEG-eye movement information.

[0093] A processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect the various components within the entire computer device. It executes instructions, programs, code sets, or instruction sets stored in memory, and accesses data stored in memory to perform various functions of the computer device and process data. Optionally, the processor may be implemented in the form of at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a communications chip.

[0094] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the terminal device during use, etc.

[0095] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A life scene assistance method based on EEG-eye movement information, characterized in that: include: Acquiring first EEG data; If it is determined based on the first EEG data that the user is in a state of Alzheimer's disease, reacquire the EEG data in a preset reliable manner and record it as reliable EEG data; If the user is judged to be in a state of concentration based on reliable EEG data and previous EEG data, real-time eye movement data is obtained according to a preset reliable mode; Input real-time eye movement data into the eye movement data extraction model to obtain reliable eye movement data; If it is determined based on the reliable EEG data, the preceding EEG data, and the reliable eye movement data that the user intends to manipulate an object in a life scene, the user's target manipulation object is determined based on the reliable eye movement data and the user's life scene map; collecting the user's limb parameters at a second preset frequency; If the limb parameter indicates that the user has not made any action within a preset time threshold, an object operation reminder is triggered.

2. The life scene assistance method based on EEG-eye movement information according to claim 1, characterized in that: The process of determining whether the user is in the onset state of Alzheimer's disease based on the first EEG data is as follows: The first EEG data is input into a trained deep learning model to determine whether the user is in a state of Alzheimer's disease.

3. The life scene assistance method based on EEG-eye movement information according to claim 2, characterized in that: The EEG data re-acquired in a preset reliable manner is recorded as reliable EEG data, specifically: Collect EEG data in real time at the third frequency; The average or median of the collected EEG data was recorded as the reliable EEG data; The third frequency is higher than the first frequency of the first EEG data acquisition.

4. The life scene assistance method based on EEG-eye movement information according to claim 2, characterized in that: The EEG data re-acquired according to the preset reliable method is recorded as reliable EEG data, specifically: EEG data of the frontal lobe area are collected at the fourth frequency, and EEG data of other areas are collected at the third frequency; The average or median of the EEG data of the frontal lobe region collected at the fourth frequency is taken, and then combined with the average or median of the EEG data of other regions collected at the third frequency to form reliable EEG data; The fourth frequency is higher than the third frequency, and the third frequency is higher than the first frequency of the first EEG data acquisition.

5. The life scene assistance method based on EEG-eye movement information according to claim 1, characterized in that: Based on reliable EEG data and previous EEG data, it is determined whether the user is in a state of concentration. Specifically: If the reliable EEG data The peak value of wave energy and the previous EEG data The increase in the peak value of the wave energy exceeds the first threshold, and the reliable EEG data Wave and The ratio of the wave to the previous EEG data Wave and If the ratio of the two waves decreases by more than a second threshold, it is determined that the user is in a state of concentration.

6. The life scene assistance method based on EEG-eye movement information according to claim 1, characterized in that: Based on reliable EEG data, previous brain data, and reliable eye movement data, it is determined that the user has the intention to manipulate objects in life scenes. Specifically: If the reliable EEG data The peak value of wave energy and the previous EEG data The increase in the peak energy of the wave exceeds the first threshold, and the reliable EEG data Wave and The ratio of the wave to the previous EEG data Wave and If the ratio of the two waves decreases by more than the second threshold, and the average gaze time in the reliable eye movement data exceeds the gaze time threshold and the number of gaze point shifts is less than the attention distraction threshold, it is determined that the user has the intention to manipulate objects in life scenes.

7. The life scene assistance method based on EEG-eye movement information according to claim 1, characterized in that: The determining of the user's target manipulation object based on the reliable eye movement data and the user's life scene map is specifically as follows: determining the user's gaze focus based on the reliable eye movement data; The user's visual focus is projected onto the user's life scene map to determine the user's target manipulation object.

8. The life scene assistance method based on EEG-eye movement information according to claim 1, characterized in that: Also includes: Acquiring first EEG data and first eye movement data; If it is determined based on the first EEG data and the first eye movement data that the user is in an Alzheimer's disease state, the EEG data is reacquired according to a preset trustworthy mode and recorded as reliable EEG data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of any one of the life scene assistance methods based on EEG-eye movement information in claims 1-8 are implemented.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it executes any one of the life scene assistance methods based on EEG-eye movement information in claims 1-8.

Images