Multi-mode portable cognitive screening device

Abstract

The invention discloses a multi-mode portable cognitive screening device which comprises a suitcase, a display screen module, a system control module, a multi-section connecting support module, a key control module, an electroencephalogram acquisition module, a camera module, a Bluetooth headset device, a power module, a voice broadcast module and a peripheral module. Wherein the display screen module, the camera module and the non-invasive eye tracker are arranged on the inner side of a case cover of the suitcase; the system control module is arranged inside a suitcase body of the suitcase and used for being connected with the tablet computer device, and the control key module, the multi-section connecting support module, the electroencephalogram collecting module, the Bluetooth headset device and the key control module are all arranged on the suitcase body part of the suitcase. According to the multi-modal portable cognitive screening device, multi-modal data such as eye movement, electroencephalogram, scales, expressions and gaits in the testing and training process can be collected, the device is portable and simple, the influence of noise is weakened, and the test result is prevented from being influenced.

Description

Technical Field

[0001] The present invention relates to the field of medical devices, and in particular to a multimodal portable cognitive screening device. Background Art

[0002] With the advancement of medical technology and the promotion of social development, the aging phenomenon is becoming more and more serious worldwide. Aging has a significant impact on society, economy, and health. Mild cognitive impairment (MCI) is a state between normal cognitive decline in the elderly and Alzheimer's disease. MCI is characterized by a slight decline in cognitive function, but it does not affect daily life activities. MCI patients may experience symptoms such as memory loss, inattention, language disorders, and spatial orientation difficulties. Neuropsychological scales are often used clinically to assess cognitive status. Although the scale can produce more accurate diagnostic results, this method requires professional physicians to perform, and it takes at least 2 hours for a single person to complete a set of paper-based scales at one time, which cannot achieve rapid screening, especially in non-medical environments.

[0003] Cognitive impairment can be detected from two aspects: neurophysiology and overt behavior. Neurophysiology, eye movement and gait can quantitatively reflect changes in cognitive state. External manifestations such as gait instability can also reflect cognitive state. Studies have shown that patients with cognitive impairment will show longer gaze time and fewer glances when looking at negative cognitive information; patients with cognitive impairment often show tension and uneasiness in facial expressions; and the walking speed, posture and stride of patients with cognitive impairment will change significantly. Existing studies have combined EEG signals with peripheral physiological signals for cognitive recognition and classification tasks. Experimental results show that both modal data contain task-related information and that the two also influence each other. Combining the above three modal cognitive assessment methods can greatly improve the ability to identify cognitive impairment. Current studies have shown the following models: a multimodal model based on individual facial expressions and EEG signals, in which a multi-task convolutional network is used to transfer the cognitive modality, and the EEG modality is classified by SVM, and the final dual-modal joint recognition accuracy is high; in cutting-edge research in the field of cognition, a BDAE (Bimodal Deep AutoEncoder) model combines EEG signals with eye movement data, and uses the complementarity between EEG and eye movement to enhance the generalization ability of the model. However, all of the above related studies only consider data from two modalities, making it difficult to achieve more comprehensive and accurate cognitive recognition. In addition, these studies have not considered the interdependence and synchronization alignment between modalities. Different modalities also have different degrees of influence on the final prediction results. It is difficult to extract an information-rich and robust multimodal feature by simple feature concatenation or addition.

[0004] In recent years, many companies at home and abroad have developed corresponding products for cognitive detection. For example, the conversational cognitive recognition platform developed by Baidu can automatically detect the cognitive features contained in the text of users' daily conversations; the artificial intelligence platform developed by Megvii Technology can obtain seven types of cognition, such as anger, disgust, and panic, by inputting pictures; the dynamic cognitive recognition system produced by Shenzhen Anshibao Technology Co., Ltd. can be used for cognitive early warning; Empower Me designed by Brain Power can help patients with cognitive impairment interpret cognition, train their social skills and cognitive skills, and provide timely feedback on their own nervous or anxious state. However, most of the above products are based on a single modality of voice or image for cognitive recognition. Due to the single acquisition modality, the simple single-channel mode in the product design process is easily affected by noise, making it difficult to accurately reflect the individual state, and has certain limitations. The fusion of multi-modal data can provide more comprehensive and rich information for cognitive state assessment and risk early warning, thereby improving the accuracy of detection.

[0005] At present, cognitive impairment can be detected from two aspects: neurophysiology and external behavior. For example, information such as EEG, pupil changes, muscle electrical signals, and facial expressions can quantitatively reflect cognitive status. However, most products on the market currently use a single modality of speech or image for cognitive recognition, which is easily affected by noise and difficult to accurately reflect individual status. It has certain limitations. Recent studies have shown that cognitive status assessment results based on multimodal fusion are better than those of a single modality, and multimodal fusion methods can improve the robustness of detection results. However, in the process of equipment development, multimodal fusion faces the following technical difficulties, which restricts the promotion of productization:

[0006] (1) During the multimodal signal acquisition process, it is difficult to synchronize signal acquisition between different acquisition modules, and it is impossible to capture subtle signal changes caused by cognitive impairment in time and space dimensions;

[0007] (2) The amount of multimodal signal data is huge, and the source signals are mixed together. How to quickly and effectively extract features and fuse them is one of the technical difficulties;

[0008] (3) There is little research on human-computer interaction and integration of multimodal devices, which limits the application scenarios of the devices. Summary of the invention

[0009] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that the existing cognitive screening relies on professional physicians to perform, which is time-consuming and cannot be completed under non-medical conditions, or the existing cognitive screening platform has a single modality, great limitations, little human-computer interaction and high noise. The present invention provides a multimodal portable cognitive screening device that can collect multimodal data such as eye movement, EEG, scale, expression, gait, etc. during testing and training. The device is portable and simple, and the influence of noise is weakened to avoid affecting the test results.

[0010] The present invention provides a multimodal portable cognitive screening device, comprising a suitcase, a display screen module, a system control module, a multi-segment connection bracket module, a button control module, an electroencephalogram acquisition module, a camera module, a Bluetooth headset device, a power module, a voice broadcast module and a peripheral module; wherein the suitcase is configured as a shell for carrying a whole set of the multimodal portable cognitive screening device, the display screen module is arranged on the inner side of the box cover of the suitcase, the camera module is arranged above the display screen module, and the non-invasive eye tracker is arranged below the display screen module; the system control module is arranged inside the box body of the suitcase, and is used to be connected with a tablet computer device, and the control button module, the multi-segment connection bracket module, the electroencephalogram acquisition module, the Bluetooth headset device and the button control module are all arranged in the box body of the suitcase; the power module and the voice broadcast module are arranged in the box body of the suitcase, and the interfaces are all arranged on the outside of the suitcase; the peripheral module includes a tablet computer device, which is connected to the system control module.

[0011] Further, the suitcase is configured such that the outer shell of the suitcase is coated with a coating for shielding interference signals.

[0012] Furthermore, the coating includes a resin diluent, an additive and nickel powder.

[0013] Furthermore, the display screen module includes a VGA interface, an audio interface, and a USB interface, and is connected to a tablet computer device via the VGA interface to achieve human-computer interaction.

[0014] Furthermore, the button control module includes a plurality of buttons and a single-chip microcomputer, and the pins of the buttons are connected to the digital input and output ports of the single-chip microcomputer and communicate via RS232.

[0015] Further, a non-invasive eye tracker is configured to record the subject's eye movement data and transmit it to a tablet device to analyze and visualize the data.

[0016] Furthermore, the EEG acquisition device includes an EEG acquisition module, a Bluetooth receiver, a single electrode patch and a strap, wherein the EEG acquisition module includes a TGAM chip, a Bluetooth transmitter and a grounded ear clip, the single electrode patch collects the frontal EEG of the subject through the strap, and inputs the signal into the TGAM chip for data processing, and transmits the signal to the tablet computer device through the Bluetooth transmitter in the EEG acquisition module to realize real-time acquisition; the Bluetooth receiver device transmits the signal to the tablet computer device through the Bluetooth transmitter in the EEG acquisition module to realize real-time wireless acquisition; the grounded ear clip protects the wearer from electric shock and keeps the EEG acquisition device operating normally.

[0017] Furthermore, the button control module also includes a potentiometer module, an RS232 module, and an RS232-to-serial port module. The potentiometer module and the RS232 module are connected to the single-chip processor via the RS232-to-serial port module.

[0018] Further, the tablet device includes a multimodal fusion training model, and in the micro-expression module, the multimodal fusion training model includes the following steps: pre-processing the vertex frame image in the micro-expression module;

[0019] Extract the optical flow features of the start frame and the vertex frame to obtain the vertical component, horizontal component and OpticalStrain component;

[0020] Each component result is fed into the Mobile VIT network as the first part of the micro-expression matrix sequence, and then another n×n convolutional layer is used to fuse the local and global features in the cascade tensor. Finally, the fused features are fed into the Transformer encoder.

[0021] Furthermore, the tablet computer device includes a multimodal fusion training model. In the eye movement part, the multimodal fusion training model includes the following steps:

[0022] The eye tracker's infrared camera collects the tester's eye data to obtain the tester's eye movement parameters - position, speed, acceleration and movement trajectory, and maps the eye movement trajectory into screen coordinates;

[0023] After obtaining relevant data and dynamic characteristics and preprocessing the collected eye movement data, we obtain the Euclidean distance ρ, eye tracking speed gain v g , eye tracking speed v i , and after data processing, it is input as the three features of the MEF model. g 、v i , ρ represents the temporal modeling function of three LSTMs, the metadata module autoencoder output is the output of LSTM; the metadata is represented by v g and v i The corresponding two scales represent that in the multi-module, the output f of the time series module, ρ of the spectrogram module, and d of the metadata module are then fused by the autoencoder; the final feature f is usually calculated as follows:

[0024]

[0025] The final output feature f is used as the classification feature of the eye movement part, enters the Transformer encoder, and finally enters a fully connected layer and Softmax to convert the output into a classification probability.

[0026] Technical Effects

[0027] The present invention uses a coating that shields interference signals and optimizes the circuit layout to integrate the timing signals collected by high-precision sensors. Before collecting signals, external interference signals that are irrelevant to the collected signals are removed by filtering, anti-jitter, isolation, shielding and other methods, thereby improving the accuracy and reliability of collecting keys, voices, and multimodal signals. At the same time, in order to ensure data synchronization between multiple acquisition channels, clock synchronization, trigger synchronization, DMA transmission and other methods are used to achieve synchronous acquisition, avoiding problems such as inconsistent data, overlap or loss due to time differences between channels, and ensuring the stability and reliability of the acquisition system. A multimodal portable cognitive screening device of the present invention is small in size, light, easy to carry, and suitable for a variety of applications. At the same time, a coating of special materials covers the outer surface of the aluminum layer, which can effectively shield signal interference and control the weight of the device, achieving true lightness and being able to be carried with you.

[0028] The algorithm of the present invention adopts pre-trained CNN and LSTM models, applies transfer learning technology to pre-train the models, and performs fine-tuning on the data set collected in this study to obtain high-throughput single-modal features. In combination with the attention mechanism of Transformer, corresponding weights are adaptively assigned to different modal features, the information of effective modalities is amplified, the influence of data noise is weakened, the organic combination of different modal information is achieved, and the robustness of the model is improved.

[0029] The cognitive recognition model in the present invention adopts self-supervised learning technology. For the multimodal data of the same subject, the features of two modalities are randomly extracted to form a positive sample pair. For the multimodal data of different subjects, the features of two modalities are randomly extracted to form a negative sample pair. The self-supervised loss function makes the mapping features between the positive sample pairs close and the mapping features between the negative sample pairs far away. The cross entropy classification loss function and the self-supervised loss function are jointly optimized to explore the correlation between the modal data of the same subject and the implicit relationship between the modal data of different subjects.

[0030] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a screening process diagram of a multimodal portable cognitive screening device according to a preferred embodiment of the present invention;

[0032] Figure 2 It is a schematic diagram of a multimodal fusion extraction process of a multimodal portable cognitive screening device according to a preferred embodiment of the present invention;

[0033] Figure 3It is a hardware system schematic diagram of a multimodal portable cognitive screening device according to a preferred embodiment of the present invention;

[0034] Figure 4 It is a structural schematic diagram of a multimodal portable cognitive screening device according to a preferred embodiment of the present invention;

[0035] Figure 5 It is a schematic diagram of a multimodal portable cognitive screening device according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] In the embodiment of the present invention, Figure 1-5As shown, a multimodal portable cognitive screening device is provided, comprising a suitcase (100), a display screen module (300), a system control module, a multi-section connection bracket module (900), a key control module (450), an electroencephalogram acquisition module (500), a camera module (350), a Bluetooth headset device (700), a power module, a voice broadcast module and a peripheral module; wherein the suitcase (100) is configured as a shell for carrying the entire multimodal portable cognitive screening device, the display screen module (300) is arranged on the inner side of the box cover of the suitcase (100), and the camera module (350) is arranged on the inner side of the box cover of the suitcase (100). (350) is arranged above the display screen module, and the non-invasive eye tracker (600) is arranged below the display screen module; the system control module is arranged inside the suitcase body and is used to connect with the tablet device, the control button module (450), the multi-section connection bracket module (900), the EEG acquisition module (500), the Bluetooth headset device (700) and the button control module (450) are all arranged in the body of the suitcase; the power module and the voice broadcast module are arranged in the body of the suitcase, and the interfaces (150) and (170) are both arranged on the outside of the suitcase. The peripheral module includes a tablet device (250), and the tablet device is connected to the system control module. The display screen module (300) is arranged on the inner side of the cover of the suitcase (100) and is connected to the tablet computer via a USB interface (101) and a VGA interface (102); the camera module (350) is arranged above the display screen module (300); the non-invasive eye tracker (600) is arranged below the display screen module (300) and is connected to the tablet computer via a USB interface (101); the key control device (400) is connected to the tablet computer via a serial port; the electroencephalogram acquisition device (500) is connected to the tablet computer via Bluetooth; the printer device (800) is connected to the tablet computer via Bluetooth; the multi-section connection bracket module (900) assists in supporting the tablet computer for use by a guardian; and the micro-expression recognition camera (350) is connected to the tablet computer via Bluetooth.

[0038] The tablet computer device (250) is provided with a multimodal cognitive screening and rehabilitation module, and the tablet computer device is mainly responsible for the installation of the software detection system and the connection and interaction with other hardware modules.

[0039] The external display screen device (300) is used for displaying and realizing human-computer interaction during the testing and training process.

[0040] The key control module (400) comprises an STM32F407VET6 single-chip processor (401), a potentiometer (402), a digital key device (460), an RS232 module (403), and an RS232-to-serial port module (404), and is used to realize human-computer interaction during testing and training. The potentiometer module and the RS232 module are connected to the single-chip processor via the RS232-to-serial port module.

[0041] The electroencephalogram (EEG) acquisition device (500) comprises a TGAM chip (510), a Bluetooth transmission module (520), an electrode patch (530), and a power module, wherein the electrode patch is guided by voice teaching to allow the subject to stick the electrode patch in front of the frontal lobe, and the patch is clicked to transmit the collected EEG information to the TGAM chip (510) via a Micro-USB interface (103) by wire. The collected EEG signal is filtered, and the EEG information between 0.5 Hz and 50 Hz is extracted using a Butterworth bandpass filter to achieve high anti-interference ability; the module adopts a single channel to acquire signals, analyze data, and directly outputs EEG band results, concentration and relaxation data after filtering and removing power frequency interference, and transmits the data to a tablet device via the Bluetooth transmission module (520).

[0042] The non-invasive eye tracker (600) is connected to a tablet device via a USB interface, and uses a built-in camera to capture the subject's line of sight and map it to corresponding screen coordinates. All coordinate data will be processed and classified by a support vector machine (SVM) algorithm based on pupil changes.

[0043] The Bluetooth headset device (700) is connected to the tablet device via Bluetooth transmission and is used for voice broadcasting and voice guidance.

[0044] The printer device (800) is connected to the tablet device via Bluetooth transmission and is used to print the report after the final test is completed.

[0045] The specific implementation process is as follows:

[0046] The first step is to connect the suitcase to the power supply and turn on the power switch; open the suitcase, connect the tablet device to the USB adapter and project the screen onto the entire external display; then open the software in the tablet and prepare to start testing.

[0047] The second step is to register. The registration content includes name, gender, age, years of education, academic qualifications, and occupation, and then officially enter the test.

[0048] Before the test, you need to complete the wearing of the EEG acquisition device according to the teaching, and complete the wearing of the EEG detection device, eye movement detection, gait analysis, and micro-expression recognition tests according to the teaching requirements before you can officially enter the digital cognitive rehabilitation system test.

[0049] The EEG test runs through the entire set of subsequent tests. When the subject wears the EEG acquisition device (500), a corresponding voice broadcast module (150) will play the teaching of wearing the EEG acquisition device, and the teaching video of wearing will be played to facilitate the subject's understanding.

[0050] In the eye tracking test, subjects will stare at the monkeys in the scene in front of the screen. The game rules will be explained through static positioning of a total of 9 monkeys with the help of the corresponding voice broadcast module. Then, a 3D eye movement model will be built based on the statically collected calibration data. The dynamic scene will then be tested to collect corresponding data for feature analysis.

[0051] In the process of detecting eye movement and EEG data, time series models such as LSTM are used to extract eye movement and EEG features, pre-trained based on a large sample data set, and then fine-tuned on the data collected in this study.

[0052] The two-dimensional time-frequency response of the EEG signal is used as input, and the short-time Fourier transform (STFT) is used to extract the time-frequency response of the ECG signal. The method involves sliding a small window on the signal and then performing a discrete Fourier transform on each corresponding window. The STFT equation is expressed as 0:3, where Sis STFT function and g(nm) is the window function, usually a Hann or Gaussian window, and the width of the window is specified by m.

[0053]

[0054]

[0055] Where k and m are called the time index and frequency index. Through a 2D RCNN network (similar to 1DRCNN, 2D RCNN is designed using 2D CNN and LSTM network). Represented as the output of the 2D RCNN module, that is, this block will also be passed to the corresponding autoencoder part.

[0056] In the eye movement test, the eye data of the tester is collected through the infrared camera of the non-invasive eye tracker. By obtaining the motion parameters of the tester's eyes - position, speed, acceleration and motion trajectory, and mapping the eye movement trajectory to screen coordinates, the eye movement trajectory can be visualized. The obtained motion parameters can calculate the tracking speed and gain of the eye movement through its trajectory and acceleration. The features are extracted through the eye movement feature extraction algorithm designed by ourselves, and the collection details include:

[0057] 1) Eye tracking speed. Eye tracking speed refers to the speed at which the subject's eyes follow a moving target. It is usually used to measure the visual tracking speed of the subject's eyes following a moving target. This indicator can provide a deeper understanding of the subject's eye tracking behavior, including tracking speed and accuracy. i According to the following formula, substitute t iand t i-1 The coordinate value of the gaze point at time t i and t i-1 Obtained after a time interval.

[0058]

[0059] Among them, (a i ,b i ) and (a i-1 ,b i-1 ) refers to the i and t i-1 The coordinates of the subject's gaze point at each moment correspond to the coordinates on the screen. The value of i ranges from 2 to n, where n is a positive integer.

[0060] 2) Eye tracking speed gain. Eye tracking speed gain is an indicator that represents the relationship between the visual tracking speed of the subject's eyes following a moving target and the speed of the moving target. It is usually used to measure the degree of matching between the eye tracking speed and the target speed. Specifically, this indicator is the ratio of the visual tracking speed of the subject's eyes tracking a moving target to the speed of the moving target. By calculating and analyzing the eye tracking speed gain, the movement trajectory and tracking behavior of the subject's eyes can be described and evaluated more specifically. In this embodiment, first calculate v according to the above formula i The value of the moving target is then calculated. i , and finally calculate v′ i With v i The ratio v g .

[0061]

[0062] 2) Based on the difference in the position of the fixation point at different times, the difference between the fixation point and the target position can be explained and calculated. This difference is usually caused by the position error between the fixation point and the moving target when the subject's eyes track the moving target. i The difference between the fixation point and the target position at the moment is expressed as the Euclidean distance ρ between the fixation point and the actual motion trajectory of the moving target. This data will be used to describe and analyze the eye movement trajectory of the subject in more detail.

[0063]

[0064] After obtaining relevant data and dynamic characteristics and preprocessing the collected eye movement data, we obtain the Euclidean distance ρ, eye tracking speed gain v g , eye tracking speed v i , and after data processing, they are input together as the three features of the MEF model.

[0065] The infrared camera of the eye tracker measures specific properties of eye movement: 1) Cursor size, by adjusting the size of the gaze point cursor in the playback video, in pixels, to record the fixed point and the scanning point. 2) Border width, we can adjust the thickness of the gaze point cursor line width, also in pixels, where the mapping point record extracts the specific gaze point position information (a i ,b i ) and mapped to the target image. 3) Opacity, responsible for adjusting the transparency of the gaze point display. 100% opacity is real, and can be adjusted as needed in actual operation. 4) Display type, we obtain direct coordinate values ​​through the cross display. Different from the cursor size and gaze point position, it is reflected as a scan point. A scan point is recorded every 40ms, and the sampling frequency is 60Hz.

[0066] These four specific attributes are used to evaluate the visual tracking performance and filter out the fixation points that meet a certain confidence range within the selected time range, including the normalized pupil coordinate position (a i ,b i ), analyze the characteristics of eye movement signals, and these characteristic values ​​are used as the second part of the eye movement sequence input and enter another part of the block CNN. This module uses a one-dimensional convolutional neural network to extract spatial information from the time series and then uses LSTM to simulate the one-dimensional time relationship. Finally, it enters the transformer for training and judgment at the same time.

[0067] Among them, gait analysis test and micro-expression recognition are auxiliary multimodal detection contents of this test.

[0068] The gait analysis test requires the subjects to imitate the standard gait walking paradigm and record the gait video through the micro-expression recognition camera, and explain the rules through the corresponding voice broadcast module. The gait analysis test includes recording the gait movements of the subjects, taking the video frames as input, and using transfer learning to pre-train and fine-tune the model. For the gait video data, the GaitPart model is used to extract gait features, and pre-train and fine-tune it on the OU-MVLP dataset.

[0069] In the micro-expression recognition test, the subjects were asked to imitate the different expressions of the little monkeys, and the facial expressions of the subjects were collected through the micro-expression recognition camera. Micro-expression recognition includes the acquisition of facial image frames after the subject's voice guidance. It mainly extracts facial expression features using the ResNet model for facial image data, and pre-trains and fine-tunes on the RAF-DB dataset. In the micro-expression module, first, the vertex frame image is pre-processed. The first step is to remove the background and extract the ROI. Then, all adjacent frames of the video are operated according to the optical flow feature. The optical flow feature obtains an image result by performing certain mathematical operations on two images, which can well reflect the transformation between the two frames.

[0070] Second, extract the optical flow features of the start frame and the vertex frame to obtain the vertical component, horizontal component and OpticalStrain component.

[0071] The OF component is approximated based on the brightness constancy equation as shown in the following formula:

[0072]

[0073]

[0074] where (a, b, t) represents the image intensity function at point (a, b) and time t, and I x , I y is the spatial variation gradient. t It is the time-varying gradient. dA, dB and dt are the displacement vectors of point and time respectively. P and q represent the horizontal and vertical motion vectors respectively.

[0075]

[0076]

[0077] The magnitude of the optical strain (OS) can be further extracted from the (OF) brightness constancy equation to provide a matrix transformation, the operation of which can be expressed as a square matrix:

[0078]

[0079] 3. Finally, we input the results of each component into Mobile VIT separately as the first part of the input of the micro-expression matrix sequence. The Mobile VIT network is a lightweight mobile visual task network.

[0080] First, the input image with a width of W and a height of H is expanded into N non-overlapping straightened Patches layers X F ∈R A×B×CWhere A=WH, B=A / P, and B is the number of patches. a ∈{1,…,P a}, by applying Transformer to encode the relationship between patches X G ∈R A×B×C :

[0081] X G (p a )=Transformer(X U (p a )), 1≤p a ≤P a

[0082] Mobile ViT does not lose either the patch order or the spatial order of pixels within each patch. Another n×n convolutional layer is then used to fuse local and global features in the concatenated tensor. The fused features are finally fed into the Transformer encoder.

[0083] The third step is to formally start the test, which includes: cognitive ability, logical ability, memory, concentration, and judgment. The health of the subject's brain is comprehensively assessed through different aspects.

[0084] The first part is a cognitive ability test, which is presented in the form of clock questions. Users need to observe the time displayed on the clock in the picture and accurately answer the time by pressing buttons 0 to 9.

[0085] After completing the cognitive ability test, the next thing to complete is the memory ability test. The user needs to remember the two bus combinations to the park as much as possible and correctly fill in the blanks by pressing keys 0 to 9.

[0086] The second part is a judgment test, which aims to observe the user's judgment. The questions require you to observe lollipops of different shapes and colors and answer questions by pressing keys 0 to 9 to answer the corresponding number.

[0087] The logic ability test is divided into sorting the animals from large to small and identifying whether the animals are the same.

[0088] After completing this test, the last test is a concentration test, which requires users to listen to whether the two numbers are the same. If they are the same, press a key, otherwise do not press. Users can only complete the test by pressing keys 0 and 1.

[0089] After completing all tests, the system will draw conclusions based on the evaluation form and the process will end.

[0090] Finally, a report will be generated to form a closed loop, and the entire set of tests will be completed through the cooperation of software and hardware.

[0091] The following is a specific example to illustrate the specific process of the micro-expression module and the eye movement module:

[0092] An eye-tracking infrared camera and a micro-expression recognition camera were used, with 50 experimental subjects.

[0093] Micro-expression and eye movement information collection process:

[0094] Step 1: 50 people rested for 3 minutes in a static state, and then watched a video displayed on the screen for 30 seconds. The video contained content that made people feel happy, sad, and surprised. The eye tracker infrared camera and micro-expression recognition camera recorded the micro-expressions and eye movement data of the subjects. The eye tracker collected the eye movement position information of the subjects (a i ,b i ), and related parameter information includes: eye tracking speed v i , eye tracking speed gain v g , Euclidean distance ρ.

[0095] Step 2: Record and archive the data of 50 people, exclude data with obvious abnormal behavior, and add other data to the database.

[0096] MEF-NET model training phase:

[0097] The data is divided into training set, validation set and test set in a ratio of 6:2:2, and ρ and v in the eye movement information are extracted. i The data is input into the MEF-NET model and trained using the K-fold method to obtain the trained model weights.

[0098] Since unimodal data is easily affected by noise during analysis, the above unimodal cognitive impairment recognition research is difficult to accurately reflect the individual status. Fusion of multimodal data can provide richer information for cognitive screening and rehabilitation training, thereby enhancing the recognition effect. The multimodal fusion training model in the present invention analyzes the results of multimodal data and unimodal data, and compares them using different methods on different data sets. It is found that the multimodal recognition results are significantly better than the unimodal ones, and the multimodal fusion method can improve the robustness of the system.

[0099] like Figure 2As shown in the figure, in the extraction process of the micro-expression and eye movement fusion recognition model, a multimodal fusion training model MEF-Net (Microexpression Eye Facial-Net) is designed and trained. It can not only utilize the task-related information contained in a variety of data, but also extract the interactive relationship between multimodal features, thereby obtaining a more accurate and better generalized prediction model. Finally, the Transformer-based attention mechanism technology is used to fuse multimodal information such as EEG, eye movement, facial image, gait video, etc., and combined with transfer learning and self-supervised learning to improve the feature extraction ability of the model. After the front camera relies on multi-angle to capture image and video information, and the eye tracker infrared camera captures the subject's eye movement information, the fusion positioning algorithm of facial micro-expression images and eye movements is specially combined to further improve the accuracy of the training model.

[0100] A multimodal portable cognitive screening device provided by the present invention is based on a hybrid model of eye movement information analysis and micro-expression information processing. It utilizes the characteristics of micro-expressions and eye movement information that are difficult to forge, extracts multiple features from the two respectively, and then uses a Transformer-based fusion model to stably achieve effective screening of cognitive impairment and construct a multimodal portable cognitive screening core model.

[0101] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A multimodal portable cognitive screening device, characterized in that: It includes a suitcase, a display screen module, a system control module, a multi-section connection bracket module, a button control module, an EEG acquisition module, a camera module, a Bluetooth headset device, a power module, a voice broadcast module and a peripheral module; wherein the suitcase is configured as a shell for carrying a whole set of multimodal portable cognitive screening devices, the display screen module is arranged on the inner side of the box cover of the suitcase, the camera module is arranged above the display screen module, and the non-invasive eye tracker is arranged below the display screen module; the system control module is arranged inside the case body of the suitcase, and is used to connect with a tablet device to control the button module, the multi-section connection bracket module, the EEG acquisition module, the Bluetooth headset device and the button control module are all arranged in the case body of the suitcase; the power module and the voice broadcast module are arranged in the case body of the suitcase, and the interfaces are all arranged on the outside of the suitcase; the peripheral module includes a tablet device, which is connected to the system control module.

2. A multimodal portable cognitive screening device as claimed in claim 1, characterized in that: The suitcase is configured such that the outer shell of the suitcase is coated with a coating for shielding interference signals.

3. A multimodal portable cognitive screening device as claimed in claim 2, characterized in that: The coating layer comprises a resin diluent, an additive and nickel powder.

4. A multimodal portable cognitive screening device as claimed in claim 1, characterized in that: The display screen module includes a VGA interface, an audio interface, and a USB interface, and is connected to a tablet computer device via the VGA interface to achieve human-computer interaction.

5. A multimodal portable cognitive screening device as claimed in claim 1, characterized in that: The button control module includes a plurality of buttons and a single-chip microcomputer. The pins of the buttons are connected to the digital input and output ports of the single-chip microcomputer and communicate via RS232.

6. A multimodal portable cognitive screening device as claimed in claim 1, characterized in that: The non-invasive eye tracker is configured to record eye movement data of a subject and transmit the data to a tablet device to analyze and visualize the data.

7. A multimodal portable cognitive screening device as claimed in claim 1, characterized in that: The EEG acquisition device includes an EEG acquisition module, a Bluetooth receiver, a single electrode patch and a strap, wherein the EEG acquisition module includes a TGAM chip, a Bluetooth transmitter and a grounded ear clip, the single electrode patch acquires the frontal EEG of the subject through the strap, and inputs the signal into the TGAM chip for data processing, and transmits the signal to the tablet computer device through the Bluetooth transmitter in the EEG acquisition module to realize real-time acquisition; the Bluetooth receiver device transmits the signal to the tablet computer device through the Bluetooth transmitter in the EEG acquisition module to realize real-time wireless acquisition; the grounded ear clip protects the wearer from electric shock and keeps the EEG acquisition device in normal operation.

8. A multimodal portable cognitive screening device as claimed in claim 5, characterized in that: The key control module also includes a potentiometer module, an RS232 module, and an RS232-to-serial port module. The potentiometer module and the RS232 module are connected to the single-chip processor via the RS232-to-serial port module.

9. A multimodal portable cognitive screening device as claimed in claim 1, characterized in that: The tablet device includes a multimodal fusion training model, and in the micro-expression module, the multimodal fusion training model includes the following steps: pre-processing the vertex frame image in the micro-expression module; Extract the optical flow features of the start frame and the vertex frame to obtain the vertical component, horizontal component and optical strain component; Each component result is fed into the Mobile VIT network separately as the first part of the micro-expression matrix sequence input, and then another n×n convolutional layer is used to fuse the local and global features in the cascade tensor, and finally the fused features are fed into the Transformer encoder.

10. A multimodal portable cognitive screening device as claimed in claim 1, characterized in that: The tablet computer device includes a multimodal fusion training model. In the eye movement part, the multimodal fusion training model includes the following steps: The eye tracker's infrared camera collects the tester's eye data to obtain the tester's eye movement parameters - position, speed, acceleration and movement trajectory, and maps the eye movement trajectory into screen coordinates; After obtaining relevant data and dynamic characteristics and preprocessing the collected eye movement data, we obtain the Euclidean distance ρ, eye tracking speed gain v g , eye tracking speed v i , and after data processing, it is input as the three features of the MEF model. g 、v i , ρ represents the temporal modeling function of three LSTMs, the metadata module autoencoder output is the output of LSTM; the metadata is represented by v g and v i The corresponding two scales represent that in the multi-module, the output f of the time series module, ρ of the spectrogram module, and d of the metadata module are then fused by the autoencoder; the final feature f is usually calculated as follows: The final output feature f is used as the classification feature of the eye movement part, enters the Transformer encoder, and finally enters a fully connected layer and Softmax to convert the output into a classification probability.

Images