Auditory cognitive dysfunction assessment and screening system

Abstract

The present invention discloses an auditory cognitive dysfunction assessment and screening system, which belongs to the field of data analysis technology. The system specifically includes: setting basic test parameters and initial stimulation parameters, and synchronously collecting speech, eye movement trajectory data and brain wave signals in combination with spatiotemporal anchor markers to form a multimodal dataset with a timestamp; conducting multi-stage cognitive testing in an adaptive test engine, and dynamically adjusting the stimulation parameters based on real-time accuracy and reaction time using a forgetting curve prediction algorithm; based on the adjusted stimulation parameters, using a dynamic time warping algorithm to perform time domain alignment on the multimodal dataset with spatiotemporal anchor markers as a benchmark, extracting features and combining them into a cross-modal feature vector; inputting the cross-modal feature vector into a pre-trained LSTM-decision tree fusion diagnostic model, and outputting a quantitative assessment report of the disorder type and degree grading, thereby improving the accuracy of cognitive impairment diagnosis.

Description

Technical Field

[0001] The present invention belongs to the technical field of data analysis, and in particular to an auditory cognitive dysfunction assessment and screening system. Background Art

[0002] Auditory cognitive dysfunction is a type of disease that affects people's processing and understanding of sound information. Its early screening and accurate assessment are crucial for the diagnosis and treatment of the disease. Existing methods for assessing auditory cognitive function have many shortcomings. For example, the data collection is single and can only collect voice data, which cannot fully reflect the patient's auditory cognitive status; the test items are fixed and the test difficulty and content cannot be intelligently adjusted according to the patient's real-time performance, resulting in inaccurate assessment results; the data processing and analysis are not comprehensive and in-depth enough to generate a comprehensive assessment report; the storage and interaction functions are not perfect, which is not conducive to the management and screening of patient information and test data by medical staff. Therefore, there is an urgent need for an innovative auditory cognitive dysfunction assessment and screening system to improve the accuracy, comprehensiveness and efficiency of the assessment. Summary of the Invention

[0003] In response to the shortcomings of the existing technology, the present invention proposes an auditory cognitive dysfunction assessment and screening system, which sets basic test parameters and initial stimulation parameters, and combines spatiotemporal anchor markers to synchronously collect voice, eye movement trajectory data and brain wave signals to form a multimodal data set with a timestamp; conducts multi-stage cognitive testing in an adaptive test engine, and dynamically adjusts the stimulation parameters based on real-time accuracy and reaction time using a forgetting curve prediction algorithm; based on the adjusted stimulation parameters, a dynamic time warping algorithm is used to perform time domain alignment on the multimodal data set with spatiotemporal anchor markers as a benchmark, extract features and combine them into a cross-modal feature vector; the cross-modal feature vector is input into a pre-trained LSTM-decision tree fusion diagnostic model, and a quantitative assessment report of the disorder type and degree classification is output, thereby improving the accuracy of cognitive impairment diagnosis.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] The auditory cognitive dysfunction assessment and screening system includes: a data acquisition module, a test engine module, a feature extraction module, and a diagnosis and evaluation module; the test engine module includes a test execution unit, a data monitoring unit, and an algorithm driving unit;

[0006] The data acquisition module sets basic test parameters and initial stimulation parameters through the medical terminal interface, and synchronously collects voice signals, eye movement trajectory data and brain wave signals under spatiotemporal anchoring marks to form a multimodal data set with time stamps, and transmits it to the test execution unit for multi-stage cognitive testing;

[0007] Dynamically adjust the stimulation parameters by using the forgetting curve prediction algorithm configured in the algorithm driving unit and combining the real-time accuracy, reaction time, and eye movement attention index monitored by the data monitoring unit;

[0008] Based on the adjusted stimulation parameters, the dynamic time warping algorithm configured in the feature extraction module is used to perform time domain alignment processing on the multimodal dataset with timestamps using the spatiotemporal anchor markers as a reference, and speech response delay, eye movement focus distribution entropy, and EEG power ratio features are extracted, and a cross-modal feature vector is formed by combining them;

[0009] The cross-modal feature vector is input into the pre-trained LSTM-decision tree fusion diagnosis model configured in the diagnosis and evaluation module, and a quantitative evaluation report of the obstacle type and degree classification is output.

[0010] Specifically, the data acquisition module integrates a microphone, an eye tracking device and an EEG monitoring device. The eye tracking device monitors the patient's eye gaze point and eye movement trajectory in real time when listening to voice information, and obtains the patient's visual attention to different auditory stimuli; the EEG monitoring device collects the electrophysiological activities of the patient's brain during the process of auditory information processing.

[0011] Specifically, the data acquisition module sets basic test parameters and initial stimulation parameters through the medical terminal interface, and synchronously collects voice signals, eye movement trajectory data, and brain wave signals under spatiotemporal anchoring marks to form a multimodal data set with time stamps, including:

[0012] A1: Set basic test parameters and initial stimulation parameters to generate a structured parameter configuration file; the basic test parameters include test type, subject information, and test environment parameters; the stimulation parameters include stimulation intensity, presentation frequency, and time interval;

[0013] A2: The master control server reads the structured parameter configuration file and sends a synchronization signal to the data acquisition device via the PTP protocol. Simultaneously, the data acquisition device calibrates the local clock after receiving the synchronization signal and records the device calibration timestamp. The data acquisition device includes an EEG monitoring device, an eye tracking device, and a microphone. The eye tracking device refers to an eye tracker.

[0014] A3: The system displays a 9-point checkerboard calibration pattern. While the subject is looking at the pattern, pupil coordinates and visual evoked potential data are collected simultaneously. The collected pupil coordinates and visual evoked potential data are used to calculate the affine transformation matrix. The affine transformation matrix is ​​solved using the least squares method to obtain the eye movement-screen coordinate mapping matrix.

[0015] A4: Configure the GPIO trigger circuit, set the stimulus presentation mechanism and pulse width parameters, and record the local trigger time when the data acquisition device detects the pulse generated by the GPIO trigger circuit.

[0016] Specifically, the data acquisition module sets basic test parameters and initial stimulation parameters through the medical terminal interface, and synchronously collects voice signals, eye movement trajectory data, and brain wave signals under spatiotemporal anchoring to form a multimodal data set with a time stamp. It also includes:

[0017] A5: Based on the calibrated data acquisition equipment, synchronously collect data to form a multimodal data stream and add a local acquisition timestamp to each frame of data;

[0018] A6: Based on the multimodal data stream and GPIO trigger events, the GPIO rising edge pulse time is detected and combined with the calibrated local trigger time recorded by the data acquisition device and the device calibration timestamp to calculate the global timestamp; the device calibration timestamp is calibrated using the PTP protocol;

[0019] A7: Determine the capture time window based on the global timestamp and the preset leading and trailing times. The leading time is the estimated length of time for data collection before the event occurs, and the trailing time is the estimated length of time for data collection after the event occurs.

[0020] A8: According to the interception time window, speech, eye movement, and EEG data are intercepted from the multimodal data stream and formed into a multimodal dataset with a time stamp after Lagrangian interpolation.

[0021] Specifically, the process of forming the multimodal data stream in A5 includes:

[0022] A5.1: A microphone is used to collect speech signals, which are then enhanced by a bandpass filter and an adaptive beamforming algorithm.

[0023] A5.2: Use pupil detection algorithms to calculate pupil coordinates and pupil diameter in real time and generate eye movement trajectory data;

[0024] A5.3: Use EEG monitoring equipment to collect brain wave signals;

[0025] A5.4: Combine the enhanced speech signal, eye movement data, and EEG signal to form a multimodal data stream.

[0026] Specifically, the forgetting curve prediction algorithm configured in the algorithm driving unit is combined with the real-time accuracy, reaction time, and eye movement attention index monitored by the data monitoring unit to dynamically adjust the stimulation parameters, including:

[0027] B1: In the multi-stage cognitive test, the accuracy, reaction time, and eye movement attention index are collected in real time and standardized to obtain the real-time state vector ,in, represents the normalized accuracy, represents the normalized reaction time, represents the eye movement attention index;

[0028] B2: Use weighted linear combination method to transform the real-time state vector Mapped to memory retention rate, we get the forgetting curve state model ,in, 、 、 、 represents the forgetting curve state model parameters.

[0029] Specifically, the forgetting curve prediction algorithm configured in the algorithm driving unit is combined with the real-time accuracy, reaction time, and eye movement attention index monitored by the data monitoring unit to dynamically adjust the stimulation parameters, and further includes:

[0030] B3: Combine time series data and use exponential smoothing to predict the memory retention rate at the next moment ; The time series data is the memory retention rate of the past m time windows ,in, represents the smoothing coefficient, represents the predicted value of memory retention rate at time t;

[0031] B4: Set target memory retention rate range ,when Trigger adjustment when deviating from the target memory retention rate range, combined with prediction deviation , through the formula Get the adjusted stimulation parameters ,in, and Respectively represent the minimum and maximum values ​​of the target memory retention rate interval, Indicates the preset target memory retention rate, represents the current stimulation parameter, and k represents the adjustment coefficient.

[0032] Specifically, based on the adjusted stimulation parameters, the dynamic time warping algorithm configured in the feature extraction module performs time domain alignment processing on the multimodal dataset with timestamps based on the spatiotemporal anchor markers, extracts speech response delay, eye movement focus distribution entropy, and EEG power ratio features, and forms a cross-modal feature vector by combining them, including:

[0033] C1: Obtain a multimodal dataset with timestamps and spatiotemporal anchor markers, and perform preprocessing to obtain a preprocessed multimodal dataset;

[0034] C2: Calculate the distance matrix between each pair of different modal data in the preprocessed multimodal dataset. Starting from the upper left corner of the distance matrix, gradually calculate the minimum distance to each point along the shortest path to form a cumulative distance matrix.

[0035] C3: Starting from the lower right corner of the cumulative distance matrix, trace back to find the optimal path, that is, the path with the minimum distance. According to the principle of the shortest distance, the correspondence between the data elements of different modalities is constructed to complete the time domain alignment.

[0036] Specifically, based on the adjusted stimulation parameters, the dynamic time warping algorithm configured in the feature extraction module performs time domain alignment processing on the multimodal dataset with timestamps using the spatiotemporal anchor markers as a reference, extracts speech response delay, eye movement focus distribution entropy, and EEG power ratio features, and forms a cross-modal feature vector by combining them, and also includes:

[0037] C4: Calculate the time interval from stimulus presentation to the start of speech response as the speech response latency feature;

[0038] C5: Use an eye tracker to obtain the gaze point coordinates and calculate the horizontal gaze entropy based on the gaze point coordinates and vertical gaze entropy , and the entropy value of eye movement focus distribution is obtained comprehensively ;

[0039] C6: Preprocessing the EEG signal and dividing the preprocessed EEG signal into different frequency bands, calculating the power of each frequency band, and calculating the power ratio between the frequency bands based on the power of each frequency band to obtain an EEG power ratio feature; the preprocessing includes filtering and denoising;

[0040] C7: Combine the extracted speech response delay features, eye movement focus distribution entropy, and EEG power ratio features to form a cross-modal feature vector.

[0041] Specifically, the cross-modal feature vector is input into a pre-trained LSTM-decision tree fusion diagnosis model configured in the diagnosis and evaluation module, and a quantitative evaluation report of the obstacle type and degree classification is output, including:

[0042] D1: Obtain a cross-modal feature vector and perform normalization on the cross-modal feature vector to obtain a normalized cross-modal feature vector;

[0043] D2: Input the standardized cross-modal feature vector into the LSTM layer of the pre-trained LSTM-decision tree fusion diagnosis model;

[0044] D3: The LSTM layer performs feature extraction and dimensionality reduction on the input standardized cross-modal feature vector to obtain the main feature vector;

[0045] D4: Pass the main feature vector output by the LSTM layer to the decision tree fusion layer;

[0046] D5: Each decision tree classifies the input main feature vector according to its own preset feature classification rules, and obtains its own judgment results on the obstacle type and severity classification. The judgment results of multiple decision trees are combined through weighted fusion to obtain the final obstacle type and severity classification.

[0047] D6: Generate a quantitative assessment report based on the determined obstacle type and degree classification, combined with pre-set assessment criteria, and output the quantitative assessment report in text or table format.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. The present invention proposes an auditory cognitive dysfunction assessment and screening system, and optimizes and improves the architecture, operating steps and processes. The system has the advantages of simple process, low investment and operating costs, and low production work costs.

[0050] 2. The present invention proposes an auditory cognitive dysfunction assessment and screening system. By setting basic test and initial stimulation parameters and combining spatiotemporal anchor markers to synchronously collect multiple signals, a multimodal dataset with timestamps is formed. Multi-stage cognitive testing is carried out in an adaptive testing engine, and stimulation parameters are dynamically adjusted based on real-time data, making the testing process more in line with the actual situation of the subjects, improving the pertinence and effectiveness of the test, and helping to more accurately assess the cognitive ability of the subjects.

[0051] 3. The present invention proposes an auditory cognitive dysfunction assessment and screening system. Based on the adjusted stimulation parameters, a dynamic time warping algorithm is used to perform time-domain alignment processing on the multimodal dataset and extract key features to form a cross-modal feature vector. The pre-trained LSTM-decision tree fusion diagnostic model is then input to output a quantitative assessment report of the type and degree of impairment. This realizes the integration of data collection, processing, and diagnostic evaluation, and can quickly and accurately provide diagnostic results, providing support for the early detection and intervention of cognitive impairment. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a diagram of the auditory cognitive dysfunction assessment and screening system architecture of the present invention;

[0053] Figure 2 This is a flow chart showing the principle of the auditory cognitive dysfunction assessment and screening system of the present invention;

[0054] Figure 3 A flow chart is provided for forming a multimodal dataset with time stamps for the auditory cognitive dysfunction assessment and screening system of the present invention. DETAILED DESCRIPTION

[0055] Example 1:

[0056] See also Figure 1 and Figure 2 , an embodiment provided by the present invention: an auditory cognitive dysfunction assessment and screening system, comprising a data acquisition module, a test engine module, a feature extraction module, and a diagnosis and assessment module;

[0057] The data acquisition module synchronously collects voice signals, eye movement data, and brain wave signals by initializing the test environment and parameter configuration; the parameter settings include basic test parameter settings and initial stimulation parameter settings;

[0058] It should be noted that the data acquisition module integrates an eye tracking device and an EEG monitoring device. The eye tracking device monitors the patient's eye gaze point and eye movement trajectory in real time when listening to voice information, and obtains the patient's visual attention to different auditory stimuli; the EEG monitoring device collects the electrophysiological activities of the patient's brain during the auditory information processing process.

[0059] A test engine module for executing multi-stage cognitive tests and dynamically adjusting test parameters to suit the subject's ability level;

[0060] The feature extraction module is used to extract speech response delay, eye movement focus distribution entropy, and EEG power ratio features, and combine them to form a cross-modal feature vector;

[0061] The diagnosis and assessment module identifies obstacle types and grades their severity based on cross-modal feature vectors, and generates a quantitative assessment report.

[0062] The data acquisition module includes: medical interaction unit, spatiotemporal anchoring unit, and signal acquisition unit;

[0063] The medical interaction unit is used to provide a medical interface and configure test parameters, including test type and difficulty level.

[0064] The spatiotemporal anchoring unit is used to generate spatiotemporal coordinate markers to provide a temporal and spatial alignment benchmark for speech signals, eye movement data, and EEG signals;

[0065] The signal acquisition unit is used to synchronously collect voice signals, eye movement trajectory data, and brain wave signals.

[0066] The test engine module includes: test execution unit, data monitoring unit, and algorithm driving unit;

[0067] A test execution unit for managing the process of multi-stage cognitive testing; the multi-stage cognitive testing includes vocabulary learning, story retelling, digit span, vocabulary recall, vocabulary re-recognition, and story recall;

[0068] Data monitoring unit, used for real-time analysis of accuracy, reaction time, and eye movement attention index;

[0069] The algorithm driving unit is used to dynamically adjust the stimulation parameters, including stimulation intensity, presentation frequency, and time interval, based on the forgetting curve prediction algorithm.

[0070] The feature extraction module includes: time domain alignment unit, feature extraction unit, and feature fusion unit;

[0071] The time domain alignment unit is used to align the time axis of multimodal datasets with timestamps using a dynamic time warping algorithm;

[0072] The feature extraction unit is used to calculate the voice response delay feature, the eye movement focus distribution entropy value feature, and the brain wave power ratio feature of a specific frequency band; the specific frequency band refers to Wave power ratio;

[0073] The feature fusion unit is used to combine the extracted speech response delay features, eye movement focus distribution entropy features, and specific frequency band EEG power ratio features into a cross-modal feature vector.

[0074] The diagnosis and evaluation module includes: model reasoning unit, evaluation generation unit, and report output unit;

[0075] The model inference unit is used to run the pre-trained LSTM-decision tree fusion diagnosis model, input the cross-modal feature vector, and output the diagnosis result;

[0076] It needs to be explained that the Long Short-Term Memory (LSTM) network is a recurrent neural network used to process sequence data.

[0077] An evaluation generation unit, used to convert the output of the LSTM-decision tree fusion diagnosis model into a barrier type and severity rating;

[0078] The report output unit is used to generate structured quantitative assessment reports, which can be viewed and exported by medical staff.

[0079] It needs to be explained that the auditory cognitive dysfunction assessment screening is a process of determining whether an individual has abnormal auditory cognitive abilities, and aims to detect disorders in the synergy between auditory and cognitive processing at an early stage.

[0080] Furthermore, the overall implementation process of the auditory cognitive dysfunction assessment and screening system includes:

[0081] S1: Medical staff set basic test parameters and initial stimulation parameters through the medical terminal interface. Combined with the spatiotemporal anchoring method, they synchronously collect voice signals, eye movement data, and brain wave signals to form a multimodal dataset with a time stamp.

[0082] S2: Executing a multi-stage cognitive test in an adaptive testing engine, dynamically adjusting stimulation parameters using a forgetting curve prediction algorithm based on real-time accuracy, reaction time, and eye movement attention index; the multi-stage cognitive test includes vocabulary learning, story retelling, digit span, vocabulary recall, vocabulary recognition, and story recall tests; the stimulation parameters include stimulation intensity, presentation frequency, and time interval;

[0083] S3: Based on the adjusted stimulation parameters, a dynamic time warping algorithm is used to perform time-domain alignment on the multimodal dataset with time stamps using spatiotemporal anchor markers as a benchmark. The speech response delay, eye movement focus distribution entropy, and EEG power ratio features are extracted and combined to form a cross-modal feature vector.

[0084] S4: Input the cross-modal feature vector into the pre-trained LSTM-decision tree fusion diagnosis model to output a quantitative assessment report of the obstacle type and severity.

[0085] Example 2:

[0086] See also Figure 3 In this embodiment, the data acquisition module integrates a microphone, an eye tracking device and an EEG monitoring device. The eye tracking device monitors the patient's eye gaze point and eye movement trajectory in real time when listening to voice information, and obtains the patient's visual attention to different auditory stimuli; the EEG monitoring device collects the electrophysiological activities of the patient's brain during the process of auditory information processing.

[0087] The data acquisition module sets basic test parameters and initial stimulation parameters through the medical terminal interface. Under the spatiotemporal anchor mark, it synchronously collects voice signals, eye movement trajectory data, and brain wave signals to form a multimodal data set with a time stamp, including:

[0088] A1: Medical staff set basic test parameters and initial stimulation parameters through the medical terminal interface to generate a structured parameter configuration file; the basic test parameters include test type, subject information, and test environment parameters; the stimulation parameters include stimulation intensity, presentation frequency, and time interval;

[0089] Among them, test types include auditory memory test and language comprehension test; subject information includes age, gender, and medical history; test environment parameters include background noise and light intensity.

[0090] A2: The master server reads the structured parameter configuration file and sends a synchronization signal to the data acquisition device through the PTP protocol. At the same time, the data acquisition device calibrates the local clock after receiving the synchronization signal and records the device calibration timestamp. ; The data acquisition equipment includes an EEG monitoring device, an eye tracking device, and a microphone; the eye tracking device refers to an eye tracker;

[0091] A3: The system displays a 9-point checkerboard calibration pattern on the screen. While the subject is looking at the pattern, pupil coordinates and visual evoked potential data are collected simultaneously. The collected pupil coordinates and visual evoked potential data are used to calculate the affine transformation matrix. The affine transformation matrix is ​​solved using the least squares method to obtain the eye movement-screen coordinate mapping matrix.

[0092] Among them, the screen refers to the display device that presents auditory cognitive function test content, 9-point checkerboard calibration patterns and other visual information to the subjects. Common types include computer monitors, tablet computer screens and special test equipment displays.

[0093] Furthermore, the specific steps of A3 include:

[0094] (1) Nine calibration targets are evenly distributed on the screen, namely Grid;

[0095] (2) Set the target parameters, including diameter, color, and interval. The diameter is set to 10 mm, the color is white for the target, black for the background, and the interval is set to 1 / 4 of the screen.

[0096] (3) Use pupil detection algorithm to collect pupil coordinates ;

[0097] (4) By calculating the brain wave signal after the jth stimulation The mean value over the number of stimulation repetitions N is used to obtain the visual evoked potential data ;

[0098] (5) Establish screen coordinates and pupil coordinates The mapping relationship is: , which is expanded into a system of linear equations: , where a, b, c, d, e, and h represent the parameters to be solved;

[0099] (6) For n calibration targets, collect the actual screen coordinates of n calibration targets and the corresponding pupil coordinates ,in, and Respectively represent the actual x-axis coordinate and y-axis coordinate of the i-th calibration target point on the screen, represents the i-th calibration target point on screen s;

[0100] (7) Construct a system of linear equations: ,in, represents the parameter vector to be solved, T represents the transpose, represents the error term, X represents the design matrix consisting of pupil coordinates, and satisfies: , Y represents the screen coordinate vector and satisfies: ,in, and represent the x-axis coordinate and y-axis coordinate of the i-th calibration target point in the physical space, respectively, and ;

[0101] (8) Solution , get the parameter vector to be solved The specific value of each parameter in;

[0102] (9) Matrixing is performed to obtain the eye movement-screen coordinate mapping matrix .

[0103] A4: Configure the GPIO trigger circuit, set the stimulus presentation mechanism and pulse width parameters, and record the local trigger time when the data acquisition device detects the pulse generated by the GPIO trigger circuit ;

[0104] A5: Based on the calibrated data acquisition equipment, synchronously collect data to form a multimodal data stream and add a local acquisition timestamp to each frame of data ;

[0105] A6: Based on multimodal data streams and GPIO trigger events, detect the GPIO rising edge pulse time, combined with the calibrated local trigger time recorded by the data acquisition device and device calibration timestamp , a global timestamp is calculated; the device calibration timestamp is obtained by calibration via the PTP protocol;

[0106] Among them, the global timestamp provides a unified time base for multimodal data, ensuring the consistency of different modal data in the time dimension.

[0107] Furthermore, the specific steps of A6 include:

[0108] (1) Obtain device calibration timestamp , local trigger time and local collection timestamp ;

[0109] (2) Associate the multimodal data stream with the GPIO trigger event and mark the starting point of the data stream corresponding to the trigger event;

[0110] (3) Obtain the device calibration timestamp based on the PTP protocol calibration results and the master server clock Deviation ;

[0111] (4) Set the local trigger time Convert to global timestamp ;

[0112] (5) Local collection timestamp Convert to global timestamp Next time.

[0113] A7: Determine the capture time window based on the global timestamp and the preset leading and trailing times. The leading time is the estimated length of time for data collection before the event occurs, and the trailing time is the estimated length of time for data collection after the event occurs.

[0114] For example, assuming that the leading time is set to 500 ms and the following time is set to 1000 ms, the data from 500 ms before to 1000 ms after the stimulus presentation moment is taken as a reference.

[0115] A8: According to the interception time window, the speech, eye movement, and EEG data within the time window are intercepted from the multimodal data stream, and Lagrangian interpolation is performed to form a multimodal data set with a timestamp. Among them, Lagrangian interpolation is the existing technical content in this field and is not an inventive solution of this application, so it will not be elaborated here.

[0116] The process of forming the multimodal data stream in A5 includes:

[0117] A5.1: A microphone is used to collect speech signals, and the enhanced speech signals are obtained after passing them through a bandpass filter and an adaptive beamforming algorithm. The bandpass filter and the adaptive beamforming algorithm are prior art in this field and do not constitute the inventive solution of this application, and are not described in detail here.

[0118] A5.2: Use pupil detection algorithms to calculate pupil coordinates and pupil diameter in real time and generate eye movement trajectory data;

[0119] Furthermore, the specific steps of A5.2 include:

[0120] (1) Collect images of the human eye area through a camera and perform preprocessing;

[0121] (2) Using Hough circle transform to detect the pupil, obtain the initial circle center and radius, and obtain the initial circle. Hough circle transform is a prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0122] (3) Optimizing pupil positioning near the initial circle using a contour matching method to calculate pupil center coordinates and diameter. The contour matching method is a prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0123] (4) Directly measuring the distance between the two farthest points and estimating the diameter using an ellipse fitting method. The ellipse fitting method is a prior art in this field and is not an inventive solution of the present application, so it will not be described in detail here.

[0124] (5) Calculate the pupil center coordinates frame by frame in the video stream, and arrange the pupil center coordinates of consecutive frames in chronological order to generate eye movement trajectory data.

[0125] A5.3: Use EEG monitoring equipment to collect brain wave signals;

[0126] A5.4: Combine the enhanced speech signal, eye movement data, and EEG signal to form a multimodal data stream.

[0127] The forgetting curve prediction algorithm configured in the algorithm driving unit is combined with the real-time accuracy, reaction time, and eye movement attention index monitored by the data monitoring unit to dynamically adjust the stimulation parameters, including:

[0128] B1: In the multi-stage cognitive test, the accuracy, reaction time, and eye movement attention index are collected in real time and standardized to obtain the real-time state vector ,in, represents the normalized accuracy, represents the normalized reaction time, represents the eye movement attention index;

[0129] B2: Use weighted linear combination method to transform the real-time state vector Mapped to memory retention rate, we get the forgetting curve state model ,in, 、 、 、 Represents the parameters of the forgetting curve state model, which is determined by training with historical data;

[0130] B3: Combine time series data and use exponential smoothing to predict the memory retention rate at the next moment ; The time series data is the memory retention rate of the past m time windows ,in, represents the smoothing coefficient, represents the predicted value of memory retention rate at time t;

[0131] B4: Set target memory retention rate range ,when Trigger adjustment when deviating from the target memory retention rate range, combined with prediction deviation , through the formula Get the adjusted stimulation parameters ,in, and Respectively represent the minimum and maximum values ​​of the target memory retention rate interval, Indicates the preset target memory retention rate, represents the current stimulation parameter, k represents the adjustment coefficient, and the present invention takes .

[0132] Based on the adjusted stimulation parameters, the dynamic time warping algorithm configured in the feature extraction module performs time domain alignment processing on the multimodal dataset with timestamps using the spatiotemporal anchor markers as a reference, extracts speech response delay, eye movement focus distribution entropy, and EEG power ratio features, and forms a cross-modal feature vector by combining them, including:

[0133] C1: Obtain a multimodal dataset with timestamps and spatiotemporal anchor markers, and perform preprocessing to obtain a preprocessed multimodal dataset;

[0134] C2: Calculate the distance matrix between each pair of different modal data in the preprocessed multimodal dataset. Starting from the upper left corner of the distance matrix, gradually calculate the minimum distance to each point along the shortest path to form a cumulative distance matrix.

[0135] Furthermore, in the present invention, the distance calculation formula adopts Euclidean distance, wherein the Euclidean distance is the existing technical content in this field and is not an inventive solution of this application, and will not be described here in detail.

[0136] Furthermore, the specific steps of calculating the distance matrix between different modal data in the multimodal dataset include:

[0137] (1) Obtain preprocessed multimodal dataset;

[0138] (2) Select data from different modalities from the preprocessed multimodal dataset and combine them in pairs to form data pairs. For example, for data from three modalities, namely speech, eye movement, and EEG, it is necessary to calculate the distance matrices of the three combinations of speech-eye movement, speech-EEG, and eye movement-EEG.

[0139] (3) Calculate the distance value for each data pair using the Euclidean distance formula;

[0140] (4) Create an empty distance matrix;

[0141] (5) Fill the calculated distance values ​​into the corresponding positions in the matrix to obtain the distance matrix. For example, for the distance values ​​of the speech-eye movement data pair, fill in the 1st row and 2nd column and the 2nd row and 1st column of the matrix.

[0142] C3: Starting from the lower right corner of the cumulative distance matrix, backtrack to find the optimal path, that is, the path with the minimum distance. Based on the principle of the shortest distance, the correspondence between the data elements of different modalities is constructed to complete the time domain alignment;

[0143] C4: Calculate the time interval from stimulus presentation to the start of speech response as the speech response latency feature;

[0144] C5: Use an eye tracker to obtain the gaze point coordinates and calculate the horizontal gaze entropy based on the gaze point coordinates and vertical gaze entropy , and the entropy value of eye movement focus distribution is obtained comprehensively ; The eye movement focus distribution entropy value is to turn the horizontal gaze entropy and vertical gaze entropy To achieve the summation;

[0145] Furthermore, horizontal gaze entropy It is calculated by calculating the horizontal coordinate value of the rth fixation point Probability of occurrence and The corresponding logarithmic value The product of and the cumulative summation of all fixation points are achieved.

[0146] Vertical gaze entropy It is calculated by calculating the vertical coordinate value of the rth fixation point Probability of occurrence and The corresponding logarithmic value The product of and the cumulative summation of all fixation points are achieved.

[0147] C6: Preprocessing the EEG signal and dividing the preprocessed EEG signal into different frequency bands, calculating the power of each frequency band, and calculating the power ratio between the frequency bands based on the power of each frequency band to obtain an EEG power ratio feature; the preprocessing includes filtering and denoising;

[0148] Furthermore, the power of each frequency band is the mean value of the discrete brain wave signal under the signal length.

[0149] Furthermore, the power ratio between frequency bands is the ratio of frequency band powers of different brain wave signals.

[0150] C7: Combine the extracted speech response delay features, eye movement focus distribution entropy, and EEG power ratio features to form a cross-modal feature vector.

[0151] The cross-modal feature vector is input into a pre-trained LSTM-decision tree fusion diagnosis model configured in the diagnosis and evaluation module, and a quantitative evaluation report of the obstacle type and degree classification is output, including:

[0152] D1: Obtaining cross-modal feature vectors and organizing them to ensure that the feature vector dimensions of each sample are consistent, including speech response delay, eye movement focus distribution entropy, and EEG power ratio features. At the same time, the cross-modal feature vectors are normalized to obtain a normalized cross-modal feature vector. The present invention adopts the Z-score normalization method, which is a prior art in this field and does not constitute an inventive solution of the present application and is not described in detail here.

[0153] D2: Input the standardized cross-modal feature vector into the LSTM layer of the pre-trained LSTM-decision tree fusion diagnostic model. The LSTM layer can process sequence data and capture the temporal dependencies and long-term memory information in the standardized cross-modal feature vector. LSTM is prior art in this field and does not constitute an inventive solution of this application, so it will not be described in detail here.

[0154] D3: The LSTM layer extracts features and reduces the dimensionality of the input normalized cross-modal feature vector to obtain the main feature vector. The LSTM layer uses a series of gating mechanisms, such as the input gate, forget gate, and output gate, to control the inflow and outflow of information and memory updates to extract feature representations.

[0155] D4: The main feature vector output by the LSTM layer is passed to the decision tree fusion layer. The decision tree fusion layer consists of multiple decision trees, each of which classifies and judges the input main feature vector based on different features and rules.

[0156] D5: Each decision tree classifies the input main feature vector according to its own preset feature classification rules, and obtains its own judgment results on the obstacle type and severity classification. The judgment results of multiple decision trees are then combined through weighted fusion to obtain the final obstacle type and severity classification. The weighted fusion is a prior art in this field and does not constitute an inventive solution of this application, so it will not be described in detail here.

[0157] For example, mild, moderate, and severe are represented by numerical values ​​of 1, 2, and 3, respectively.

[0158] D6: Generate a quantitative assessment report based on the determined obstacle type and degree classification, combined with pre-set assessment criteria, and output the quantitative assessment report in text or table format.

[0159] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also change, modify, replace and modify the above-mentioned embodiments without departing from the purpose and scope of protection of the present invention. These are all protected by the present invention.

[0160] If the technical solution disclosed herein involves personal information, the product using the technical solution disclosed herein has clearly informed the individual of the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution disclosed herein involves sensitive personal information, the product using the technical solution disclosed herein has obtained the individual's separate consent before processing the sensitive personal information and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the individual has entered the personal information collection scope and that personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information. The personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. Auditory cognitive dysfunction assessment and screening system, characterized by: include: Data acquisition module, test engine module, feature extraction module, diagnosis and evaluation module; The test engine module includes a test execution unit, a data monitoring unit, and an algorithm driving unit; The data acquisition module sets basic test parameters and initial stimulation parameters through the medical terminal interface, and synchronously collects voice signals, eye movement trajectory data and brain wave signals under spatiotemporal anchoring marks to form a multimodal data set with time stamps, and transmits it to the test execution unit for multi-stage cognitive testing; Dynamically adjust the stimulation parameters by using the forgetting curve prediction algorithm configured in the algorithm driving unit and combining the real-time accuracy, reaction time, and eye movement attention index monitored by the data monitoring unit; Based on the adjusted stimulation parameters, the dynamic time warping algorithm configured in the feature extraction module is used to perform time domain alignment processing on the multimodal dataset with timestamps using the spatiotemporal anchor markers as a reference, and speech response delay, eye movement focus distribution entropy, and EEG power ratio features are extracted, and a cross-modal feature vector is formed by combining them; Input the cross-modal feature vector into the pre-trained LSTM-decision tree fusion diagnosis model configured in the diagnosis and evaluation module, and output a quantitative evaluation report of the obstacle type and severity classification; The forgetting curve prediction algorithm configured in the algorithm driving unit is combined with the real-time accuracy, reaction time, and eye movement attention index monitored by the data monitoring unit to dynamically adjust the stimulation parameters, including: B1: In the multi-stage cognitive test, the accuracy, reaction time, and eye movement attention index are collected in real time and standardized to obtain the real-time state vector ,in, represents the normalized accuracy, represents the normalized reaction time, represents the eye movement attention index; B2: Use weighted linear combination method to transform the real-time state vector Mapped to memory retention rate, we get the forgetting curve state model ,in, 、 、 、 represents the forgetting curve state model parameters.

2. The auditory cognitive dysfunction assessment and screening system according to claim 1, wherein: The data acquisition module integrates a microphone, an eye tracking device and an EEG monitoring device. The eye tracking device monitors the patient's eye gaze point and eye movement trajectory in real time when listening to voice information, and obtains the patient's visual attention to different auditory stimuli; the EEG monitoring device collects the electrophysiological activities of the patient's brain during the process of auditory information processing.

3. The auditory cognitive dysfunction assessment and screening system according to claim 2, wherein: The data acquisition module sets basic test parameters and initial stimulation parameters through the medical terminal interface. Under the spatiotemporal anchor mark, it synchronously collects voice signals, eye movement trajectory data, and brain wave signals to form a multimodal data set with a time stamp, including: A1: Set basic test parameters and initial stimulation parameters to generate a structured parameter configuration file; the basic test parameters include test type, subject information, and test environment parameters; the stimulation parameters include stimulation intensity, presentation frequency, and time interval; A2: The master control server reads the structured parameter configuration file and sends a synchronization signal to the data acquisition device via the PTP protocol. Simultaneously, the data acquisition device calibrates the local clock after receiving the synchronization signal and records the device calibration timestamp. The data acquisition device includes an EEG monitoring device, an eye tracking device, and a microphone. The eye tracking device refers to an eye tracker. A3: The system displays a 9-point checkerboard calibration pattern. While the subject is looking at the pattern, pupil coordinates and visual evoked potential data are collected simultaneously. The collected pupil coordinates and visual evoked potential data are used to calculate the affine transformation matrix. The affine transformation matrix is ​​solved using the least squares method to obtain the eye movement-screen coordinate mapping matrix. A4: Configure the GPIO trigger circuit, set the stimulus presentation mechanism and pulse width parameters, and record the local trigger time when the data acquisition device detects the pulse generated by the GPIO trigger circuit.

4. The auditory cognitive dysfunction assessment and screening system according to claim 3, wherein: The data acquisition module sets basic test parameters and initial stimulation parameters through the medical terminal interface, and synchronously collects voice signals, eye movement trajectory data, and brain wave signals under spatiotemporal anchor marks to form a multimodal data set with a time stamp. It also includes: A5: Based on the calibrated data acquisition equipment, synchronously collect data to form a multimodal data stream and add a local acquisition timestamp to each frame of data; A6: Based on the multimodal data stream and GPIO trigger events, the GPIO rising edge pulse time is detected and combined with the calibrated local trigger time recorded by the data acquisition device and the device calibration timestamp to calculate the global timestamp; the device calibration timestamp is calibrated using the PTP protocol; A7: Determine the capture time window based on the global timestamp and the preset leading and trailing times. The leading time is the estimated length of time for data collection before the event occurs, and the trailing time is the estimated length of time for data collection after the event occurs. A8: According to the interception time window, speech, eye movement, and EEG data are intercepted from the multimodal data stream and formed into a multimodal dataset with a time stamp after Lagrangian interpolation.

5. The auditory cognitive dysfunction assessment and screening system according to claim 4, wherein: The process of forming the multimodal data stream in A5 includes: A5.1: A microphone is used to collect speech signals, which are then enhanced by a bandpass filter and an adaptive beamforming algorithm. A5.2: Use pupil detection algorithms to calculate pupil coordinates and pupil diameter in real time and generate eye movement trajectory data; A5.3: Use EEG monitoring equipment to collect brain wave signals; A5.4: Combine the enhanced speech signal, eye movement data, and EEG signal to form a multimodal data stream.

6. The auditory cognitive dysfunction assessment and screening system according to claim 5, wherein: The method of dynamically adjusting the stimulation parameters by using the forgetting curve prediction algorithm configured in the algorithm driving unit in combination with the real-time accuracy, reaction time, and eye movement attention index monitored by the data monitoring unit further includes: B3: Combine time series data and use exponential smoothing to predict the memory retention rate at the next moment ; The time series data is the memory retention rate of the past m time windows ,in, represents the smoothing coefficient, represents the predicted value of memory retention rate at time t; B4: Set target memory retention rate range ,when Trigger adjustment when deviating from the target memory retention rate range, combined with prediction deviation , through the formula Get the adjusted stimulation parameters ,in, and Respectively represent the minimum and maximum values ​​of the target memory retention rate interval, Indicates the preset target memory retention rate, represents the current stimulation parameter, and k represents the adjustment coefficient.

7. The auditory cognitive dysfunction assessment and screening system according to claim 6, wherein: Based on the adjusted stimulation parameters, the dynamic time warping algorithm configured in the feature extraction module performs time domain alignment processing on the multimodal dataset with timestamps using the spatiotemporal anchor markers as a reference, extracts speech response delay, eye movement focus distribution entropy, and EEG power ratio features, and forms a cross-modal feature vector by combining them, including: C1: Obtain a multimodal dataset with timestamps and spatiotemporal anchor markers, and perform preprocessing to obtain a preprocessed multimodal dataset; C2: Calculate the distance matrix between each pair of different modal data in the preprocessed multimodal dataset. Starting from the upper left corner of the distance matrix, gradually calculate the minimum distance to each point along the shortest path to form a cumulative distance matrix. C3: Starting from the lower right corner of the cumulative distance matrix, trace back to find the optimal path, that is, the path with the minimum distance. According to the principle of the shortest distance, the correspondence between the data elements of different modalities is constructed to complete the time domain alignment.

8. The auditory cognitive dysfunction assessment and screening system according to claim 7, wherein: Based on the adjusted stimulation parameters, the dynamic time warping algorithm configured in the feature extraction module performs time domain alignment processing on the multimodal dataset with timestamps using the spatiotemporal anchor markers as a reference, extracts speech response delay, eye movement focus distribution entropy, and brain wave power ratio features, and forms a cross-modal feature vector by combining them, further comprising: C4: Calculate the time interval from stimulus presentation to the start of speech response as the speech response latency feature; C5: Use an eye tracker to obtain the gaze point coordinates and calculate the horizontal gaze entropy based on the gaze point coordinates and vertical gaze entropy , and the entropy value of eye movement focus distribution is obtained comprehensively ; C6: Preprocessing the EEG signal and dividing the preprocessed EEG signal into different frequency bands, calculating the power of each frequency band, and calculating the power ratio between the frequency bands based on the power of each frequency band to obtain an EEG power ratio feature; the preprocessing includes filtering and denoising; C7: Combine the extracted speech response delay features, eye movement focus distribution entropy, and EEG power ratio features to form a cross-modal feature vector.

9. The auditory cognitive dysfunction assessment and screening system according to claim 8, wherein: The cross-modal feature vector is input into a pre-trained LSTM-decision tree fusion diagnosis model configured in the diagnosis and evaluation module, and a quantitative evaluation report of the obstacle type and degree classification is output, including: D1: Obtain a cross-modal feature vector and perform normalization on the cross-modal feature vector to obtain a normalized cross-modal feature vector; D2: Input the standardized cross-modal feature vector into the LSTM layer of the pre-trained LSTM-decision tree fusion diagnosis model; D3: The LSTM layer performs feature extraction and dimensionality reduction on the input standardized cross-modal feature vector to obtain the main feature vector; D4: Pass the main feature vector output by the LSTM layer to the decision tree fusion layer; D5: Each decision tree classifies the input main feature vector according to its own preset feature classification rules, and obtains its own judgment results on the obstacle type and severity classification. The judgment results of multiple decision trees are combined through weighted fusion to obtain the final obstacle type and severity classification. D6: Generate a quantitative assessment report based on the determined obstacle type and degree classification, combined with pre-set assessment criteria, and output the quantitative assessment report in text or table format.

Images