Auditory cognitive impairment evaluating and screening system

Abstract

The invention discloses an auditory cognitive impairment evaluation and screening system, which belongs to the technical field of data analysis, and specifically comprises the following steps: setting basic test parameters and initial stimulation parameters, and synchronously acquiring voice, eye movement trajectory data and brain wave signals by combining space-time anchoring marks to form a multi-modal data set with a timestamp; a multi-stage cognitive test is carried out in an adaptive test engine, and stimulation parameters are dynamically adjusted by using a forgetting curve prediction algorithm according to real-time accuracy and response time; based on the adjusted stimulation parameters, performing time domain alignment on the multi-modal data set by adopting a dynamic time warping algorithm and taking a space-time anchoring mark as a benchmark, extracting features and combining the features into a cross-modal feature vector; and the cross-modal feature vectors are input into a pre-trained LSTM-decision tree fusion diagnosis model, and a quantitative evaluation report of obstacle type and degree grading is output, so that the accuracy of cognitive disorder diagnosis is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data analysis, specifically an auditory cognitive dysfunction assessment and screening system. Background Art

[0002] Auditory cognitive dysfunction is a type of disease that affects people's processing and understanding of sound information. Its early screening and accurate assessment are crucial for the diagnosis and treatment of the disease. Existing auditory cognitive function assessment methods have many deficiencies. For example, data collection is single, only voice data can be collected, and it cannot comprehensively reflect the patient's auditory cognitive state; the test items are fixed, and the test difficulty and content cannot be intelligently adjusted according to the patient's real-time performance, resulting in inaccurate assessment results; data processing and analysis are not comprehensive and in-depth enough to generate a comprehensive assessment report; the storage and interaction functions are imperfect, which is not conducive to the management and screening of patient information and test data by medical staff. Therefore, there is an urgent need for an innovative auditory cognitive dysfunction assessment and screening system to improve the accuracy, comprehensiveness, and efficiency of assessment. Summary of the Invention

[0003] Aiming at the deficiencies of the existing technology, the present invention proposes an auditory cognitive dysfunction assessment and screening system, which sets basic test parameters and initial stimulus parameters, synchronously collects voice, eye movement trajectory data, and electroencephalogram signals under the spatio-temporal anchor marker to form a multi-modal data set with time stamps; conducts multi-stage cognitive tests in the adaptive test engine, and dynamically adjusts the stimulus parameters according to the real-time correct rate and reaction time using the forgetting curve prediction algorithm; based on the adjusted stimulus parameters, uses the dynamic time warping algorithm to perform time domain alignment on the multi-modal data set with the spatio-temporal anchor marker as the reference, extracts features and combines them into cross-modal feature vectors; inputs the cross-modal feature vectors into a pre-trained LSTM-decision tree fusion diagnosis model to output a quantitative assessment report on the type and degree of impairment, improving the accuracy of cognitive impairment diagnosis.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] An auditory cognitive dysfunction assessment and screening system, including: a data collection module, a test engine module, a feature extraction module, and a diagnosis and evaluation module; the test engine module includes a test execution unit, a data monitoring unit, and an algorithm driving unit;

[0006] The data collection module sets basic test parameters and initial stimulus parameters through the medical staff interface, and synchronously collects voice signals, eye movement trajectory data, and electroencephalogram signals under the spatio-temporal anchor marker to form a multi-modal data set with time stamps, and transmits it to the test execution unit for multi-stage cognitive tests;

[0007] Through the forgetting curve prediction algorithm configured in the algorithm driving unit, combined with the real-time accuracy rate, reaction time, and eye movement attention index monitored by the data monitoring unit, the stimulation parameters are dynamically adjusted;

[0008] Based on the adjusted stimulation parameters, through the dynamic time warping algorithm configured in the feature extraction module, the time-domain alignment processing of the timestamped multi-modal data set is carried out with the spatio-temporal anchor mark as the reference, and the speech response delay, eye movement focus distribution entropy value, and brain wave power ratio features are extracted, and a cross-modal feature vector is formed by combination;

[0009] The cross-modal feature vector is input into the pre-trained LSTM-decision tree fusion diagnosis model configured in the diagnosis and evaluation module, and a quantitative evaluation report on the obstacle type and degree classification is output.

[0010] Specifically, the data acquisition module integrates a microphone, an eye movement tracking device, and an electroencephalogram monitoring device. The eye movement tracking device monitors the eye fixation points and eye movement trajectories of the patient in real time when listening to voice information, and obtains the visual attention of the patient to different auditory stimuli; the electroencephalogram monitoring device collects the electrophysiological activities of the patient's brain during the processing of auditory information.

[0011] Specifically, the data acquisition module sets basic test parameters and initial stimulation parameters through the medical staff interface. Under the spatio-temporal anchor mark, voice signals, eye movement trajectory data, and electroencephalogram signals are synchronously collected to form a timestamped multi-modal data set, including:

[0012] A1: Set basic test parameters and initial stimulation parameters to generate a structured parameter configuration file; the basic test parameters include test type, subject information, and test environment parameters; the stimulation parameters include stimulation intensity, presentation frequency, and time interval;

[0013] A2: The main control server reads the structured parameter configuration file and sends a synchronization signal to the data acquisition device through the PTP protocol. At the same time, after receiving the synchronization signal, the data acquisition device calibrates the local clock and records the device calibration timestamp; the data acquisition device includes an electroencephalogram monitoring device, an eye movement tracking device, and a microphone; the eye movement tracking device refers to an eye tracker;

[0014] A3: The system displays a 9-point checkerboard calibration pattern. During the process of the subject's fixation on the pattern, pupil coordinates and visual evoked potential data are synchronously collected. Using the collected pupil coordinates and visual evoked potential data, an affine transformation matrix is calculated, and the affine transformation matrix is solved by the least squares method to obtain an eye movement-screen coordinate mapping matrix;

[0015] A4: Configure the GPIO trigger circuit, set the stimulus presentation mechanism and pulse width parameters, and record the local trigger time when the data acquisition device detects the pulse generated by the GPIO trigger circuit.

[0016] Specifically, the data acquisition module sets the basic test parameters and initial stimulus parameters through the medical staff interface, synchronously acquires voice signals, eye movement trajectory data, and electroencephalogram signals under the spatio-temporal anchor marker to form a multi-modal data set with time stamps, and further includes:

[0017] A5: Based on the calibrated data acquisition device, synchronously acquire data to form a multi-modal data stream, and add a local acquisition time stamp to each frame of data;

[0018] A6: Based on the multi-modal data stream and GPIO trigger events, detect the GPIO rising edge pulse time, and calculate the global time stamp by combining the calibrated local trigger time recorded by the data acquisition device and the device calibration time stamp; the device calibration time stamp is calibrated through the PTP protocol;

[0019] A7: According to the global time stamp, combine the preset leading time and subsequent time to determine the interception time window; the leading time is the expected data acquisition time length before the event occurs; the subsequent time is the expected data acquisition time length after the event occurs;

[0020] A8: According to the interception time window, intercept voice, eye movement, and electroencephalogram data from the multi-modal data stream, and form a multi-modal data set with time stamps after Lagrange interpolation.

[0021] Specifically, the formation process of the multi-modal data stream in A5 includes:

[0022] A5.1: Use a microphone to collect voice signals, and obtain enhanced voice signals after passing through a band-pass filter and an adaptive beamforming algorithm;

[0023] A5.2: Use the pupil detection algorithm to calculate the pupil coordinates and pupil diameter in real time to generate eye movement trajectory data;

[0024] A5.3: Use an electroencephalogram monitoring device to collect electroencephalogram signals;

[0025] A5.4: Combine the enhanced voice signals, eye movement trajectory data, and electroencephalogram signals to form a multi-modal data stream.

[0026] Specifically, through the forgetting curve prediction algorithm configured in the algorithm driving unit, combined with the real-time accuracy rate, reaction time, and eye movement attention index monitored by the data monitoring unit, dynamically adjust the stimulus parameters, including:

[0027] B1: During the multi-stage cognitive test, the correct rate, reaction time, and eye movement attention index are collected in real time and standardized to obtain the real-time state vector , where represents the normalized correct rate, represents the normalized reaction time, represents the eye movement attention index;

[0028] B2: The weighted linear combination method is used to map the real-time state vector to the memory retention rate to obtain the forgetting curve state model , where , , , represent the forgetting curve state model parameters.

[0029] Specifically, through the forgetting curve prediction algorithm configured in the algorithm driving unit, combined with the real-time correct rate, reaction time, and eye movement attention index monitored by the data monitoring unit, the stimulation parameters are dynamically adjusted, and it further includes:

[0030] B3: Combining time series data, the exponential smoothing method is used to predict the memory retention rate at the next moment ; the time series data is the memory retention rate of the past m time windows , where represents the smoothing coefficient, represents the predicted value of the memory retention rate at time t;

[0031] B4: Set the target memory retention rate interval , when deviates from the target memory retention rate interval, an adjustment is triggered. Combining the prediction deviation , through the formula the adjusted stimulation parameter is obtained, where and respectively represent the minimum and maximum values of the target memory retention rate interval, represents the preset target memory retention rate, represents the current stimulation parameter, and k represents the adjustment coefficient.

[0032] Specifically, based on the adjusted stimulation parameter, through the dynamic time warping algorithm configured in the feature extraction module, the time-domain alignment processing of the timestamped multi-modal data set is performed with the spatio-temporal anchor marker as the reference, and the speech response delay, eye movement focus distribution entropy value, and brain wave power ratio features are extracted, and the cross-modal feature vector is formed by combination, including:

[0033] C1: Obtain a multimodal dataset with timestamps and spatio-temporal anchor markers, and perform preprocessing to obtain a preprocessed multimodal dataset;

[0034] C2: Calculate the distance matrix between different modal data in the preprocessed multimodal dataset pairwise, and starting from the upper left corner of the distance matrix, gradually calculate the minimum distance when reaching each point along the shortest path to form an accumulated distance matrix;

[0035] C3: Starting from the lower right corner of the accumulated distance matrix, backtrack an optimal path, that is, the path with the minimum distance, and construct the corresponding relationship between different modal data elements according to the principle of the closest distance to complete the time domain alignment.

[0036] Specifically, based on the adjusted stimulus parameters, the dynamic time warping algorithm configured in the feature extraction module is used to perform time domain alignment processing on the multimodal dataset with timestamps based on the spatio-temporal anchor markers, extract features such as speech response delay, eye movement focus distribution entropy value, and brain wave power ratio, and form a cross-modal feature vector through combination. It further includes:

[0037] C4: Calculate the time interval from stimulus presentation to the start of speech response as the speech response delay feature;

[0038] C5: Use an eye tracker to obtain the fixation point coordinates, and calculate the horizontal fixation entropy and the vertical fixation entropy , and comprehensively obtain the eye movement focus distribution entropy value ;

[0039] C6: Preprocess the electroencephalogram signal, divide the preprocessed electroencephalogram signal into different frequency bands, calculate the power of each frequency band, and calculate the power ratio between frequency bands according to the power of each frequency band to obtain the brain wave power ratio feature; the preprocessing includes filtering and denoising;

[0040] C7: Combine the extracted speech response delay feature, eye movement focus distribution entropy value, and brain wave power ratio feature to form a cross-modal feature vector.

[0041] Specifically, inputting the cross-modal feature vector into the pre-trained LSTM-decision tree fusion diagnosis model configured in the diagnosis and evaluation module, and outputting a quantitative evaluation report on the obstacle type and degree classification, including:

[0042] D1: Obtain the cross-modal feature vector, and perform standardization processing on the cross-modal feature vector to obtain a standardized cross-modal feature vector;

[0043] D2: Input the standardized cross-modal feature vector into the LSTM layer of the pre-trained LSTM-decision tree fusion diagnosis model;

[0044] D3: The LSTM layer extracts features and reduces the dimension of the standardized cross-modal feature vectors of the input to obtain the main feature vectors.

[0045] D4: Transfer the main feature vectors output by the LSTM layer to the decision tree fusion layer.

[0046] D5: Each decision tree classifies the input main feature vectors according to the preset feature division rules of itself to obtain the judgment results of the obstacle type and degree classification for each, and comprehensively combines the judgment results of multiple decision trees through weighted fusion to obtain the final obstacle type and degree classification.

[0047] D6: Generate a quantitative evaluation report according to the judged obstacle type and degree classification, combined with the preset evaluation criteria, and output the quantitative evaluation report in the form of text or table.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] 1. The present invention proposes an auditory cognitive dysfunction evaluation and screening system, and has optimized and improved the architecture, operation steps and processes. The system has the advantages of simple process, low investment and operation costs, and low production and working costs.

[0050] 2. The present invention proposes an auditory cognitive dysfunction evaluation and screening system. By setting basic tests and initial stimulus parameters, and synchronously collecting multiple signals in combination with spatio-temporal anchor markers to form a multi-modal data set with time stamps; conducting multi-stage cognitive tests in an adaptive test engine, and dynamically adjusting the stimulus parameters according to real-time data, making the test process more in line with the actual situation of the subject, improving the pertinence and effectiveness of the test, and helping to more accurately evaluate the cognitive ability of the subject.

[0051] 3. The present invention proposes an auditory cognitive dysfunction evaluation and screening system. Based on the adjusted stimulus parameters, the dynamic time warping algorithm is used to perform time domain alignment processing on the multi-modal data set and extract key features to form cross-modal feature vectors, and then input them into the pre-trained LSTM-decision tree fusion diagnosis model to output a quantitative evaluation report of the obstacle type and degree classification, realizing the integration from data collection, processing to diagnosis and evaluation, and being able to quickly and accurately give a diagnosis result, providing support for the early detection and intervention of cognitive disorders. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is the architecture diagram of the auditory cognitive dysfunction evaluation and screening system of the present invention;

[0053] Figure 2 is the principle flow chart of the auditory cognitive dysfunction evaluation and screening system of the present invention;

[0054] Figure 3 This is a flowchart for forming a multimodal dataset with timestamps for the auditory cognitive function disorder assessment and screening system of the present invention. Specific implementation manners

[0055] Example 1:

[0056] Please refer to Figure 1 and Figure 2 An example provided by the present invention: An auditory cognitive function disorder assessment and screening system includes a data acquisition module, a test engine module, a feature extraction module, and a diagnosis and evaluation module;

[0057] The data acquisition module synchronously acquires voice signals, eye movement trajectory data, and electroencephalogram signals by initializing the test environment and parameter configuration; the parameter settings include basic test parameter settings and initial stimulus parameter settings;

[0058] It should be noted that the data acquisition module integrates an eye movement tracking device and an electroencephalogram monitoring device. The eye movement tracking device monitors the eye fixation points and eye movement trajectories of the patient when listening to voice information in real time, and obtains the visual attention of the patient to different auditory stimuli; the electroencephalogram monitoring device acquires the electrophysiological activities of the patient's brain during the processing of auditory information.

[0059] The test engine module is used to execute multi-stage cognitive tests and dynamically adjust test parameters to adapt to the ability level of the subject;

[0060] The feature extraction module is used to extract features such as voice response delay, entropy value of eye movement focus distribution, and electroencephalogram power ratio, and form a cross-modal feature vector through combination;

[0061] The diagnosis and evaluation module identifies the type of disorder and grades the degree based on the cross-modal feature vector, and generates a quantitative evaluation report.

[0062] The data acquisition module includes: a medical staff interaction unit, a spatio-temporal anchoring unit, and a signal acquisition unit;

[0063] The medical staff interaction unit is used to provide a medical staff interface and configure test parameters, including test type and difficulty reference value;

[0064] The spatio-temporal anchoring unit is used to generate spatio-temporal coordinate markers to provide a time and space alignment reference for voice signals, eye movement trajectory data, and electroencephalogram signals;

[0065] The signal acquisition unit is used to synchronously acquire voice signals, eye movement trajectory data, and electroencephalogram signals.

[0066] The test engine module includes: a test execution unit, a data monitoring unit, and an algorithm driving unit;

[0067] A test execution unit for managing the process of a multi-stage cognitive test; the multi-stage cognitive test includes vocabulary learning, story retelling, digit span, vocabulary recall, vocabulary recognition, and story recall;

[0068] A data monitoring unit for real-time analysis of the correct rate, reaction time, and eye movement attention index;

[0069] An algorithm-driven unit for dynamically adjusting stimulus parameters, including stimulus intensity, presentation frequency, and time interval, based on the forgetting curve prediction algorithm.

[0070] The feature extraction module includes: a time-domain alignment unit, a feature extraction unit, and a feature fusion unit;

[0071] A time-domain alignment unit for aligning the time axis of a timestamped multi-modal data set using the dynamic time warping algorithm;

[0072] A feature extraction unit for calculating speech response delay features, eye movement focus distribution entropy value features, and specific frequency band brain wave power ratio features; the specific frequency band refers to Wave power ratio;

[0073] A feature fusion unit for combining the extracted speech response delay features, eye movement focus distribution entropy value features, and specific frequency band brain wave power ratio features into a cross-modal feature vector.

[0074] The diagnosis and evaluation module includes: a model inference unit, an evaluation generation unit, and a report output unit;

[0075] A model inference unit for running a pre-trained LSTM-decision tree fusion diagnosis model, inputting the cross-modal feature vector, and outputting a diagnosis result;

[0076] It should be noted that the Long Short-Term Memory (LSTM) network belongs to the recurrent neural network and is used to process sequential data.

[0077] An evaluation generation unit for converting the output of the LSTM-decision tree fusion diagnosis model into a classification of the type and degree of the disorder;

[0078] A report output unit for generating a structured quantitative evaluation report to support viewing and exporting by medical staff.

[0079] It should be noted that the assessment and screening of auditory cognitive function disorder is a process of determining whether an individual has abnormal auditory cognitive abilities, aiming to detect early the disorders in the synergistic effect of hearing and cognitive processing.

[0080] Furthermore, the overall implementation process of the auditory cognitive function disorder assessment and screening system includes:

[0081] S1: Medical staff set basic test parameters and initial stimulation parameters through the medical staff interface, and synchronously collect voice signals, eye movement trajectory data, and electroencephalogram signals in combination with the spatio-temporal anchoring marking method to form a time-stamped multi-modal data set;

[0082] S2: Perform a multi-stage cognitive test in the adaptive test engine, and dynamically adjust the stimulation parameters based on the real-time accuracy rate, reaction time, and eye movement attention index; the multi-stage cognitive test includes vocabulary learning, story retelling, digit span, vocabulary recall, vocabulary recognition, and story recall tests; the stimulation parameters include stimulation intensity, presentation frequency, and time interval;

[0083] S3: Based on the adjusted stimulation parameters, use the dynamic time warping algorithm to perform time-domain alignment processing on the time-stamped multi-modal data set with the spatio-temporal anchoring mark as the benchmark, extract voice response delay, eye movement focus distribution entropy value, and electroencephalogram power ratio features, and form a cross-modal feature vector through combination;

[0084] S4: Input the cross-modal feature vector into the pre-trained LSTM-decision tree fusion diagnosis model to output a quantitative evaluation report on the type and degree classification of the disorder.

[0085] Embodiment 2:

[0086] Please refer to Figure 3 , in this embodiment, the data acquisition module integrates a microphone, an eye movement tracking device, and an electroencephalogram monitoring device. The eye movement tracking device monitors the eye fixation points and eye movement trajectories of the patient when listening to voice information in real time to obtain the patient's visual attention to different auditory stimuli; the electroencephalogram monitoring device collects the electrophysiological activities of the patient's brain during the auditory information processing.

[0087] The data acquisition module sets basic test parameters and initial stimulation parameters through the medical staff interface, and synchronously collects voice signals, eye movement trajectory data, and electroencephalogram signals under the spatio-temporal anchoring mark to form a time-stamped multi-modal data set, including:

[0088] A1: Medical staff set basic test parameters and initial stimulation parameters through the medical staff interface to generate a structured parameter configuration file; the basic test parameters include test type, subject information, and test environment parameters; the stimulation parameters include stimulation intensity, presentation frequency, and time interval;

[0089] Among them, the test type includes auditory memory test and language comprehension test; the subject information includes age, gender, and medical history; the test environment parameters include background noise and light intensity.

[0090] A2: The master server reads the structured parameter configuration file and sends a synchronization signal to the data acquisition device via the PTP protocol. Meanwhile, after receiving the synchronization signal, the data acquisition device calibrates its local clock and records the device calibration timestamp. ; The data acquisition device includes an electroencephalogram monitoring device, an eye movement tracking device, and a microphone; the eye movement tracking device refers to an eye tracker.

[0091] A3: The system displays a 9-point checkerboard calibration pattern on the screen. During the process of the subject gazing at the pattern, pupil coordinates and visual evoked potential data are synchronously collected. Using the collected pupil coordinates and visual evoked potential data, an affine transformation matrix is calculated, and the affine transformation matrix is solved by the least squares method to obtain the eye movement-screen coordinate mapping matrix.

[0092] Among them, the screen refers to the display device that presents visual information such as auditory cognitive function test content and 9-point checkerboard calibration pattern to the subject. Common types include computer monitors, tablet computer screens, and dedicated test device display screens.

[0093] Furthermore, the specific steps of A3 include:

[0094] (1) Uniformly distribute 9 calibration target points on the screen, that is, grid.

[0095] (2) Set the target point parameters, including diameter, color, and interval. Among them, the diameter is set to 10 millimeters, the color is white for the target point and black for the background, and the interval is set to 1 / 4 of the screen.

[0096] (3) Use the pupil detection algorithm to collect pupil coordinates ;

[0097] (4) By calculating the mean value of the brain wave signal after the jth stimulus over the stimulus repetition times N, visual evoked potential data is obtained ;

[0098] (5) Establish the mapping relationship between the screen coordinates and the pupil coordinates : , expanded into a linear equation system: , where a, b, c, d, e, h represent the parameters to be solved;

[0099] (6) For n calibration target points, collect the actual screen coordinates and the corresponding pupil coordinates , where and respectively represent the actual x-axis coordinate and y-axis coordinate of the ith calibration target point on the screen. Denote the \(i\)-th calibration target on the screen \(s\);

[0100] (7) Construct a system of linear equations: , where denotes the parameter vector to be solved, \(T\) denotes transpose, denotes the error term, \(X\) denotes the design matrix composed of pupil coordinates, and satisfies: , \(Y\) denotes the screen coordinate vector, and satisfies: , where and respectively denote the \(x\)-axis coordinate and \(y\)-axis coordinate of the \(i\)-th calibration target in physical space, and ;

[0101] (8) Solve to obtain the specific values of each parameter in the parameter vector to be solved;

[0102] (9) Matrixize to obtain the eye movement - screen coordinate mapping matrix .

[0103] A4: Configure the GPIO trigger circuit, set the stimulus presentation mechanism and pulse width parameters, and record the local trigger time when the data acquisition device detects the pulse generated by the GPIO trigger circuit ;

[0104] A5: Based on the calibrated data acquisition device, synchronously collect data to form a multi - modal data stream, and add a local acquisition timestamp to each frame of data ;

[0105] A6: Based on the multi - modal data stream and GPIO trigger events, detect the GPIO rising - edge pulse time, and combine the calibrated local trigger time recorded by the data acquisition device and the device calibration timestamp to calculate the global timestamp; the device calibration timestamp is calibrated through the PTP protocol;

[0106] Among them, the global timestamp provides a unified time reference for the multi - modal data, ensuring the consistency of different modal data in the time dimension.

[0107] Furthermore, the specific steps of A6 include:

[0108] (1) Obtain the device calibration timestamp , the local trigger time and the local acquisition timestamp ;

[0109] (2) Associate the multi-modal data stream with the GPIO trigger event, and mark the starting point of the data stream corresponding to the trigger event;

[0110] (3) Obtain the device calibration timestamp according to the PTP protocol calibration result and the deviation of the master control server clock ; ;

[0111] (4) Convert the local trigger time into a global timestamp ;

[0112] (5) Convert the local acquisition timestamp into a global timestamp at the time of.

[0113] A7: According to the global timestamp, combined with the preset leading time and subsequent time, determine the intercepted time window; the leading time is the expected data acquisition time length before the event occurs; the subsequent time is the expected data acquisition time length after the event occurs;

[0114] Exemplarily, assuming that the leading time is set to 500 ms and the subsequent time is set to 1000 ms, based on the stimulus presentation moment, intercept the data from 500 ms before to 1000 ms after this moment.

[0115] A8: According to the intercepted time window, intercept the voice, eye movement, and electroencephalogram data within this time window from the multi-modal data stream, and form a time-stamped multi-modal data set after Lagrange interpolation. Among them, Lagrange interpolation is the prior art content in the field and is not the creative solution of this application, so it will not be elaborated here.

[0116] The formation process of the multi-modal data stream in A5 includes:

[0117] A5.1: Use a microphone to collect voice signals, and obtain enhanced voice signals after passing through a band-pass filter and an adaptive beamforming algorithm. Among them, the band-pass filter and the adaptive beamforming algorithm are the prior art content in the field and are not the creative solution of this application, so it will not be elaborated here;

[0118] A5.2: Use a pupil detection algorithm to calculate the pupil coordinates and pupil diameter in real time, and generate eye movement trajectory data;

[0119] Further, the specific steps of A5.2 include:

[0120] (1) Collect the human eye region image through a camera and perform preprocessing;

[0121] (2)Detect the pupil using the Hough circle transform, obtain the initial center and radius, and obtain the initial circle. The Hough circle transform is the prior art in this field and is not the creative solution of this application, so it will not be elaborated here;

[0122] (3)Optimize the pupil positioning using the contour matching method near the initial circle, and calculate the pupil center coordinates and diameter. The contour matching method is the prior art in this field and is not the creative solution of this application, so it will not be elaborated here;

[0123] (4)Directly measure the distance between the two farthest points, and use the ellipse fitting method to estimate the diameter. The ellipse fitting method is the prior art in this field and is not the creative solution of this application, so it will not be elaborated here;

[0124] (5)Calculate the pupil center coordinates frame by frame in the video stream, and arrange the pupil center coordinates of consecutive frames in chronological order to generate eye movement trajectory data.

[0125] A5.3: Collect electroencephalogram signals using an electroencephalogram monitoring device;

[0126] A5.4: Combine the enhanced speech signal, eye movement trajectory data, and electroencephalogram signals to form a multi-modal data stream.

[0127] The forgetting curve prediction algorithm configured in the algorithm driving unit is used to dynamically adjust the stimulation parameters in combination with the real-time accuracy rate, reaction time, and eye movement attention index monitored by the data monitoring unit, including:

[0128] B1: In the multi-stage cognitive test, collect the accuracy rate, reaction time, and eye movement attention index in real time, and perform standardization processing to obtain the real-time state vector , where, represents the normalized accuracy rate, represents the normalized reaction time, represents the eye movement attention index;

[0129] B2: Use the weighted linear combination method to map the real-time state vector to the memory retention rate to obtain the forgetting curve state model , where, , , , represent the forgetting curve state model parameters, which are determined by training with historical data;

[0130] B3: Combine the time series data and use the exponential smoothing method to predict the memory retention rate at the next moment ; The time series data is the memory retention rate of the past m time windows , where, represents the smoothing coefficient, represents the predicted value of the memory retention rate at time t;

[0131] B4: Set the target memory retention rate interval , when deviates from the target memory retention rate interval, triggering an adjustment. Combining the prediction deviation , through the formula obtain the adjusted stimulation parameter , where and represent the minimum and maximum values of the target memory retention rate interval respectively, represents the preset target memory retention rate, represents the current stimulation parameter, k represents the adjustment coefficient, and in the present invention, is taken.

[0132] Based on the adjusted stimulation parameter, through the dynamic time warping algorithm configured in the feature extraction module, the time-domain alignment processing of the timestamped multi-modal dataset is performed with the spatio-temporal anchor marker as the reference, and the speech response delay, eye movement focus distribution entropy value, and electroencephalogram power ratio features are extracted, and a cross-modal feature vector is formed by combination, including:

[0133] C1: Obtain the timestamped multi-modal dataset and the spatio-temporal anchor marker, and perform preprocessing to obtain the preprocessed multi-modal dataset;

[0134] C2: Calculate the distance matrix between every two different modal data in the preprocessed multi-modal dataset, and starting from the upper left corner of the distance matrix, gradually calculate the minimum distance when reaching each point along the shortest path to form the cumulative distance matrix;

[0135] Further, in the present invention, the distance calculation formula uses the Euclidean distance, where the Euclidean distance is the prior art content in the field and is not the creative solution of this application, so it will not be elaborated here.

[0136] Further, the specific steps for calculating the distance matrix between every two different modal data in the multi-modal dataset include:

[0137] (1) Obtain the preprocessed multi-modal dataset;

[0138] (2) Select pairs of different modal data from the preprocessed multi-modal dataset to form data pairs. For example, for data of three modalities: speech, eye movement, and electroencephalogram, the distance matrices of three combinations: speech-eye movement, speech-electroencephalogram, and eye movement-electroencephalogram need to be calculated;

[0139] (3) Use the Euclidean distance formula to calculate the distance value for each data pair;

[0140] (4) Create an empty distance matrix;

[0141] (5) Fill the calculated distance values into the corresponding positions in the matrix to obtain the distance matrix. For example, for the distance values of the speech - eye movement data pairs, fill them into the first row and second column and the second row and first column of the matrix.

[0142] C3: Starting from the lower - right corner of the cumulative distance matrix, backtrack an optimal path, that is, the path with the minimum distance. According to the principle of the closest distance, construct the correspondence between different - modality data elements to complete the time - domain alignment;

[0143] C4: Calculate the time interval from the stimulus presentation to the start of the speech response as the speech response delay feature;

[0144] C5: Use an eye tracker to obtain the fixation point coordinates, and calculate the horizontal fixation entropy and the vertical fixation entropy , and comprehensively obtain the eye movement focus distribution entropy value ; The eye movement focus distribution entropy value is achieved by summing the horizontal fixation entropy and the vertical fixation entropy ;

[0145] Further, the horizontal fixation entropy is obtained by calculating the product of the probability that the abscissa value of the r - th fixation point appears and the corresponding logarithm value , and then summing up for all fixation points.

[0146] The vertical fixation entropy is obtained by calculating the product of the probability that the ordinate value of the r - th fixation point appears and the corresponding logarithm value , and then summing up for all fixation points.

[0147] C6: Pre - process the electroencephalogram (EEG) signal, divide the pre - processed EEG signal into different frequency bands, calculate the power of each frequency band, and calculate the power ratio between the frequency bands according to the power of each frequency band to obtain the EEG power ratio feature; The pre - processing includes filtering and denoising;

[0148] Further, the power of each frequency band is the mean value of the discrete EEG signal over the signal length.

[0149] Further, the power ratio between the frequency bands is the ratio of the frequency - band powers of different EEG signals.​​

[0150] C7: Combine the extracted voice response delay features, eye movement focus distribution entropy values, and electroencephalogram power ratio features to form a cross-modal feature vector.

[0151] Input the cross-modal feature vector into the pre-trained LSTM-decision tree fusion diagnosis model configured in the diagnosis and evaluation module, and output a quantitative evaluation report on the type and degree classification of the disorder, including:

[0152] D1: Obtain the cross-modal feature vector, organize the cross-modal feature vector to ensure that the feature vector dimensions of each sample are consistent, including voice response delay, eye movement focus distribution entropy value, and electroencephalogram power ratio features. At the same time, perform standardization processing on the cross-modal feature vector to obtain the standardized cross-modal feature vector. In the present invention, the Z-score standardization method is adopted, and the Z-score standardization method is the prior art content in the field and not the creative solution of this application, so it will not be elaborated here;

[0153] D2: Input the standardized cross-modal feature vector into the LSTM layer of the pre-trained LSTM-decision tree fusion diagnosis model. Among them, the LSTM layer can process sequence data, capture the time-dependent relationship and long-term memory information in the standardized cross-modal feature vector, and the LSTM is the prior art content in the field and not the creative solution of this application, so it will not be elaborated here;

[0154] D3: The LSTM layer performs feature extraction and dimensionality reduction on the input standardized cross-modal feature vector to obtain the main feature vector. Among them, the LSTM layer controls the inflow, outflow, and memory update of information through a series of gating mechanisms, such as input gates, forget gates, and output gates, to extract feature representations;

[0155] D4: Transmit the main feature vector output by the LSTM layer to the decision tree fusion layer. Among them, the decision tree fusion layer consists of multiple decision trees, and each decision tree classifies and judges the input main feature vector according to different features and rules;

[0156] D5: Each decision tree classifies the input main feature vector according to the preset feature division rules of itself to obtain its own judgment results on the type and degree classification of the disorder, and comprehensively combines the judgment results of multiple decision trees through a weighted fusion method to obtain the final type and degree classification of the disorder. Among them, the weighted fusion is the prior art content in the field and not the creative solution of this application, so it will not be elaborated here;

[0157] Exemplarily, for example, mild, moderate, and severe correspond to the numerical values 1, 2, and 3 respectively.

[0158] D6: Based on the determined obstacle type and degree grading, combined with the pre-set evaluation criteria, generate a quantitative evaluation report and output the quantitative evaluation report in the form of text or table.

[0159] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments without departing from the purpose and scope of the present invention, and these all fall within the protection scope of the present invention.

[0160] If the technical solution of the present disclosure involves personal information, before the product applying the technical solution of the present disclosure processes personal information, it has clearly informed the personal information processing rules and obtained the individual's independent consent. If the technical solution of the present disclosure involves sensitive personal information, before the product applying the technical solution of the present disclosure processes sensitive personal information, it has obtained the individual's separate consent and at the same time meets the requirement of "express consent". For example, at a personal information collection device such as a camera, a clear and prominent sign is set to inform that the personal information collection range has been entered and personal information will be collected. If an individual voluntarily enters the collection range, it is regarded as consenting to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are informed by obvious signs / information, personal authorization is obtained through pop-up messages or by asking the individual to upload their personal information by themselves; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. An auditory cognitive function disorder assessment and screening system, characterized in that, It includes: a data acquisition module, a test engine module, a feature extraction module, and a diagnosis and evaluation module; the test engine module includes a test execution unit, a data monitoring unit, and an algorithm driving unit; the data acquisition module sets basic test parameters and initial stimulus parameters through the medical staff interface. Under the spatio-temporal anchoring mark, it synchronously acquires voice signals, eye movement trajectory data, and electroencephalogram signals, forms a multi-modal data set with time stamps, and transmits it to the test execution unit for multi-stage cognitive tests; through the forgetting curve prediction algorithm configured in the algorithm driving unit, combined with the real-time accuracy rate, reaction time, and eye movement attention index monitored by the data monitoring unit, the stimulus parameters are dynamically adjusted; based on the adjusted stimulus parameters, through the dynamic time warping algorithm configured in the feature extraction module, the multi-modal data set with time stamps is subjected to time-domain alignment processing with the spatio-temporal anchoring mark as the reference, and features such as voice response delay, eye movement focus distribution entropy value, and electroencephalogram power ratio are extracted, and a cross-modal feature vector is formed through combination; the cross-modal feature vector is input into the pre-trained LSTM-decision tree fusion diagnosis model configured in the diagnosis and evaluation module, and a quantitative evaluation report on the type and degree of the disorder is output.

2. The auditory cognitive dysfunction assessment and screening system according to claim 1, wherein the data acquisition module integrates a microphone, an eye movement tracking device, and an electroencephalogram monitoring device. The eye movement tracking device monitors the eye fixation points and eye movement trajectories of the patient during listening to voice information in real time, and obtains the visual attention of the patient to different auditory stimuli; the electroencephalogram monitoring device collects the electrophysiological activities of the patient's brain during the processing of auditory information.

3. The auditory cognitive dysfunction assessment and screening system according to claim 2, wherein the data acquisition module sets basic test parameters and initial stimulus parameters through the medical staff interface. Under the spatio-temporal anchoring mark, it synchronously acquires voice signals, eye movement trajectory data, and electroencephalogram signals, forms a multi-modal data set with time stamps, including: A1: Set basic test parameters and initial stimulus parameters to generate a structured parameter configuration file; the basic test parameters include test type, subject information, and test environment parameters; the stimulus parameters include stimulus intensity, presentation frequency, and time interval; A2: The main control server reads the structured parameter configuration file, sends a synchronization signal to the data acquisition device through the PTP protocol. At the same time, after receiving the synchronization signal, the data acquisition device calibrates its local clock and records the device calibration time stamp; the data acquisition device includes an electroencephalogram monitoring device, an eye movement tracking device, and a microphone; the eye movement tracking device refers to an eye tracker; A3: The system displays a 9-point checkerboard calibration pattern. During the process of the subject's fixation on the pattern, pupil coordinates and visual evoked potential data are synchronously acquired. Using the acquired pupil coordinates and visual evoked potential data, an affine transformation matrix is calculated, and the affine transformation matrix is solved by the least squares method to obtain an eye movement-screen coordinate mapping matrix; A4: Configure the GPIO trigger circuit, set the stimulus presentation mechanism and pulse width parameters, and when the data acquisition device detects the pulse generated by the GPIO trigger circuit, record the local trigger time.

4. The auditory cognitive dysfunction assessment and screening system according to claim 3, wherein The data acquisition module sets basic test parameters and initial stimulation parameters through the medical staff interface. Under the spatio-temporal anchor marker, it synchronously acquires voice signals, eye movement trajectory data, and electroencephalogram signals to form a multi-modal data set with timestamps, and further includes: A5: Based on the calibrated data acquisition device, synchronously acquire data to form a multi-modal data stream, and add a local acquisition timestamp to each frame of data; A6: Based on the multi-modal data stream and GPIO trigger events, detect the GPIO rising edge pulse time, and combine the calibrated local trigger time and device calibration timestamp recorded by the data acquisition device to calculate the global timestamp; the device calibration timestamp is calibrated through the PTP protocol; A7: According to the global timestamp, combine the preset leading time and subsequent time to determine the interception time window; the leading time is the expected data acquisition time length before the event occurs; the subsequent time is the expected data acquisition time length after the event occurs; A8: According to the interception time window, intercept voice, eye movement, and electroencephalogram data from the multi-modal data stream, and form a multi-modal data set with timestamps after Lagrangian interpolation.

5. The auditory cognitive dysfunction assessment and screening system according to claim 4, wherein The formation process of the multi-modal data stream in A5 includes: A5.1: Use a microphone to collect voice signals, and obtain enhanced voice signals after passing through a band-pass filter and an adaptive beamforming algorithm; A5.2: Use a pupil detection algorithm to calculate pupil coordinates and pupil diameter in real time to generate eye movement trajectory data; A5.3: Use an electroencephalogram monitoring device to collect electroencephalogram signals; A5.4: Combine the enhanced voice signals, eye movement trajectory data, and electroencephalogram signals to form a multi-modal data stream.

6. The auditory cognitive dysfunction assessment and screening system according to claim 5, wherein The dynamic adjustment of the stimulation parameters by combining the forgetting curve prediction algorithm configured in the algorithm driving unit with the real-time accuracy rate, reaction time, and eye movement attention index monitored by the data monitoring unit includes: B1: During the multi-stage cognitive test, the correct rate, reaction time, and eye movement attention index are collected in real time and standardized to obtain the real-time state vector , where represents the normalized correct rate, represents the normalized reaction time, represents the eye movement attention index; B2: Using the weighted linear combination method to map the real-time state vector to the memory retention rate, obtaining the forgetting curve state model , where , , , represent the forgetting curve state model parameters.

7. The auditory cognitive dysfunction assessment and screening system according to claim 6, characterized in that, The dynamic adjustment of the stimulation parameters by combining the forgetting curve prediction algorithm configured in the algorithm driving unit with the real-time accuracy rate, reaction time, and eye movement attention index monitored by the data monitoring unit further includes: B3: Combine time series data and use exponential smoothing method to predict the memory retention rate at the next moment ; The time series data is the memory retention rate in the past m time windows , where represents the smoothing coefficient represents the predicted value of the memory retention rate at time t B4: Set the target memory retention rate range When deviates from the target memory retention rate range, trigger an adjustment. Combine the prediction deviation and obtain the adjusted stimulation parameter through the formula where , and represent the minimum and maximum values of the target memory retention rate range respectively, represents the preset target memory retention rate, represents the current stimulation parameter, and k represents the adjustment coefficient.

8. The auditory cognitive dysfunction assessment and screening system according to claim 7, characterized in that, Based on the adjusted stimulation parameters, through the dynamic time warping algorithm configured in the feature extraction module, perform time-domain alignment processing on the multi-modal data set with timestamps based on the spatio-temporal anchor marker, extract features such as voice response delay, eye movement focus distribution entropy value, and electroencephalogram power ratio, and form a cross-modal feature vector through combination, including: C1: Obtain the multi-modal data set with timestamps and the spatio-temporal anchor marker, and perform preprocessing to obtain the preprocessed multi-modal data set; C2: Calculate the distance matrix between different modal data in the preprocessed multi-modal data set pairwise, and starting from the upper left corner of the distance matrix, gradually calculate the minimum distance when reaching each point along the shortest path to form an accumulated distance matrix; C3: Starting from the lower right corner of the accumulated distance matrix, backtrack an optimal path, that is, the path with the minimum distance, and construct the corresponding relationship between different modal data elements according to the principle of the closest distance to complete the time-domain alignment.

9. The auditory cognitive dysfunction assessment and screening system according to claim 8, wherein, Based on the adjusted stimulation parameters, the dynamic time warping algorithm configured in the feature extraction module is used to perform time-domain alignment processing on the timestamped multi-modal dataset with the spatio-temporal anchor marker as the reference, extract speech response delay, eye movement focus distribution entropy value, and electroencephalogram power ratio features, and form a cross-modal feature vector through combination. It further includes: C4: Calculate the time interval from the presentation of the stimulus to the start of the speech response as the speech response delay feature; C5: Obtain the fixation point coordinates using an eye tracker, calculate the horizontal fixation entropy based on the fixation point coordinates and the vertical fixation entropy , and comprehensively obtain the entropy value of the eye movement focus distribution ; C6: Preprocess the electroencephalogram signal, divide the preprocessed electroencephalogram signal into different frequency bands, calculate the power of each frequency band, and calculate the power ratio between the frequency bands according to the power of each frequency band to obtain the electroencephalogram power ratio feature; the preprocessing includes filtering and denoising; C7: Combine the extracted speech response delay feature, eye movement focus distribution entropy value, and electroencephalogram power ratio feature to form a cross-modal feature vector.

10. The auditory cognitive dysfunction assessment and screening system according to claim 9, wherein Input the cross-modal feature vector into the pre-trained LSTM-decision tree fusion diagnosis model configured in the diagnosis and evaluation module, and output a quantitative evaluation report on the type and degree classification of the disorder, including: D1: Obtain the cross-modal feature vector and perform standardization processing on the cross-modal feature vector to obtain the standardized cross-modal feature vector; D2: Input the standardized cross-modal feature vector into the LSTM layer of the pre-trained LSTM-decision tree fusion diagnosis model; D3: The LSTM layer performs feature extraction and dimensionality reduction on the input standardized cross-modal feature vector to obtain the main feature vector; D4: Transmit the main feature vector output by the LSTM layer to the decision tree fusion layer; D5: Each decision tree classifies the input main feature vector according to its preset own feature division rules to obtain its own judgment results on the type and degree classification of the disorder, and comprehensively combines the judgment results of multiple decision trees through a weighted fusion method to obtain the final type and degree classification of the disorder; D6: According to the determined type and degree classification of the disorder, combine the preset evaluation criteria to generate a quantitative evaluation report, and output the quantitative evaluation report in the form of text or table.

Images