Answer answer validity discrimination method based on multi-modal signal

Abstract

The invention provides an answer validity judgment method based on a multi-modal signal, which belongs to the technical field of medical treatment, and comprises the following steps: collecting brain wave data of a testee by using a data acquisition device, extracting a P300 component in the brain wave data, judging whether the extracted P300 component is taken as a validity mark or not based on the extracted P300 component, and judging whether the P300 component is taken as the validity mark or not. The method comprises the following steps: selecting physiological data corresponding to a P300 component, carrying out correlation analysis on the physiological data and the P300 component, selecting data with a correlation coefficient greater than or equal to 0.5 as an input variable, constructing a neural network model, carrying out model training, selecting a neural network to construct an identification model for training, and inputting the collected data into the model for prediction after the model is trained. The problem that a lie detection mechanism as an objective evaluation method still has the defects of poor applicability, weak effectiveness, long evaluation time and the like is solved.

Description

Technical Field

[0001] The present invention belongs to the field of medical technology, and in particular, relates to a method for determining the validity of an answer based on multimodal signals. Background Art

[0002] The accelerated pace of social life has made mental health one of the key issues. Rapid and accurate screening of mental illness is the prerequisite for solving this problem. Psychological scale assessment has become the most common screening method at this stage due to its low cost and portability. However, the polygraph mechanism as an objective assessment method still has shortcomings such as poor applicability, weak effectiveness, and long assessment time.

[0003] In order to solve the evaluation accuracy problem caused by the effectiveness of answering questions, this application proposes a method for determining the effectiveness of answering questions based on multimodal signals. This method can effectively detect the answering behavior during the psychological scale evaluation process by using physiological signals, and ultimately generate a more objective and accurate confidence level for the psychological scale evaluation results. Summary of the invention

[0004] The embodiment of the present invention provides a method for determining the validity of answers based on multimodal signals, which solves the problem that the lie detection mechanism as an objectivity evaluation method still has the disadvantages of poor applicability, weak validity, and long evaluation time.

[0005] In view of the above problems, the technical solution proposed by the present invention is:

[0006] The present invention provides a method for determining the validity of an answer based on a multimodal signal, comprising the following steps:

[0007] Step S1, data extraction, using a data acquisition device to collect the brain wave data of the test subject and extract the P300 component therein;

[0008] Step S2, physiological data collection, using a data acquisition device to collect the test subject's electrocardiogram, respiratory rate, heart rate, blood oxygen saturation and answering time parameters at the same time;

[0009] Step S3, data judgment, judging whether the P300 component extracted in step S1 occurs as a sign of effectiveness;

[0010] Step S4, data correlation analysis, selecting physiological data corresponding to the P300 component, calculating the mean, variance, mean square error, covariance and median of the physiological data collected in step S1 and step S2, and performing correlation analysis with the P300 component;

[0011] Step S5, data fusion, aligning the data based on the correlation analysis in step S4, identifying the same entities from different data sources, matching the data using a matching algorithm, and integrating the matched data;

[0012] Step S6, constructing a model, selecting data with a correlation coefficient greater than or equal to 0.5 from the data set fused in step S4 as input variables, and constructing a neural network model;

[0013] Step S7, model training, selecting a neural network to build a recognition model for training, and when the accuracy does not meet the requirements, continue to supplement data for training;

[0014] Step S8, data prediction, after the model is trained, the collected data is input into the model for prediction. If the output value is 0, it is determined that lying has occurred and the validity of the answer is not high. If the output value is 1, it is determined that no lying has occurred and the validity of the answer is high. The validity of the answer is judged based on this.

[0015] As a preferred technical solution of the present invention, the method for extracting the P300 component in step S1 is as follows:

[0016] Establish a linear model of EEG signals, calculate the covariance matrix of signals and noise, design a Wiener filter to minimize the mean square error between signals and noise, use the filter to filter EEG signals, analyze the filtered signals, and identify the characteristics of the P300 component;

[0017] The filtered signal is segmented, each time corresponds to a time window, the signal in each time window is baseline corrected, and all signals are averaged to obtain the ERP waveform. The P300 component is represented as a positive peak in the ERP waveform. The characteristics of the P300 component are extracted to obtain the data of the P300 component.

[0018] As a preferred technical solution of the present invention, the judgment of the P300 component in step S3 is: when the stimulus is generated, that is, when lying behavior occurs, the brain wave will generate a P300 signal. If the test subject has lying behavior when answering the questions, the validity is recorded as 0; if it does not occur, the answering validity is considered to be high and the validity is recorded as 1.

[0019] As a preferred technical solution of the present invention, in step S5, after the data matching is completed, the data is integrated into a unified data set using data merging, data aggregation or data conversion methods, and a feature selection algorithm is used to extract features useful for the prediction model from the integrated data.

[0020] As a preferred technical solution of the present invention, the data acquisition device includes a user information module, a facial data acquisition module, a psychological evaluation module, a blood oxygen respiration module, a facial analysis module, an electrocardiogram analysis module and an evaluation effectiveness module;

[0021] The user information module is used to store the basic information of the test subject, including name, gender, height, weight, ID number, date of birth, age, serial number, medical number, medical department, clinical diagnosis, self-reported symptoms, whether or not to smoke, and whether or not to have diabetes. The user should try to fill in the correct information;

[0022] The facial data collection module allows the subject to sit in front of the test device and collects the subject's facial data through a camera;

[0023] The psychological assessment module uses a display to display the selected test scale and real-time blood pressure data during the blood pressure test, and displays the scale questions, the options selected by the test subject, and the final answer results of the scale in real time;

[0024] The blood oxygen respiration module displays the blood oxygen and respiration values ​​of the subject in real time during the evaluation process through a display and displays them through a waveform graph;

[0025] The facial analysis module uses the test subject's facial expression data from the facial data acquisition module to analyze the test subject's smile degree, dark circle index and color palette index;

[0026] The electrocardiogram analysis module collects the electrocardiogram data of the subject using an electrocardiograph and plots it into a pattern for intuitive display;

[0027] The evaluation validity module evaluates and determines whether the test taker's answers are valid based on the data calculation of the above modules.

[0028] As a preferred technical solution of the present invention, the test scales in the psychological assessment module are divided into five categories, including cognitive function, quality of life, mental and psychological state, personality characteristics and emotional characteristics and sleep quality;

[0029] The facial analysis module uses a facial detection algorithm to locate the collected facial information, uses a facial landmark location algorithm to mark facial feature points on the facial area information, analyzes the degree of smile by analyzing the position and shape changes of the landmark points around the mouth, analyzes the degree of dark circles by analyzing the color and brightness of the eye socket area, and evaluates the quality of skin color by analyzing the color distribution and brightness changes of the facial area.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] (1) The present invention uses the P300 component of the brain wave as a judgment indicator for lying, collects basic parameters such as the electrocardiogram, respiratory rate, heart rate, blood oxygen saturation and answering time of the test subject, integrates the physiological parameters of the multimodal dimension, and combines them with the analysis structure of the P300 component to judge the validity of the test subject's answer data, generate more accurate psychological scale results, and the measurement data is stable, has strong anti-interference ability, and is easy to use.

[0032] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a method for judging the validity of a question answer based on a multimodal signal disclosed in the present invention;

[0034] Figure 2 It is a block diagram of a data acquisition device for a method for determining the validity of an answer based on multimodal signals disclosed in the present invention;

[0035] Figure 3 It is a schematic diagram of a model training of a method for determining the validity of answers based on multimodal signals disclosed in the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by the tester of ordinary technology in this field without creative work are within the scope of protection of the present invention.

[0037] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by the tester of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0038] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0039] In the description of the present invention, it is necessary to understand that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "up," "down," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inside," "outside," "clockwise," "counterclockwise," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0040] In addition, the terms "first," "second," and "first" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first," "second" may explicitly or implicitly include one or more of the feature. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0041] Embodiment 1

[0042] Refer to the attached Figure 1-3 As shown, the present invention provides a technical solution: a method for determining the validity of an answer based on a multimodal signal, comprising the following steps:

[0043] Step S1, data extraction, using a data acquisition device to collect the EEG data of the subject, and extracting the P300 component. When the human brain is stimulated by a low-probability related event, a positive peak with a latency of about 300ms will appear in the EEG signal;

[0044] Step S2, physiological data collection, using a data acquisition device to collect the test subject's electrocardiogram, respiratory rate, heart rate, blood oxygen saturation and answering time parameters at the same time;

[0045] Step S3, data judgment, based on whether the P300 component extracted in step S1 occurs as a sign of effectiveness;

[0046] Step S4, data correlation analysis, select the physiological data corresponding to the P300 component, calculate the mean, variance, mean square error, covariance and median of the physiological data collected in steps S1 and S2, and perform correlation analysis on it with the P300 component. The correlation calculation formula is as follows:

[0047]

[0048] Among them, p is the Pearson correlation coefficient, which indicates the degree of correlation between two variables. The value range is from -1 to 1, where 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no correlation.i is the observed value of variable X, that is, the specific value of variable X. is the mean value of variable X, that is, the average of all observed values ​​in variable X, Y i is the observed value of variable Y, that is, the specific value of the Y variable, is the mean value of variable Y, that is, the average of all observed values ​​in the Y variable;

[0049] Step S5, data fusion, aligning the data based on the correlation analysis in step S4, including time alignment, space alignment, feature alignment, etc., identifying the same entities from different data sources, using matching algorithms to match the data, including string matching, fuzzy matching, rule-based matching, etc., and integrating the matched data;

[0050] Step S6, constructing a model, selecting data with a correlation coefficient greater than or equal to 0.5 from the fused data set in step S4 as input variables, and constructing a neural network model. Before construction, the input variables need to be normalized and calculated according to the following formula:

[0051]

[0052] Among them, X j is the normalized value, that is, the original value X i After normalization, X min is the minimum value in the data set, used to determine the lower limit of normalization, X max is the maximum value in the data set, which is used to determine the upper limit of normalization;

[0053] Step S7, model training, select a neural network to build a recognition model for training. When the accuracy rate does not meet the requirements, it should generally reach more than 95%. Continue to supplement data for training, such as Figure 3 As shown, the variables that meet the conditions are used as input data X i Whether the P300 EEG signal occurs is used as the output label. If it occurs, the output record is 0, and if it does not occur, the output record is 1, and the model is trained.

[0054] Step S8, data prediction, after the model is trained, the collected data is input into the model for prediction. If the output value is 0, it is determined that lying has occurred and the validity of the answer is not high. If the output value is 1, it is determined that no lying has occurred and the validity of the answer is high. The validity of the answer is judged based on this.

[0055] The embodiment of the present invention is also implemented through the following technical solutions.

[0056] In an embodiment of the present invention, the method for extracting the P300 component in step S1 is as follows:

[0057] Establish a linear model of EEG signals, calculate the covariance matrix of signals and noise, which is used to describe the statistical characteristics of signals and noise, design a Wiener filter to minimize the mean square error between signals and noise, use the filter to filter EEG signals, analyze the filtered signals, and identify the characteristics of the P300 component;

[0058] The filtered signal is segmented, each time corresponds to a time window, the signal in each time window is baseline corrected, and all signals are averaged to obtain the ERP waveform. The P300 component is represented as a positive peak in the ERP waveform. The characteristics of the P300 component are extracted to obtain the data of the P300 component.

[0059] In the embodiment of the present invention, the judgment of the P300 component in step S3 is: when the stimulus is generated, that is, when lying behavior occurs, the brain wave will generate a P300 signal. If the test subject has lying behavior when answering the questions, the validity is recorded as 0. If it does not occur, the answering validity is considered to be high and the validity is recorded as 1.

[0060] In an embodiment of the present invention, after data matching is completed in step S5, the data is integrated into a unified data set using data merging, data aggregation or data conversion methods, and a feature selection algorithm is used to extract features useful for the prediction model from the integrated data.

[0061] In an embodiment of the present invention, the data acquisition device includes a user information module, a facial data acquisition module, a psychological evaluation module, a blood oxygen respiration module, a facial analysis module, an electrocardiogram analysis module, and an evaluation effectiveness module;

[0062] The user information module is used to store the basic information of the test subject, including name, gender, height, weight, ID number, date of birth, age, serial number, medical number, medical department, clinical diagnosis, self-reported symptoms, whether or not to smoke, and whether or not to have diabetes. Users should try to fill in the correct information;

[0063] The facial data collection module allows the subject to sit in front of the test device and collects the subject's facial data through a camera;

[0064] The psychological assessment module uses a display to display the selected test scale and real-time blood pressure data during the blood pressure test, and displays the scale questions, the options selected by the test subject, and the final answer results of the scale in real time;

[0065] The blood oxygen and respiration module displays the blood oxygen and respiration values ​​of the subject in real time during the evaluation process through a display and displays them through a waveform graph;

[0066] The facial analysis module uses the test subject's facial expression data from the facial data acquisition module to analyze the test subject's smile degree, dark circle index, and color palette index;

[0067] The ECG analysis module uses an electrocardiograph to collect the ECG data of the subject and draws it into a pattern for intuitive display;

[0068] The effectiveness evaluation module evaluates the data from the above modules and determines whether the test taker's answers are effective.

[0069] In addition, the test device is to integrate the display, blood pressure measurement equipment and other physiological data collection equipment on the seat. The actual installation structure and position are reasonably installed according to the size of the selected equipment.

[0070] In the embodiment of the present invention, the test scales in the psychological assessment module are divided into five categories, including cognitive function, quality of life, mental and psychological state, personality characteristics and emotional characteristics and sleep quality, including the following scales:

[0071] Cognitive function includes the Mini-Mental State Examination (MMSE) scale, which is the most commonly used scale for cognitive examinations and is very simple and convenient for elderly people with cardiovascular diseases. It mainly evaluates whether the cognitive function of the test subject is normal;

[0072] The aspects of quality of life generally include: comprehensive health, body function, symptoms and side effects, cognitive function, emotional function, psychological well-being, role function, sexual function, social well-being and function, spirit / belief, etc., including: WHOQOL and SAQ. WHOQOL mainly measures the quality of life in the last two weeks, and SAQ mainly measures the quality of life and body function status.

[0073] The mental and psychological state mainly reflects that the test subject may have mental and psychological problems such as anxiety and depression, but it cannot give a clear diagnosis. It includes 9 scales, including the General Hospital Anxiety and Depression Scale (HAD), Depression Screening (PHQ-9), Anxiety Screening (GAD-7), Self-Assessment of Anxiety (SAS), Self-Assessment of Depression Scale (SDS), Self-Assessment of Somatization Symptom Scale (SSS), Personality, Affection and Mood Measurement Scale (PAD), Symptom Checklist (SCL-90), Brief Psychiatric Rating Scale (BPRS), and the General Hospital Anxiety and Depression Scale (HAD) to assess the anxiety and depression conditions respectively;

[0074] The assessment of personality traits and emotional characteristics includes the A-line personality assessment and the assessment of hostility, anger, and aggression. Mainly, the Eysenck Personality Questionnaire (EPQ-RSC), the Eysenck Personality Questionnaire (EPQ-RSC) mainly evaluates the psychological condition, personality, emotions, and authenticity of the test subject;

[0075] The sleep quality assessment is mainly to understand the sleep quality, sleep status and whether there is sleep apnea problem of the test subject. It includes a scale of Pittsburgh Sleep Quality Index (PSQI). Pittsburgh Sleep Quality Index (PSQI) mainly evaluates the sleep quality of the test subject;

[0076] The facial analysis module uses a facial detection algorithm to locate the collected facial information, and uses a facial landmark positioning algorithm on the facial area information to mark facial feature points, such as eyes, nose, mouth, etc. The degree of smile is analyzed by analyzing the position and shape changes of the landmark points around the mouth. For example, the degree of upturned corners of the mouth and the width of the mouth opening can be used to quantify the intensity of the smile. The degree of dark circles is analyzed by analyzing the color and brightness of the eye socket area. This requires color space conversion (such as RGB to HSV) and histogram analysis. The quality of the skin color is evaluated by analyzing the color distribution and brightness changes in the facial area. The analysis is performed using color histogram analysis, color space conversion, and skin color segmentation technology.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0078] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0079] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0080] The technical testee of this field should also be understood that the various illustrative logic boxes, modules, circuits and algorithm steps described in conjunction with the embodiments of this paper can all be implemented as electronic hardware, computer software or its combination. In order to clearly illustrate the interchangeability between hardware and software, various illustrative parts, frames, modules, circuits and steps are generally described above around their functions. Whether this function is implemented as hardware or software depends on specific application and the design constraints imposed on the entire system. Skilled technical testees can realize the described function in an alternative manner for each specific application, but this realization decision should not be interpreted as departing from the protection scope of the present disclosure.

Claims

1. A method for determining the validity of an answer based on multimodal signals, characterized in that: The following steps are involved: Step S1, data extraction, using a data acquisition device to collect the brain wave data of the test subject and extract the P300 component therein; Step S2, physiological data collection, using a data acquisition device to collect the test subject's electrocardiogram, respiratory rate, heart rate, blood oxygen saturation and answering time parameters at the same time; Step S3, data judgment, judging whether the P300 component extracted in step S1 occurs as a sign of effectiveness; Step S4, data correlation analysis, selecting physiological data corresponding to the P300 component, calculating the mean, variance, mean square error, covariance and median of the physiological data collected in step S1 and step S2, and performing correlation analysis with the P300 component; Step S5, data fusion, aligning the data based on the correlation analysis in step S4, identifying the same entities from different data sources, matching the data using a matching algorithm, and integrating the matched data; Step S6, constructing a model, selecting data with a correlation coefficient greater than or equal to 0.5 from the data set fused in step S4 as input variables, and constructing a neural network model; Step S7, model training, selecting a neural network to build a recognition model for training, and when the accuracy does not meet the requirements, continue to supplement data for training; Step S8, data prediction, after the model is trained, the collected data is input into the model for prediction. If the output value is 0, it is determined that lying has occurred and the validity of the answer is not high. If the output value is 1, it is determined that no lying has occurred and the validity of the answer is high. The validity of the answer is judged based on this.

2. The method for determining the validity of an answer based on a multimodal signal according to claim 1, characterized in that: The extraction method of the P300 component in step S1 is as follows: Establish a linear model of EEG signals, calculate the covariance matrix of signals and noise, design a Wiener filter to minimize the mean square error between signals and noise, use the filter to filter EEG signals, analyze the filtered signals, and identify the characteristics of the P300 component; The filtered signal is segmented, each time corresponds to a time window, the signal in each time window is baseline corrected, and all signals are averaged to obtain the ERP waveform. The P300 component is represented as a positive peak in the ERP waveform. The characteristics of the P300 component are extracted to obtain the data of the P300 component.

3. The method for determining the validity of an answer based on multimodal signals according to claim 2, characterized in that: The judgment of the P300 component in step S3 is: when the stimulus is generated, that is, when lying behavior occurs, the brain wave will generate a P300 signal. If the test subject has lying behavior when answering the questions, the validity is recorded as 0. If it does not happen, the answer is considered to be highly valid and the validity is recorded as 1.

4. The method for determining the validity of an answer based on multimodal signals according to claim 3, characterized in that: In step S5, after the data matching is completed, the data is integrated into a unified data set using data merging, data aggregation or data conversion methods, and a feature selection algorithm is used to extract features useful for the prediction model from the integrated data.

5. The method for determining the validity of an answer based on multimodal signals according to claim 1, characterized in that: The data acquisition device includes a user information module, a facial data acquisition module, a psychological evaluation module, a blood oxygen respiration module, a facial analysis module, an electrocardiogram analysis module and an evaluation effectiveness module; The user information module is used to store the basic information of the test subject, including name, gender, height, weight, ID number, date of birth, age, serial number, medical number, medical department, clinical diagnosis, self-reported symptoms, whether or not to smoke, and whether or not to have diabetes. The user should try to fill in the correct information; The facial data collection module allows the subject to sit in front of the test device and collects the subject's facial data through a camera; The psychological assessment module uses a display to display the selected test scale and real-time blood pressure data during the blood pressure test, and displays the scale questions, the options selected by the test subject, and the final answer results of the scale in real time; The blood oxygen respiration module displays the blood oxygen and respiration values ​​of the subject in real time during the evaluation process through a display and displays them through a waveform graph; The facial analysis module uses the test subject's facial expression data from the facial data acquisition module to analyze the test subject's smile degree, dark circle index and color palette index; The electrocardiogram analysis module collects the electrocardiogram data of the subject using an electrocardiograph and plots it into a pattern for intuitive display; The evaluation validity module evaluates and determines whether the test taker's answers are valid based on the data calculation of the above modules.

6. A method for determining the validity of an answer based on multimodal signals according to claim 5, characterized in that: The test scales in the psychological assessment module are divided into five categories, including cognitive function, quality of life, mental and psychological state, personality traits, emotional characteristics and sleep quality; The facial analysis module uses a facial detection algorithm to locate the collected facial information, uses a facial landmark location algorithm to mark facial feature points on the facial area information, analyzes the degree of smile by analyzing the position and shape changes of the landmark points around the mouth, analyzes the degree of dark circles by analyzing the color and brightness of the eye socket area, and evaluates the quality of skin color by analyzing the color distribution and brightness changes of the facial area.

Images