Questionnaire credibility detection method and system based on visible light and infrared images

Abstract

A questionnaire credibility detection method and system based on visible light and infrared images relates to the field of graphic detection and analysis technology. This invention utilizes a pre-set non-contact sensor, along with detection algorithms and strategies, to assess the credibility of survey conclusions from computer-generated questionnaires. This eliminates the need to include redundant information in the questionnaire, reducing the number of questions required and avoiding inaccurate reference information. Furthermore, the method is contactless, reducing the difficulty of detection. Furthermore, the micro-expression and ocular blood flow characteristics selected by this method are universal, making it applicable to various countries, ethnic groups, and cultural backgrounds, demonstrating promising application prospects and value.

Description

Technical Field

[0001] The present invention relates to the technical field of graphic detection and analysis, and in particular to a method and system for detecting the credibility of a questionnaire based on visible light and infrared images. Background Art

[0002] Questionnaires are an effective tool for collecting large amounts of data. Through questionnaires, researchers can quickly and efficiently obtain information about respondents' opinions, perspectives, attitudes, and behaviors, thereby obtaining rich data on specific topics. By statistically analyzing respondents' answers, researchers can draw quantitative conclusions and better understand the characteristics, trends, and relationships of the research subjects.

[0003] The credibility of survey results refers to the degree of trust or reliability in assessing these results. In statistics and research methodology, credibility is often used to measure the consistency and accuracy of data to determine the trustworthiness of survey results. The level of credibility affects the credibility and reliability of research conclusions and is crucial for correctly understanding the significance of the research question. To assess the credibility of survey results, the following methods are generally used: Internal consistency: This refers to the consistency between questions in the same questionnaire. Statistical analysis can be used to test the consistency of answers to questions within the questionnaire. Repeated measurements: Repeated measurements can assess the stability and consistency of questionnaire results. By repeating the questionnaire at different times or under different conditions, the reliability and repeatability of the results can be tested. External consistency: This refers to the consistency between survey results and other relevant data or known facts. The accuracy and credibility of survey results can be verified by comparing them with existing research results, official data, or data from other independent sources.

[0004] Chinese patent CN115659129B discloses a method for analyzing the credibility of a questionnaire survey based on the VARK scale. It uses an external consistency method to verify the user description and questionnaire information to determine feasibility. Chinese patent CN108829839A discloses a method, device, storage medium, and processor for verifying sample credibility. It uses internal consistency to verify the credibility of the questionnaire through three parameters. All of the above methods artificially introduce redundant information, and calculate the credibility of the subjects' answers in the questionnaire through the mutual verification of the redundant information. The disadvantage of this type of method is that the introduction of redundant information greatly increases the workload of the subjects, which not only increases the difficulty of detection, but also the credibility of the introduced redundant information cannot be guaranteed. Summary of the Invention

[0005] One of the objectives of the present invention is to provide a questionnaire credibility detection method based on visible light and infrared images to reduce the detection difficulty and improve the reliability of the assessment.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solution: a questionnaire credibility detection method based on visible light and infrared images, comprising the following steps:

[0007] Step 1: Data Collection and Segmentation: The questionnaire was deployed on a computer, and a visible light camera and an infrared camera were installed on the computer to collect facial micro-expression data and eye blood flow infrared image data of the subjects answering each question;

[0008] Step 2: Data Analysis: Analyze the micro-expression facial image data, extract and detect facial key points to determine whether the subject has made the predetermined micro-expression movements, and obtain the micro-expression judgment result; analyze the eye blood flow infrared image data, and determine whether the subject has a large emotional fluctuation by analyzing the changes in eye blood flow, and obtain the eye blood flow change judgment result;

[0009] Step 3: Result fusion and judgment: The micro-expression judgment results are combined with the eye blood flow change judgment results. If abnormal micro-expressions and significant changes in eye blood flow appear at the same time, it is determined that the subject's credibility in answering the corresponding questions is low.

[0010] Preferably, in step 2, the step of analyzing the micro-expression facial image data includes:

[0011] (1) Extract key points: Detect and extract facial key points using convolutional neural networks or facial key point extraction tools;

[0012] (2) Face alignment: aligning facial features in a face image to a standard position or shape;

[0013] (3) Detecting each AU region: Assume that a set of features x is extracted from facial feature points, and there is a set of labels y indicating whether the corresponding AU exists, where 1 indicates existence and 0 indicates non-existence. AU detection is performed by using a linear SVM classifier. The decision function of SVM is expressed as:

[0014] f(x)=sign(w·x+b);

[0015] Where w is the weight vector and b is the bias term;

[0016] (4) Detection of abnormal micro-expressions: By detecting each AU, it is determined whether the subject has a predetermined abnormal micro-expression movement, including biting lips, frequent swallowing, frequent blinking, and pupil dilation.

[0017] In the above step (2), let (x, y) be the coordinates of the feature point in the original image, and (x', y') be the coordinates of the feature point after alignment, then:

[0018]

[0019] Among them, the matrix T is the affine transformation matrix, which describes the changes of a geometric object under linear transformations such as translation, rotation, scaling and shearing. For face alignment, the affine transformation matrix can be calculated by the following steps: first, based on the feature points or feature areas in the original image, its center of gravity is calculated; then the corresponding reference points in the target position or shape are calculated, and finally, based on these corresponding points, the affine transformation matrix T is calculated.

[0020] Preferably, in step 2, the step of analyzing the eye blood flow infrared image data includes:

[0021] (5) Image preprocessing: image enhancement and smoothing are used to reduce interference and highlight blood vessels in the eye area;

[0022] (6) Eye region detection: The position of the eyes can be found from the pre-processed image through template matching or deep learning methods, and the region of interest to be analyzed is reversely selected based on the positions of the two inner corners of the eyes;

[0023] (7) Eye area analysis: Let the eye brightness of the starting frame image be Brt onset , the maximum brightness of the eye image in this infrared video stream is Brt apex , we can use the following formula to judge whether the subject has emotional fluctuations:

[0024]

[0025] Where Threshold is a hyperparameter threshold. When the peak brightness and the initial brightness of the eye exceed the threshold, it means that the subject has a large emotional fluctuation.

[0026] In addition, the present invention also provides a questionnaire credibility detection system based on visible light and infrared images, which includes:

[0027] The data acquisition and segmentation module is used to collect micro-expression facial image data and eye blood flow infrared image data of the subjects answering each question after deploying the questionnaire on the computer and installing a visible light camera and an infrared camera on the computer;

[0028] The data analysis module is used to analyze micro-expression facial image data, determine whether the subject has made predetermined micro-expression movements by extracting and detecting facial key points, and obtain micro-expression judgment results; and is used to analyze eye blood flow infrared image data, determine whether the subject has significant emotional fluctuations by observing changes in eye blood flow, and obtain eye blood flow change judgment results;

[0029] The result fusion and judgment module is used to fuse the micro-expression judgment results with the eye blood flow change judgment results. If abnormal micro-expressions and significant changes in eye blood flow appear at the same time, it is determined that the subject's credibility in answering the corresponding questions is low.

[0030] The system operates using the above-mentioned questionnaire credibility detection method based on visible light and infrared images.

[0031] Compared with the prior art, the advantages of the present invention are as follows:

[0032] 1. No redundant information is added to the questionnaire, which reduces the number of questions the subjects need to answer and avoids incorrect reference information; 2. The method is contactless, which reduces the difficulty of detection; 3. The micro-expressions and eye blood flow change characteristics selected by this method are universal and can be applied to various countries, nationalities, and cultural backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the process of the present invention;

[0034] Figure 2 Schematic diagram of the location of facial key points;

[0035] Figure 3 Schematic diagram of face alignment before and after;

[0036] Figure 4 Schematic diagram of the main AU areas of the human face;

[0037] Figure 5 Schematic diagram of the effect before and after infrared image preprocessing;

[0038] Figure 6 Schematic diagram of eye area detection. DETAILED DESCRIPTION

[0039] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0040] like Figure 1 As shown, a questionnaire credibility detection method based on visible light and infrared images includes the following steps:

[0041] Step 1: Data collection and segmentation.

[0042] 1. Deploy the questionnaire on the computer and set it up with one test question per page;

[0043] 2. Install a visible light camera and an infrared camera on the display, with their directions aligned with the normal direction of the display;

[0044] 3. Synchronously capture visible light video and infrared video. Each time the subject completes a test question, a visible light and infrared video segment will be automatically saved.

[0045] 4. Analyze the facial video data of each test question separately to detect the facial features of the subjects.

[0046] Step 2: Data analysis.

[0047] The micro-expression facial image data is analyzed, and the facial key points are extracted and detected to determine whether the subject has made the predetermined micro-expression movement, and the micro-expression judgment result is obtained; the eye blood flow infrared image data is analyzed, and the changes in eye blood flow are analyzed to determine whether the subject has large emotional fluctuations, and the eye blood flow change judgment result is obtained.

[0048] Specifically, the steps of analyzing micro-expression facial image data include:

[0049] 1. Extract key points.

[0050] Facial key points are points with special meaning or specific location in facial images. They are usually used in applications such as facial recognition, expression analysis, and posture estimation. Common facial key points include eye-related key points, nose-related key points, mouth-related key points, and overall facial contour key points (such as Figure 2 These key points can be used to describe and represent the shape, structure, and posture of the face, thereby helping to implement applications such as face recognition, expression recognition, and posture estimation.

[0051] Facial key points can be detected and extracted through convolutional neural networks or related tools (Dlib, etc.).

[0052] 2. Facial alignment.

[0053] Face alignment refers to the process of aligning facial features in a face image to a standard position or shape (such as Figure 3 In tasks such as face recognition, expression recognition, and facial landmark detection, face alignment is a key step that can improve the accuracy and robustness of subsequent processing.

[0054] Let (x, y) be the coordinates of the feature points in the original image, and (x', y') be the coordinates of the feature points after alignment, then:

[0055]

[0056] Among them, the matrix T is the affine transformation matrix, which describes the changes of a geometric object under linear transformations such as translation, rotation, scaling and shearing. For facial alignment, the common affine transformation matrix can be calculated through the following steps: first, based on the feature points or feature areas in the original image, calculate its center of gravity (or other geometric center); then calculate the corresponding reference points in the target position or shape (such as the standard eye position or mouth position), and finally calculate the affine transformation matrix T based on these corresponding points.

[0057] 3. Detect each AU area.

[0058] Based on the anatomical characteristics of the human face, scientists have divided it into several independent yet interconnected motor units (AUs), and analyzed the movement characteristics of these motor units, the main areas they control, and the expressions related to them. Figure 4 Shown are some main AU schematics.

[0059] Assume that a set of features x are extracted from facial feature points, and there is a set of labels y indicating whether the corresponding AU exists, where 1 indicates existence and 0 indicates non-existence. AU detection is performed using a linear SVM classifier. The decision function of SVM is expressed as:

[0060] f(x)=sign(w·x+b);

[0061] Where w is the weight vector and b is the bias term. These parameters are determined by training the model and optimizing the model.

[0062] 4. Detection of abnormal micro-expressions.

[0063] By detecting each AU, it is determined whether the subject has a predetermined micro-expression, including biting lips, frequent swallowing, frequent blinking, pupil dilation, etc.

[0064] Lip biting: Lip biting usually occurs when the lower lip is lightly bitten by the upper teeth or when the lips are bitten. By analyzing facial images or videos, changes in the lip area, such as tightening of the lips or slight movements of the jaw, can be detected.

[0065] Frequent swallowing: Frequent swallowing may indicate that the person is nervous or anxious. The frequency of swallowing can be detected by analyzing the movements in the throat area.

[0066] Frequent blinking: Some studies have shown that when people are nervous or anxious, their blinking rate increases, suggesting they are nervous and, in turn, may be associated with lying. Blinking frequency can be detected by analyzing the movements of the eye area, specifically the opening and closing of the eyelids.

[0067] Pupil dilation: Pupil dilation is often associated with emotional excitement or concentration. Pupil dilation can be detected by analyzing changes in pupil size.

[0068] Combining the above methods, a facial expression analysis system can be designed to determine whether the subject has a predetermined micro-expression by detecting and analyzing each AU and combining it with the pre-defined micro-expression movements.

[0069] Analysis of infrared image data of eye blood flow reveals that facial blood flow changes may exhibit certain characteristics when subjects lie. Because the skin around the eyes is thinnest, analyzing blood flow changes around the eyes provides more accurate results. When subjects experience emotional fluctuations, the eye area exhibits the following changes.

[0070] Ocular vasodilation: In some cases, the liar may feel nervous or anxious, which leads to increased activity in the sympathetic nervous system and vasodilation. This can cause blood vessels in the eye area to dilate, increasing blood flow.

[0071] Changes in skin temperature around the eyes: When we are emotionally aroused, this can cause changes in local blood flow, which in turn affects the skin temperature around the eyes. Some studies have shown that lying can cause small changes in the skin temperature around the eyes.

[0072] Therefore, this method analyzes the changes in the subjects' ocular blood flow through the following steps.

[0073] 1. Image preprocessing.

[0074] Preprocessing infrared images can ensure image quality and consistency. Image enhancement and smoothing can be performed to reduce interference and highlight blood vessels in the eye area. For example, filtering algorithms such as Gaussian filtering can be used to remove noise information from infrared images. At the same time, methods such as histogram equalization can be used to improve the contrast of facial images, making it easier to extract subsequent feature information. Figure 5 A schematic diagram showing the effects of infrared images before and after preprocessing is shown.

[0075] 2. Eye area detection.

[0076] The position of the eyes can be found from the processed infrared image by template matching or deep learning methods, and the subsequent regions of interest to be analyzed (such as Figure 6 shown).

[0077] 3. Eye area analysis.

[0078] When emotions fluctuate, it may cause changes in local blood flow, which in turn affects the skin temperature around the eyes. Some studies have shown that lying may cause slight changes in the skin temperature around the eyes. Let the eye brightness of the starting frame image be Brt onset , the maximum brightness of the eye image in this infrared video stream is Brt apex , we can use the following formula to judge whether the subject has emotional fluctuations:

[0079]

[0080] Where Threshold is a hyperparameter threshold. When the peak brightness and the initial brightness of the eye exceed the threshold, it means that the subject has a large emotional fluctuation.

[0081] Step 3: Result fusion and judgment.

[0082] To reduce the system's false positive rate, we combined visible light micro-expression measurements with infrared eye blood flow changes to determine the credibility of each questionnaire answer. Specifically, if both an unusual micro-expression and significant eye blood flow changes are detected while a subject is answering a question, the confidence level of the answer is considered low.

[0083] The results of the questionnaire can then be processed based on the confidence level of each question, such as eliminating questions with low confidence levels or asking the subjects to re-answer.

[0084] The questionnaire credibility detection method based on visible light and infrared images provided in the above embodiment utilizes a preset non-contact sensor, detection algorithm and strategy to evaluate the credibility of the survey conclusions of subjects who use computers to complete questionnaires. Since no redundant information is added to the questionnaire, the number of questions answered by the subjects is reduced, and the situation where the reference information is incorrect is avoided. Moreover, the method is contactless, which reduces the difficulty of detection. In addition, the micro-expression and eye blood flow change characteristics selected by the method have a certain degree of universality, so it can be applied to various countries, nationalities and cultural backgrounds, and has good application prospects and value.

[0085] In order to make it easier for ordinary technicians in this field to understand the improvements of the present invention over the prior art, some drawings and descriptions of the present invention have been simplified, and the above-mentioned embodiments are preferred implementation schemes of the present invention. In addition, the present invention can also be implemented in other ways. Any obvious replacement without departing from the concept of the present technical solution is within the scope of protection of the present invention.

Claims

1. A questionnaire credibility detection method based on visible light and infrared images includes the following steps: Step 1: Data Collection and Segmentation: The questionnaire was deployed on a computer, and a visible light camera and an infrared camera were installed on the computer to collect facial micro-expression data and eye blood flow infrared image data of the subjects answering each question; Step 2: Data Analysis: Analyze the micro-expression facial image data, extract and detect facial key points to determine whether the subject has made the predetermined micro-expression movements, and obtain the micro-expression judgment result; analyze the eye blood flow infrared image data, and determine whether the subject has a large emotional fluctuation by analyzing the changes in eye blood flow, and obtain the eye blood flow change judgment result; Step 3: Result fusion and judgment: The micro-expression judgment results are combined with the ocular blood flow change judgment results. If abnormal micro-expressions and significant changes in ocular blood flow are present at the same time, the subject's credibility in answering the corresponding question is considered low. In step 2, the steps of analyzing the micro-expression facial image data include: (1) Extract key points: Detect and extract facial key points using convolutional neural networks or facial key point extraction tools; (2) Face alignment: aligning facial features in a face image to a standard position or shape; (3) Detecting each AU region: Assume that a set of features x is extracted from facial feature points, and there is a set of labels y indicating whether the corresponding AU exists, where 1 indicates existence and 0 indicates non-existence. AU detection is performed by using a linear SVM classifier. The decision function of SVM is expressed as: f(x)=sign(w·x+b); Where w is the weight vector and b is the bias term; (4) Detection of abnormal micro-expressions: By detecting each AU, it is determined whether the subject has a predetermined abnormal micro-expression movement, including lip biting, frequent swallowing, frequent blinking, and pupil dilation; In step 2, the step of analyzing the eye blood flow infrared image data includes: (5) Image preprocessing: image enhancement and smoothing are used to reduce interference and highlight blood vessels in the eye area; (6) Eye region detection: Use template matching or deep learning methods to find the position of the eyes from the preprocessed image, and reversely select the region of interest to be analyzed based on the positions of the two inner corners of the eyes; (7) Eye area analysis: Let the eye brightness of the starting frame image be Brt onset , the maximum brightness of the eye image in the infrared video stream is Brt apex , then the following formula is used to determine whether the subject has emotional fluctuations: Where Threshold is a hyperparameter threshold. When the peak brightness and the initial brightness of the eye exceed the threshold, it means that the subject has a large emotional fluctuation.

2. The questionnaire credibility detection method based on visible light and infrared images according to claim 1 is characterized in that: In step (2), let (x, y) be the coordinates of the feature point in the original image, and (x', y') be the coordinates of the feature point after alignment, then: Among them, the matrix T is the affine transformation matrix, which describes the changes of a geometric object under translation, rotation, scaling and shear linear transformation. For face alignment, the affine transformation matrix is ​​calculated by the following steps: first, based on the feature points or feature areas in the original image, its center of gravity is calculated; then the corresponding reference points in the target position or shape are calculated, and finally, based on these corresponding points, the affine transformation matrix T is calculated.

3. A questionnaire credibility detection system based on visible light and infrared images, characterized in that: include: The data acquisition and segmentation module is used to collect micro-expression facial image data and eye blood flow infrared image data of the subjects answering each question after deploying the questionnaire on the computer and installing a visible light camera and an infrared camera on the computer; The data analysis module is used to analyze micro-expression facial image data, determine whether the subject has made predetermined micro-expression movements by extracting and detecting facial key points, and obtain micro-expression judgment results; and is used to analyze eye blood flow infrared image data, determine whether the subject has significant emotional fluctuations by analyzing eye blood flow changes, and obtain eye blood flow change judgment results; The result fusion and judgment module is used to fuse the micro-expression judgment results with the eye blood flow change judgment results. If abnormal micro-expressions and significant changes in eye blood flow appear at the same time, it is determined that the subject's credibility in answering the corresponding questions is low; The method for detecting the credibility of a questionnaire based on visible light and infrared images as described in claim 1 or 2 is used for operation.

Images