Non-contact lie detection-oriented clue labeling method

Abstract

The invention relates to the technical field of artificial intelligence, in particular to the field of image processing, voice processing and text processing, and provides a clue labeling method for non-contact lie detection. The objective of the invention is to solve the problems of scarcity of Chinese lie detection data, poor interpretability and high manual evaluation cost. According to the main scheme, the method comprises the following steps: performing visual cue description generation processing on video content to obtain a visual cue description text; performing audio clue description generation processing on the audio content to obtain an audio clue description text; performing text clue description generation processing on the text content to obtain a text clue description text; according to the visual clue description, the audio clue description and the text clue description, generating a final clue description by using a large model; and performing quality evaluation on the generated clue description.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, in particular to the fields of image processing, speech processing and text processing, and provides a clue annotation method for contactless lie detection. Background Art

[0002] With the rapid development of artificial intelligence technology, especially in the fields of machine learning and deep learning, a variety of models and algorithms have made significant progress. These advances have provided a technical foundation for various complex tasks in image processing, speech processing, and text processing. Especially in high-risk application scenarios, such as lie detection, the interpretability of models and algorithms is increasingly required.

[0003] Currently, a variety of technologies have been applied to the field of lie detection, including but not limited to:

[0004] Basic models and algorithms: Deep learning models, such as the audio pre-training model Hubert, the visual pre-training model VIT, and the text pre-training model BERT, perform well in pattern recognition and prediction tasks, and provide a technical basis for data feature extraction and classification in lie detection.

[0005] Large language models: Large language models, such as the GPT series, perform well in processing complex language tasks such as text generation, information extraction, semantic understanding, etc. Based on the powerful world knowledge of large models, they generate interpretable descriptions for multimodal data and improve the transparency of lie detection.

[0006] Multimodal data processing:

[0007] Feature extraction and alignment: Cross-modal feature extraction and temporal alignment techniques are used to extract relevant information from different modalities and unify them into the same semantic space to improve the accuracy of lie detection.

[0008] Modal fusion: Multimodal data processing technology provides more comprehensive information for lie detection by integrating visual (image / video), voice and text data.

[0009] Reinforcement Learning with Human Feedback (RLHF): RLHF is a method that combines human feedback and reinforcement learning. It optimizes the model's behavior strategy by allowing the model to receive reward signals (such as preferences, evaluations, etc.) provided by humans during task execution. In this invention, RLHF is used to simulate the preferences of human evaluations, optimize the model's weights, and improve its reliability in generating clues in lie detection scenarios.

[0010] However, despite the significant progress made in multimodal data processing, large language models, and human feedback reinforcement learning, the above technologies still have obvious shortcomings in the scarcity of Chinese lie detection data, poor interpretability, and high manual evaluation costs. Specifically:

[0011] Existing audio and facial lie detection methods require large-scale, high-quality deceptive behavior data samples, but the number of data sets in real scenarios is limited and mainly in English, which limits the research on Chinese lie detection technology.

[0012] Existing deep learning models can usually only output a "lie" result in lie detection, which lacks interpretability. It is difficult for the model to clearly explain the specific basis for triggering the "deception" judgment, such as which audio features or facial micro-expressions play a key role.

[0013] Existing technologies rely on manual evaluation to determine the accuracy of lie detection, which is not only costly and inefficient, but also difficult to adapt to the evaluation needs of large-scale data.

[0014] In response to the above problems and challenges, this application proposal proposes a new interpretable annotation method for lie detection, provides an interpretable Chinese lie detection data annotation method, reduces the difficulty of Chinese lie detection data annotation and the labor cost required for evaluation, and promotes the development of Chinese interpretable lie detection research. Summary of the invention

[0015] The present invention aims to provide an explainable clue annotation method for contactless lie detection to solve the problems of scarcity of Chinese lie detection data, poor explainability and high cost of manual evaluation.

[0016] The present invention provides a clue annotation method for contactless lie detection, comprising the following steps:

[0017] Step 1: Cluster the video frames to be tested, extract key frames, encode the key frames using the pre-trained model, compress and align the encoded visual features using Q-Former, perform the same operation on the video clips that are not lying to obtain the reference video feature vector, and input the video feature vector to be tested and the reference video feature vector into Video-LLAMA2 to generate visual clue description text;

[0018] Step 2: feature encode the audio to be tested, separate the acoustic features and content features in the audio to be tested by optimizing the objective function, compress and align the acoustic features using Q-Former, perform the same operation on the audio clips that are not lying to obtain the reference audio feature vector, and input the audio feature vector to be tested and the reference audio feature vector into SECap to generate the audio clue description text;

[0019] Step 3: Analyze the test question information and personality traits, and input the test question information, personality trait analysis results and the tester's answer into the LLAMA2 model to generate a text clue description;

[0020] Step 4: Generate the final clue description using the large model based on the visual clue description, the audio clue description, and the text clue description;

[0021] Step 5: Build a quality assessment model to perform quality assessment on the generated clue descriptions.

[0022] In the above scheme, the steps of step 1 include:

[0023] Step 1.1, clustering the video frames to obtain clustering results;

[0024] Step 1.2, extract key frames from the clustering results to obtain a key frame list;

[0025] Step 1.3, performing visual feature encoding processing on the key frame list to obtain a visual feature sequence;

[0026] Step 1.4, perform Q-Former-based visual information compression and alignment processing on the visual feature sequence to obtain a compressed visual information representation;

[0027] Step 1.5: Extract key features from the interview clips that never lie, and concatenate them with the key features of the video to be tested to obtain the input for visual description generation;

[0028] Step 1.6: Use the Video-LLAMA2 model to perform visual prompt reasoning and generate the final visual clue description text.

[0029] In the above scheme, the steps of step 2 include:

[0030] Step 2.1, feature encode the audio input to obtain an audio feature sequence;

[0031] Step 2.2: Use two independent linear layers to extract acoustic representation and content representation respectively, and obtain acoustic features and semantic content representation;

[0032] Step 2.3, minimize the mutual information between acoustic representation and content representation by optimizing the objective, impose orthogonal constraints, and optimize acoustic features and content representation by contrastive learning loss and CTC loss;

[0033] Step 2.4: Use Q-former to compress and align the acoustic features to obtain a compressed acoustic information representation;

[0034] Step 2.5, randomly select an acoustic feature from the interview clip that never lies, and concatenate it with the acoustic feature to be tested to obtain the input for acoustic description generation;

[0035] Step 2.6: Take the acoustic prompt and the final acoustic description generation input as the input of SECap to generate the final interpretable description D a , and obtain the final audio clue description text.

[0036] In the above scheme, the steps of step 3 include:

[0037] Step 3.1, analyze the test question information to obtain the type of question and the difficulty of answer verification;

[0038] Step 3.2, analyze personality traits and obtain the Big Five personality test scale vector, covering five dimensions: openness, conscientiousness, extraversion, agreeableness, and emotional instability;

[0039] Step 3.3: Use the LLAMA2 model to generate descriptions, generate lie clue descriptions of the text content according to different text prompt strategies, and obtain an interpretable description D t ;

[0040] Step 3.4: Generate the generated interpretable description D t Generate the final text clue description text.

[0041] In the above scheme, the steps of step 4 include:

[0042] Step 4.1, integrating the prompt template of the final clue description of text, voice and vision to obtain the input information of the multimodal clue description;

[0043] Step 4.2: Use the large model to integrate multimodal clues and conduct correlation analysis to generate the final lie clue description text.

[0044] In the above scheme, the steps of step 5 include:

[0045] Step 5.1: Build a dataset for training the automatic quality assessment model, including raw data features, generated descriptions and their corresponding manual scores;

[0046] Step 5.2, establishing a scoring model, including a clue clarity scoring module, an accuracy scoring module, and a label alignment scoring module;

[0047] Step 5.3: Use the scoring model to automatically score the generated clue description to obtain a reward score;

[0048] Step 5.4: Evaluate the quality of the generated clue description based on the reward score.

[0049] In the above scheme, the steps of step 1 include:

[0050] Step 1.1, clustering the video frames to obtain clustering results;

[0051] Step 1.1.1: For video frames V = F1, F2, ..., F n Use the pre-trained visual encoder VIT to extract the visual features of each frame and obtain the video frame feature vector set f v1 , f v2 , ..., f vn ;

[0052] f vi =VIT(F vi ), f vi ∈R d , i∈{1, 2, ..., n}

[0053] Where VIT(*) represents the visual encoder VIT, f vi Indicates frame F i The visual feature vector of .

[0054] Step 1.1.2: Use K-Means method to analyze the video frame feature vector set f v1 , f v2 , ..., f vn Perform clustering, generate k clusters, and obtain cluster centers c1, c2, ..., c k ;

[0055] Step 1.2, extract key frames from the clustering results to obtain a key frame list;

[0056] Step 1.2.1: For each cluster C i , select the closest to the centroid c i The frame is used as the key frame

[0057] Step 1.2.2: Keyframe Input descriptor BLIP2 to generate the corresponding text description t i ;

[0058] Step 1.2.3: Cluster description t1, t2, ..., t k The query Q is input into the large language model LLAMA2, and the relevance score r between the query and each cluster is calculated. i ;

[0059] Step 1.2.4: Score r based on relevance i, set the number threshold of highly correlated clusters to rele_num_thresh. If the condition is met, stop the iteration and add the highly correlated cluster C to the list Tree; otherwise, update the number of clusters k: k = 2k;

[0060] Step 1.3, performing visual feature encoding processing on the key frame list to obtain a visual feature sequence;

[0061] Step 1.3.1, use VIVIT to encode the video frames in the key frame list;

[0062] Step 1.3.2: Each image frame F T Divide into fixed blocks and calculate the number of blocks Np;

[0063] Step 1.3.3, F T After flattening all blocks, we get P T , use a linear layer W to map each block to the feature space, and get P T The embedding representation E T ;

[0064] Step 1.3.4: For each embedding representation E T Add spatial location information to get E′ T ;

[0065] Step 1.3.5: Embedding representation E′ for each frame T Add the corresponding time position code Ptemporal, and we get

[0066] Step 1.3.6: Send the video frame embedding to the Transformer encoding module for deep feature encoding to obtain the final feature embedding

[0067] Step 1.4, perform Q-Former-based visual information compression and alignment processing on the visual feature sequence to obtain a compressed visual information representation;

[0068] Step 1.4.1. Initialize the learnable query vector:

[0069] Definition N q A learnable query vector Q q ={q1, q2, ..., q Nq},in

[0070] Among them, N q is the number of query vectors, which is usually much smaller than the visual sequence encoding length, d q is the dimension of the query vector, which is similar to the target feature space d V Be consistent.

[0071] Step 1.4.2, Cross-Attention Mechanism: Calculate the query vector Q q With visual feature sequence Attention score:

[0072]

[0073] Among them, W q , W k ∈R dq×dq is the linear projection matrix of query and key, V∈R Nq×L is the attention distribution matrix;

[0074] Step 1.4.2, calculate the context representation Z:

[0075]

[0076] Among them, W v ∈R dv×dq is the linear projection matrix of the value vector, and Z represents the visual feature sequence By N q A compressed representation of the query vectors summarized;

[0077] Step 1.4.3: Visual information compression:

[0078] The query vector Q is fused via residual connections and layer normalization. q The features of the self and the context Z are used to obtain the compressed visual information representation Q v_out :

[0079] Q v_out =LayerNorm(Z+Q q ), Q v_out ∈R Nq×dq

[0080] Step 1.4.4, feature space alignment:

[0081] Q v_out Project it into the text feature space to get the aligned feature representation Q′ out , that is, we get the visual information representation Q′ v_out :

[0082] Q′ v_out =Q out W+b,Q′ v_out ∈R Nq×dq

[0083] Where W is the learnable projection matrix and b is the bias term;

[0084] Step 1.5: Extract key features from the interview clips that never lie, and concatenate them with the key features of the video to be tested to obtain the input for visual description generation;

[0085] Step 1.5.1: Extract key features of the segments that do not lie:

[0086] From the pre-labeled interview clips, the same visual feature extraction method (such as VIVIT) is used to extract key features to obtain the visual feature representation Q′ of the non-lying clips v_out_base ;

[0087] Step 1.5.2, feature stitching:

[0088] The compressed visual information obtained in step 1.4 is represented as Q′ v_out and the visual feature representation Q′ of the true lying segment v_out_base Concatenate and obtain the input features for visual description generation:

[0089] Image final =Concat(Q′ v_out_base , Q′ v_out )Step 1.6, use the Video-LLAMA2 model to perform visual prompt reasoning and generate the final visual clue description text;

[0090] Step 1.6.1, build the visual prompt template:

[0091] According to the input visual feature sequence, a Prompt template for generating visual lie clue description is set;

[0092] The content of the visual prompt template is: based on the comprehensive analysis of facial features, hand movements, and posture performance, generate an interpretable description of whether the frame under test contains lies, combine with the reference frame to provide comparative reasoning, explain the key visual differences and their supporting label basis, and align with the real label results;

[0093] Step 1.6.2, input information preparation:

[0094] The input information includes: the input feature Image obtained in step 1.5 final , where Image final It is composed of two parts of video sequence splicing, namely the reference video frame sequence features and the test video frame sequence features;

[0095] The lie labels corresponding to the features of the video frame sequence to be tested;

[0096] Step 1.6.3, fill in the visual prompt template:

[0097] Fill the input information into the visual prompt template for subsequent visual prompt reasoning;

[0098] Step 1.6.4, visual prompt reasoning:

[0099] The filled visual prompt template is input into the Video-LLAMA2 model for visual prompt reasoning. The model will combine the guidance of the visual prompt template and the image representation to generate visual clue description text related to lies.

[0100] The model inference formula is expressed as:

[0101] D v =Video-LLAMA2(Prompt,Image final ).

[0102] Where Video-LLAMA2(*) represents the Video-LLAMA2 model, and Prompt represents the visual Prompt template.

[0103] In the above scheme, the steps of step 2 include:

[0104] Step 2.1, feature encode the audio input to obtain an audio feature sequence;

[0105] Step 2.1.1. The audio input x uses the pre-trained audio feature encoder hubert(·) to extract features from the pre-processed audio and obtain the audio feature sequence F:

[0106]

[0107] Among them, T represents the length of the encoded feature sequence, d a Represents the feature dimension of each sequence;

[0108] Step 2.2: Use two independent linear layers to extract acoustic representation and content representation respectively, and obtain acoustic features and semantic content representation;

[0109] Step 2.2.1. Acoustic representation extraction: Apply a linear layer E a To the audio feature sequence F, we get the acoustic representation F a :

[0110]

[0111] Step 2.2.2. Content representation extraction: Apply another linear layer E c To the audio feature sequence F, we get the content representation F c :

[0112]

[0113] Step 2.3, minimize the mutual information between acoustic representation and content representation by optimizing the objective, impose orthogonal constraints, and optimize acoustic features and content representation by contrastive learning loss and CTC loss;

[0114] Step 2.3.1, mutual information minimization:

[0115] Step 2.3.1.1: Use the mutual information minimization strategy to reduce the acoustic representation F a and content representation F c The correlation between them, mutual information I(F a ; F c ) measures the correlation between two variables and is calculated as:

[0116]

[0117] p(F a , F c ) indicates F a and content representation F c The joint probability distribution of a ) and p(F c ) are F a and F c The marginal probability distribution of Indicates the degree of difference between the joint distribution and the independent distribution;

[0118] Step 2.3.1.2: Use the vCLUB method to estimate the upper bound of the mutual information. The calculation formula of the vCLUB method is:

[0119]

[0120] q(F c |F a ) is a conditional probability distribution used to approximate the joint probability distribution p(F a , F c ), by minimizing Lower F a and F c The mutual information between them can be used to decouple acoustic representation and content representation.

[0121] Step 2.3.2: Acoustic Characterization F a and content representation F c The learning process imposes orthogonal constraints to reduce F a and F c The linear correlation between them improves their independence in the representation space, and its orthogonal loss is defined as:

[0122]

[0123] For calculating the matrix sum of squares, Indicates F a The transpose of the i-th row of Indicates F c The i-th row of

[0124] Step 2.3.3: Use the contrastive learning strategy to move the acoustic features of lie samples close to each other in the feature space, while the real samples are far away from the lie samples and close to each other. The loss function of contrastive learning is as follows:

[0125]

[0126] Where: S(a i , p i ) indicates F a Sample a i and its positive sample p i The similarity of S(a i , n i ) indicates F a Sample a i and its negative sample n j , τ represents the contrastive learning temperature hyperparameter, N represents the batch size, and K represents the number of negative samples for each sample;

[0127] Step 2.3.4: Introduce the speech transcription task c Mapped to the corresponding transcribed text representation, F is optimized by minimizing the loss of transcription prediction c , the specific formula is as follows:

[0128] P(Y|F c )=Softmax(WF c ), P(Y|F c )∈R T×V

[0129] Among them, P(Y|F c ) represents the predicted probability distribution of the transcribed characters at each time step t, Softmax(·) is an activation function used to convert any real vector into a probability distribution, V represents the vocabulary size of the transcribed text including the special character blank, and the target transcribed sequence is Y true , corresponding to a label sequence of length U:

[0130] Y true =y1,y2,...,y U ,y u ∈V

[0131] Among them, U≤T, that is, the length of the label sequence is less than or equal to the length of the frame sequence, and y U Represents a sequence of transcribed text; (For example, the translation of a voice is "The weather is really good today". In the truth value sequence Ytrue, y1 is "today", y2 is "day", y3 is "sky", y4 is "air", y5 is "true", and y6 is "good". These words all belong to the vocabulary table V.)

[0132] Step 2.3.5: Use the CTC model to train F c and Y true To align, the alignment path is:

[0133] π=π1,π2,…,π T , π t ∈V∪blank

[0134] The alignment path that satisfies π can be mapped to Y by removing repeated characters and blanks. true , the CTC loss is calculated by summing over all possible alignment paths, as follows:

[0135]

[0136] By maximizing L content To optimize the content representation F c , align(·) represents the alignment operation, P(·) represents the probability that the tth element of the generated sequence corresponds to the true sequence given a path π and a time step t, and t represents the time step;

[0137] Step 2.3.6: The final loss function is as follows:

[0138] L=αL MI +βL CL +γL orth -δL content

[0139] Among them, α, β, γ, δ are weight hyperparameters used to balance the contribution of each part of the loss to the overall optimization, L content is the CTC loss for speech transcription.

[0140] Step 2.4: Use Q-former to compress and align the acoustic features to obtain a compressed acoustic information representation;

[0141] Step 2.4.1: Define N q Learnable query vectors:

[0142]

[0143] Where N qis the number of query vectors, which is usually much smaller than L(N q =512), d q is the dimension of the query vector, which is the same as the target feature dimension d a Stay consistent;

[0144] Step 2.4.2: Through the cross-attention mechanism, query vector Q q From the acoustic feature sequence F a Extract global information from the query Q. The specific calculation formula is as follows: q With the acoustic feature sequence F a Attention score:

[0145]

[0146] Among them, W q , W k is the linear projection matrix of query and key, A∈R Nq×L is the attention distribution matrix, which represents the correlation between each query vector and each time step in the acoustic feature sequence;

[0147] Step 2.4.3: Compute the context representation Z:

[0148]

[0149] in, is the linear projection matrix of the value vector, Z represents the acoustic feature sequence F a By N q A compressed representation of the query vectors summarized;

[0150] Step 2.4.4: Set the query Q q The features of itself are further fused with the extracted context Z through residual connection and layer normalization (LayerNorm):

[0151]

[0152] Among them, Q a_out is the compressed acoustic feature sequence, with a length of N q , dimension and acoustic feature dimension d a same;

[0153] Step 2.4.5: Pass Q through a linear layer a_out Projected into the text feature space, the specific formula is as follows:

[0154]

[0155] in, is the learnable projection matrix and b is the bias term.

[0156] Step 2.5, randomly select an acoustic feature from the interview clip that never lies, and concatenate it with the acoustic feature to be tested to obtain the input for acoustic description generation;

[0157] Step 2.5.1: Randomly select an acoustic feature from the interview segment that never lied, and use the same method as steps 2.1-2.4 to extract the acoustic feature, denoted as Q′ a_out_base ;

[0158] Step 2.5.2: Represent the compressed acoustic information obtained in step 2.4 as Q′ a_out The acoustic feature representation Q′ of the non-lying segment a_out_base Concatenate and obtain the input features for acoustic description generation:

[0159] Audio final =Concat(Q′ a_out_base , Q′ a_out ).

[0160] Step 2.6: Take the acoustic prompt and the final acoustic description generation input as the input of SECap to generate the final interpretable description D a , and obtain the final audio clue description text.

[0161] Step 2.6.1, build the acoustic prompt template:

[0162] According to the input audio feature sequence, a Prompt template is designed for generating audio lie clue description.

[0163] The content is: Based on the comprehensive analysis of acoustic features such as pitch, energy, speaking speed, hesitation, and silence, an interpretable description of whether the test segment contains lies is generated, combined with the benchmark segment to provide clear comparison and reasoning basis, and aligned with the real lie label;

[0164] Step 2.6.2, input information preparation:

[0165] The input information includes:

[0166] The input feature Audio obtained in step 2.5 final , where Audio final It is composed of two parts of audio sequences, namely the reference audio sequence features and the audio sequence features to be tested;

[0167] The lie label of the audio segment to be tested;

[0168] Step 2.6.3: Integrate the acoustic prompt with the final acoustic description generation input:

[0169] Combine the filled acoustic prompt template with the final audio representation Audio final Integrate them to ensure that the formats of the two match, so that the model can process both prompt information and audio features at the same time;

[0170] Step 2.6.4: SECap model reasoning to generate the final interpretable description D a , the generated interpretable description D a Arranged into a complete audio clue description text:

[0171] The integrated acoustic prompt and audio representation are used as input to the SECap model;

[0172] Use the SECap model to reason about the input acoustic prompt and audio representation;

[0173] The model will combine the guidance of acoustic prompts and audio representation to generate audio clue description text related to lies. The model reasoning formula is expressed as:

[0174] D a =SECap(prompt, Audio final )

[0175] Where SECap(*) indicates the SECap model, and Prompt indicates the acoustic prompt template;

[0176] The generated interpretable description D a Organize into complete audio clue description text.

[0177] In the above scheme, the steps of step 3 include:

[0178] Step 3.1, analyze the test questions to obtain the prior information that may be helpful for lie identification;

[0179] Step 3.1.1: Set up the text prompt template for question analysis, including the following steps:

[0180] Step 3.1.1.1. Whether the questions are likely to induce the respondents to give brief or vague answers;

[0181] Step 3.1.1.1. Whether the questions are likely to induce the respondents to give brief or vague answers;

[0182] Step 3.1.1.2: Is the question likely to make the respondent use common sense answers to cover up a lie?

[0183] Step 3.1.1.3: If the respondent lies on the question, is it possible to verify the lie through follow-up questions or background knowledge?

[0184] Step 3.1.1.4: Will the question significantly increase the cognitive load of the subject?

[0185] Step 3.1.2, input the text prompt template in 3.1.1 together with the question into the LLAMA2 model to obtain the lie clue analysis results about the question;

[0186] Step 3.2, analyze the personality characteristics of the subjects to obtain descriptions of lie clues related to personality;

[0187] Step 3.2.1, set the text prompt template of "Personality Traits", including the following steps:

[0188] Step 3.2.1.1, combining the five dimensions of the Big Five personality, generate a summary description for the input Big Five personality test scale vector;

[0189] Step 3.2.2, obtain the Big Five personality test scale vector of the test subject;

[0190] Step 3.2.3, fill the subject's personality vector into the text prompt template of "Personality Traits" and input it into the LLAMA2 model for description generation, and obtain the lie clue analysis results about the subject's personality traits;

[0191] Step 3.3: Final text clue description generation:

[0192] Step 3.3.1, set the text prompt template generated by the final text clue description;

[0193] Step 3.3.1.1, prepare input information, including: analysis results of test questions, personality characteristics analysis results, test taker's answer text and corresponding lie label;

[0194] Step 3.3.1.2: Prepare the analysis content requirements text prompt, including:

[0195] The compatibility or potential contradiction between the test taker's answer and the characteristics of the question;

[0196] The degree to which the test taker's answers match the Big Five personality traits and any abnormalities;

[0197] Language clues in the test-taker’s answers that may indicate a lie, including excessive embellishment, abnormal emotions, and logical inconsistency;

[0198] Step 3.3.1.3: Prepare the text prompt required for output, the content is:

[0199] Integrate the input information into clues and generate the final lie clue description according to the analysis requirements. The final description must conform to the given lie label, that is, generate a corresponding reasoning description for the "true" or "false" label;

[0200] Step 3.3.2, fill the analysis results obtained in steps 3.1 and 3.2 together with the test subject's answer text and the real label into the text prompt template of step 3.3.1 and then input it into the LLAMA2 model for description generation to obtain the final text clue description;

[0201] Use the LLAMA2 model to perform text prompt reasoning and generate text clues related to lies to describe text D t , the model inference formula is expressed as:

[0202] D t =LLAMA2(Prompt[Input])

[0203] Wherein LLAMA2(*) represents the LLAMA2 model, Prompt represents the text prompt template, and Input represents the prompt of "test question" filled with corresponding content, the prompt of "personality characteristics" or the prompt instruction generated by the final text clue description.

[0204] Because the present invention adopts the above technical means, it has the following beneficial effects:

[0205] 1. Visual clue description generation:

[0206] The present invention uses a large model to extract video key frames and generate descriptions. It provides concise and key descriptions of video content, reduces redundant information, and enhances interpretability. It improves the efficiency and accuracy of video analysis, allowing the model to focus more on key information related to lie detection. Compared with traditional video analysis methods, this proposal uses key frame extraction and description generation to more effectively utilize video information and improve the efficiency and accuracy of lie detection.

[0207] 2. Audio clue description generation:

[0208] The present invention adopts acoustic and content information decoupling technology, combines mutual information minimization and orthogonal loss, and contrastive learning strategy to improve the discrimination of audio features and reduce the interference of speech content on lie detection.

[0209] The role of audio features in lie detection is enhanced, and the accuracy and reliability of the model are improved. Compared with traditional methods that rely only on speech content, this proposal enhances the role of audio features in lie detection and reduces the false positive rate by decoupling acoustic and content information.

[0210] 3. Text clue description generation:

[0211] The present invention combines prior information such as question type and personality traits to design a prompt template to guide the generation of text description. The pertinence and accuracy of the text description are improved, and a more comprehensive analysis is performed in combination with contextual information. The generated text description is more comprehensive and reliable, and can be comprehensively analyzed in combination with the question background and personality traits. Compared with the analysis method that relies solely on text content, this proposal improves the comprehensiveness and accuracy of the text description and enhances the reliability of lie detection by introducing question types and personality traits.

[0212] IV. Lead Generation Quality Assessment:

[0213] The present invention uses an automatic evaluation method based on RLHF to evaluate from three perspectives: clue clarity, accuracy, and label alignment. It reduces the reliance on manual evaluation and improves the efficiency and accuracy of evaluation. It realizes automatic and accurate evaluation of the quality of generated clues, reduces the cost of manual evaluation, and improves evaluation efficiency. Compared with traditional fine-grained text evaluation methods, this proposal uses RLHF automatic evaluation, which is more in line with human judgment standards and improves the accuracy and efficiency of evaluation.

[0214] In summary, this application proposal effectively solves the problems of data scarcity, poor interpretability, and high manual evaluation costs in existing technologies through advanced technical means such as multimodal information processing, application of large language models, and automated evaluation strategies, and achieves technological breakthroughs and improvements. BRIEF DESCRIPTION OF THE DRAWINGS

[0215] Figure 1 , Figure 2 It is a simplified flow chart of the present invention;

[0216] Figure 3 is a sample data example;

[0217] Figure 4 It is a keyframe tree search. DETAILED DESCRIPTION

[0218] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.

[0219] In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0220] 1.1 Visual cue description generation

[0221] 1.1. Key frame extraction

[0222] In order to reduce the interference of a large amount of redundant information in the video and improve the accuracy of the explainable clue description, we first need to extract the key frames in the video. The specific process is as follows:

[0223] 1.1.1 Adaptive Breadth Search

[0224] 1.1.1.1) Video frame clustering:

[0225] For video frames V = F1, F2, ..., F n Use the pre-trained visual encoder VIT to extract the visual features of each frame:

[0226] f v_i =VIT(F v_i ), f v_i ∈R d , i∈{1, 2, ..., n}

[0227] Among them, f v_i Indicates frame F v_i The visual feature vector of .

[0228] Use K-Means method to analyze the video frame feature vector f v_1 , f v_2 , ..., f v_n Perform clustering and generate k clusters:

[0229] (C1, C2, ..., C k ), (c1, c2, ..., c k )=K-Means({f1, f2,..., f n}, k)

[0230] Among them, C i is the i-th cluster, c i ∈R d represents the centroid of the ith cluster.

[0231] 1.1.1.2) Cluster description generation:

[0232] For each cluster C i , select the closest to the centroid c i The frame is used as the key frame F i *:

[0233]

[0234] Set the key frame F i *Input the descriptor BLIP2(·) to generate the corresponding description t i :

[0235] t i =BLIP2(F i * )

[0236] 1.1.1.3) Relevance score:

[0237] Describe the clusters t1, t2, ..., t k The query Q is input into the large language model LLAMA2, and the relevance score r between the query and each cluster is calculated. i :

[0238] r i =LLAMA2(t i , Q), r i ∈1,2,3

[0239] Among them, 1 means "irrelevant", 2 means "partially relevant", 3 means "highly relevant", and Q is the prompt used to calculate the relevance with the key frame description, and its format is as follows:

[0240] "Please choose from the following candidate descriptions [t1, t2, ..., t n ] to identify possible deceptive behavior characteristics. Focus on descriptions containing changes in facial expressions, postures, eye movements, etc. and give these descriptions a relevance score.

[0241] 1.1.1.4) Adaptive expansion:

[0242] In order to find enough key frames, we set the number threshold of highly correlated clusters to rele_num_thresh. If:

[0243] count(r i |r i =3)≥rele n um t hresh

[0244] Then stop the iteration and cluster the high correlation C = C i |r i =3 is added to the list Tree in the order of the classes.

[0245] Tree = Tree.add(C)

[0246] Otherwise, update the number of clusters k: k=2k.

[0247] 1.1.2 Keyframe Depth Extension

[0248] 1.1.2.1) Sub-clustering:

[0249] For each cluster C i , if the correlation r i ≥2, the category is further clustered to find more key frames.

[0250] Let C i =F1, F2, ..., F m , re-clustered into w sub-clusters:

[0251] (C i1 , C i2 , ..., C iw ), (c i1 , c i2 , …, c iw )=K-Means({f v_1 , f v_2 , ..., f v_m},w)

[0252] Where w represents the number of categories of secondary clustering.

[0253] Finally, based on the secondary clustering, the correlation score is used to obtain more key frames.

[0254] 1.1.2.2) Depth expansion:

[0255] For high correlation clustering r i =3, a secondary sub-clustering will be performed on the basis of the sub-clustering to obtain the final sub-clustering and find the final key frame

[0256] 1.1.2.3) Finally, all key frames Add to the final list in order t In ree:

[0257] Final tree =Final tree .add(F i *)

[0258] 1.1.3 Video Coding

[0259] Using VIVIT for Final tree The specific process is as follows:

[0260] 1.1.3.1) We define the final key video frame sequence V = F1, F2, F3, F4, ..., F T , F T ∈RH ×W×C Where T is the total number of frames in the input video, H and W are the height and width of each frame image, and C is the number of channels.

[0261] 1.1.3.2) Each image frame F T Will be divided into fixed blocks with dimension size P 2 ×C, the number of blocks is N p , and its calculation formula is as follows:

[0262]

[0263] 1.1.3.3) F T After flattening all blocks, we get P T ,

[0264] 1.1.3.4) Using a linear layer Map each block to the feature space and get P T The embedding representation E T :

[0265]

[0266] Among them, N p is the number of blocks, d V is the feature dimension of the block.

[0267] 1.1.3.5) For each embedding representation E t Add spatial location information, the calculation formula is as follows:

[0268] E′ T =E T +P spatial

[0269]

[0270] Among them, P spatial is automatically learned through training, pos is the position index of the block, i is the dimension index, d V is the dimension of the embedding vector.

[0271] 1.1.3.6) In order to preserve the temporal information of the video, we embed each frame as E′ T , add the corresponding time position code P temporal :

[0272]

[0273]

[0274] Among them, Ptemporal With P spatial The difference between the encoding methods is that different scales are used in time and space position encoding. By changing the scale of time position encoding, it is ensured that the model can accurately capture the temporal characteristics and spatial structure of the video.

[0275] 1.1.3.7) Finally, the video frame embedding is fed into the Transformer encoding module for deep feature encoding to obtain the final feature embedding:

[0276]

[0277] in, is the encoded video embedding, with a dimension of T×N p ×d V .

[0278] 1.2 Visual Information Compression and Alignment Based on Q-Former

[0279] 1.2.1 Learnable Query Vector Q q initialization

[0280] Q-Former defines N q A learnable query vector for Extract the visual information related to the lie from the query vector. The set of these query vectors is defined as:

[0281]

[0282] Among them, N q is the number of query vectors, which is usually much smaller than the visual sequence encoding length, d q is the dimension of the query vector, which is similar to the target feature space d V Be consistent.

[0283] 1.2.2 Cross-Attention Mechanism

[0284] Through the cross attention mechanism, the query vector Q q From the visual feature sequence Extract the visual clue information related to lies. The specific calculation formula is as follows:

[0285] Compute query Q q With visual feature sequence Attention score:

[0286]

[0287] in, is the linear projection matrix of the query and key. is the attention distribution matrix, which represents the relevance of each query vector to each time step in the visual feature sequence.

[0288] 1.2.3 Contextual Representation Computation

[0289]

[0290] Where Wv∈R dv×dq is the linear projection matrix of the value vector, and Z represents the visual feature sequence By N q The compressed representation of the query vectors.

[0291] 1.2.4 Visual Information Compression

[0292] The query Q q The features of the image are further fused with the extracted context Z through residual connection and layer normalization (LayerNorm) to obtain the compressed visual information representation Q v_out :

[0293]

[0294] Among them, Q v_out is the compressed acoustic feature sequence, with a length of N q , dimension d q With visual features d V same.

[0295] 1.2.5 Feature Space Alignment

[0296] Through a linear layer, Q v_out Projected into the text feature space, the specific formula is as follows:

[0297]

[0298] Among them, W is the learnable projection matrix and b is the bias term.

[0299] 1.3 Visual Reference Fiducial Stitching

[0300] For each subject, we randomly select a segment from the interview that never lied and extract its key features, denoted as Q′ v_out_base , the key feature of the video to be tested is recorded as Q′ v_out Then we will Q′ v_out_base and Q′ v_out Concatenate to obtain the final input for visual description generation.

[0301] Image final =Concat(Q′ v_out_base Q′ v_out ).

[0302] 1.4 Visual prompt reasoning

[0303] 1.4.1 Prompt template with joint true label

[0304] A template specifically designed for generating visual lie clue descriptions is designed:

[0305] ###**Prompt template: Visual lie clue description generation**

[0306] Please generate a description of visual clues related to lies based on the input visual feature sequence and the comparison between the reference frame and the frame to be tested. The input information is as follows:

[0307] ####**Input information**:

[0308] 1. **Visual feature sequence**:

[0309] - The first Nq sequences: record the test subject's performance when he / she is not lying (reference frames).

[0310] - The next Nq sequences: record the performance of the video frame to be tested (frame to be tested).

[0311] 2. **Lie Label**:

[0312] -Label of the frame to be tested: Tru e (True) or Fals e (lie).

[0313] ####**Output requirements**:

[0314] 1. Compare the reference frame with the frame to be tested and analyze the differences between them from the following visual perspectives:

[0315] -Facial Features:

[0316] - Eyes (blinking frequency, gaze direction, eye movement).

[0317] - Mouth (lip movements, tense pursing, micro-expression changes).

[0318] - Eyebrows (frowning, raising eyebrows, frowning, etc.).

[0319] -**Hand movements**:

[0320] -Hand gesture changes, unusual shaking, rubbing, or hiding behavior.

[0321] -**Posture and body language**:

[0322] - Changes in body posture (stiffness, overly relaxed, or frequent movement).

[0323] - Whether there are any abnormal body movements (such as trembling, leaning forward / backward, etc.).

[0324] 2. Extract visual lie clues from the above analysis:

[0325] -Are there visual abnormalities associated with tension, anxiety, or avoidance?

[0326] -Does the frame under test show significant deviations compared to the reference frame?

[0327] -Does the clue support a lie label (True / False)?

[0328] 3. Generate a summary description:

[0329] - Based on a comprehensive analysis of facial features, hand movements, and posture, an interpretable description of whether the frame under test contains lying behavior is generated.

[0330] - Comparative reasoning needs to be provided in conjunction with the reference frames, explaining the key visual differences and their supporting labeling rationale.

[0331] -Need to be aligned with the true label results.

[0332] 1.4.2 Model Reasoning

[0333] Finally, we use Video-LLAMA2 as the video description generation model and combine Prompt with the final Image final As input, it guides the visual model to generate the final interpretable description D v :

[0334] D v =Video-LLAMA2(Prompt,Image final ).

[0335] 2. Audio cue description generation

[0336] 2.1 Decoupling of Acoustic and Content Information

[0337] In lie detection, acoustic features (such as speaking speed, intonation, pauses, etc.) may reveal lying behavior, but the specific content of the speech (such as the semantic information expressed) may interfere with the model's extraction of lie-related features. For example:

[0338] Voice A (lie): The speaker said nervously and quickly: "I didn't see him today." Voice B (truth): The speaker said calmly and slowly: "I didn't see him today." The content of the two voices is exactly the same, but the acoustic features are obviously different: Voice A's acoustic features: rising tone, fast speaking speed, irregular pauses. Voice B's acoustic features: steady tone, normal speaking speed, natural pauses. In this case,

[0339] It is impossible to judge lies only by semantic content, and it needs to be judged in combination with contextual information. Therefore, when performing lie detection based on audio, without providing more contextual information, we should focus on using acoustic features to detect lies. That is, when extracting acoustic clues related to lies, the content and acoustic information should be decoupled to reduce the interference of irrelevant information. The present invention adopts mutual information minimization and feature orthogonalization strategies to achieve the decoupling of content and acoustic information. The specific steps are as follows:

[0340] 2.1.1 Input Representation

[0341] The audio input x is passed through the pre-trained audio feature encoder hubert(·) to obtain the audio feature F:

[0342] F = hubert(x)

[0343]

[0344] Among them, T represents the length of the encoded feature sequence, d a Represents the feature dimension of each sequence.

[0345] 2.1.2 Acoustic Representation and Content Representation Learning

[0346] 2.1.2.1) Use a linear layer that does not change the dimension to act as the acoustic representation extractor E a , the formula is as follows:

[0347]

[0348] Among them, F a Represents the acoustic characteristics of audio.

[0349] 2.1.2.2) We use a linear layer that does not change the dimension size to act as the audio content representation extractor E c , the formula is as follows:

[0350]

[0351] Among them, F c Represents the semantic content of audio.

[0352] 2.1.3 Optimization Objectives

[0353] 2.1.3.1) Minimizing Mutual Information

[0354] In order to reduce the acoustic characterization F a and content representation F c The correlation between them needs to minimize the mutual information between them. The calculation formula of mutual information is as follows:

[0355]

[0356] Among them, I(F a ; F c ) represents the acoustic representation F a and content representation F c The mutual information between them is used to measure the correlation between the two.

[0357] p(F a , F c ) indicates F a and content representation F c The joint probability distribution of a ) and p(F c ) are F a and F c The marginal probability distribution of . Indicates the degree of difference between the joint distribution and the independent distributions.

[0358] Minimize I(F a ; F c ) aims to make the acoustic representation and content representation as independent as possible, thereby reducing the redundant information and correlation between them and achieving the effect of feature decoupling.

[0359] Since it is not feasible to directly calculate the mutual information, we use the vCLUB (Variational Contrastive Log-ratio Upper Bound) method to estimate its upper bound. The vCLUB method is a technique for estimating the upper bound of mutual information. It constructs a conditional probability distribution q(F c |F a ) to approximate the joint probability distribution p(F a , F c ), thus avoiding the direct calculation of the difficult-to-solve mutual information. The calculation formula is as follows:

[0360]

[0361] Among them, the conditional probability distribution q(F c |F a ) is fitted by a multi-layer perceptron MLP.

[0362] 2.1.3.2) Orthogonal loss

[0363] In addition, to ensure that F a With F c Independence in the representation space reduces redundant information and linear correlation between each other. We a With F c The learning process imposes an orthogonal constraint, and its orthogonal loss is defined as:

[0364]

[0365] in, Computes the matrix sum of squares.

[0366] 2.1.3.3) Contrastive Learning Loss

[0367] To ensure that F a Contains acoustic information related to lies, uses the acoustic features of positive samples of the same type (acoustic sample pairs constructed by lies and lies, truth and truth) to be as close as possible in the feature space, and the acoustic features of samples of different categories (such as acoustic sample pairs constructed by lies and truth) are more separated in the feature space. The loss function of contrastive learning is as follows:

[0368]

[0369] Where: S(a i , p i ): F a Sample a i and its positive sample p i The similarity (cosine similarity).

[0370] S(a i , n i ): F a Sample a i and its negative sample n j similarity.

[0371] τ: Contrastive learning temperature hyperparameter, used to adjust the gradient size of the similarity distribution.

[0372] N: batch size.

[0373] K: The number of negative samples for each example.

[0374] Through the above contrastive learning strategy, the acoustic features of lie samples are close to each other in space, while the true samples are far away from the samples and close to each other.

[0375] 2.1.3.4) CTC Loss

[0376] To ensure that F ccontains content information, we introduce the speech transcription task to convert F c Mapped to the corresponding transcribed text representation, F is optimized by minimizing the loss of transcription prediction c The specific formula is as follows:

[0377] P(Y|F c )=Softmax(WF c ), P(Y|F c )∈R T×V

[0378] Among them, P(Y|F c ) represents the predicted probability distribution of the transcribed characters at each time step t, V represents the vocabulary size of the transcribed text (including the special character blank for ctc),

[0379] The target transcription sequence is Y true , corresponding to a label sequence of length U:

[0380] Y true =y1,y2,...,y u ,y u ∈V

[0381] Among them, U≤T, that is, the length of the tag sequence is less than or equal to the length of the frame sequence.

[0382] The CTC model allows for the c and Y true To align, the alignment path is:

[0383] π=π1,π2,…,π T , π t ∈V∪blank

[0384] The alignment path that satisfies π can be mapped to Y by removing repeated characters and blanks. true .

[0385] The CTC loss is calculated by summing over all possible alignment paths, as follows:

[0386]

[0387] By maximizing L content To optimize the content representation F c .

[0388] 2.1.3.5) The final loss function is as follows:

[0389] L=αL MI +βL CL +γL or -δL content

[0390] Among them, α, β, γ, δ are weight hyperparameters used to balance the contribution of each part of the loss to the overall optimization, L content is the CTC loss for speech transcription.

[0391] 2.2 Acoustic Feature Compression and Alignment Based on Q-former

[0392] 2.2.1 Query representation initialization

[0393] Q-Former defines N q A learnable query vector for a Extract the acoustic information related to the lie from the query vector. The set of these query vectors is defined as:

[0394]

[0395] Among them, N q is the number of query vectors, usually much smaller than L, such as (N q =512), d q is the dimension of the query vector, which is the same as the target feature dimension d a Be consistent.

[0396] 2.2.2 Cross-Attention Mechanism

[0397] Through the cross attention mechanism, the query vector Q q From the acoustic feature sequence F a Extract global information from the . The specific calculation formula is as follows:

[0398] Compute query Q q With the acoustic feature sequence F a Attention score:

[0399]

[0400] Among them, W q , W k is the linear projection matrix of the query and key. A∈R Nq×L is the attention distribution matrix, which represents the relevance of each query vector to each time step in the acoustic feature sequence.

[0401] 2.2.3 Contextual Representation Computation

[0402]

[0403] Among them, W v ∈R da×dq is the linear projection matrix of the value vector, Z represents the acoustic feature sequence F a By N qThe compressed representation of the query vectors.

[0404] 2.2.4 Output

[0405] The query Q q The features of itself are further fused with the extracted context Z through residual connection and layer normalization (LayerNorm):

[0406]

[0407] Among them, Q a_out is the compressed acoustic feature sequence, with a length of N q , dimension and acoustic feature dimension d a same.

[0408] 2.2.5 Feature Space Alignment

[0409] In this paper, we transform Q out Projected into the text feature space, the specific formula is as follows:

[0410]

[0411] in, is the learnable projection matrix and b is the bias term.

[0412] 2.3 Acoustic reference datum splicing

[0413] For each subject, we randomly selected an acoustic feature from the interview segment that never lied and recorded it as Q′ a_out_base . Then, we transform the baseline acoustic feature Q′ a_out_base and the acoustic characteristic Q' to be measured a_out Concatenate and use as final input.

[0414] Audio final =Concat(Q′ a_out_base , Q′ a_out )

[0415] 2.4 Acoustic prompt reasoning

[0416] 2.4.1 Acoustic lie clue description to generate prompt template

[0417] ####**Input information**:

[0418] 1. **Acoustic feature sequence**:

[0419] - The first Nq sequences: record the subject's acoustic performance when he / she is not lying (baseline acoustic features).

[0420] - The next Nq sequences: record the acoustic features to be measured (the segments to be measured).

[0421] 2. **Lie Label**:

[0422] - The label of the segment to be tested: True (truth) or False (lie).

[0423] ####**Output requirements**:

[0424] 1. **Comparative analysis of the baseline acoustic characteristics and the acoustic characteristics to be measured**:

[0425] -Analyze the differences between the two from the following perspectives:

[0426] - **Pitch**: Is there any abnormal fluctuation, significant rise or fall?

[0427] -**Energy**: The intensity change of the audio, whether it shows a tense or urgent trend.

[0428] - Speaking Rate: Whether the speaking rate has significantly increased or decreased, and whether there is a difference from the baseline performance.

[0429] -Speaking Style: Whether the speaker is speaking in a hasty, incoherent or too steady manner.

[0430] - Hesitation: Are there any unusual pauses or repetitions, which may reflect hesitation.

[0431] - Silence Duration: Is there any long or irregular silence, which may reflect tension or information concealment?

[0432] 2. Extracting acoustic cues related to lies:

[0433] - Compare the baseline segment with the segment to be tested and analyze whether the segment to be tested deviates significantly from normal performance.

[0434] - Whether to support lie labels for input (True / False).

[0435] 3. Generate summary description:

[0436] - Based on a comprehensive analysis of acoustic features such as pitch, energy, speech rate, hesitation, silence, etc., an interpretable description of whether the test segment contains lying behavior is generated.

[0437] -Need to combine with benchmark clips to provide clear comparison and reasoning.

[0438] -Needs to be aligned with the True Lies label.

[0439] 2.4.2 Model Reasoning

[0440] Finally, Prompt and the final Audio final As the input of SECap, SECap is a speech-text model that guides the audio-text model to generate the final interpretable description D a .

[0441] D a =SECap(prompt, Audio final ).

[0442] 3.3 Text clue description generation

[0443] Inferring lies from textual content is a complex task that involves capturing potential deception from language expressions. Studies have shown that people tend to believe intuitive answers more easily, which provides a greater chance for liars to successfully deceive the questioner. However, most current text-based lie detection studies focus on multi-level analysis of the content of the answer, such as judging the possibility of a lie by evaluating the logic, conciseness, and fluency of the answer, but ignore the in-depth analysis of the question itself and prior information such as individual personality traits. This may lead to a decrease in the accuracy of lie discrimination, especially when the answer is highly related to the question context or individual characteristics. In order to improve the accuracy of discrimination, in addition to the content of the subject's answer, we also introduced additional prior information - test questions and personality information, to provide more available auxiliary information for lie discrimination.

[0444] The following is the process of obtaining and modeling text lie clues:

[0445] A. Test question information

[0446] (1) Type of question

[0447] The type of question has a direct impact on the way of answering. Different types of questions may induce different answer characteristics, which also means that there should be different focuses when analyzing the content.

[0448] Yes or No (closed questions): Yes or No questions usually require short and clear answers, such as "Did you do this?". The brevity and directness of the answers to these questions reduce the complexity of language features, allowing lie detection to focus on the consistency and logic of the answers. By analyzing whether the answers are consistent, contradictory or unreasonable, false answers can be effectively identified.

[0449] Narrative questions (open-ended questions): Narrative questions require the subject to provide a more detailed description, such as "Please describe your behavior last night." Unlike true or false questions, narrative questions prompt the subject to provide more background information and details. This type of question not only increases the complexity of the answer, but may also reveal more hidden clues to lies. Usually, potential contradictions or inconsistencies can be discovered through detailed questioning.

[0450] Hypothetical questions: Hypothetical questions allow the subject to enter into an imaginary situation, which helps to detect the individual's true emotional expression in a hypothetical scenario. The answers to hypothetical questions may be inconsistent with the subject's actual behavior and personality traits, thereby revealing lies or false statements.

[0451] (2) Difficulty of answer verification

[0452] Whether the answer is easy to verify is an important basis for judging the possibility of lying. For questions that are difficult to verify, the subjects may be more inclined to lie, while for questions that are easy to verify, the subjects are less likely to lie. Therefore, the analysis of the difficulty of answer verification can provide prior information for lie discrimination:

[0453] Answers that are easily verified: answers that involve clear facts or can be verified by external data (such as "Where were you last night?"). For such answers, the subject needs to provide highly specific and true information, otherwise it is easy for the interrogator to verify and find contradictions. Lie detection can be judged by features such as the logic of the answer, consistency of details, and fluency of language.

[0454] Answers that are difficult to verify: such as questions that involve subjective feelings or cannot be directly verified (such as "How did you feel at the time?"). Such answers usually lack objective basis, and subjects are more likely to provide vague or ambiguous answers to cover up the facts. Lie detection can identify potential lies by analyzing vague words, emotional words (such as "maybe", "probably") and the specificity of the content in the answers. In addition, for answers that are difficult to verify, people are generally more likely to believe answers that are consistent with common sense, such as "There are four people in my family, me, my father, my mother and my brother." Therefore, evaluating whether the answer is consistent with common sense can help to be vigilant against the possibility that subjects may use answers that are consistent with common sense to deceive, so as to make judgments from other entry points.

[0455] B. Personality traits

[0456] The Big Five Personality Traits provides a scientific and widely used personality structure framework, covering openness, conscientiousness, extraversion, agreeableness, and neuroticism. Different personality dimensions have significant differences in the manifestation of lying behavior. Through modeling and analysis of personality traits, we can better evaluate whether the answer is consistent with the individual's behavior pattern and improve the personalization and accuracy of lie identification.

[0457] C. Description Generation

[0458] For the text lie clue description generation task, we use LLAMA2 as our inference model. We design different prompt strategies to guide the model to generate lie clue descriptions of text content.

[0459] (1) Prompt template

[0460] 1) "Test Question" prompt

[0461] Please answer the following questions, summarize your answers, and return a description that is helpful for lie detection:

[0462] 1. Does the question tend to induce respondents to give brief or vague answers?

[0463] 2. Does the question make it easy for the respondent to use a common-sense answer to cover up a lie?

[0464] 3. If the respondent is lying, is it possible to verify the lie through follow-up questions or background knowledge?

[0465] 4. Is the question psychologically stressful (e.g., will it significantly increase the cognitive load on the subject)?

[0466] The content of the questions asked is as follows:

[0467] [Question text]

[0468] 2) Prompt for “Personality Traits”

[0469] Please combine the five dimensions of the Big Five personality and generate a summary description for the input [Big Five Personality Test Scale Vector] to judge the potential abnormal patterns of the subjects in the process of answering questions.

[0470] 3) Final text clue description generated prompt

[0471] ### Input information:

[0472] 1. **Analysis of lying behavior characteristics based on test questions: [Analysis results of test questions]**:

[0473] 2. **Analysis of lying behavior characteristics based on the Big Five personality dimensions: [Personality characteristics analysis results]**:

[0474] 3. **Tester's answer and lie label**:

[0475] -Tester's answer: [Text of tester's answer]

[0476] -Lie Tags: [Tag: True / False]

[0477] ###**Analysis content requirements**:

[0478] 1. The matching or potential contradiction between the characteristics of the test-taker’s answer and the question asked.

[0479] 2. The degree of conformity of the test-taker’s answers with the Big Five personality traits and any abnormal performance.

[0480] 3. Linguistic clues in the test-taker’s answers that may indicate a lie (such as excessive embellishment, abnormal emotions, and logical inconsistency).

[0481] ###**Output requirements**:

[0482] The input information is integrated with clues, and the final lie clue description is generated according to the analysis requirements. The final description must conform to the given lie label, that is, a corresponding reasoning description is generated for the "true" or "false" label.

[0483] (IV) Final clue description generation

[0484] Prompt that integrates text, voice, and visual final clue description

[0485] ###**Input Information**

[0486] 1. **Final text clue description result**:

[0487] 2. **Final acoustic clue description results**:

[0488] 3. **Final visual clue description result**:

[0489] ###**Output Requirements**

[0490] Integrate the above input information to generate the final lie clue description. The description must meet the following requirements:

[0491] ####**1. Modal analysis summary**:

[0492] Lie clues in three modalities: text, voice, and vision are summarized independently.

[0493] ####**2. Multimodal cue integration and relevance**:

[0494] Analyze the relevance of text, speech, and visual features, including consistency and contradiction, and combine them to determine whether these clues collectively point to lying behavior or support a true description.

[0495] ####**3. Lie label alignment and reasoning conclusion**:

[0496] Give the inference conclusion and make it clear whether the subject's answer is a lie.

[0497] 5. Lead generation quality assessment

[0498] Quantifying the annotation quality of generated data can indirectly optimize processes such as prompt engineering and candidate screening, thereby improving the generation of interpretable descriptions, reducing reliance on manual evaluation, and improving overall annotation efficiency.

[0499] This paper proposes an automatic evaluation method based on reinforcement learning with human feedback (RLHF). The core of the method is to simulate the preference of manual scoring and automatically score the generated multimodal descriptions from three perspectives: clue clarity, accuracy, and label alignment. The specific steps are as follows:

[0500] 5.1 Data Preparation and Manual Labeling

[0501] First, we need to build a dataset for training the automatic quality assessment model, which includes the original data features, generated descriptions and their corresponding manual scores.

[0502] The data format is where x i is the original data feature, y i is the description generated by the model, s i It is a manually annotated rating vector, which is the weighted average of the three dimensions. Its value range is [0,1]. The larger the value, the higher the quality.

[0503] Among them, the manual scoring rules refer to the following three aspects:

[0504] Cue clarity: evaluates whether the source of each modality clue is clearly indicated in the generated description.

[0505] Accuracy: Evaluates whether the generated description truly and accurately reflects the characteristics of the input data.

[0506] Label alignment: evaluates whether the generated description is consistent with the input label (True / False).

[0507] 5.2 Establishing a scoring model

[0508] The scoring model inputs the generated description y and the corresponding input data x, and outputs a reward score R(y|x) to indicate the quality of the generated description. The score range of the generated quality is from [0, 1], and the larger the value, the higher the quality. The reward model consists of the following three sub-modules: ClarityEncoder(·), AccuracyEncoder(·), and AlignmentEncoder(·).

[0509] Among them, ClarityEncoder(·), AccuracyEncoder(·), and AlignmentEncoder(·) all have pre-trained language model T5 as the input. The difference is that the input of AccuracyEncoder(·) contains the original data in addition to the generated clue description data, while the other two only contain the generated clue description data.

[0510] (1) Clue clarity scoring module R clarity (y):

[0511] R clarity (y) = σ(ClarityEncoder(y))

[0512] Among them, ClarityEncoder(·) is played by T5, which automatically scores the input by learning human scoring preferences (whether the clue description clearly points out the modal source and whether specific features, values ​​or events are used to describe the clues). σ is a Sigmoid function, which is used to map the output of ClarityEncoder(·) to a value between [0, 1].

[0513] (2) Accuracy scoring module R accuracy (y|x):

[0514] The accuracy score scores the generated clues by comparing the consistency between the clues in the generated description and the input data. The specific formula is as follows:

[0515] R accuracy (y|x)=σ(AccuracyEncoder(x,y))

[0516] Among them, AccuracyEncoder(·) is played by T5. The visual and audio modal features in the multimodal data x are compressed using Q-former and aligned to the text space for easy processing by T5. AccuracyEncoder(·) learns human scoring preferences: it compares whether the clues of each modality in the generated description are consistent with those in the original modal data. If they are consistent, they are given high scores, otherwise they are given low scores.

[0517] (3) Label alignment scoring module R alignment (y):

[0518] The label alignment scoring module is used to evaluate the consistency between the described features and the input features.

[0519] R alignment (y) = σ(AlignmentEncoder(y))

[0520] Among them, AlignmentEncoder(·) is played by BERT, which learns human rating preferences: it determines whether the semantic information described by the clue is consistent with the label information to decide whether to give a high score.

[0521] 5.3 Overall Reward Function

[0522] Finally, the reward function combines the outputs of the three sub-scoring models:

[0523] R(y|x)=α·R clarity (y)+β·R accuracy (y|x)+γ·R alignment (y)

[0524] Among them, α, β, and γ are weight parameters used to adjust the relative importance of the three dimensions in the overall score, and α+β+γ=1.

[0525] This application proposal proposes a reliable and explainable data annotation strategy for non-invasive lie detection research. This is the first automated data annotation solution proposed in this field, filling the gap in data annotation research in the field and laying the foundation for non-invasive lie detection research. The specific reliability is reflected in the following technical advantages:

[0526] 1. Introducing a reference benchmark for explainable lie reasoning:

[0527] Previous non-intrusive lie detection methods only extracted features from the input visual, audio, and text, and finally sent them to the classification network to achieve lie detection. This method lacks interpretability and cannot effectively utilize prior information, resulting in poor results. In contrast, introducing non-lying features as a reference benchmark at the front end of the data to be tested provides more prior information for the large model, making it possible to more effectively utilize the comparison and reasoning capabilities of the large model to achieve reliable lie detection.

[0528] 2. Designed a comprehensive prompt reasoning template:

[0529] Based on domain prior knowledge, this paper proposes a series of prompts to use the reasoning ability of the large model to obtain clue descriptions related to lies in each modality. Compared with previous studies, the prompt scheme proposed in this paper is more comprehensive and provides a better reference strategy for subsequent related research.

[0530] 3. Automated quality assessment

[0531] This paper proposes a new automatic evaluation method for reinforced learning with human feedback (RLHF), which simulates the preference of manual scoring and automatically scores the generated multimodal descriptions from three perspectives: clue clarity, accuracy, and label alignment. Compared with the fine-grained text generation quality evaluation method, the method proposed in this paper is more in line with human judgment standards and has better performance. It provides a new idea for solving the problem of high cost of manual evaluation.

[0532] Finally, the above advantages directly correspond to the problems existing in the existing technology, such as data scarcity, poor interpretability, high cost of manual evaluation, etc. This application proposal effectively solves these problems and achieves technical breakthroughs and improvements by adopting advanced technical means, such as multimodal information processing, large language models, and automated evaluation strategies.

[0533] Example

[0534] (I) Sample labeling example

[0535] Assume that the scenario of this embodiment is "answering a question during an interview". Our goal is to determine whether the interviewee has lied in his answer by analyzing video, audio and text information. The specific content of the input sample of this embodiment is as follows: Figure 3 shown in the table.

[0536] Step 1: Data preprocessing and feature extraction

[0537] First, obtain the sample data of people who are not lying, the video data to be tested, and the related text data. Assume that the video sample to be tested is a response during an interview, and the video length is less than 2 minutes. In the data preprocessing stage, first extract the frames of the video. Assume that the frame rate is 30 frames per second, so the video has a maximum of 3600 frames.

[0538] 1.1 Visual Feature Extraction

[0539] Through the visual encoder, we use VIVIT as a visual feature encoder to extract visual features from each input frame. In order to reduce computational complexity and improve processing efficiency, we first collect frame samples for 3600 frames according to a time unit (such as collecting 2 frames per second) to reduce the number of samples, so that the features corresponding to the collected frames can be clustered in subsequent operations.

[0540] 1.2 Audio Feature Extraction

[0541] At the same time, the audio part of the interview video is extracted. We use the audio feature extraction network Hubert to extract the feature representation of the audio and obtain an audio feature vector sequence. Among them, in Hubert, the audio duration is proportional to the sequence length. For an audio with an average length of 25s, its sequence length is 1250.

[0542] 1.3 Text Feature Extraction

[0543] The personality data, question asking and answering are input into the feature encoder of a large language model (such as LLAMA2) together with the corresponding prompt template to obtain a text feature vector dedicated to natural language processing for subsequent operations.

[0544] Step 2: Modal feature post-processing

[0545] 2.1 Obtaining visual features of key frames

[0546] Use VIVIT to encode the features of the video frames, use K-Means to perform preliminary clustering on these video frame features, and select the sample closest to the centroid from each clustered category as the key frame of the current category to reduce the large amount of redundant information in the video for subsequent analysis. The corresponding feature f i * Input to the descriptor model (BLIP) to obtain the corresponding description information Get description information After that, it is input into the large language model (LLAMA2) and the relevance analysis is performed according to the preset questions in the following picture. Finally, the relevance score is annotated on the frame sample for subsequent more fine-grained analysis and interpretation. So far, the adaptive breadth search is completed. Among them, the score 1 represents that the cluster keyframe has the lowest relevance to the question, 2 represents a certain relevance to the question, and 3 represents a high relevance to the question. If the relevance of the current cluster keyframes is less than 2, adjust the K-Means threshold parameter to re-cluster to obtain more keyframes for relevance score calculation.

[0547] After clustering the key frame samples to obtain the relevance score, the categories of the frame samples with a relevance score higher than 1 are clustered again to obtain a more fine-grained category division. The key frame category with a score of 2 is clustered once, and the key frame category with a score of 3 is clustered twice. Each clustering generates two additional key frames. Finally, all the key frames obtained are stored in chronological order for subsequent interpretable reasoning.

[0548] 2.2 Decoupling of Audio Acoustic and Semantic Features

[0549] After extracting features from the audio data, in order to more accurately capture the acoustic features of the interviewee's narration, two linear transformation layers that do not change the feature dimension are used to extract acoustic features and semantic features respectively. In order to effectively separate the acoustic features and content features, the mutual information and orthogonal loss are minimized to ensure the independence of the two. Among them, the mutual information estimation adopts the vCLUB method. In addition, in order to ensure that the separated acoustic features and content features respectively have information related to lies and corresponding acoustic content information. The present invention adopts a contrastive learning method to achieve alignment between acoustic information and lie labels. For content features, the present invention adopts audio transcription text technology and CTC alignment to achieve alignment between content features and corresponding text labels. Finally, the separated acoustic features are retained for subsequent operations.

[0550] For example, if the interviewee speaks faster and the intonation of his voice changes significantly in the audio when answering, the decoupled feature extractor can better capture these changes without having to pay attention to the interference caused by the semantic information in the audio.

[0551] Step 3: Generate explainable lie detection results

[0552] During the detection process, the system not only gives a judgment result of "lie" or "truth", but also generates explainable clues. Specifically, the system generates the following important clue descriptions through comparative analysis:

[0553] Visual cues: After obtaining the fine-grained key frames of the test sample after multiple clustering of correlation scores, input them into the video frame encoder VIVIT to obtain features, and then input the features into Q-Former to compress and align the visual information using the attention query mechanism. Finally, the processed test video sample features and the features of the benchmark video that are not lying are spliced ​​and input into the visual description model Video-LLAMA2 to generate prompts according to the corresponding visual clue descriptions to obtain visually interpretable description information D v .

[0554] Audio clues: After obtaining the acoustic features after correctly separating the semantic information, they are input into Q-former, and the acoustic features are compressed and aligned using the attention query mechanism. Then, the Q-former output features are concatenated with the acoustic features of the baseline audio that is not lying to obtain the acoustic features Audio. final , the acoustic clue description generates the corresponding prompt and the final audio final Together, they are used as input to the large language model SECap dedicated to acoustic clues to explain descriptions, generating the final acoustically interpretable description D a

[0555] Text clues: Fill the personality information and text information into the text clues and their corresponding prompts, input them into the LLAMA2 tokenizer, and then obtain the corresponding feature vectors, which are input into the large language model LLAMA2 to obtain the descriptive information D for reasoning about the text information. t .

[0556] Final result: After obtaining D t , D a and D v After obtaining the text of the clue description information, it is input into the LLAMA2 tokenizer together with the Prompt dedicated to integrating multimodal lie clues to obtain the feature vector, and then input into the LLAMA2 model to obtain the final lie inference description and final discrimination result of the video to be tested.

[0557] Step 4: RLHF-based model optimization and result evaluation

[0558] The RLHF strategy is introduced to replace manual evaluation of the quality of the final generated interpretable description and guide the model to optimize. Specifically, the present invention proposes three scoring modules, Clarity Encoder, Accuracy Encoder and Alignment Encoder, which are used to simulate the three aspects that the human evaluation process focuses on, namely clue clarity, description accuracy and label correctness. After each module obtains the score of the generated description, it outputs the final score through weighted average. These scores are used to locate the source of the problem later: optimize the Prompt design when the clarity is low; improve the modal feature extraction (such as Q-former) when the accuracy is insufficient; and adjust the logic of generation and label matching when the alignment is poor. By combining feedback optimization with modular diagnosis, the quality and efficiency of the labeling process are comprehensively improved.

[0559] Summary of Examples

[0560] This embodiment successfully achieves reliable and interpretable description generation for lie discrimination tasks in interview scenarios through multimodal representation learning, interpretable lie reasoning, multimodal fusion, and RLHF technologies. Ultimately, the present invention can not only comprehensively judge whether the subject is lying based on the facial expressions, audio features, and text content in the video, but also generate interpretable deception descriptions, providing reliable decision support for related applications.

Claims

1. A clue annotation method for contactless lie detection, characterized in that: The following steps are involved: Step 1: Cluster the video frames to be tested, extract key frames, encode the key frames using the pre-trained model, compress and align the encoded visual features using Q-Former, perform the same operation on the video clips that are not lying to obtain the reference video feature vector, and input the video feature vector to be tested and the reference video feature vector into Video-LLAMA2 to generate visual clue description text; Step 2: feature encode the audio to be tested, separate the acoustic features and content features in the audio to be tested by optimizing the objective function, compress and align the acoustic features using Q-Former, perform the same operation on the audio clips that are not lying to obtain the reference audio feature vector, and input the audio feature vector to be tested and the reference audio feature vector into SECap to generate the audio clue description text; Step 3: Analyze the test question information and personality traits, and input the test question information, personality trait analysis results and the tester's answer into the LLAMA2 model to generate a text clue description; Step 4: Generate the final clue description using the large model based on the visual clue description, the audio clue description, and the text clue description; Step 5: Build a quality assessment model to perform quality assessment on the generated clue descriptions.

2. The clue annotation method for contactless lie detection according to claim 1, characterized in that: The steps in step 1 include: Step 1.1, clustering the video frames to obtain clustering results; Step 1.2, extract key frames from the clustering results to obtain a key frame list; Step 1.3, performing visual feature encoding processing on the key frame list to obtain a visual feature sequence; Step 1.4, perform Q-Former-based visual information compression and alignment processing on the visual feature sequence to obtain a compressed visual information representation; Step 1.5: Extract key features from the interview clips that never lie, and concatenate them with the key features of the video to be tested to obtain the input for visual description generation; Step 1.6: Use the Video-LLAMA2 model to perform visual prompt reasoning and generate the final visual clue description text.

3. The clue annotation method for contactless lie detection according to claim 1, characterized in that: The steps in step 2 include: Step 2.1, feature encode the audio input to obtain an audio feature sequence; Step 2.2: Use two independent linear layers to extract acoustic representation and content representation respectively, and obtain acoustic features and semantic content representation; Step 2.3, minimize the mutual information between acoustic representation and content representation by optimizing the objective, impose orthogonal constraints, and optimize acoustic features and content representation by contrastive learning loss and CTC loss; Step 2.4: Use Q-former to compress and align the acoustic features to obtain a compressed acoustic information representation; Step 2.5, randomly select an acoustic feature from the interview clip that never lies, and concatenate it with the acoustic feature to be tested to obtain the input for acoustic description generation; Step 2.6: Take the acoustic prompt and the final acoustic description generation input as the input of SECap to generate the final interpretable description D a , and obtain the final audio clue description text.

4. The clue annotation method for contactless lie detection according to claim 1, characterized in that: The steps in step 3 include: Step 3.1, analyze the test questions to obtain the prior information that may be helpful for lie identification; Step 3.2, analyze the personality characteristics of the subjects to obtain descriptions of lie clues related to personality; Step 3.3: Take the analysis results of the test questions, the personality characteristics analysis results, the tester's answer text and the corresponding lie label as input, and generate the lie clue description of the text content according to the prompt generated by the final text clue description to obtain the interpretable description D t .

5. The clue annotation method for contactless lie detection according to claim 1, characterized in that: The steps in step 4 include: Step 4.1, construct a Prompt template of the final clue description of text, voice, and vision, and fill the interpretable clue description obtained in steps 1, 2, and 3 into the Prompt template as the input for generating the final clue description; Step 4.2: Use the large model to integrate multimodal clues and conduct correlation analysis to generate the final lie clue description text.

6. The clue annotation method for contactless lie detection according to claim 1, characterized in that: The steps in step 5 include: Step 5.1: Build a dataset for training the automatic quality assessment model, including raw data features, generated descriptions and their corresponding manual scores; Step 5.2, establishing a scoring model, including a clue clarity scoring module, an accuracy scoring module, and a label alignment scoring module; Step 5.3: Use the scoring model to automatically score the generated clue description to obtain a reward score; Step 5.4: Evaluate the quality of the generated clue description based on the reward score.

7. The clue annotation method for contactless lie detection according to claim 1, characterized in that: The steps in step 1 include: Step 1.1, clustering the video frames to obtain clustering results; Step 1.1.1: For video frames V = F1, F2, ..., F n Use the pre-trained visual encoder VIT to extract the visual features of each frame and obtain the video frame feature vector set f v1 , f v2 , …, f vn ; f vi =VIT(F vi ),f vi ∈R d ,i∈{1,2,...,n} Where VIT(*) represents the visual encoder VIT, f vi Indicates frame F i The visual feature vector of Step 1.1.2: Use K-Means method to analyze the video frame feature vector set f v1 , f v2 , …, f vn Perform clustering, generate k clusters, and obtain cluster centers c1, c2, ..., c k ; Step 1.2, extract key frames from the clustering results to obtain a key frame list; Step 1.2.

1. For each cluster C i , select the closest to the centroid c i The frame is used as the key frame Step 1.2.2: Keyframe Input descriptor BLIP2 to generate the corresponding text description t i ; Step 1.2.3: Cluster description t1, t2, ..., t k The query Q is input into the large language model LLAMA2, and the relevance score r between the query and each cluster is calculated. i ; Step 1.2.4: Score r based on relevance i , set the number threshold of highly correlated clusters to rele_num_thresh. If the condition is met, stop the iteration and add the highly correlated cluster C to the list Tree; otherwise, update the number of clusters k: k = 2k; Step 1.3, performing visual feature encoding processing on the key frame list to obtain a visual feature sequence; Step 1.3.1, use VIVIT to encode the video frames in the key frame list; Step 1.3.2: Each image frame F T Divide into fixed blocks and calculate the number of blocks Np; Step 1.3.3, F T After flattening all blocks, we get P T , use a linear layer W to map each block to the feature space, and get P T The embedding representation E T ; Step 1.3.4: For each embedding representation E T Add spatial location information to get E′ T ; Step 1.3.5: Embedding representation E′ for each frame T Add the corresponding time position code Ptemporal, and we get Step 1.3.6: Send the video frame embedding to the Transformer encoding module for deep feature encoding to obtain the final feature embedding Step 1.4, perform Q-Former-based visual information compression and alignment processing on the visual feature sequence to obtain a compressed visual information representation; Step 1.4.

1. Initialize the learnable query vector: Definition N q A learnable query vector Q q ={q1, q2, ..., q Nq },in Among them, N q is the number of query vectors, which is usually much smaller than the visual sequence encoding length, d q is the dimension of the query vector, which is similar to the target feature space d V Stay consistent; Step 1.4.2, Cross-Attention Mechanism: Calculate the query vector Q g With visual feature sequence Attention score: Among them, W q , W k ∈R dq×dq is the linear projection matrix of query and key, V∈R Nq×L is the attention distribution matrix; Step 1.4.2, calculate the context representation Z: Among them, W v ∈R dv×dq is the linear projection matrix of the value vector, and Z represents the visual feature sequence By N q A compressed representation of the query vectors summarized; Step 1.4.3: Visual information compression: The query vector Q is fused via residual connections and layer normalization. q The features of the self and the context Z are used to obtain the compressed visual information representation Q v_out : Q v_out =LayerNorm(Z+Q q ),Q v_out ∈R Nq×dq Step 1.4.4, feature space alignment: Q v_out Project it into the text feature space to get the aligned feature representation Q′ out , that is, we get the visual information representation Q′ v_out : Q′ v_out =Q out W+b,Q′ v_out ∈R Nq×dq Where W is the learnable projection matrix and b is the bias term; Step 1.5: Extract key features from the interview clips that never lie, and concatenate them with the key features of the video to be tested to obtain the input for visual description generation; Step 1.5.1: Extract key features of the segments that do not lie: From the pre-labeled interview clips, the same visual feature extraction method (such as VIVIT) is used to extract key features to obtain the visual feature representation Q′ of the non-lying clips v_out_base ; Step 1.5.2, feature stitching: The compressed visual information obtained in step 1.4 is represented as Q′ v_out and the visual feature representation Q′ of the true lying segment v_out_base Concatenate and obtain the input features for visual description generation: Image final =Concat(Q′ v_out_base , Q′ v_out )Step 1.6, use the Video-LLAMA2 model to perform visual prompt reasoning and generate the final visual clue description text; Step 1.6.1, build the visual prompt template: According to the input visual feature sequence, a Prompt template for generating visual lie clue description is set; The content of the visual prompt template is: based on the comprehensive analysis of facial features, hand movements, and posture performance, generate an interpretable description of whether the frame under test contains lies, combine with the reference frame to provide comparative reasoning, explain the key visual differences and their supporting label basis, and align with the real label results; Step 1.6.2, input information preparation: The input information includes: the input feature Image obtained in step 1.5 final , where Image final It is composed of two parts of video sequence splicing, namely the reference video frame sequence features and the test video frame sequence features; The lie labels corresponding to the features of the video frame sequence to be tested; Step 1.6.3, fill in the visual prompt template: Fill the input information into the visual prompt template for subsequent visual prompt reasoning; Step 1.6.4, visual prompt reasoning: The filled visual prompt template is input into the Video-LLAMA2 model for visual prompt reasoning. The model will combine the guidance of the visual prompt template and the image representation to generate visual clue description text related to lies. The model inference formula is expressed as: D v =Video-LLAMA2(Prompt,Image final ) Where Video-LLAMA2(*) represents the Video-LLAMA2 model, and Prompt represents the visual Prompt template.

8. The clue annotation method for contactless lie detection according to claim 1, characterized in that: The steps in step 2 include: Step 2.1, feature encode the audio input to obtain an audio feature sequence; Step 2.1.1, audio input x uses the pre-trained audio feature encoder hubert (·) to extract features from the pre-processed audio and obtain the audio feature sequence F: Among them, T represents the length of the encoded feature sequence, d a Represents the feature dimension of each sequence; Step 2.2: Use two independent linear layers to extract acoustic representation and content representation respectively, and obtain acoustic features and semantic content representation; Step 2.2.

1. Acoustic representation extraction: Apply a linear layer E a To the audio feature sequence F, we get the acoustic representation F a : Step 2.2.

2. Content representation extraction: Apply another linear layer E c To the audio feature sequence F, we get the content representation F c : Step 2.3, minimize the mutual information between acoustic representation and content representation by optimizing the objective, impose orthogonal constraints, and optimize acoustic features and content representation by contrastive learning loss and CTC loss; Step 2.3.1, mutual information minimization: Step 2.3.1.1: Use the mutual information minimization strategy to reduce the acoustic representation F a and content representation F c The correlation between them, mutual information I(F a ; F c ) measures the correlation between two variables and is calculated as: p(F a , F c ) indicates F a and content representation F c The joint probability distribution of a ) and p(F c ) are F a and F c The marginal probability distribution of Indicates the degree of difference between the joint distribution and the independent distribution; Step 2.3.1.2: Use the vCLUB method to estimate the upper bound of the mutual information. The calculation formula of the vCLUB method is: q(F c |F a ) is a conditional probability distribution used to approximate the joint probability distribution p(F a , F c ), by minimizing L VCLUB , reduce F a and F c The mutual information between them can decouple acoustic representation and content representation. Step 2.3.2: Acoustic Characterization F a and content representation F c The learning process imposes orthogonal constraints to reduce F a and F c The linear correlation between them improves their independence in the representation space, and its orthogonal loss is defined as: For calculating the matrix sum of squares, Indicates F a The transpose of the i-th row of Indicates F c The i-th row of Step 2.3.3: Use the contrastive learning strategy to move the acoustic features of lie samples close to each other in the feature space, while the real samples are far away from the lie samples and close to each other. The loss function of contrastive learning is as follows: Where: S(a i , p i ) indicates F a Sample a i and its positive sample p i The similarity of S(a i , n i ) indicates F a Sample a i and its negative sample n j , τ represents the contrastive learning temperature hyperparameter, N represents the batch size, and K represents the number of negative samples for each sample; Step 2.3.4: Introduce the speech transcription task c Mapped to the corresponding transcribed text representation, F is optimized by minimizing the loss of transcription prediction c , the specific formula is as follows: P(Y|F c )=Softmax(WF c ),P(Y|F c )∈R T×V Among them, P(Y|F c ) represents the predicted probability distribution of the transcribed characters at each time step t, Softmax(·) is an activation function used to convert any real vector into a probability distribution, V represents the vocabulary size of the transcribed text including the special character blank, and the target transcribed sequence is Y true , corresponding to a label sequence of length U: Y true =y1,y2,...,y U ,y u ∈V Among them, U≤T, that is, the length of the label sequence is less than or equal to the length of the frame sequence, and y U represents a sequence of transcribed text; Step 2.3.5: Use the CTC model to train F c and Y true To align, the alignment path is: π=π1,π2,...,π T ,p t ∈V∪blank The alignment path that satisfies π can be mapped to Y by removing repeated characters and blanks. true , the CTC loss is calculated by summing over all possible alignment paths, as follows: By maximizing L content To optimize the content representation F c , align(·) represents the alignment operation, P(·) represents the probability that the tth element of the generated sequence corresponds to the true sequence given a path π and a time step t, and t represents the time step; Step 2.3.6: The final loss function is as follows: L=αL MI +βL CL +γL orth -δL content Among them, α, β, γ, δ are weight hyperparameters used to balance the contribution of each part of the loss to the overall optimization, L content is the CTC loss of speech transcription; Step 2.4: Use Q-former to compress and align the acoustic features to obtain a compressed acoustic information representation; Step 2.4.1: Define N q Learnable query vectors: Where N q is the number of query vectors, which is usually much smaller than L(N q =512), d q is the dimension of the query vector, which is the same as the target feature dimension d a Stay consistent; Step 2.4.2: Through the cross-attention mechanism, query vector Q q From the acoustic feature sequence F u Extract global information from, and the specific calculation formula is as follows: Calculate the query Q q With the acoustic feature sequence F a Attention score: Among them, W q , W k is the linear projection matrix of query and key, A∈R Nq×L is the attention distribution matrix, which represents the correlation between each query vector and each time step in the acoustic feature sequence; Step 2.4.3: Compute the context representation Z: in, is the linear projection matrix of the value vector, Z represents the acoustic feature sequence F a By N q A compressed representation of the query vectors summarized; Step 2.4.4: Set the query Q q The features of Z are further fused with the extracted context Z through residual connection and layer normalization: Among them, Q a_out is the compressed acoustic feature sequence, with a length of N q , dimension and acoustic feature dimension d a same; Step 2.4.5: Pass Q through a linear layer a_out Projected into the text feature space, the specific formula is as follows: in, is the learnable projection matrix, b is the bias term; Step 2.5, randomly select an acoustic feature from the interview clip that never lies, and concatenate it with the acoustic feature to be tested to obtain the input for acoustic description generation; Step 2.5.1: Randomly select an acoustic feature from the interview segment that never lied, and use the same method as steps 2.1-2.4 to extract the acoustic feature, denoted as Q′ a_out_base ; Step 2.5.2: Represent the compressed acoustic information obtained in step 2.4 as Q′ a_out The acoustic feature representation Q′ of the non-lying segment a_out_base Concatenate and obtain the input features for acoustic description generation: Listen final =Concat(Q′ a_out_base ,Q′ a_out ) Step 2.6: Take the acoustic prompt and the final acoustic description generation input as the input of SECap to generate the final interpretable description D a , get the final audio clue description text; Step 2.6.1, build the acoustic prompt template: According to the input audio feature sequence, a Prompt template is designed for generating audio lie clue description. The content is: Based on the comprehensive analysis of acoustic features such as pitch, energy, speaking speed, hesitation, and silence, an interpretable description of whether the test segment contains lies is generated, combined with the benchmark segment to provide clear comparison and reasoning basis, and aligned with the real lie label; Step 2.6.2, input information preparation: The input information includes: The input feature Audio obtained in step 2.5 final , where Audio final It is composed of two parts of audio sequences, namely the reference audio sequence features and the audio sequence features to be tested; The lie label of the audio segment to be tested; Step 2.6.3: Integrate the acoustic prompt with the final acoustic description generation input: Combine the filled acoustic prompt template with the final audio representation Audio final Integrate them to ensure that the formats of the two match, so that the model can process both prompt information and audio features at the same time; Step 2.6.4: SECap model reasoning to generate the final interpretable description D a , the generated interpretable description D a Arranged into a complete audio clue description text: The integrated acoustic prompt and audio representation are used as input to the SECap model; Use the SECap model to reason about the input acoustic prompt and audio representation; The model will combine the guidance of acoustic prompts and audio representation to generate audio clue description text related to lies. The model reasoning formula is expressed as: D a =SECap(prompt,Audio final ) Where SECap(*) indicates the SECap model, and Prompt indicates the acoustic prompt template; The generated interpretable description D a Organize into complete audio clue description text.

9. The clue annotation method for contactless lie detection according to claim 1, characterized in that: The steps in step 3 include: Step 3.1, analyze the test questions to obtain the prior information that may be helpful for lie identification; Step 3.1.1: Set up the text prompt template for question analysis, including the following steps: Step 3.1.1.

1. Whether the questions are likely to induce the respondents to give brief or vague answers; Step 3.1.1.

1. Whether the questions are likely to induce the respondents to give brief or vague answers; Step 3.1.1.2: Is the question likely to make the respondent use common sense answers to cover up a lie? Step 3.1.1.3: If the respondent lies on the question, is it possible to verify the lie through follow-up questions or background knowledge? Step 3.1.1.4: Will the question significantly increase the cognitive load of the subject? Step 3.1.2, input the text prompt template in 3.1.1 together with the question into the LLAMA2 model to obtain the lie clue analysis results about the question; Step 3.2, analyze the personality characteristics of the subjects to obtain descriptions of lie clues related to personality; Step 3.2.1, set the text prompt template of "Personality Traits", including the following steps: Step 3.2.1.1, combining the five dimensions of the Big Five personality, generate a summary description for the input Big Five personality test scale vector; Step 3.2.2, obtain the Big Five personality test scale vector of the test subject; Step 3.2.3, fill the subject's personality vector into the text prompt template of "Personality Traits" and input it into the LLAMA2 model for description generation, and obtain the lie clue analysis results about the subject's personality traits; Step 3.3: Final text clue description generation: Step 3.3.1, set the text prompt template generated by the final text clue description; Step 3.3.1.1, prepare input information, including: analysis results of test questions, personality characteristics analysis results, test taker's answer text and corresponding lie label; Step 3.3.1.2: Prepare the analysis content requirements text prompt, including: The compatibility or potential contradiction between the test taker's answer and the characteristics of the question; The degree to which the test taker's answers match the Big Five personality traits and any abnormalities; Language clues in the test-taker’s answers that may indicate a lie, including excessive embellishment, abnormal emotions, and logical inconsistency; Step 3.3.1.3: Prepare the text prompt required for output, the content is: Integrate the input information into clues and generate the final lie clue description according to the analysis requirements. The final description must conform to the given lie label, that is, generate a corresponding reasoning description for the "true" or "false" label; Step 3.3.2, fill the analysis results obtained in steps 3.1 and 3.2 together with the test subject's answer text and the real label into the text prompt template of step 3.3.1 and then input it into the LLAMA2 model for description generation to obtain the final text clue description; Use the LLAMA2 model to perform text prompt reasoning and generate text clues related to lies to describe text D t , the model inference formula is expressed as: D t =LLAMA2(Prompt[Input]) Where LLAMA2(*) represents the LLAMA2 model, Prompt represents the text prompt template, and Input represents the prompt of "test question" filled with corresponding content, the prompt of "personality characteristics" or the prompt instruction generated by the final text clue description.

Images