Remote interview anti-fraud system and method based on multi-mode depth forgery detection

Abstract

The invention discloses a remote interview anti-fraud system and method based on multi-modal deep counterfeiting detection, and relates to the technical field of data analysis. According to the method, face hash and voiceprint hash are generated by collecting identity document information and biological characteristics of job seekers, and after zero-knowledge identity matching is achieved, hash triples are stored in a block chain; in the interview process, three layers of confidence coefficients are calculated by analyzing physiological feature pairs, deep forging abnormal indexes, voiceprint synthesis feature values and lip sound maximum offset, and weights are adjusted based on historical attack rates to generate fusion scores; optimizing an RNN behavior analysis network, outputting a real-time abnormal behavior index, obtaining a final risk score in combination with the fusion score, and triggering a three-level response mechanism; according to the method, the fraud interception rate is increased, the misjudgment rate is reduced, and the problems of identity false use, deep counterfeiting and behavior fraud are effectively solved on the premise of ensuring privacy security.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data analysis, and in particular, relates to a remote interview anti-fraud system and method based on multi-modal deep fake detection. BACKGROUND

[0002] With the popularization of remote recruitment, the abuse of AI deep fake technology (such as Deepfake and voice cloning) has led to a surge in "virtual job seeker" fraud incidents. In existing solutions, the centralized storage of raw biological data can easily lead to privacy leaks; the live detection, fake analysis, and lip sync modules operate independently and lack collaborative decision-making; manual review is inefficient, while automated interception can easily harm real users; and the industry still needs a fraud prevention system that integrates zero-trust architecture, dynamic multi-modal analysis, and adaptive decision-making.

[0003] The prior art has problems of identity verification vulnerabilities, insufficient deep fake defense, and rigid risk decision-making. SUMMARY

[0004] (I) Technical problems solved To solve the problems in the related art, the present application provides a remote interview anti-fraud method based on multi-modal deep fake detection to overcome the above technical problems existing in the prior art.

[0005] (II) Technical solutions To solve the above technical problems, the present application is implemented by the following technical solutions: S1, collect the identity information and biological characteristics of the job seeker; hash the identity information and biological characteristics of the job seeker and perform identity matching, and upload the matching result to the blockchain for storage; S2, use multi-modal dynamic detection combined with the data stored in the blockchain to perform live physiological signal detection, deep fake spatiotemporal analysis, voiceprint frequency domain diagnosis, and cross-modal synchronization verification on the job seeker, and obtain a pair of physiological characteristics, an anomaly index, a synthetic feature value, and a maximum offset; S3, calculate the live confidence, fake detection confidence, and cross-modal confidence based on the pair of physiological characteristics, anomaly index, synthetic feature value, and maximum offset; Dynamically adjust the weights of the live confidence, fake detection confidence, and cross-modal confidence to obtain a fusion score; S4, construct an RNN neural network, optimize the RNN neural network using historical remote interview behavior feature data combined with an optimization algorithm, and obtain an RNN behavior analysis network; Use the behavior analysis network to obtain the real-time anomaly behavior index of the job seeker; S5, obtaining a final risk score according to the fusion score combined with the real-time abnormal behavior index; and executing a hierarchical response scheme according to the final risk score; The application takes zero-knowledge verification, multi-modal collaborative sensing and dynamic decision-making as cores, solves technical problems of identity impersonation, deep forgery and behavior fraud in remote interviews, effectively solves the problems of identity impersonation, deep forgery and behavior fraud under the premise of ensuring privacy security, and provides an end-to-end trusted solution for recruitment security.

[0006] Preferably, the S1 comprises the following steps: S11, collecting a high-definition image of an identity certificate of a job seeker, extracting a name and an ID number through OCR to obtain job seeker certificate information, and hashing the job seeker certificate information to obtain job seeker information hash; S12, collecting photoplethysmogram signal data, 3D structured light face point cloud data and voiceprint sample data of the job seeker through an encryption SDK to obtain job seeker biological characteristics; S13, performing anti-collision fuzzy hash processing on the job seeker biological characteristics to obtain a job seeker biological characteristic hash pair; the biological characteristic hash pair comprises a face hash and a voiceprint hash, and the anti-collision fuzzy hash processing comprises the face hash and the voiceprint hash; S14, using zkSNARKs to prove that the job seeker biological characteristic hash matches the job seeker certificate information hash and that the original biological characteristics are not leaked; S15, after successful matching, constructing a job seeker biological characteristic hash pair and a job seeker certificate information hash into a triple to obtain a job seeker information triple, and writing the job seeker information triple into a Chia blockchain; the Chia blockchain comprises a BLS signature + space-time proof mechanism; The application extracts certificate information through OCR and hashes it, collects photoplethysmogram, 3D face point cloud and voiceprint sample through an encryption SDK, generates an anti-collision face hash by fusing curvature gradient and physiological band energy, and forms a biological characteristic hash pair in combination with a voiceprint mutation feature hash; the application verifies the matching of the certificate and the biological hash by using zk-SNARKs zero-knowledge proof, and finally writes the hash triple into a Chia blockchain; the original biological data is not stored throughout the process, and the hashing + zero-knowledge proof completely eliminates the risk of leakage; the 3D point cloud curvature gradient and the living pulse energy are fused, effectively resisting photo / mask attacks; the Chia blockchain space-time proof ensures that the hash triple is not tamperable, and provides an auditable traceability chain.

[0007] Preferably, the S2 comprises the following steps: S21, extract the signal related to physiological activity by analyzing the dual-band video stream of the face of the job seeker captured by the camera, obtain the living physiological signal; obtain the physiological feature pair by combining the living physiological signal through the living physiological detection algorithm; the physiological feature pair includes the signal-to-noise ratio and the main pulse frequency; S22, collect the continuous frames of the job seeker interview video to obtain a video frame sequence; the video frame sequence includes all continuous frames of the real-time interview video of the job seeker; extract the abnormal features of each frame in the video frame sequence to obtain a video frame abnormal feature sequence; the video frame abnormal feature sequence includes the abnormal features of each frame in the video frame sequence; obtain the abnormal index by combining the video frame abnormal feature sequence through the deep fake abnormal index algorithm; S23, extract the speech phase spectrum of the sound signal in the interview process of the job seeker, calculate the synthesized speech feature to obtain a synthesized feature value; S24, verify the lip sound synchronization of the job seeker by using the DTW algorithm to obtain the maximum offset; The application extracts the subcutaneous blood pulsation signal through the dual-band video stream, calculates the signal-to-noise ratio and the main pulse frequency to verify the physiological living body; analyzes the micro-expression, muscle movement and other abnormal features of the video frame sequence, calculates the abnormal index based on the real data set statistics, extracts the second-order mixed partial derivative of the speech phase spectrum to quantify the synthesized feature value; aligns the lip shape and the sound event stamp by using the DTW algorithm, and outputs the maximum offset; the dual-band physiological signal accurately locks the human blood pulsation and resists the 3D mask attack; the abnormal index reveals the space-time violation of the Deepfake video; the second-order partial derivative of the phase spectrum captures the frequency domain flaws of the synthesized speech; and the maximum offset quantifies the asynchronous flaws of the AI lip sound forgery.

[0008] Preferably, the S3 comprises the following steps: S31, according to the physiological feature pair, the abnormal index, the synthesized feature value and the maximum offset, calculate the living body confidence, the forgery detection confidence and the cross-modal confidence; S32, collect the historical attack rate; combine the living body confidence, the forgery detection confidence, the cross-modal confidence with the historical attack rate, and obtain the living body confidence weight, the forgery detection confidence weight and the cross-modal confidence weight through the dynamic weight adjustment algorithm; vectorize the living body confidence, the forgery detection confidence and the cross-modal confidence to obtain a confidence vector; vectorize the living body confidence weight, the forgery detection confidence weight and the cross-modal confidence weight to obtain a weight vector; calculate the fusion score according to the confidence vector combined with the weight vector; The application realizes intelligent risk decision-making through a dynamic weight fusion mechanism; physiological living bodies, video forgery, voiceprint synthesis and lip sound synchronization are quantified into confidence degrees; the confidence degree weights are dynamically adjusted based on historical attack rates to construct a self-adaptive fusion scoring model; the static weight limitation is broken, the misjudgment rate is reduced in the new deep forgery attack scene, the real-time defense response speed is improved, and the ability of the system to resist evolving attacks is significantly enhanced.

[0009] Preferably, the S31 comprises the following steps: S311, setting a parameter for controlling the steepness of a logic function and a threshold parameter of the logic function; and calculating a living body confidence degree according to the parameter for controlling the steepness of the logic function and the threshold parameter of the logic function in combination with physiological characteristics; S312, setting a weight parameter of video abnormality and a weight parameter of voice abnormality; and calculating a forgery detection confidence degree according to the abnormality index, the synthetic feature value in combination with the weight parameter of video abnormality and the weight parameter of voice abnormality; S313, calculating a cross-modal confidence degree according to the maximum offset; The application realizes dynamic risk determination through a three-level confidence quantification system; the physiological feature parameterization logic function is based on living body detection to improve the living body verification accuracy; the video abnormality index and the voiceprint synthesis feature value are fused to enhance the forgery recognition robustness through weight distribution; the lip sound offset is directly mapped to the cross-modal consistency; a multi-dimensional adjustable threshold mechanism is established to reduce the false alarm rate while improving the attack detection rate, and the adaptive ability of the anti-fraud system is significantly improved.

[0010] Preferably, the S4 comprises the following steps: S41, collecting behavior data in historical job seeker remote interviews and corresponding behavior abnormality indexes to obtain historical remote interview behavior data; and extracting features of the historical remote interview behavior data to obtain historical remote interview behavior feature data; S42, constructing an RNN neural network, setting an initial weight value of the RNN neural network; and setting a prediction accuracy of the RNN neural network and a prediction accuracy threshold; training the RNN neural network using the historical remote interview behavior feature data, in the training process, finding an optimal weight value of the RNN neural network by using an ant algorithm in combination with a training accuracy and a training accuracy threshold to obtain an optimal solution; using the optimal solution as the weight value of the RNN neural network to obtain an RNN behavior analysis network; S43, collecting interview behavior data of real-time job seekers and extracting features to obtain real-time job seeker behavior feature data; inputting the real-time job seeker behavior feature data into the RNN behavior analysis network to obtain a real-time abnormal behavior index; The application realizes behavior abnormality dynamic monitoring by optimizing an RNN model; historical behavior data is used to train a network, an ant colony algorithm is used to optimize weights to avoid local optimization; real-time analysis is used to output an abnormality index of the behavior of a job seeker; the accuracy of abnormality detection is improved, the false positive rate is reduced, the model training efficiency is improved, and new deep fake accompanying behaviors are effectively identified.

[0011] Preferably, the historical remote interview behavior data comprises behavior data of abnormal job seekers and non-abnormal job seekers.

[0012] Preferably, the S5 comprises the following steps: S51, a hierarchical response scheme is set; the hierarchical response scheme is divided into a first-level execution scheme, a second-level execution scheme and a third-level execution scheme according to a final risk score; S52, a final risk score is calculated according to the fusion score combined with a real-time abnormality index; S53, a hierarchical response scheme is executed according to the final risk score; The application combines dynamic fusion scoring and behavior abnormality indexes to calculate a final risk value through a quantitative hierarchical response mechanism, and automatically triggers a differentiated disposal process based on a threshold value; the application realizes accurate interception of deep fake attacks while minimizing interference to real users, and significantly improves the reliability and execution efficiency of an anti-fraud system.

[0013] The remote interview anti-fraud system based on multi-modal deep fake detection is used to realize the remote interview anti-fraud method based on multi-modal deep fake detection, and comprises an identity authentication and blockchain storage module, a multi-modal dynamic detection module, a confidence fusion and dynamic scoring module, a behavior analysis network module and a risk decision and response module. The identity authentication and blockchain storage module is responsible for collecting identity information and biological characteristics of a job seeker, generating a pair of job seeker information hash and biological characteristic hash through an anti-collision fuzzy hash algorithm, realizing zero-knowledge proof of hash matching through zkSNARKs technology to ensure privacy security, and writing a hash triple (certificate information hash + face hash + voiceprint hash) into a Chia blockchain after successful matching to realize tamper-proof distributed storage. The multi-modal dynamic detection module extracts physiological signals through double-band video streams, calculates signal-to-noise ratio and main pulse frequency, verifies whether physiological activity conforms to human characteristics, analyzes abnormal characteristics of a video frame sequence, combines a real data set statistical quantity to calculate an abnormality index, extracts a second-order mixed partial derivative of a speech phase spectrum, calculates a synthetic characteristic value to identify synthetic speech, and uses a DTW algorithm to align timestamps of lip shapes and sounds to output a maximum offset. The confidence fusion and dynamic scoring module generates a living body confidence through a logical function combined with physiological characteristics; a fake detection confidence is obtained by weighted fusion of a video anomaly index and a speech synthesis characteristic value; a cross-modal confidence is generated according to an exponential decay function of a lip sound offset; through a dynamic weight adjustment algorithm, each confidence weight is updated in real time in combination with a historical attack rate, and finally a fusion score is output, and the calculation process combines a confidence vector, a weight vector and a minimum value adjustment mechanism; The behavior analysis network module uses historical interview behavior data, optimizes RNN weights through an ant algorithm, and outputs an optimal RNN behavior analysis network; interview behavior characteristics of a job seeker are extracted and input into the trained network to generate a real-time abnormal behavior index; The risk decision and response module integrates the fusion score and the behavior abnormality index, and finally calculates a risk score through weighted calculation; and performs a graded response according to a score threshold.

[0014] (Three) beneficial effects The present application has the following beneficial effects: The present application takes zero-knowledge verification, multi-modal collaborative sensing and dynamic decision as the core, and solves the technical problems of identity impersonation, deep forgery and behavior fraud in remote interviews, and provides an end-to-end trusted solution for recruitment security.

[0015] The present application constructs a zero-leakage identity certificate chain, realizes trusted storage and privacy protection, and breaks through the dual breakthroughs; through the anti-collision fuzzy hash technology, the curvature gradient of the 3D face point cloud and the physiological band energy of the photoelectric volume diagram are fused to generate an irreversible face hash, and the absolute value derivative maximum feature of the voiceprint is extracted to generate a voiceprint hash, which eliminates the risk of original biological data leakage; in combination with zk-SNARKs, the "hash matching zero-knowledge proof" is realized, and in the process of verifying that the certificate information and the biological characteristics belong to the same person, the sensitive information is completely hidden; the Chia block chain is used to store the hash triple, and an unalterable and auditable storage chain is constructed, which meets the highest privacy compliance standard.

[0016] The present application multi-modal collaborative detection deep forgery, accurately identifies cross-modal attacks; innovatively fuses four-dimensional anti-forgery verification; the living body detection extracts the subcutaneous blood pulsation signal through the double-band video stream, locks the physiological living body according to the signal-to-noise ratio and the main pulse frequency, and effectively resists 3D mask attacks; the deep forgery detection is based on real video data set statistics, and quantifies unnatural micro-expression and muscle movement through abnormal index; the voiceprint diagnosis utilizes the second-order mixed partial derivative of the speech phase spectrum to capture the frequency domain flaws of the synthesized speech; the cross-modal verification adopts the DTW algorithm to calculate the maximum offset of the lip shape and the sound event, and completely exposes the AI-generated false synchronization attack.

[0017] The dynamic weight fusion mechanism of the application realizes adaptive risk decision-making; a three-layer confidence model is constructed; the living confidence dynamically responds to the physiological signal quality through a logic function; the counterfeit confidence weightedly fuses video and voice abnormal indexes; the cross-modal confidence quantifies the lip synchronization tolerance with an exponential decay; a weight evolution algorithm driven by the historical attack rate is introduced, so that the system automatically increases the counterfeit detection weight in the high attack period; and finally, the final fusion score combined with the minimum value adjustment mechanism significantly reduces the false negative rate of complex attacks.

[0018] The hierarchical response closed loop of the application improves the risk control efficiency; based on the dynamic fusion score and the behavior index, the final risk value is calculated by weighting; a multi-level response mechanism is executed, which improves the fraud interception rate and the interview process efficiency, and reduces the misjudgment rate.

[0019] Of course, implementing any product of the application does not necessarily need to achieve all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the description of the embodiments, obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, the drawings obtained from these drawings can also be obtained without creative labor.

[0021] Figure 1 The flowchart of the remote interview anti-fraud method based on multi-modal deep fake detection of the application; Figure 2 The flowchart of multi-modal dynamic detection in the remote interview anti-fraud method based on multi-modal deep fake detection of the application; Figure 3 The module schematic diagram of the remote interview anti-fraud system based on multi-modal deep fake detection of the application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application, obviously, the described embodiments are only some embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0023] In the description of the application, it should be understood that the terms "opening", "upper", "lower", "top", "middle", "inner" and the like indicate the orientation or positional relationship, which are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the indicated components or elements must have a particular orientation, be constructed and operated in a particular orientation, therefore, it cannot be understood as a limitation of the application.

[0024] Embodiment one

[0025] Please refer to Figure 1 、 Figure 2 The application discloses a remote interview anti-fraud method based on multi-modal deep fake detection, comprising the following steps: S1, collecting the identity information and biological characteristics of the job seeker; the identity information and biological characteristics of the job seeker are hashed and then identity matching is performed, and after the matching is completed, the result is uploaded to a block chain for storage; The S1 comprises the following steps: S11, collecting a high-definition image of the job seeker's identity card, extracting the name and ID number through OCR to obtain the job seeker's certificate information; the job seeker's certificate information is hashed to obtain the job seeker's information hash; S12, collecting the job seeker's photoplethysmogram signal data, 3D structured light face point cloud data (≥30000 feature points) and voiceprint sample data (obtained by reading a random encrypted text) through an encryption SDK to obtain the job seeker's biological characteristics; S13, performing anti-collision fuzzy hash processing on the job seeker's biological characteristics to obtain a job seeker's biological characteristic hash pair; the biological characteristic hash pair comprises a face hash and a voiceprint hash, and the anti-collision fuzzy hash processing comprises the face hash and the voiceprint hash; The face hash formula and the voiceprint hash formula are as follows, ; Among them, H face represents the face hash, BLAKE 3() represents a hash function, delta represents a gradient operator, epsilon represents the curvature of the point cloud in the 3D structured light face point cloud data, delta ε (P 3D ) represents the curvature gradient calculated for the 3D structured light face point cloud, β { S rPPG} represents the complex frequency spectrum of the output photoplethysmogram signal data, df represents a frequency differential variable, represents the total energy sum of the photoplethysmogram signal data in the 0.8-2.0 Hz frequency band, which is used as a physiological living body feature; Among them, H voice represents the voiceprint hash, and Sphincs+ represents a hash function for voiceprint hash, V MFCC represents the voiceprint sample data, Indicates the maximum value of the absolute derivative over the entire time series (used to extract the most significant change features in the voiceprint); S14. Use zkSNARKs to verify that the job applicant’s biometric hash matches the job applicant’s ID hash without revealing the original biometric. S15. After a successful match, the job applicant's biometric hash and the job applicant's ID information hash are constructed into a triple to obtain the job applicant information triple. The job applicant information triple is written into the Chia blockchain, which includes a BLS signature + time-space proof mechanism. S2. Use multimodal dynamic detection combined with data stored in the blockchain to perform live physiological signal detection, deepfake spatiotemporal analysis, voiceprint frequency domain diagnosis, and cross-modal synchronous verification on job applicants to obtain physiological feature pairs, anomaly index, synthetic feature value, and maximum offset; The S2 comprises the following steps: S21. Analyze the dual-band video stream of the job applicant's face captured by the camera to extract signals related to physiological activities and obtain a living physiological signal. Combine the living physiological signal with a living physiological detection algorithm to obtain a physiological feature pair. The physiological feature pair includes a signal-to-noise ratio and a main pulse frequency, which are used to verify whether the blood pulsation frequency is within the human physiological range of 0.82 Hz. The formula of the living physiological detection algorithm is as follows: ; in, SNR represents the signal-to-noise ratio (i.e., the ratio of the signal fluctuation intensity caused by blood pulsation to the ambient noise intensity). I 530 、 I 660 Respectively represent the 530nm and 660nm band signals in the living body physiological signals, sigma represents the standard deviation, f p Indicates the main pulse frequency, Indicates the frequency value corresponding to the peak of the power spectrum; S22. Collect continuous frames of the job seeker interview video to obtain a video frame sequence ,in v i Indicates the first i frame, N Indicates the total number of frames in the video; the video frame sequence includes all consecutive frames of the job applicant's real-time interview video; Extracting abnormal features of each frame in the video frame sequence to obtain a video frame abnormal feature sequence; the video frame abnormal feature sequence includes abnormal features of each frame in the video frame sequence; the abnormal features include micro-expression duration, facial muscle movement amplitude, and head rotation acceleration; The anomaly index is obtained by combining the deep fake anomaly index algorithm with the abnormal feature sequence of the video frame; the deep fake anomaly index algorithm formula is as follows: ; in, T abn represents the abnormality index, N Represents the total number of video frames in the video frame sequence, lambda i Indicates the first i Abnormal feature values, represents the mean value of the i-th feature in the real human video dataset, Indicates the i The standard deviation of the features in the real human video dataset; S23, extracting the speech phase spectrum of the sound signal during the job applicant interview, calculating the synthetic speech features, and obtaining the synthetic feature value; the calculation formula for calculating the synthetic speech features is as follows: ; Among them, Pfake represents the synthetic feature value, f Indicates frequency, t Indicates time, etaf eta Indicates frequency f and time t The partial derivative, Psi The speech phase spectrum represented by represents the second-order mixed partial derivative of the speech phase spectrum, d represents differential; S24. Use the DTW algorithm to verify the lip synchronization of the job applicant and obtain the maximum offset; the calculation formula is as follows: ; in, E max Indicates the maximum offset, k Indicates the event sequence number of the job applicant during the interview process, t k lip Indicates in k The timestamp of the significant change in lip shape under each event, t k voice Indicates in k The timestamp of the significant sound change under each event; S3. Calculate liveness confidence, forgery detection confidence, and cross-modal confidence based on the physiological feature pair, abnormality index, synthetic feature value, and maximum offset; The weights of the living body confidence, the fake detection confidence and the cross-modal confidence are dynamically adjusted to obtain a fusion score. The S31 includes the following steps: The S31 includes the following steps: The S31 includes the following steps: The S311 sets a parameter for controlling the steepness of a logical function and a threshold parameter of the logical function, and calculates the living body confidence according to the parameter for controlling the steepness of the logical function and the threshold parameter of the logical function in combination with the physiological feature pair; the calculation formula is as follows, ; C live represents the living body confidence, omega represents the parameter for controlling the steepness of the logical function, chi represents the threshold parameter of the logical function; The S312 sets a weight parameter of a video anomaly and a weight parameter of a voice anomaly, and calculates the fake detection confidence according to the anomaly index and the synthetic feature value in combination with the weight parameter of the video anomaly and the weight parameter of the voice anomaly; the calculation formula is as follows, ; C C fake represents the fake detection confidence, xi 1 represents the weight parameter of the video anomaly, xi 2 represents the weight parameter of the voice anomaly; The S313 calculates the cross-modal confidence according to the maximum offset; the calculation formula is as follows, ; C C sync represents the cross-modal confidence, g represents the parameter for controlling the decay speed of the index; The S32 collects a historical attack rate; the historical attack rate represents the frequency or proportion of attacks (such as fake, living body fraud, etc.) detected by the system in the past period of time. It reflects the threat level of the current environment; the living body confidence weight, the fake detection confidence weight and the cross-modal confidence weight are obtained according to the living body confidence, the fake detection confidence, the cross-modal confidence and the historical attack rate, and through a dynamic weight adjustment algorithm; the formula of the dynamic weight adjustment algorithm is as follows, ; C w it+1 represents the confidence score corresponding to the time t +1th confidence score at the time i C live , C fake , C sync corresponding to the weight of the w i t represents the confidence score corresponding to the time t th confidence score at the time i gamma represents the set learning rate, r represents the historical attack rate, C i t represents the confidence score corresponding to the time i th confidence score at the time vectorize the live body confidence, the fake detection confidence, and the cross-modal confidence to obtain a confidence vector; vectorize the live body confidence weight, the fake detection confidence weight, and the cross-modal confidence weight to obtain a weight vector; According to the confidence vector combined with the weight vector, a fusion score is calculated; the calculation formula is as follows, ; Among them, Score indicates fusion score, w i represents the vector of the i th weight in the weight vector, C i represents the vector of the i th confidence in the confidence vector, tau represents the set adjustment coefficient (used to control the influence degree of the minimum value item on the final score), C represents the confidence vector; S4, constructing an RNN neural network, using historical remote interview behavior feature data combined with an optimization algorithm to optimize the RNN neural network to obtain an RNN behavior analysis network; Using the behavior analysis network, the real-time abnormal behavior index of the job seeker is obtained; The S4 includes the following steps: S41, collecting behavior data and corresponding behavior abnormality index in historical job seeker remote interviews to obtain historical remote interview behavior data; the historical remote interview behavior data contains behavior data of abnormal job seekers and non-abnormal job seekers; Extracting features of the historical remote interview behavior data to obtain historical remote interview behavior feature data; ​​S42, construct the RNN neural network, set the initial weight value of the RNN neural network; set the prediction accuracy and the prediction accuracy threshold of the RNN neural network; The RNN neural network is trained using historical remote interview behavior feature data, and in the training process, the optimal weight value of the RNN neural network is found by combining the training accuracy and the training accuracy threshold through an optimization algorithm to obtain an optimal solution; The optimal solution is used as the weight value of the RNN neural network to obtain an RNN behavior analysis network; The optimization algorithm can be the ternary optimization algorithm, the genetic algorithm, the fish swarm algorithm and the ant algorithm in the prior art; This embodiment takes the ant algorithm as an example for introduction: The optimal weight value of the RNN neural network is found by combining the training accuracy and the training accuracy threshold through the ant algorithm to obtain an optimal solution, including the following steps: S421, construct an ant colony, and set the size of the ant colony to m The ant colony is represented as , wherein n i The first ant in the ant colony is represented as i The maximum number of iterations is set; S422, according to the weight value of the RNN neural network, the initial position of the ant colony is randomly set, and the initial position set of the ant colony is obtained as , wherein u i The initial position of the first ant in the ant colony is represented as i The position of the ant can reflect the distance of the ant from the food; S423, according to the prediction accuracy and the prediction accuracy threshold of the RNN neural network, the fitness function of the initial position of the ant in the ant colony is defined, and the fitness function formula is as follows, ; , wherein ，o The fitness function is represented as z1, the prediction accuracy is represented as z2, the prediction accuracy threshold is represented as b The bias is represented as S424, the initial position set of the ant colony is iterated, the higher the fitness value, the closer the position of the ant to the food; in each iteration process, the fitness value of each position in the initial position set of the ant colony is calculated according to the fitness function, the position concentration of each ant in the initial position set of the ant colony is updated from high to low according to the fitness value, and the best ant individual position in the ant colony and the global best ant position are obtained in each iteration process; S425, repeat 424, when the maximum number of iterations is reached, stop iteration, and take the global best ant position as the optimal solution; The intelligent behavior analysis optimizes the abnormal recognition accuracy; the ant algorithm is used to optimize the RNN neural network; the prediction accuracy is taken as the objective function, the optimal weight combination is searched through the ant colony intelligence, and the training efficiency is improved; the optimized RNN behavior analysis network can capture the hidden fraud features such as micro-expression timing contradiction and response delay in real time, output an abnormal behavior index, and make up for the blind area of biological detection.

[0026] S43, collect real-time job seeker interview behavior data, and extract features to obtain real-time job seeker behavior feature data; The real-time job seeker behavior feature data is input into the RNN behavior analysis network to obtain a real-time abnormal behavior index; S5, obtaining a final risk score according to the fusion score combined with the real-time abnormal behavior index; and executing a hierarchical response scheme according to the final risk score; The S5 includes the following steps: S51, setting a hierarchical response scheme; the hierarchical response scheme is divided into a first-level execution scheme, a second-level execution scheme and a third-level execution scheme according to the final risk score; for example, the first-level execution scheme is that the final risk score is less than or equal to 0.1, a verifiable digital resume is generated, the resume is hashed and stored in IPFS, and a CID+blockchain is returned for storage, the second-level execution scheme is that 0.1 is less than the final risk score and the final risk score is less than or equal to 0.3, a dynamic challenge response is started, and the first-level execution scheme is executed after passing, and the third-level execution scheme is that the final risk score is greater than 0.3, and multi-party video arbitration is triggered; S52, calculating a final risk score according to the fusion score combined with the real-time abnormal behavior index; the calculation formula is as follows, ; Wherein, RiskScore represents the final risk score, q 1 represents the weight coefficient of the fusion score, q 2 represents the weight coefficient of the real-time abnormal behavior index, and represents B SCORE The real-time abnormal behavior index; S53, executing a hierarchical response scheme according to the final risk score.

[0027] Embodiment two: Please refer to Figure 3 , the remote interview anti-fraud system based on multi-modal deep fake detection is used to realize the remote interview anti-fraud method based on multi-modal deep fake detection, and includes an identity authentication and blockchain storage module, a multi-modal dynamic detection module, a confidence fusion and dynamic scoring module, a behavior analysis network module and a risk decision and response module. The identity authentication and blockchain storage module is responsible for collecting the identity information and biological characteristics of the job seeker, generating a job seeker information hash and biological characteristic hash pair through an anti-collision fuzzy hash algorithm, realizing zero-knowledge proof of hash matching by using zkSNARKs technology to ensure privacy security, and writing the hash triplet (certificate information hash + face hash + voiceprint hash) into the Chia blockchain after successful matching to realize tamper-proof distributed storage. The multi-modal dynamic detection module extracts physiological signals through double-band video streams, calculates the signal-to-noise ratio and the main pulse frequency, verifies whether the physiological activity conforms to the human characteristics, analyzes the abnormal features of the video frame sequence, calculates the abnormal index in combination with the real data set statistics, extracts the second-order mixed partial derivative of the speech phase spectrum, calculates the composite eigenvalue to identify the synthesized speech, and uses the DTW algorithm to align the timestamps of the lip shape and the sound to output the maximum offset. The confidence fusion and dynamic scoring module generates a living body confidence by combining physiological characteristics through a logical function, obtains a forgery detection confidence by weightedly fusing the video abnormal index and the speech synthesis eigenvalue, generates a cross-modal confidence according to an exponential decay function of the lip sound offset, and finally outputs a fusion score by using a dynamic weight adjustment algorithm in combination with the historical attack rate to update the weight of each confidence in real time, and the calculation process fuses the confidence vector, the weight vector and the minimum value adjustment mechanism. The behavior analysis network module uses historical interview behavior data to optimize the RNN weight through the ant algorithm, and outputs the optimal RNN behavior analysis network; extracts the interview behavior characteristics of the job seeker, and inputs the trained network to generate a real-time abnormal behavior index. The risk decision and response module integrates the fusion score and the behavior abnormal index, calculates the final risk score by weighting, and executes a graded response according to the score threshold.

[0028] In the description of the present specification, the description of the terms "one embodiment", "example", "specific example" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the invention. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0029] The preferred embodiments of the above disclosed invention are only used to help explain the invention. The preferred embodiments do not describe all the details and limit the invention to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the present specification. The present specification selects and describes these embodiments in order to better explain the principles and practical applications of the invention, so that those skilled in the art can well understand and utilize the invention.

Claims

1. A remote interview anti-fraud method based on multimodal deep fake detection, characterized by: The following steps are involved: S1. Collect the identity information and biometrics of job applicants; The job applicant’s identity information and biometrics are hashed and then matched. Once the match is complete, it is uploaded to the blockchain for storage. S2. Use multimodal dynamic detection combined with data stored in the blockchain to perform live physiological signal detection, deepfake spatiotemporal analysis, voiceprint frequency domain diagnosis, and cross-modal synchronous verification on job applicants to obtain physiological feature pairs, anomaly index, synthetic feature value, and maximum offset; S3. Calculate liveness confidence, forgery detection confidence, and cross-modal confidence based on the physiological feature pair, abnormality index, synthetic feature value, and maximum offset; Dynamically adjust the weights of liveness confidence, forgery detection confidence, and cross-modal confidence to obtain a fusion score; S4. Construct an RNN neural network and use historical remote interview behavior feature data combined with an optimization algorithm to optimize the RNN neural network to obtain an RNN behavior analysis network. Use behavioral analysis networks to obtain real-time abnormal behavior indexes of job seekers; S5. Obtain the final risk score based on the fusion score combined with the real-time abnormal behavior index; Execute a graded response plan based on the final risk score.

2. The remote interview anti-fraud method based on multimodal deep fake detection according to claim 1 is characterized in that: Said S1 comprises the following steps: S11. Collect high-definition images of the job seeker's ID card, extract the name and ID number through OCR, and obtain the job seeker's ID information; hash the job seeker's ID information to obtain the job seeker information hash; S12. Collect the job applicant's photoelectric volumetric image signal data, 3D structured light facial point cloud data, and voiceprint sample data through the encryption SDK to obtain the job applicant's biometric characteristics; S13. Performing collision-resistant fuzzy hashing on the biometrics of the job applicant to obtain a biometric hash pair of the job applicant; the biometric hash pair includes a face hash and a voiceprint hash, and the collision-resistant fuzzy hashing includes a face hash and a voiceprint hash; S14. Use zkSNARKs to verify that the job applicant’s biometric hash matches the job applicant’s ID information hash without leaking the original biometric. S15. After the match is successful, the job applicant's biometric hash pair and the job applicant's ID information hash are constructed into a triple to obtain the job applicant information triple; the job applicant information triple is written into the Chia blockchain, which includes a BLS signature + time-space proof mechanism.

3. The remote interview anti-fraud method based on multimodal deep fake detection according to claim 1 is characterized in that: The S2 comprises the following steps: S21. Analyzing a dual-band video stream of the job applicant's face captured by a camera to extract signals related to physiological activities and obtain a living physiological signal; combining the living physiological signal with a living physiological detection algorithm to obtain a physiological feature pair; the physiological feature pair includes a signal-to-noise ratio and a main pulse frequency; S22. Collect continuous frames of the job applicant's interview video to obtain a video frame sequence; the video frame sequence includes all continuous frames of the job applicant's real-time interview video; Extracting abnormal features of each frame in the video frame sequence to obtain a video frame abnormal feature sequence; the video frame abnormal feature sequence includes the abnormal features of each frame in the video frame sequence; The anomaly index is obtained by combining the deep fake anomaly index algorithm with the abnormal feature sequence of the video frame; S23, extracting the speech phase spectrum of the voice signal of the job applicant during the interview process, calculating the synthesized speech features, and obtaining the synthesized feature value; S24. Use the DTW algorithm to verify the lip synchronization of the job applicant and obtain the maximum offset.

4. The remote interview anti-fraud method based on multimodal deep fake detection according to claim 1 is characterized in that: The S3 includes the following steps: S31. Calculate a liveness confidence, a forgery detection confidence, and a cross-modal confidence based on the physiological feature pair, the abnormality index, the synthetic feature value, and the maximum offset; S32. Collect historical attack rates; combine the liveness confidence, forgery detection confidence, and cross-modal confidence with the historical attack rates, and use a dynamic weight adjustment algorithm to obtain liveness confidence weight, forgery detection confidence weight, and cross-modal confidence weight; Vectorize the liveness confidence, forgery detection confidence, and cross-modal confidence to obtain a confidence vector; vectorize the liveness confidence weight, forgery detection confidence weight, and cross-modal confidence weight to obtain a weight vector; The fusion score is calculated based on the confidence vector and the weight vector.

5. The remote interview anti-fraud method based on multimodal deep fake detection according to claim 4 is characterized in that: The S31 includes the following steps: S311, setting a parameter for controlling the steepness of a logic function and a threshold parameter of the logic function; calculating a living body confidence level based on the parameter for controlling the steepness of the logic function and the threshold parameter of the logic function in combination with physiological characteristics; S312. Setting a weight parameter for video anomaly and a weight parameter for voice anomaly; calculating a forgery detection confidence level based on the anomaly index, the synthetic feature value, and the weight parameter for video anomaly and the weight parameter for voice anomaly; S313: Calculate a cross-modal confidence score according to the maximum offset.

6. The remote interview anti-fraud method based on multimodal deep fake detection according to claim 1 is characterized in that: The S4 comprises the following steps: S41. Collecting historical behavioral data of job seekers in remote interviews and corresponding behavioral anomaly indexes to obtain historical remote interview behavioral data; extracting features of the historical remote interview behavioral data to obtain historical remote interview behavioral feature data; S42, constructing an RNN neural network, setting the initial weight value of the RNN neural network; setting the prediction accuracy and prediction accuracy threshold of the RNN neural network; The RNN neural network is trained using historical remote interview behavioral feature data. During the training process, the ant algorithm is combined with the training accuracy and training accuracy threshold to find the optimal weight value of the RNN neural network and obtain the optimal solution; Using the optimal solution as the weight value of the RNN neural network to obtain an RNN behavior analysis network; S43. Collecting real-time job seeker interview behavior data and extracting features to obtain real-time job seeker behavior feature data; The real-time job seeker behavior feature data is input into the RNN behavior analysis network to obtain the real-time abnormal behavior index.

7. The remote interview anti-fraud method based on multimodal deep fake detection according to claim 6 is characterized in that: The historical remote interview behavior data includes behavior data of abnormal job seekers and non-abnormal job seekers.

8. The remote interview anti-fraud method based on multimodal deep fake detection according to claim 1 is characterized in that: The S5 comprises the following steps: S51. Setting a hierarchical response plan; the hierarchical response plan is divided into a first-level implementation plan, a second-level implementation plan, and a third-level implementation plan according to the final risk score; S52. Calculate the final risk score based on the fusion score and the real-time abnormal behavior index; S53. Implement a graded response plan based on the final risk score.

9. Remote interview anti-fraud system based on multimodal deep fake detection, characterized by: A remote interview anti-fraud method based on multimodal deep fake detection as described in any one of claims 1 to 8 is implemented, wherein the system includes an identity authentication and blockchain storage module, a multimodal dynamic detection module, a confidence fusion and dynamic scoring module, a behavior analysis network module, and a risk decision and response module.

10. A storage medium, characterized in that: A program is stored thereon, and when the program is executed by the processor, the remote interview anti-fraud method based on multimodal deep fake detection as described in any one of claims 1-8 is implemented.

Images