A Facial Fraud Action Recognition Method Based on Visual Optical Flow Features

Abstract

The present invention relates to the field of Internet financial risk control technology, and specifically discloses a method for identifying facial fraud actions based on visual optical flow features, including: obtaining a sequence of face images from an in-person review video; performing face detection on the first and last face images in the sequence of face images respectively; determining the facial fraud regions of the first and last face images respectively according to the detected face key points, and filling the background regions other than the facial fraud regions in the first and last face images respectively to obtain optical flow feature maps; extracting the face ROI regions from the corrected optical flow feature maps; inputting the target optical flow feature maps within the face ROI regions into a facial fraud action recognition model to obtain the facial fraud action recognition results. The method for identifying facial fraud actions based on visual optical flow features provided by the present invention can reduce background interference in complex environments, enhance the extraction of features of facial fraud actions, and improve the accuracy of the anti-fraud model.

Description

Technical Field

[0001] The present invention relates to the field of Internet financial risk control technology, and more specifically, to a facial fraud action recognition method based on visualized optical flow features. Background Art

[0002] Internet finance is a new financial business model in which traditional financial institutions and Internet companies use Internet technology and information and communication technology to achieve financing, payment, investment and information intermediary services. In recent years, Internet finance has been a hot topic. While the new business model has brought new growth points, it has also brought new challenges to risk control. Among them, the external fraud risk in operational risk is particularly worthy of attention.

[0003] In the business scenario of Internet finance, the risk of customer fraud is more serious than offline business due to the characteristics of the Internet, customer resource issues, and product defects. Among them, video interview is an important pre-loan process and loan basis for online approval. Therefore, it is particularly important to establish necessary risk control models in video interview.

[0004] Most risk control models in video review are anti-fraud models based on micro-expressions. They mainly process single-frame images in the video, capture face candidate frames through face detection, and use them as inputs to the deep learning network model to obtain feature sequences, and then obtain the final anti-fraud results through classification methods. On the one hand, this single-frame processing method cannot extract facial action features well, and on the other hand, micro-expressions cannot cover all facial fraud actions. At the same time, the complex and changeable environment of the customer cannot be restricted in video review, resulting in a low recognition rate of the actual anti-fraud model. Therefore, how to reduce background interference in complex environments and enhance the features of extracting facial fraud actions is the key to improving the robustness of the anti-fraud model. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a facial fraud action recognition method based on visualized optical flow features, which can reduce background interference in complex environments, enhance the features of facial fraud actions, and improve the accuracy of anti-fraud models.

[0006] As a first aspect of the present invention, a method for identifying facial fraud actions based on visualized optical flow features is provided, comprising the following steps:

[0007] Step S1: obtaining a face-to-face review video, and obtaining a face image sequence of a fixed length from the face-to-face review video;

[0008] Step S2: performing face detection on the first and last two frames of face images in the face image sequence at the current moment, respectively, obtaining two face coordinate frames, and performing face key point detection on the face images in the two face coordinate frames respectively;

[0009] Step S3: Determine the facial fraud regions of the first and last frame face images respectively according to the detected face key points, and perform filling processing on the background regions other than the facial fraud regions in the first and last frame face images respectively;

[0010] Step S4: Operate on the first and last frame face images after background filling processing to obtain an optical flow feature map;

[0011] Step S5: Correct the optical flow feature map to obtain a corrected optical flow feature map, and extract the face ROI region from the corrected optical flow feature map;

[0012] Step S6: Train a facial fraud action recognition model, input the target optical flow feature map within the face ROI region into the facial fraud action recognition model, and obtain a facial fraud action recognition result.

[0013] Further, in the step S1, it further includes:

[0014] The face review video is obtained by real-time collection of an arbitrary user device camera.

[0015] Further, in the step S2, it further includes:

[0016] Perform face detection on the first and last frame face images respectively through RetinaFace to obtain the corresponding face coordinate box bbox(x lt , y lt , x rb , y rb ) and the five-point face key points Point k (x k , y k ), (k = 5);

[0017] Input the face coordinate box bbox(x lt , y lt , x rb , y rb ) into the 106-point face key point detection model, and output the 106-point face key points Point l (x l , y l ), (l = 106);

[0018] Save the face coordinate box bbox, the five-point face key points Point k and the 106-point face key points Point l as the attribute table of the current face image.

[0019] Further, in the step S3, it further includes:

[0020] Determine the 2D facial fraud candidate region A of the first and last frame face images respectively according to the 106 key points of the face;

[0021] Determine the 3D facial fraud candidate region B of the first and last frame face images respectively through 3D face pose estimation;

[0022] Calculate the IOU value of the 2D facial fraud candidate region A and the 3D facial fraud candidate region B of the first frame face image, and at the same time calculate the IOU value of the 2D facial fraud candidate region A and the 3D facial fraud candidate region B of the last frame face image;

[0023] Determine the facial fraud region of the first frame face image by comparing the bilateral confidence and IOU value of the 2D facial fraud candidate region A and the 3D facial fraud candidate region B of the first frame face image; at the same time, determine the facial fraud region of the last frame face image by comparing the bilateral confidence and IOU value of the 2D facial fraud candidate region A and the 3D facial fraud candidate region B of the last frame face image;

[0024] Among them, the confirmation processes of the facial fraud regions of the first frame face image and the last frame face image are the same.

[0025] Further, in the step of determining the 2D facial fraud candidate region A of the first and last frame face images respectively according to the 106 key points of the face, it further includes:

[0026] Determine the coordinates P of both sides of the face cheeks and the chin edge of the first and last frame face images respectively through the 106 key points of the face n (x, y), (n = 32);

[0027] Calculate the coordinates of the forehead region of the first and last frame face images respectively through formula (1), and formula (1) is as follows:

[0028] (x - a) 2 +(y - b) 2 = r 2

[0029] Among them, Point1(x1, y1), Point 17 (x 17 , y 17 ) are the corresponding rectangular coordinate system coordinates in the face key points, and a and b are the face center point coordinates;

[0030] Determine the 2D facial fraud candidate region A of the first and last frame face images respectively through the coordinates of both sides of the cheeks and the chin edge and the coordinates of the forehead region of the face.

[0031] Further, it further includes:

[0032] Calculate the IOU values of the first-frame and last-frame face images respectively through formula (2), and formula (2) is as follows:

[0033]

[0034] Among them, P 2D is the 2D facial fraud candidate area A of the first-frame or last-frame face image, and P 3D is the 3D facial fraud candidate area B of the first-frame or last-frame face image;

[0035] If the bilateral confidence levels of the 2D facial fraud candidate area A and the 3D facial fraud candidate area B of the first-frame or last-frame face image are both higher than the threshold H, then judge the IOU value of the first-frame or last-frame face image; if the IOU value is higher than the threshold K, then select the 3D facial fraud candidate area B as the facial fraud area of the first-frame or last-frame face image, and if the IOU value is less than the threshold K, then select the 2D facial fraud candidate area A as the facial fraud area of the first-frame or last-frame face image;

[0036] If the bilateral confidence levels of the 2D facial fraud candidate area A and the 3D facial fraud candidate area B of the first-frame or last-frame face image are both lower than the threshold L, then skip the judgment of the current moment's face image sequence;

[0037] If the bilateral confidence levels of the 2D facial fraud candidate area A and the 3D facial fraud candidate area B of the first-frame or last-frame face image do not both exceed the threshold H and do not both fall below the threshold L at the same time, then compare the confidence levels of the 2D facial fraud candidate area A and the 3D facial fraud candidate area B of the first-frame or last-frame face image, and select the facial fraud candidate area with the higher confidence level as the facial fraud area of the first-frame or last-frame face image;

[0038] Among them, the threshold H is 0.8, the threshold K is 0.5, and the threshold L is 0.3.

[0039] Furthermore, in the step S4, it further includes:

[0040] Perform grayscale conversion on the first and last face images after background filling processing respectively;

[0041] According to the grayscale images of the first and last face images, calculate the dense optical flow of each pixel point, and obtain the h*w*2-dimensional optical flow feature, where the h*w*2-dimensional optical flow feature represents the displacement bias corresponding to each pixel point as coffset (h，w) (d x , d y );

[0042] Through coordinate system transformation, the displacement offset coffset (h，w) (d x, d y ) Convert from the rectangular coordinate system to the polar coordinate system;

[0043] Substitute the polar coordinate system into the HSV color space, and through the visualization of the optical flow field, convert it into the optical flow feature map, where the H channel represents the direction and the V channel represents the motion intensity.

[0044] Further, in the step S5, it further includes:

[0045] According to the five key points of the human face Point k (x k , y k ), (k = 5) calculate the transformation matrix;

[0046] Perform an affine transformation on the optical flow feature map according to the transformation matrix to obtain the corrected optical flow feature map;

[0047] Extract the human face ROI region from the corrected optical flow feature map.

[0048] Further, in the step S6, it further includes:

[0049] Screen out the visual optical flow feature maps that meet the facial fraud behavior from the target optical flow feature maps within the human face ROI region to construct the facial fraud action recognition model;

[0050] Input the target optical flow feature maps within the human face ROI region into the facial fraud action recognition model to obtain multi-dimensional feature results, and then obtain the facial fraud action recognition result score of the current frame human face image sequence through the softmax function;

[0051] Perform a weighted sum of the facial fraud action recognition result scores of the N-frame human face image sequence, calculate the final score of the facial fraud action recognition result of the face review video, and determine whether there is a facial fraud action behavior of the person in the face review video;

[0052] Among them, the calculation formula of the final score Q of the facial fraud action recognition result of the face review video is as follows:

[0053]

[0054] Among them, Q is the final score of the facial fraud action recognition result of the face review video, w i is the weight of the current frame human face image sequence, and S i is the facial fraud action recognition result score of the current frame human face image sequence.

[0055] Further, it further includes:

[0056] Based on the MobileNetV2 network model framework, the facial fraud action recognition model is constructed by adding convolutional layers and pooling layers to the network model and modifying the fully connected layer in the output layer.

[0057] The facial fraud action recognition method based on visual optical flow features provided by the present invention has the following advantages:

[0058] (1) By extracting optical flow features from an image sequence of a fixed time length, it is possible to solve the problem of different frame rates caused by different user devices during video face review without changing the video frame rate of the user, thereby increasing the stability of facial fraud action recognition.

[0059] (2) Before extracting optical flow features, through background processing, the interference of complex and variable backgrounds on optical flow features can be eliminated, the accuracy of facial fraud action recognition can be improved, and the stability of the system can be improved by using 2D and 3D methods to extract facial fraud regions.

[0060] (3) According to the facial expression coding system, starting directly from facial movement units, compared with other methods such as micro-expressions, facial fraud actions are improved from the root, the coverage of facial fraud actions is expanded, and the probability of missed recognition of facial fraud actions is reduced.

[0061] (4) By fusing the recognition results of multiple frames, the stability of facial fraud action recognition is effectively improved.

[0062] (5) Therefore, the present invention can effectively reduce the influence of complex environments on facial fraud action recognition, improve the accuracy of facial fraud action recognition, be applicable to complex business scenarios such as video face review and loan approval in all financial risk control fields, and has good promotion and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification, and are used together with the following specific embodiments to explain the present invention, but do not limit the present invention.

[0064] Figure 1 It is a flowchart of the facial fraud action recognition method based on visual optical flow features provided by the present invention.

[0065] Figure 2 It is a flowchart of the specific implementation of the facial fraud action recognition method based on visual optical flow features provided by the present invention.

[0066] Figure 3 It is a flowchart of visual optical flow feature extraction provided by the present invention. SPECIFIC EMBODIMENTS

[0067] To further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation manners, structures, features and their effects of the facial fraud action recognition method based on visual optical flow features proposed according to the present invention. Obviously, the described embodiments are a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0068] In this embodiment, a facial fraud action recognition method based on visual optical flow features is provided. As Figure 1 shown, the facial fraud action recognition method based on visual optical flow features includes:

[0069] Step S1: Obtain an in-person review video, and obtain a sequence of face images with a fixed duration from the in-person review video;

[0070] Among them, the sequence of face images list{P F 、P F+1 、P F+2 …P L-1 、P L} includes multiple frames of face images, P F is the first frame of face image in the sequence of face images at the current moment, and P L is the last frame of face image in the sequence of face images at the current moment;

[0071] Step S2: Perform face detection on the first and last frames of face images in the sequence of face images at the current moment respectively to obtain two face coordinate frames, and perform face key point detection on the face images within the two face coordinate frames respectively;

[0072] It should be noted that face detection is performed on the first and last frames of face images P F (former) and P L (latter) at both ends of the sequence of face images at the current moment respectively;

[0073] Step S3: Determine the facial fraud regions of the first and last frames (P F frame and P L frame) of face images according to the detected face key points respectively, and perform pixel-level filling processing on the background regions other than the facial fraud regions in the first and last frames of face images respectively;

[0074] Step S4: Operate on the first and last frames of face images after background filling processing to obtain an optical flow feature map;

[0075] Step S5: Correct the optical flow feature map to obtain a corrected optical flow feature map, and extract the face ROI region from the corrected optical flow feature map;

[0076] Step S6: Train a facial fraud action recognition model, input the target optical flow feature map within the face ROI region into the facial fraud action recognition model, and obtain a facial fraud action recognition result.

[0077] It should be noted that the target optical flow feature map within the face ROI region is first subjected to image preprocessing operations, and then the preprocessed target optical flow feature map is input into the facial fraud action recognition model.

[0078] Preferably, in step S1, it further includes:

[0079] The face review video is obtained by real-time collection from the camera of any user device.

[0080] Specifically, in step S1, it further includes:

[0081] Read the face review video frame by frame, and regard each frame as a P L (latter) face image, and save continuous image sequences of a fixed duration in segments.

[0082] Preferably, as Figure 2 shown, in step S2, it further includes:

[0083] Perform face detection on the first and last frame face images respectively through RetinaFace (Single-stage Dense Face Localisation in the Wild) to obtain the corresponding face coordinate box bbox(x lt , y lt , x rb , y rb ) and five-point face key points Point k (x k , y k ), (k = 5);

[0084] Input the face coordinate box bbox(x lt , y lt , x rb , y rb ) into the 106-point face key point detection model provided by InsightFace, and output 106-point face key points Point l (x l , y l ), (l = 106);

[0085] Save the face coordinate box bbox, the five-point face key points Point k and the 106-point face key points Point l as the attribute table of the current face image; this attribute table not only prepares for subsequent face alignment and facial fraud area extraction in the current image sequence, but also can be reused when turning P L into P F images in the next image sequence, saving costs.

[0086] Preferably, in the step S3, it further includes:

[0087] Respectively determine the 2D facial fraud candidate areas A of the first and last frame face images according to the 106-point face key points;

[0088] Respectively determine the 3D facial fraud candidate areas B of the first and last frame face images through 3D face pose estimation;

[0089] Calculate the IOU values of the 2D facial fraud candidate area A and the 3D facial fraud candidate area B of the first frame face image, and at the same time calculate the IOU values of the 2D facial fraud candidate area A and the 3D facial fraud candidate area B of the last frame face image;

[0090] Determine the facial fraud area of the first frame face image by comparing the bilateral confidence and IOU values of the 2D facial fraud candidate area A and the 3D facial fraud candidate area B of the first frame face image; at the same time, determine the facial fraud area of the last frame face image by comparing the bilateral confidence and IOU values of the 2D facial fraud candidate area A and the 3D facial fraud candidate area B of the last frame face image;

[0091] Among them, the confirmation processes of the facial fraud areas of the first frame face image and the last frame face image are the same.

[0092] Specifically, through the Img2Pose (Face Alignment and Detection via 6DoF) algorithm, estimate P F 、P L the 3D face poses of the two frame face images and convert them into the facial fraud candidate areas B.

[0093] It should be noted that IOU is the Intersection over Union.

[0094] Preferably, in the process of respectively determining the 2D facial fraud candidate areas A of the first and last frame face images according to the 106-point face key points, it further includes:

[0095] Respectively determine the coordinates P on both sides of the cheeks and the edge of the chin of the first and last frame face images through the 106-point face key pointsn (x, y), (n = 32);

[0096] Assume that the forehead area of the human face is a quasi-circular area, and calculate the coordinates of the forehead area of the human face in the first and last frames of the human face images respectively through formula (1). Formula (1) is as follows:

[0097] (x - a) 2 + (y - b) 2 = r 2

[0098] Among them, Point1(x1, y1), Point 17 (x 17 , y 17 ) are the corresponding rectangular coordinate system coordinates among the key points of the human face, and a and b are the coordinates of the center point of the human face;

[0099] Determine the 2D facial fraud candidate area A of the first and last frames of the human face images respectively through the coordinates of both sides of the cheeks and the edge of the chin and the coordinates of the forehead area of the human face.

[0100] Preferably, it further includes:

[0101] Calculate the IOU values of the first and last frames of the human face images respectively through formula (2). Formula (2) is as follows:

[0102]

[0103] Among them, P 2D is the 2D facial fraud candidate area A of the first or last frame of the human face image, and P 3D is the 3D facial fraud candidate area B of the first or last frame of the human face image;

[0104] If the bilateral confidence levels of the 2D facial fraud candidate area A and the 3D facial fraud candidate area B of the first or last frame of the human face image are both higher than the threshold H, then judge the IOU value of the first or last frame of the human face image; if the IOU value is higher than the threshold K, select the 3D facial fraud candidate area B as the facial fraud area of the first or last frame of the human face image, and if the IOU value is less than the threshold K, select the 2D facial fraud candidate area A as the facial fraud area of the first or last frame of the human face image;

[0105] If the bilateral confidence levels of the 2D facial fraud candidate area A and the 3D facial fraud candidate area B of the first or last frame of the human face image are both lower than the threshold L, skip the judgment of the current moment human face image sequence;

[0106] If the bilateral confidence levels of the 2D facial fraud candidate region A and the 3D facial fraud candidate region B in the first or last frame face image are not both higher than the threshold H and not both lower than the threshold L, then compare the confidence levels of the 2D facial fraud candidate region A and the 3D facial fraud candidate region B in the first or last frame face image, and select the facial fraud candidate region with the higher confidence level as the facial fraud region of the first or last frame face image;

[0107] Among them, the threshold H is 0.8, the threshold K is 0.5, and the threshold L is 0.3.

[0108] Specifically, perform a unified color filling method on the background regions of the first and last frame face images except for the facial fraud regions to remove the background environment of the non-facial fraud regions, and select black as the filling color; this method can well preserve the features of the facial fraud regions of the human face while removing background interference, and improve the system stability through the combination of the 2D and 3D methods for extracting facial fraud regions.

[0109] Preferably, as Figure 3 shown, in the step S4, it further includes:

[0110] Convert the first and last frame face images after background filling processing into grayscale images respectively;

[0111] According to the grayscale images of the first and last frame face images, calculate the dense optical flow of each pixel point and obtain the h*w*2-dimensional optical flow feature, where the h*w*2-dimensional optical flow feature represents that the displacement offset corresponding to each pixel point is coffset (h，w) (d x ,d y );

[0112] Through coordinate system transformation, convert the displacement offset coffset (h，w) (d x ,d y ) from the rectangular coordinate system to the polar coordinate system;

[0113] Bring the polar coordinate system into the HSV color space, and through the visualization of the optical flow field, convert it into the optical flow feature map, where the H channel represents the direction and the V channel represents the motion intensity.

[0114] It should be noted that through the optical flow field visualization operation, the change trend of facial movements can be expressed more intuitively.

[0115] Preferably, in the step S5, it further includes:

[0116] According to the five key points of the human face Point k (x k ,y k ), (k = 5) calculate the transformation matrix;

[0117] Perform an affine transformation on the optical flow feature map according to the transformation matrix to obtain a corrected optical flow feature map;

[0118] Extract the face ROI region from the corrected optical flow feature map.

[0119] Preferably, in step S6, it further includes:

[0120] Screen out the visual optical flow feature maps that conform to facial fraud behaviors from the target optical flow feature maps within the face ROI region through the Facial Action Coding System to construct the facial fraud action recognition model; this visual optical flow feature map is used as the input of the deep learning model, and training samples are enhanced by methods such as random flipping, adding noise, and randomly modifying pixel intensities;

[0121] Input the target optical flow feature maps within the face ROI region into the facial fraud action recognition model to obtain multi-dimensional feature results, and then obtain the facial fraud action recognition result score of the current frame face image sequence through the softmax function;

[0122] During training, the traditional cross-entropy (CrossEntropyLoss) is used as the loss function, and the formula (3) of the loss function is as follows:

[0123]

[0124] Among them, p is the true value, s is the predicted value of the facial fraud action recognition result, s is obtained by calculating through the softmax formula, and the softmax formula (4) is as follows:

[0125]

[0126] Among them, i represents the current element, j represents all elements, and the softamx value of the current element represents the ratio of the exponent of the current element to the sum of the exponents of all elements;

[0127] Perform preprocessing operations such as normalizing the face optical flow feature map, and then input it into the facial fraud action recognition model to obtain a 1*N-dimensional feature result, where N is 2; and input the feature result into the above softmax formula to obtain the facial fraud action recognition result score of the current face image sequence;

[0128] Finally, perform a weighted sum of the facial fraud action recognition result scores of the N-frame face image sequence to calculate the final score of the facial fraud action recognition of the face review video to determine whether there is a facial fraud action behavior of the person in the face review video;

[0129] Among them, the calculation formula for the final score Q of the facial fraud action recognition result of the in-person review video is as follows:

[0130]

[0131] Among them, Q is the final score of the facial fraud action recognition result of the in-person review video, w i is the weight of the current frame face image sequence, and S i is the score of the facial fraud action recognition result of the current frame face image sequence.

[0132] Preferably, it further includes: Based on the MobileNetV2 network model framework, in order to make the model more suitable for the current actual needs, by adding the convolutional layer and pooling layer of the network model, the situation of sudden drop in network feature dimensions is avoided, and the fully connected layers in the output layer are modified to adapt to the current output dimension requirements, and the facial fraud action recognition model is constructed.

[0133] The facial fraud action recognition method based on visual optical flow features provided by the present invention can extract optical flow features from an image sequence of a fixed time length, and can solve the problem of different frame rates brought by different user devices during video in-person review without changing the video frame rate of the user, thereby increasing the stability of facial fraud action recognition; before optical flow feature extraction, through background processing, the interference of complex and variable backgrounds on optical flow features can be eliminated, improving the accuracy of facial fraud action recognition. At the same time, the method of extracting facial fraud regions through 2D and 3D can improve the stability of the system; according to the facial expression coding system, starting directly from facial action units, compared with other methods such as micro-expressions, facial fraud actions are improved from the root, expanding the coverage of facial fraud actions, thereby reducing the probability of missed recognition of fraud actions; by fusing the recognition results of multiple frames, the stability of facial fraud action recognition is effectively improved; therefore, the present invention can effectively reduce the influence of complex environments on facial fraud action recognition, improve the accuracy of facial fraud action recognition, be applicable to complex business scenarios such as video in-person review and loan approval in all financial risk control fields, and has good promotion and application value.

[0134] The above are only the preferred embodiments of the present invention and do not impose any formal restrictions on the present invention. Although the present invention has been disclosed above with the preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments by using the disclosed technical content within the scope of the technical solution of the present invention. However, as long as it does not depart from the technical content of the technical solution of the present invention, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A method for facial fraud action recognition based on visual optical flow features, characterized in that, It includes the following steps: Step S1: Obtain the in-person interview video, and obtain a sequence of face images of a fixed duration from the in-person interview video; Step S2: Perform face detection on the first and last frames of face images in the sequence of face images at the current moment respectively to obtain two face coordinate frames, and perform face key point detection on the face images within the two face coordinate frames respectively; Step S3: Determine the facial fraud regions of the first and last frames of face images respectively according to the detected face key points, and perform filling processing on the background regions other than the facial fraud regions in the first and last frames of face images respectively; Step S4: Operate on the first and last frames of face images after background filling processing to obtain an optical flow feature map; Step S5: Correct the optical flow feature map to obtain a corrected optical flow feature map, and extract the face ROI region from the corrected optical flow feature map; Step S6: Train a facial fraud action recognition model, input the target optical flow feature map within the face ROI region into the facial fraud action recognition model to obtain a facial fraud action recognition result; Among them, in the step S2, it further includes: Face detection is performed on the first and last frame face images respectively through RetinaFace to obtain the corresponding face coordinate boxes bbox(x lt , y lt , x rb , y rb ) and the five-point face key points Point k (x k , y k ), (k = 5); Input the face coordinate box bbox(x lt , y lt , x rb , y rb ) into the 106-point face key point detection model, and output the 106-point face key point Point l (x l , y l ), (l = 106); Save the face coordinate box bbox, the five-point face key points Point k and the 106-point face key points Point l as the attribute table of the current face image; Among them, in the step S3, it further includes: Determine the 2D facial fraud candidate region A of the first and last frames of face images respectively according to the 106-point face key points; Determine the 3D facial fraud candidate region B of the first and last frames of face images respectively through 3D face pose estimation; Calculate the IOU value of the 2D facial fraud candidate region A and the 3D facial fraud candidate region B of the first frame of face image, and calculate the IOU value of the 2D facial fraud candidate region A and the 3D facial fraud candidate region B of the last frame of face image at the same time; Determine the facial fraud region of the first frame of face image by comparing the bilateral confidence and IOU value of the 2D facial fraud candidate region A and the 3D facial fraud candidate region B of the first frame of face image; at the same time, determine the facial fraud region of the last frame of face image by comparing the bilateral confidence and IOU value of the 2D facial fraud candidate region A and the 3D facial fraud candidate region B of the last frame of face image; Among them, the confirmation processes of the facial fraud regions of the first frame of face image and the last frame of face image are the same.

2. The facial fraud action recognition method based on visual optical flow features according to claim 1, characterized in that In the step S1, it further includes: The in-person interview video is obtained by real-time collection from the camera of any user device.

3. The facial fraud action recognition method based on visual optical flow features according to claim 1, wherein In the determination of the 2D facial fraud candidate region A of the first and last frames of face images respectively according to the 106-point face key points, it further includes: Determine the coordinates P of the two sides of the face cheeks and the chin edge of the face images in the first and last frames respectively through the 106 key points of the face n (x, y), (n = 32); Calculate the face forehead region coordinates of the first and last frames of face images respectively through formula (1), and formula (1) is as follows: (x - a) 2 +(y - b) 2 = r 2 Among them, Point1(x1,y1), Point 17 (x 17 ,y 17 ) are the corresponding rectangular coordinate system coordinates of the facial key points, and a and b are the coordinates of the facial center point; Determine the 2D facial fraud candidate region A of the first and last frames of face images respectively through the coordinates of both sides of the cheeks and the chin edge and the face forehead region coordinates.

4. The method for facial fraud action recognition based on visual optical flow features according to claim 1, wherein It further includes: Calculate the IOU value of the first and last frames of face images respectively through formula (2), and formula (2) is as follows: Among them, P 2D is the 2D facial fraud candidate area A of the first frame or the last frame face image, and P 3D is the 3D facial fraud candidate area B of the first frame or the last frame face image; If the bilateral confidence levels of the 2D face fraud candidate region A and the 3D face fraud candidate region B in the first frame or the last frame face image are both higher than the threshold H, then judge the IOU value of the first frame or the last frame face image; if the IOU value is higher than the threshold K, select the 3D face fraud candidate region B as the face fraud region of the first frame or the last frame face image, and if the IOU value is less than the threshold K, select the 2D face fraud candidate region A as the face fraud region of the first frame or the last frame face image; If the bilateral confidence levels of the 2D face fraud candidate region A and the 3D face fraud candidate region B in the first frame or the last frame face image are both lower than the threshold L, then skip the judgment of the current moment face image sequence; If the bilateral confidence levels of the 2D face fraud candidate region A and the 3D face fraud candidate region B in the first frame or the last frame face image do not both exceed the threshold H and do not both fall below the threshold L, then compare the confidence levels of the 2D face fraud candidate region A and the 3D face fraud candidate region B in the first frame or the last frame face image, and select the face fraud candidate region with the higher confidence level as the face fraud region of the first frame or the last frame face image; Among them, the threshold H is 0.8, the threshold K is 0.5, and the threshold L is 0.

3.

5. The method for facial fraud action recognition based on visual optical flow features according to claim 1, characterized in that In the step S4, it further includes: Perform grayscale conversion on the first and last frame face images after background filling processing respectively; According to the grayscale images of the first and last face images, calculate the dense optical flow of each pixel point, and obtain the h*w*2-dimensional optical flow feature, where the h*w*2-dimensional optical flow feature represents that the displacement offset corresponding to each pixel point is coffset (h,w) (d x ,d y ); Through coordinate system transformation, the displacement offset coffset (h,w) (d x ,d y ) is converted from the rectangular coordinate system to the polar coordinate system; Bring the polar coordinate system into the HSV color space, and through the visualization of the optical flow field, convert it into the optical flow feature map, where the H channel represents the direction and the V channel represents the motion intensity.

6. The facial fraud action recognition method based on visual optical flow features according to claim 1, wherein In the step S5, it further includes: According to the five key points of the human face Point k (x k ,y k ),(k = 5) to calculate the transformation matrix; Perform affine transformation on the optical flow feature map according to the transformation matrix to obtain the corrected optical flow feature map; Extract the face ROI region from the corrected optical flow feature map.

7. The method for facial fraud action recognition based on visual optical flow features according to claim 1, wherein In the step S6, it further includes: Screen out the visual optical flow feature maps that meet the face fraud behavior from the target optical flow feature maps within the face ROI region to construct the face fraud action recognition model; Input the target optical flow feature maps within the face ROI region into the face fraud action recognition model to obtain multi-dimensional feature results, and then obtain the face fraud action recognition result score of the current frame face image sequence through the softmax function; Perform weighted summation on the face fraud action recognition result scores of the N-frame face image sequence, calculate the final score of the face fraud action recognition result of the face review video, so as to determine whether there is a face fraud action behavior of the person in the face review video; Among them, the calculation formula of the final score Q of the face fraud action recognition result of the face review video is as follows: Among them, Q is the final score of the facial fraud action recognition result of the face review video, and w i is the weight of the face image sequence of the current frame, and S i is the score of the facial fraud action recognition result of the face image sequence of the current frame.

8. The method for facial fraud action recognition based on visual optical flow features according to claim 7, wherein, It further includes: Based on the MobileNetV2 network model framework, construct the face fraud action recognition model by adding the convolutional layer and the pooling layer of the network model and modifying the fully connected layer in the output layer.

Images