Multimodal face attack detection method, device, electronic device and storage medium

Abstract

This application discloses a multimodal face attack detection method, device, electronic device, and storage medium, relating to the field of face recognition technology. The multimodal face attack detection method comprises: inputting an initial matrix corresponding to multimodal data consisting of a combination of cloud-network behavioral data and facial image data of a mobile internet user into an anomaly perception matrix generation deep network to obtain a multimodal anomaly perception data matrix; inputting the multimodal anomaly perception data matrix into a multimodal anomaly risk measurement deep network to generate a risk measurement vector; determining a dynamic threshold based on fuzzy calculation rules, the risk measurement vector, and a baseline threshold; selecting multiple face attack detection models for detection based on the dynamic threshold to obtain the attack probability of each face attack detection model; and determining an attack detection result based on each of the attack probabilities and the dynamic threshold. This application solves the technical problem of the difficulty of multi-dimensional data collection in multimodal face attack detection in mobile scenarios.

Description

Technical Field

[0001] The present application relates to the field of face recognition technology, and in particular to a multimodal face attack detection method, device, electronic device, and storage medium. Background Art

[0002] Facial attack detection technology is a core key technology in the field of trusted face authentication. It is mainly used to defend against facial recognition. Criminals use photos, reshot videos, masks, etc. of the attacked person to carry out fake face attacks, or use tools such as face-changing software to carry out deep video forgery attacks, or add disturbance noise to guide the system to misidentify facial images. Deep adversarial attacks have created major risks to public safety, economic security, information security, and personal privacy.

[0003] Current facial attack detection technologies can be divided into visible light, near-infrared, short-wave infrared and other anti-attack technologies. On the other hand, depending on the data type, existing facial attack detection models can be divided into two technical routes: single-modal and multi-modal. Single-modal data and multi-modal data are used for facial attack detection respectively. Among them, multi-modal data facial attack detection technology has higher precision and accuracy. However, facial attack detection using multi-modal data in mobile scenarios also requires specific sensors, such as depth, near-infrared, and light field cameras to capture data. These devices are relatively expensive, and many users do not have such sensors on their mobile devices. The above-mentioned multi-modal data facial attack detection technical means are also difficult to effectively implement in mobile scenarios. Summary of the Invention

[0004] The main purpose of this application is to provide a multimodal face attack detection method, device, electronic device and storage medium, aiming to solve the technical problem of difficulty in multi-dimensional data collection in multimodal face attack detection solutions in mobile scenarios.

[0005] To achieve the above objectives, the present application provides a multimodal face attack detection method, which includes:

[0006] Input the initial matrices corresponding to the multimodal data of the mobile Internet user's cloud network terminal behavior data and facial image data into the preset abnormality perception matrix generation deep network to obtain the multimodal abnormality perception data matrix;

[0007] Inputting the multimodal anomaly perception data matrix into a preset multimodal anomaly risk measurement deep network to generate a risk measurement vector;

[0008] Determining a dynamic threshold value according to a preset fuzzy calculation rule, the risk measurement vector, and a preset benchmark threshold value;

[0009] Selecting multiple face attack detection models according to the dynamic threshold to detect the face image data, and obtaining attack probabilities corresponding to the respective face attack detection models;

[0010] An attack detection result is determined based on the attack probability corresponding to each of the face attack detection models and the dynamic threshold.

[0011] Optionally, the multimodal data includes at least terminal data, location data, communication data, network data, and face image data; the abnormality perception matrix generation deep network includes at least a first subnetwork, a second subnetwork, a third subnetwork, a fourth subnetwork, and a fifth subnetwork; and the multimodal abnormality perception data matrix includes at least a terminal abnormality perception vector, a location abnormality perception vector, a communication abnormality perception vector, a network abnormality perception vector, and a face abnormality perception vector;

[0012] The step of inputting the initial matrices corresponding to the multimodal data of the mobile internet user's cloud network terminal behavior data and facial image data into a preset abnormality perception matrix generation deep network to obtain the multimodal abnormality perception data matrix includes:

[0013] Preprocessing the collected multimodal data to generate initial matrices corresponding to the respective modal data, wherein the initial matrices include at least a first initial matrix, a second initial matrix, a third initial matrix, a fourth initial matrix, and a fifth initial matrix;

[0014] Comparing the historical state and the current state of the first initial matrix corresponding to the terminal data and the second initial matrix corresponding to the position data by the first sub-network and the second sub-network, respectively, to obtain the terminal anomaly perception vector and the position anomaly perception vector corresponding to the terminal data and the position data, respectively;

[0015] Calculating, by a third sub-network, the correlation between the time series and the absolute value in the third initial matrix corresponding to the communication data and the abnormal behavior to obtain a communication anomaly perception vector corresponding to the communication data;

[0016] Calculating the correlation between each application and the number of accesses and abnormal behavior in the fourth initial matrix corresponding to the network data through the fourth sub-network to obtain a network anomaly perception vector for the network data pair;

[0017] The image features in the fifth initial matrix corresponding to the facial image data are extracted through the fifth sub-network and the image features are calculated to obtain the facial anomaly perception vector corresponding to the facial image data.

[0018] Optionally, the step of comparing a historical state and a current state of a first initial matrix corresponding to the terminal data and a second initial matrix corresponding to the position data by using the first subnetwork and the second subnetwork, and obtaining a terminal anomaly perception vector and a position anomaly perception vector corresponding to the terminal data and the position data, respectively, includes:

[0019] Perform word vectorization on the first initial matrix and the second initial matrix respectively to obtain corresponding two-dimensional matrices;

[0020] Inputting each of the two-dimensional matrices into a preset low-level feature calculation network to obtain a plurality of feature maps corresponding to the first initial matrix and the second initial matrix respectively;

[0021] Each of the feature maps is input into a preset fully connected network to obtain a terminal anomaly perception vector and a position anomaly perception vector.

[0022] Optionally, the multimodal anomaly risk measurement deep network includes at least a forward discriminant network, a reverse discriminant network, and a forward and reverse competitive decision network, and the risk measurement vector includes at least a forward discriminant score, a reverse discriminant score, and an anomaly risk probability, wherein the forward discriminant network and the reverse discriminant network are trained using samples consisting of normal users and samples consisting of abnormal users, respectively;

[0023] The step of inputting the multimodal anomaly perception data matrix into a preset multimodal anomaly risk measurement deep network to generate a risk measurement vector includes:

[0024] Inputting the multimodal anomaly perception data matrix into the forward discriminant network and the reverse discriminant network respectively to obtain a forward feature map, a reverse feature map, a forward discriminant score, and a reverse discriminant score;

[0025] Inputting the forward feature map and the reverse feature map into a forward and reverse competitive decision network to obtain an abnormal risk probability;

[0026] A risk measurement vector is constructed according to the positive discriminant score, the negative discriminant score, and the abnormal risk probability.

[0027] Optionally, before the step of inputting the multimodal abnormality perception data matrix into a forward discriminant network and a reverse discriminant network respectively to obtain a forward feature map, a reverse feature map, a forward discriminant score, and a reverse discriminant score, the method further includes:

[0028] The multimodal anomaly perception data matrix corresponding to the training sample set is divided into blocks and converted into a corresponding linear embedding sequence;

[0029] Inputting the linear embedding sequence into three preset linear transformation matrices respectively to obtain corresponding target matrices;

[0030] Calculating a representation vector of the multimodal anomaly perception data matrix according to each of the target matrices;

[0031] Optimizing the classification loss of the representation vector according to a preset target optimization network to obtain parameters corresponding to the forward discriminant network and the reverse discriminant network respectively;

[0032] Based on the parameters corresponding to the forward discriminant network and the reverse discriminant network, a forward discriminant network, a reverse discriminant network and a forward and reverse competitive decision network are constructed.

[0033] Optionally, the step of determining the dynamic threshold according to a preset fuzzy calculation rule, the risk measurement vector and a preset reference threshold includes:

[0034] Performing a grade quantification calculation on the risk measurement vector to obtain the risk grades corresponding to the positive discriminant score, the negative discriminant score, and the abnormal risk probability in the risk measurement vector;

[0035] Calculating a fuzzy function calculation result of the risk measurement vector according to a preset fuzzy calculation rule;

[0036] Determining a risk degree score based on the risk measurement vector and a corresponding fuzzy function calculation result;

[0037] A dynamic threshold is calculated based on the risk level score and a preset baseline threshold.

[0038] Optionally, the face attack detection model includes at least a visible light silent biopsy model, a dazzling color biopsy model, a dazzling pupil biopsy model, a moiré detection model, a screen border detection model, and a photo detection model;

[0039] The step of selecting multiple face attack detection models according to the dynamic threshold to detect the face image data and obtaining the attack probability corresponding to each face attack detection model comprises:

[0040] Determining a risk level according to the dynamic threshold and the preset threshold, wherein the risk level includes at least a first risk level, a second risk level, and a third risk level, the second risk level being higher than the first risk level and lower than the third risk level;

[0041] If the risk level is the first risk level, selecting a first preset number of face attack detection models to detect the face image data in the multimodal data, and obtaining the attack probability corresponding to each of the face attack detection models;

[0042] If the risk level is the second risk level, selecting a second preset number of face attack detection models to detect the face image data in the multimodal data to obtain the attack probability corresponding to each of the face attack detection models;

[0043] If the risk level is the third risk level, a third preset number of face attack detection models are selected to detect the facial image data in the multimodal data to obtain the attack probability corresponding to each of the face attack detection models, wherein the second preset number is greater than the first preset number and less than the third preset number.

[0044] Optionally, the step of determining the attack detection result based on the attack probability corresponding to each of the face attack detection models and the dynamic threshold comprises:

[0045] If the attack probabilities corresponding to the face attack detection models are all lower than the dynamic threshold, the attack detection result is determined to be normal;

[0046] Otherwise, the attack detection result is determined to be abnormal.

[0047] The present application also provides a multimodal face attack detection device, which is applied to a multimodal face attack detection device. The multimodal face attack detection device includes:

[0048] The abnormality perception matrix generation module is used to input the initial matrices corresponding to the multimodal data of the mobile Internet user's cloud network terminal behavior data and facial image data into the preset abnormality perception matrix generation deep network to obtain the multimodal abnormality perception data matrix;

[0049] A risk measurement vector determination module is used to input the multimodal anomaly perception data matrix into a preset multimodal anomaly risk measurement deep network to generate a risk measurement vector;

[0050] A dynamic threshold determination module, configured to determine a dynamic threshold based on a preset fuzzy calculation rule, the risk measurement vector, and a preset reference threshold;

[0051] a multi-model detection module, configured to select a plurality of face attack detection models according to the dynamic threshold to detect the face image data, and obtain attack probabilities corresponding to the respective face attack detection models;

[0052] The detection result determination module is used to determine the attack detection result based on the attack probability corresponding to each of the face attack detection models and the dynamic threshold.

[0053] The present application also provides an electronic device, which is a physical device, comprising: a memory, a processor, and a program of the multimodal face attack detection method stored in the memory and executable on the processor. When the program of the multimodal face attack detection method is executed by the processor, the steps of the multimodal face attack detection method described above can be implemented.

[0054] The present application also provides a computer-readable storage medium, on which is stored a program for implementing a multimodal face attack detection method. When the program of the multimodal face attack detection method is executed by a processor, the steps of the multimodal face attack detection method as described above are implemented.

[0055] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multimodal face attack detection method as described above.

[0056] The present application provides a multimodal face attack detection method, device, electronic device and storage medium. First, the initial matrices corresponding to the multimodal data of the mobile Internet user's cloud network terminal behavior data and facial image data are input into a preset abnormality perception matrix generation deep network to obtain a multimodal abnormality perception data matrix. The technical solution of the present application combines the user's cloud network terminal behavior data and the facial image data collected by the image sensor to form multimodal data as a new data discrimination dimension, thereby realizing the corresponding cloud network terminal behavior data in the mobile scene as a broader data field outside the existing biometric dimension, becoming a superimposed new data discrimination dimension, overcoming the technical defect that it is difficult to collect multi-dimensional data due to insufficient sensors on mobile devices. The multimodal abnormality perception data matrix is ​​then input into a preset multimodal abnormality risk measurement deep network to generate a risk measurement vector. The risk measurement vector is used to characterize the risk factors corresponding to each modal data. Then, according to the preset fuzzy calculation rule, the risk measurement vector and the preset benchmark threshold, a dynamic threshold is determined, and then multiple face attack detection models are selected according to the dynamic threshold to detect the human face. The face attack detection model is used to detect the face image data and obtain the attack probability corresponding to each face attack detection model. The technical solution of the present application can flexibly select different numbers of face attack detection models according to different dynamic thresholds to detect the face image data, which can not only ensure accurate detection under different risk levels, but also ensure computational efficiency. Finally, based on the attack probability corresponding to each face attack detection model and the dynamic threshold, the attack detection result is determined. In summary, the technical solution of the present application solves the technical problem of multi-dimensional data collection difficulties in multimodal face attack detection schemes in mobile scenarios by using the cloud network behavior data of mobile Internet users as a component of multimodal data. At the same time, it overcomes the defect of insufficient single visual feature information, realizes the creation of a complete chain of evidence for face attack detection, and combines the dynamic threshold adaptive calculation method to improve the credibility of face attack detection. At the same time, due to the proposed strategy adaptive adjustment scheme for selecting different numbers of face attack models for face attack detection based on multimodal fusion dynamic thresholds, a balance is achieved between the accuracy and complexity of face attack detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0059] Figure 1 This is a flowchart of the first embodiment of the multimodal face attack detection method of the present application;

[0060] Figure 2 Schematic diagram of the process of constructing a multimodal abnormal perception feature matrix in the multimodal face attack detection method of this application;

[0061] Figure 3 This is a schematic diagram of the multimodal anomaly perception feature matrix deep network in the multimodal face attack detection method of this application;

[0062] Figure 4 This is a neural network diagram of the first sub-network and the second sub-network in the multimodal face attack detection method of this application;

[0063] Figure 5 Schematic diagram of the multimodal anomaly risk measurement deep network in the multimodal face attack detection method of this application;

[0064] Figure 6 This is a schematic diagram of the fuzzy calculation rules in the multimodal face attack detection method of this application;

[0065] Figure 7 Schematic diagram of the structure of a multimodal face attack detection device in an embodiment of the present application;

[0066] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the multimodal face attack detection method in the embodiment of the present application.

[0067] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0068] To make the above-mentioned purposes, features, and advantages of the present application more clearly understood, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making any creative work are within the scope of protection of this application.

[0069] Example 1

[0070] Facial attacks by criminals during facial recognition are a common problem. Current facial attack detection methods and techniques are relatively comprehensive. However, with the continuous emergence and upgrading of new attack methods, especially in mobile scenarios, attacks are becoming more complex, diverse, unpredictable, and difficult to defend against. Cases of existing attack detection technologies being breached are common. For example, with the recent development of AI technology, the attack methods used by the black market have become more sophisticated. Virtual mobile internet users have become a hotbed of video forgery attacks, seriously endangering users' property and even personal safety. The facial recognition industry has reached a consensus that relying solely on image data for facial attack detection has many shortcomings, including limited information and representation dimensions, insufficient generalization, and significant variations in background environments, lighting conditions, and recording devices. Furthermore, due to limitations in the accumulation of new attack data and the time required to train models, it is difficult to meet the industry's demand for trusted facial recognition. To this end, multimodal biometric authentication technologies that fuse face, iris, fingerprint, and voiceprint have emerged in recent years. However, these technologies also have drawbacks, such as the high cost and difficulty of collecting multiple biometric images in mobile scenarios, the expensive hardware equipment, the lack of mobile devices, and the demanding environment for collection. Furthermore, these multimodal image fusion technologies require specialized sensors, such as depth, near-infrared, and light-field cameras, to capture data. These devices are relatively expensive, and many users lack such sensors on their mobile devices, making such technologies difficult to apply in mobile scenarios. In summary, these traditional multimodal fusion detection technologies are unsuitable for face attack detection and facial identity authentication on mobile devices, whether due to high equipment costs, difficulty in obtaining collection conditions, or limited improvements in reliability and accuracy.

[0071] In summary, in the face of the rapid offensive of the black industry, the industry pain points and difficult problems that are difficult to resist and easy to break, there is an urgent need to explore the expansion of existing data dimensions in mobile scenarios to a broader data field beyond the above-mentioned biometric features, thereby superimposing new data discrimination dimensions associated with identity for collaborative defense, and realizing the ability of trusted face authentication in mobile scenarios. The technical solution of the embodiment of the present application introduces a new data discrimination dimension through the cloud network terminal, and it is multimodal data that is low-cost and easy to obtain. By fusing multimodal data with visual features, a complete chain of evidence is provided to realize the trusted face authentication capability of the multimodal cross-domain fusion model based on the new data dimension. Among them, based on the rich cloud network terminal multimodal user behavior data resources of mobile Internet users, including: network data, communication data, location data, and terminal data, it can support the trusted face authentication technology that integrates existing visual modalities.

[0072] The present application provides a multimodal face attack detection method. In the first embodiment of the multimodal face attack detection method of the present application, referring to Figure 1, the multimodal face attack detection method includes:

[0073] Step S10: Inputting the initial matrices corresponding to the multimodal data of the mobile Internet user's cloud-network-end behavior data and facial image data into a preset abnormality perception matrix generation deep network to obtain a multimodal abnormality perception data matrix;

[0074] Step S20, inputting the multimodal anomaly perception data matrix into a preset multimodal anomaly risk measurement deep network to generate a risk measurement vector;

[0075] Step S30, determining a dynamic threshold value according to a preset fuzzy calculation rule, the risk measurement vector, and a preset reference threshold value;

[0076] Step S40, selecting multiple face attack detection models according to the dynamic threshold to detect the face image data, and obtaining attack probabilities corresponding to the respective face attack detection models;

[0077] Step S50: determining an attack detection result based on the attack probability corresponding to each of the face attack detection models and the dynamic threshold.

[0078] In the embodiment of the present application, it should be noted that the embodiment of the present application is a face attack detection method involving cloud-network-end collaboration, face recognition, face attack detection, and multimodal information fusion technologies, which can be applied to face attack detection in mobile scenarios to deal with various fake face attack methods and provide attack detection results with accuracy that meets the requirements. Among them, the mobile device is equipped with an image sensor, such as a camera on a mobile phone or a laptop computer. When these mobile devices are in a face authentication scenario, that is, when face attack detection is required, the user's cloud-network-end behavior data and face image data can be obtained. Among them, the cloud-network-end behavior data can include terminal data, location data, communication data, and network data, etc., which are all modal data that are relatively easy to obtain from the cloud-network end corresponding to the mobile device.

[0079] In addition, the initial matrix is ​​used to characterize the characteristics of cloud network terminal behavior data and facial image data, and the abnormal perception matrix generates a deep network for converting the five-dimensional data modalities of face, terminal, network, communication and location from complex heterogeneous, ultra-high-dimensional, discrete, sparse, weakly correlated and semantically misaligned data states to generate low-dimensional, dense and semantically aligned multi-modal abnormal perception vectors. Each of the modal abnormal perception vectors has the ability to quantify and perceive user risks in real time, which specifically includes source data preprocessing, vector extraction neural network, target optimization loss function design, etc., and further constructs a multi-modal abnormal perception matrix as the input matrix of the subsequent abnormal risk perception model to calculate the abnormal risk probability of the current user, so as to solve the problem of deep fusion of multiple heterogeneous and semantically misaligned data. The multi-modal abnormal perception data matrix includes multiple modal abnormal perception vectors.

[0080] Among them, the multimodal anomaly risk measurement deep network is used to calculate the abnormal risk probability of the current customer based on the input multimodal anomaly perception data matrix, and to construct an abnormal risk measurement vector by combining the forward discriminant score and reverse discriminant score calculated simultaneously by the network to enhance the reliability of risk grading judgment in the next step.

[0081] The preset fuzzy calculation rules, the risk metric vector, and the preset baseline threshold in the embodiment of the present application are used to adaptively calculate an appropriate dynamic threshold based on the quantified risk status corresponding to each user. That is, high risk corresponds to a strict threshold to increase the attack threshold, while general users correspond to a normal threshold to ensure the pass rate, thereby solving the problem of the difficult balance between the pass rate and the interception rate and achieving reliable and accurate attack judgments. In addition, the multiple face attack detection models therein can include various forms of currently mature face attack detection models. The higher the dynamic threshold, the more face attack detection models can be selected for detection. The lower the dynamic threshold, the fewer face attack models can be selected for detection. In this way, the face attack detection strategy is dynamically adjusted for users with different dynamic thresholds, which has the technical effect of saving detection time for low-risk users and improving detection accuracy for high-risk users. Finally, the attack probability corresponding to the collected face image data is compared with the dynamic threshold based on each selected face attack model to determine the attack detection result.

[0082] As an example, steps S10 to S50 include: when a face attack detection instruction is triggered, facial image data of the target user is collected through an image sensor (such as a camera) on a mobile device, and cloud network behavior data of the mobile Internet user is obtained; the multimodal data of the facial image data and the cloud network behavior data are pre-processed to obtain initial matrices corresponding to the multimodal data; the initial matrix is ​​input into a pre-trained abnormality perception matrix generation deep network, and the sub-models corresponding to the multiple modal data in the abnormality perception matrix generation deep network are used to perform feature extraction and abnormality perception on each of the initial matrices to obtain a multimodal abnormality perception data matrix corresponding to the multimodal data and the facial image data; the multimodal abnormality perception data matrix is ​​input into a preset multimodal abnormality risk measurement deep network, and the multimodal abnormality perception data matrix is ​​calculated respectively by the forward discriminant network, the reverse discriminant network and the forward and reverse competitive decision network in the multimodal abnormality risk measurement deep network, and the forward discriminant score, the reverse discriminant score and the reverse discriminant score are output. The method comprises the following steps: first, determining a positive discrimination score and a negative discrimination score and an abnormal risk probability; constructing a risk measurement vector based on the positive discrimination score, the negative discrimination score and the abnormal risk probability; processing the positive discrimination score, the negative discrimination score and the abnormal risk probability in the risk measurement vector by a preset fuzzy calculation rule to obtain a fuzzy function calculation result; calculating a risk degree score according to the fuzzy function calculation result and a preset dynamic threshold calculation formula; taking the risk degree score as a weight, and calculating the product of the risk degree score and the preset baseline threshold to obtain the dynamic threshold of the current target user; selecting a corresponding number of face attack models to detect the collected face image data according to the size relationship between the dynamic threshold and the preset first preset threshold and the second preset threshold, and obtaining the attack probability corresponding to each face attack detection model; if the attack probability corresponding to each face attack detection model is lower than the dynamic threshold, then determining that the attack detection result is normal; if there is an attack highlight detected by a face attack detection model that is higher than the dynamic threshold, then determining that the attack detection result is abnormal.

[0083] Since the existing technology only uses the detection technology of facial visual features, there is a problem of insufficient information required for attack discrimination, or it uses multimodal detection technology such as multiple biometric features, which has problems such as high collection cost, difficulty in collection, and lack of collection equipment, and is not suitable for mobile terminals. The above two situations will lead to the existing technology's lack of credibility, generalization, and accuracy in mobile terminal face attack detection. The beneficial effects of the embodiments of the present application include: compared with the existing technology, the face attack detection method of the cloud-network-end multimodal fusion of face-terminal-network-communication-location proposed in the embodiments of the present application are all data that are highly relevant to the user's normal / abnormal conditions. Deep fusion can provide sufficient and complete evidence chain for attack discrimination, and the collection cost is low, the collection is easy to implement, and no special collection equipment is required. It is very suitable for mobile terminal scenarios to achieve credibility, generalization, and accuracy of face attack detection.

[0084] Furthermore, the step of inputting the initial matrices corresponding to the multimodal data of the mobile internet user's cloud-network terminal behavior data and facial image data into a preset abnormality perception matrix generation deep network to obtain the multimodal abnormality perception data matrix includes:

[0085] Step S11, preprocessing the collected multimodal data to generate initial matrices corresponding to each modal data, wherein the initial matrices include at least a first initial matrix, a second initial matrix, a third initial matrix, a fourth initial matrix, and a fifth initial matrix;

[0086] Step S12: Comparing the historical state and the current state of the first initial matrix corresponding to the terminal data and the second initial matrix corresponding to the position data by the first sub-network and the second sub-network, respectively, to obtain a terminal anomaly perception vector and a position anomaly perception vector corresponding to the terminal data and the position data, respectively;

[0087] Step S13, calculating the correlation between the time series and absolute values ​​in the third initial matrix corresponding to the communication data and the abnormal behavior through the third sub-network to obtain a communication anomaly perception vector corresponding to the communication data;

[0088] Step S14, calculating the correlation between each application and the number of accesses and abnormal behavior in the fourth initial matrix corresponding to the network data through the fourth sub-network to obtain a network anomaly perception vector of the network data pair;

[0089] Step S15: extracting image features in a fifth initial matrix corresponding to the facial image data through a fifth sub-network and calculating the image features to obtain a facial anomaly perception vector corresponding to the facial image data.

[0090] In the embodiment of the present application, it should be noted that the multimodal data includes at least terminal data, location data, communication data, network data and face image data, the abnormality perception matrix generation deep network includes at least a first subnetwork, a second subnetwork, a third subnetwork, a fourth subnetwork and a fifth subnetwork, and the multimodal abnormality perception data matrix includes at least a terminal abnormality perception vector, a location abnormality perception vector, a communication abnormality perception vector, a network abnormality perception vector and a face abnormality perception vector.

[0091] Reference Figure 2 The embodiment of the present application provides a method for constructing a multimodal abnormality perception feature matrix, which mainly addresses the problems of heterogeneity, semantic misalignment, ultra-high dimension, sparseness, discreteness, lack of association, and different modalities of the five types of original data: face, terminal, location, network, and communication. The embodiment of the present application proposes an abnormality perception matrix generation network for face-terminal-network-communication-location multimodal information, as follows Figure 2 As shown, a technical principle diagram of a method for constructing a multimodal anomaly perception feature matrix is ​​given. In the feature matrix construction principle section, the embodiment of the present application first inputs multimodal original information, preprocesses the disordered source data, and then generates low-dimensional, dense, and semantically aligned five modal anomaly perception vectors through an anomaly perception neural network composed of five sub-networks (terminal vector quantum model, position vector quantum model, communication vector quantum model, network vector quantum model, and face vector quantum model). In the network, a target optimization loss function for normal / abnormal classification is designed to achieve iterative training and learning of each sub-network. Then, based on the five anomaly perception vectors obtained through training, a multimodal anomaly perception matrix is ​​further constructed as the input matrix of the subsequent multimodal anomaly risk perception model to calculate the abnormal risk probability of the current user.

[0092] In a feasible embodiment, the steps of preprocessing the collected multimodal data to generate initial matrices corresponding to the respective modal data, wherein the initial matrices include at least a first initial matrix, a second initial matrix, a third initial matrix, a fourth initial matrix, and a fifth initial matrix, include:

[0093] Input phase. Multimodal information primarily comes from telecom users' terminal data, location data, communication data, network data, and facial image data collected when users conduct business. Before entering the model, each of these data undergoes pre-modeling data processing, as shown in steps ①, ②, ③, ④, and ⑤ below.

[0094] ① User terminal data is mainly divided into historical user data and current user real-time data. Historical data includes device model, operating system, app version, camera parameters, and various sensor data. Current user real-time data includes two categories: the first category includes the current device model, current operating system, current network status, current various sensor data, current app version, root risk, hook risk, and simulator risk. The second category includes app behavior data, such as page views and clicks.

[0095] The principle of neural network design for user terminal data feature extraction is to achieve consistency or correlation comparison between historical state and current state. The corresponding input data is designed and organized into a 2×m-dimensional matrix, denoted as: T,

[0096] ;

[0097] The first line is the historical data, and the second line is the current status.

[0098] ② User location data, like terminal data, is divided into historical data and current data. Taking location data as an example, it can also be divided into historical user data and current real-time user data. Historical data includes: historical trajectory, permanent residence, active locations, stays at special venues, and number ownership; current data includes: current IP location, current GPS location, etc. The design principle of the neural network for feature extraction of user location data is also to achieve consistency or correlation comparison between historical and current states. The corresponding input data is also designed as a 2×m-dimensional matrix.

[0099] Denoted as P,

[0100] ;

[0101] The first row contains historical location data, while the second row contains current location status. Location data is a temporal data type. Therefore, before constructing the input matrix, an algorithm must be used to pre-calculate the user's historical trajectory, permanent locations, active locations, and special venues.

[0102] ③ For user communication data, such as call circle, number of calls (calling), number of calls (called), call duration, and industry SMS data, which belong to the s data type with time attributes, the principle of the neural network design for feature extraction is to calculate the correlation between each communication data dimension and abnormal behavior in terms of time series and absolute value. The corresponding input data is designed as an n×12-dimensional matrix, denoted as C.

[0103] ;

[0104] Among them, each line represents a communication dimension and corresponds to 12 months of data, and n communication data dimensions are designed.

[0105] ④ For user network data, such as websites on various negative lists, the principle of neural network design for feature extraction is to calculate the correlation between each APP and the number of visits in the abnormal APP list and abnormal behavior. The corresponding input data is designed as a 2×k-dimensional matrix, denoted as W

[0106] ;

[0107] The first row is for each abnormal APP, and the second row is for the cumulative number of visits within a specified time period corresponding to each APP. For example, the time period can be defined as 1 year or half a year.

[0108] ⑤ For facial image visual data, the three-channel color matrix itself is directly used as the input to the model, denoted as Face

[0109] ;

[0110] like Figure 3 As shown, for ①, ②, ③, ④, and ⑤, several different types of input matrices are given before each source data enters the model, as well as neural network designs for different targets. The overall model can be subdivided into 4 types of sub-models. The first sub-network and the second sub-network are the same type of neural networks, which are mainly used to achieve consistency or correlation comparison between historical states and current states; sub-model 3 belongs to the second type of neural network, which calculates the correlation between each communication data dimension and abnormal behavior in terms of time series and absolute values; sub-model 4 belongs to the third type of neural network, which calculates the correlation between each APP and the number of visits and abnormal behavior; sub-model 5 belongs to the fourth type of neural network, which is a network used for facial visual feature extraction, such as a convolutional neural network.

[0111] Specifically, refer to Figure 3 Each sub-model includes a low-level feature calculation network, a high-level feature extraction network, and a target optimization network, which are respectively used to calculate the low-level features in the initial matrix and extract the high-level features in the initial matrix, that is, output the perception vector. The target optimization network also includes a target loss function, which is used to train and construct each sub-model before use and iteratively optimize during use.

[0112] As an example, steps S11 to S15 include: organizing the collected original data of five modes, namely, face, terminal, location, network, and communication, in the form of a matrix to obtain the initial matrix corresponding to each modal data; calculating the low-level features of the historical state and the current state in the first initial matrix corresponding to the terminal data through the low-level feature calculation module in the first sub-network to obtain the corresponding first feature map; calculating the first feature map through the high-level feature extraction module in the first sub-network to obtain the terminal abnormality perception vector; calculating the low-level features of the historical state and the current state in the second initial matrix corresponding to the location data through the low-level feature calculation module in the second sub-network to obtain the corresponding second feature map; calculating the second feature map through the high-level feature extraction module in the second sub-network to obtain the location abnormality perception vector; calculating the low-level feature calculation module in the third sub-network to obtain the location abnormality perception vector. The block calculates the correlation between the time series and absolute values ​​in the third initial matrix corresponding to the communication data and the abnormal behavior to obtain a third feature map corresponding to the communication data; the high-level feature extraction module in the third sub-network calculates the third feature map to obtain a communication anomaly perception vector; the low-level feature calculation module in the fourth sub-network calculates the correlation between each application and the number of accesses in the fourth initial matrix corresponding to the network data and the abnormal behavior to obtain a corresponding fourth feature map; the high-level feature extraction module in the fourth sub-network calculates the fourth feature map to obtain a network anomaly perception vector; the low-level feature calculation module in the fifth sub-network extracts the image features in the fifth initial matrix corresponding to the facial image data; the high-level feature extraction module in the fifth sub-network calculates the image features to obtain a facial anomaly perception vector.

[0113] Furthermore, the step of comparing the historical state and the current state of the first initial matrix corresponding to the terminal data and the second initial matrix corresponding to the position data by the first sub-network and the second sub-network, and obtaining the terminal anomaly perception vector and the position anomaly perception vector corresponding to the terminal data and the position data respectively includes:

[0114] Step S121, performing word vectorization on the first initial matrix and the second initial matrix respectively to obtain corresponding two-dimensional matrices;

[0115] Step S122: inputting each of the two-dimensional matrices into a preset low-level feature calculation network to obtain a plurality of feature maps corresponding to the first initial matrix and the second initial matrix respectively;

[0116] In step S123, each of the feature maps is input into a preset fully connected network to obtain a terminal anomaly perception vector and a position anomaly perception vector.

[0117] In the embodiment of the present application, it should be noted that, before step S12, the steps of pre-processing the terminal data and the location data respectively include: performing data processing on the original data before inputting into the model to obtain the input data matrix, which is recorded as:

[0118] ;

[0119] in represents the historical data of the terminal's i-th dimension, and Represents the current data of the terminal's i-th dimension.

[0120] As an example, see Figure 5 , steps S121 to S123 include: constructing two two-dimensional matrices, namely, two two-dimensional matrices T1 and T2 constructed by word vectorization of the first initial matrix and the second initial matrix, i.e., two vectors [t11, t12, ⋯, t1m] and [t21, t22, ⋯, t2m]; Figure 4 As shown, d represents the dimension of the word vector and m represents the dimension of the terminal data or location data. For the two two-dimensional matrices T1 and T2 designed above as input, the corresponding neural network is designed to represent the correlation between the two. The first is the low-level feature calculation network, such as Figure 5 As shown in the figure, it is composed of a 64-channel 7×7 convolutional layer combined with 4 residual blocks; and the feature map output by the low-level feature calculation network, for example: 512 7×7 feature maps, are then calculated through the fully connected networks FC1 and FC2 to obtain high-level dense features, and the high-level dense features are used as the extracted terminal anomaly perception vector and position anomaly perception vector.

[0121] In addition, for the design of the target optimization network in each sub-model, the embodiment of this application uses two types of user labels, normal users and abnormal users, to construct the loss function calculation of the optimization target. The five sub-models all use the same loss function calculation method. Taking the terminal class feature vector as an example, the feature vector output by the high-level feature extraction network is: , as the input of the target optimization network, where the loss function of the target optimization is calculated as follows:

[0122] ;

[0123] After the training, the neural network generates 5 corresponding abnormal perception vectors for the 5 modal input data. 、…、 Further construct a multimodal anomaly perception matrix:

[0124] ;

[0125] Compared with the existing technical means, the face-terminal-network-communication-position multimodal anomaly perception matrix generation method proposed in the technical solution of the embodiment of the present application not only cleverly designs the overall fusion idea and framework, but also subdivides and designs 4 different types of neural network sub-models according to the characteristics of each modal data. Combined with the target optimization network design, it successfully realizes the generation of low-dimensional, dense and semantically aligned 5 modal anomaly perception vectors from complex heterogeneous, ultra-high-dimensional, discrete, sparse, weakly correlated, and semantically misaligned multi-data, and then constructs a multimodal matrix of anomaly perception.

[0126] In addition, in step S20, the multimodal anomaly risk measurement deep network includes at least a forward discriminant network, a reverse discriminant network, and a forward and reverse competitive decision network, and the risk measurement vector includes at least a forward discriminant score, a reverse discriminant score, and an abnormal risk probability, wherein the forward discriminant network and the reverse discriminant network are trained using samples consisting of normal users and samples consisting of abnormal users, respectively;

[0127] The step of inputting the multimodal anomaly perception data matrix into a preset multimodal anomaly risk measurement deep network to generate a risk measurement vector includes:

[0128] Step S21, inputting the multimodal abnormality perception data matrix into the forward discriminant network and the reverse discriminant network respectively to obtain a forward feature map, a reverse feature map, a forward discriminant score, and a reverse discriminant score;

[0129] Step S22, inputting the forward feature graph and the reverse feature graph into a forward and reverse competitive decision network to obtain an abnormal risk probability;

[0130] Step S23: constructing a risk measurement vector according to the positive discriminant score, the negative discriminant score, and the abnormal risk probability.

[0131] In the embodiment of the present application, it should be noted that the multimodal abnormal risk measurement deep network in the embodiment of the present application is subdivided into two parts: the first part includes two independent discriminant networks, forward and reverse, which are trained in advance; the second part jointly designs a forward and reverse competitive decision network and a target optimization function for the two discriminant networks trained in the first part, thereby constructing an abnormal risk measurement deep network to generate a risk measurement vector, wherein the risk measurement vector includes: abnormal risk probability, forward discriminant score, and reverse discriminant score.

[0132] Reference Figure 5The multimodal anomaly risk measurement deep network in the embodiment of the present application is composed of four sub-networks, namely a forward discriminant network, a reverse discriminant network, a forward and reverse competitive decision network, and a target optimization network. In the above-mentioned multimodal anomaly risk measurement deep network, the forward discriminant network C1, the reverse discriminant network C2, and the forward and reverse competitive decision network C3 jointly build the anomaly risk measurement network. The multimodal anomaly perception feature matrix is ​​the initial input. The forward discriminant network C1 is used to calculate and output the forward feature map of the current sample. The reverse discriminant network C2 is used to calculate and output the reverse feature map of the current sample. The forward and reverse feature maps are used as inputs to the network C3. The feature map after the forward and reverse competitive decision is calculated by the network C3. Finally, through the target optimization network, the estimated result is compared with the actual result by using the classification loss to calculate the loss function, where the target is defined as {normal user: 0; abnormal user: 1}. Through the target optimization gradient calculation, on the one hand, the parameters of the forward discriminant network and the reverse discriminant network are continuously adjusted, and on the other hand, the parameters of the C3 network are trained. The target optimization network can be used to train each network in advance, or it can be optimized in real time according to the actual results during the process of risk measurement vector. Finally, the output forward feature map and reverse feature map are processed by the forward and reverse competitive decision network in the multimodal abnormal risk measurement network to calculate the abnormal risk probability of the current sample, which is recorded as , in addition, the network also outputs a positive discriminant score , reverse discriminant score To enhance the reliability of risk classification, the risk measurement vector is composed of three parts, namely, abnormal risk probability , positive discriminant score and reverse discriminant score ,Right now:

[0133] ;

[0134] Among them, Vec is the risk measurement vector.

[0135] As an example, steps S21 to S23 include: inputting the multimodal abnormality perception data matrix into a forward discriminant network, calculating and predicting the multimodal abnormality perception data matrix through the forward discriminant network, and obtaining a corresponding forward feature map and a forward discriminant score, wherein the forward discriminant score is used to characterize the degree of matching between the target user corresponding to the multimodal abnormality perception data matrix and the normal user in face attack detection; inputting the multimodal abnormality perception data matrix into a reverse discriminant network, calculating and predicting the multimodal abnormality perception data through the reverse discriminant network, and obtaining a corresponding reverse feature map and a reverse discriminant score, wherein the reverse discriminant score is used to characterize the degree of matching between the target user corresponding to the multimodal abnormality perception data matrix and the abnormal user in face attack detection; inputting the forward feature map and the reverse feature map into a forward and reverse competitive decision network, and predicting the forward feature map and the reverse feature map through the forward and reverse decision network to obtain an abnormal risk probability; aggregating the forward discriminant score, the reverse discriminant score, and the abnormal risk probability to obtain a risk measurement vector.

[0136] In addition, it should be noted that before the step of inputting the multimodal abnormality perception data matrix into the forward discriminant network and the reverse discriminant network respectively to obtain the forward feature map, the reverse feature map, the forward discriminant score and the reverse discriminant score, the method further includes:

[0137] Step A10: performing matrix element block processing on the multimodal anomaly perception data matrix corresponding to the training sample set and converting it into a corresponding linear embedding sequence;

[0138] Step A20: inputting the linear embedding sequence into three preset linear transformation matrices respectively to obtain corresponding target matrices;

[0139] Step A30, calculating a representation vector of the multimodal abnormality perception data matrix according to each of the target matrices;

[0140] Step A40, performing classification loss optimization on the representation vector according to a preset target optimization network to obtain parameters corresponding to the forward discriminant network and the reverse discriminant network respectively;

[0141] Step A50: constructing a forward discriminant network, a reverse discriminant network, and a forward and reverse competitive decision network based on the parameters corresponding to the forward discriminant network and the reverse discriminant network, respectively.

[0142] In the embodiment of the present application, it should be noted that the embodiment of the present application provides a method for training and optimizing a multimodal anomaly risk measurement deep network based on a multimodal anomaly perception data matrix. Specifically, the target loss function in the target optimization network is used to calculate the corresponding function loss based on the predicted value (forward discriminant score, reverse discriminant score, and abnormal risk probability) and the actual true value each time a risk measurement vector is generated to train and optimize the parameters of the forward discriminant network, the reverse discriminant network, and the forward and reverse competitive decision network in the multimodal anomaly risk measurement deep network. It should be noted that the training and optimization can be performed both when the multimodal anomaly risk measurement deep network is constructed and when the multimodal anomaly risk measurement deep network is used to generate a risk measurement vector.

[0143] For the multimodal anomaly risk measurement deep network, the embodiment of the present application proposes to construct two discriminant subnetworks, namely the forward discriminant network and the reverse discriminant network, and design two corresponding training sample sets, namely sample set 1 and sample set 2. Sample set 1 is composed entirely of samples of normal users, and sample set 2 is composed entirely of samples of abnormal users.

[0144] As an example, steps A10 to A50 are as follows: performing matrix element block processing on the multimodal anomaly perception matrix corresponding to the training sample set and converting it into a linear embedding sequence, wherein the training sample set includes a normal sample set and an abnormal sample set, and the classification label includes a normal label and an abnormal label. Specifically, the linear embedding sequence is: in, Indicates user classification label, Represents the i-th patch, E represents the mapping vectorization; three linear transformation matrices are used to act on the input linear embedding sequence, namely Q=X*WQ, K=X*WK, L=X*WL, where W is a preset coefficient; according to the target matrices Q, K and L, the representation vector y corresponding to the multimodal anomaly perception matrix corresponding to the training sample set is calculated. Specifically, the formula for calculating the representation vector is as follows:

[0145] ;

[0146] Among them, MLP is a multi-layer perceptron, The representation vector y is optimized with softmax classification loss, and the network is trained to obtain the forward discriminant network parameters and the reverse discriminant network parameters respectively; and then the forward and reverse discriminant networks are trained according to the forward discriminant network parameters and the reverse discriminant network parameters, wherein the representation vector is used to characterize the characteristics of the multimodal abnormal perception matrix, which is a fusion feature vector; wherein the forward discriminant network is trained according to the normal sample set to obtain the network parameter set 1, which is recorded as The reverse discriminant network is trained according to the abnormal sample set to obtain the network parameter set 2, which is recorded as , embed the above two network parameter sets into the abnormal risk measurement deep network, and construct a forward and reverse competitive decision network based on the parameters of the forward discriminant network and the reverse discriminant network. Finally, the model is trained and optimized in conjunction with the forward and reverse competitive decision network and the target optimization network in the embodiment of the present application.

[0147] Furthermore, the step of determining the dynamic threshold according to the preset fuzzy calculation rule, the risk measurement vector and the preset reference threshold includes:

[0148] Step S31, performing a grade quantification calculation on the risk measurement vector to obtain the risk grades corresponding to the positive discriminant score, the negative discriminant score and the abnormal risk probability in the risk measurement vector;

[0149] Step S32, calculating the fuzzy function calculation result of the risk measurement vector according to a preset fuzzy calculation rule;

[0150] Step S33, determining a risk degree score based on the risk measurement vector and the corresponding fuzzy function calculation result;

[0151] Step S34: Calculate a dynamic threshold value based on the risk level score and a preset reference threshold value.

[0152] In the embodiment of the present application, it should be noted that the forward discriminant score, reverse discriminant score and abnormal risk probability in the risk measurement vector need to be graded first, so as to determine the corresponding fuzzy calculation result according to the fuzzy calculation rule. As a feasible embodiment, the forward discriminant score, reverse discriminant score and abnormal risk probability in the risk measurement vector can be graded by presetting a trigonometric function or other function; as another feasible embodiment, a first risk threshold can be preset. If the parameter in the risk measurement vector is lower than the first risk threshold, its grade can be quantified as low risk. The risk measurement vector includes a forward discriminant score, a reverse discriminant score and abnormal risk probability; if the parameter in the risk measurement vector is higher than the first risk threshold, its grade can be quantified as high risk. The grade quantification method in the embodiment of the present application can select a feasible grade quantification method in the prior art, and there is no restriction on the specific grade quantification calculation method.

[0153] The technical solution of the embodiment of the present application dynamically calculates the attack detection threshold through abnormal risk measurement, and adaptively calculates a relatively suitable dynamic threshold for each user according to the quantified risk level, thereby solving the problem of difficult balance between pass rate and interception rate.

[0154] As a feasible embodiment, the fuzzy calculation rule is as follows: Figure 6 Steps S31 to S34 include: quantifying the positive discriminant score, negative discriminant score, and abnormal risk probability in the risk measurement vector to obtain the risk levels corresponding to the positive discriminant score, negative discriminant score, and abnormal risk probability in the risk measurement vector; after determining the risk levels of each parameter in the risk measurement vector, determining the risk level corresponding to the overall risk measurement vector, and then outputting a risk level score based on the risk level corresponding to the overall risk measurement vector. Specifically, based on the three input features described above: the positive discriminant score SP, the negative discriminant score SN, and the abnormal probability RS, N pre-designed fuzzy calculation rules are finally combined to obtain a risk level score through a calculation formula, denoted as SCORE, which is used as a weight coefficient for calculating the dynamic biopsy threshold:

[0155] ;

[0156] in, represents the jth input, Represents the corresponding fuzzy function calculation result. First, calculate the sum of the products of each input and the corresponding fuzzy function calculation result, and then divide it by the fuzzy function calculation result of each input to obtain the risk level score; calculate the actual dynamic threshold corresponding to the current user's risk status. The risk level score SCORE is used as a weight and multiplied by the uniformly set benchmark threshold T to obtain the dynamic threshold of the current user under the current risk status. ,Right now: ; Thus, based on the risk measurement vector as input, the joint calculation algorithm is used to calculate the dynamic threshold according to the actual risk situation , which is used as the dynamic threshold for the subsequent face attack detection model judgment and the quantitative basis for the adaptive adjustment of the dynamic strategy.

[0157] Compared with the existing technology, the technical solution of the embodiment of the present application uses the attack detection model based on facial visual features as the basis for distinguishing facial attacks, while the multimodal data of the cloud network end provides an important chain of evidence to complete the chain of evidence of facial feature clues. By designing an abnormal risk measurement deep network, taking the multimodal abnormal perception matrix of the cloud network end as input, the network outputs an abnormal risk measurement vector, and through the risk degree score fuzzy algorithm designed, the dynamic threshold that meets the current user risk status is finally calculated, which is used for the judgment of the facial attack detection model and serves as a quantitative basis for the adaptive adjustment of the attack detection dynamic strategy. The multimodal fusion method in the embodiment of the present application can not only give play to the advantages of facial visual features as the main feature, but also make up for the problem of insufficient information for attack detection based on a single facial feature by introducing multimodal data of the cloud network end. Moreover, the technical solution of the embodiment of the present application is based on the dynamic threshold calculation method for face attack detection based on abnormal risk measurement. According to the calculation method with the risk measurement vector as input, the actual threshold that can correctly reflect the current risk situation is dynamically calculated. That is, for high-risk users, a more stringent high threshold is automatically adjusted to increase the threshold for risky users to pass, while for general users, a normal threshold is used to ensure the pass rate of normal users. For each user, a suitable dynamic threshold is adaptively calculated based on the quantified risk situation, solving the problem of difficult balance between pass rate and interception rate, and realizing reliable and accurate face attack detection.

[0158] Furthermore, the step of selecting multiple face attack detection models according to the dynamic threshold to detect the face image data and obtaining the attack probability corresponding to each face attack detection model includes:

[0159] Step S41, determining a risk level based on the dynamic threshold and the preset threshold, wherein the risk level includes at least a first risk level, a second risk level, and a third risk level, wherein the second risk level is higher than the first risk level and lower than the third risk level;

[0160] Step S42: If the risk level is the first risk level, a first preset number of face attack detection models are selected to detect the face image data in the multimodal data to obtain the attack probability corresponding to each of the face attack detection models;

[0161] Step S43: If the risk level is the second risk level, a second preset number of face attack detection models are selected to detect the face image data in the multimodal data to obtain the attack probability corresponding to each of the face attack detection models;

[0162] In step S44, if the risk level is the third risk level, a third preset number of face attack detection models are selected to detect the facial image data in the multimodal data to obtain the attack probability corresponding to each face attack detection model, wherein the second preset number is greater than the first preset number and less than the third preset number.

[0163] In the embodiment of the present application, it should be noted that the multiple face attack models to be selected may include visible light silent biopsy model, colorful biopsy model, dazzling pupil biopsy model, moiré detection model, screen border detection model and photo detection model and other technically mature face attack detection models, which can be detected based on the collected face image data to determine the corresponding attack probability. The embodiment of the present application proposes dynamic strategy adaptive adjustment, that is, according to the dynamic threshold and risk level score calculated in the previous step, dynamically adjust the attack detection strategy and complexity of multiple factors. Among them, the preset threshold includes the first preset threshold and a second preset threshold The first risk level is general risk, the second risk level is relatively high risk, and the third risk level is high risk.

[0164] As an example, steps S41 to S44 include: when the dynamic threshold is less than the first preset threshold, it is a general risk, and 1 to 2 face attack detection models are used to perform face attack detection on the collected face image data to obtain a corresponding attack probability, wherein the response time of the face attack detection is short; when the dynamic threshold is higher than the first preset threshold and lower than the second preset threshold, it is a higher risk, and 2 to 3 face attack detection models are used to perform face attack detection on the face image data to obtain a corresponding attack probability, wherein the face attack detection response time is longer; when the dynamic threshold is higher than the second preset threshold, it is a high risk, and more than 3 face attack detection models are used to perform face attack detection on the face image data to obtain a corresponding attack probability.

[0165] In the embodiments of the present application, experiments show that the accuracy and time response corresponding to "visual feature attack detection only" and "cloud-network multimodal fusion feature attack detection" in the embodiments of the present application are shown in the following table.

[0166]

[0167] Specifically, using only visual features for facial attack detection presents a risk of missed detections due to the limited information provided. For simple attacks in closed scenarios, the accuracy rate is less than or equal to 95%. However, for complex attacks in open environments, especially mobile scenarios, where the attack forms are more complex and diverse, the detection accuracy is even lower, with existing technologies achieving detection accuracy rates below 90%. For scenarios with high security requirements, such as payment, finance, and important business processing, the accuracy of facial attack detection is very high, reaching over 99.9%. By adopting the methods of the embodiments of the present application, cloud-network information supplements the insufficient information provided by single visual features, achieving a complete chain of evidence for facial attack detection. Combined with a dynamic threshold adaptive calculation method, this method improves the performance to a reliable and highly accurate attack detection capability, with an accuracy rate of 99.9% or higher. Furthermore, the proposed dynamic strategy adaptive adjustment technology based on multimodal fusion achieves a balance between detection accuracy and complexity, resulting in a time response that is roughly the same as that of attack detection using only visual features, slightly higher than that of the previous approach, but still maintaining a time response of around 1 second. Since the scenario in which the technical solution of the embodiment of the present application is located requires a very low fault tolerance rate, the test data shows that this method has a significant improvement in the technical effect of face attack detection.

[0168] Since the methods of multimodal image fusion and multi-biometric modality fusion are costly and have high requirements for equipment, mobile terminals such as mobile phones currently do not have the acquisition conditions required for the above two types of multimodal fusion. Some customized machines have them, but the high cost and threshold make their application cases rare. Currently, the vast majority of application scenarios for face identity authentication are mobile terminals, and the attacks on mobile terminal scenarios are complex, with numerous cases of breaches and great challenges. Therefore, the analysis and comparison of application effects are mainly for the two technologies in the table. Unlike the situation where traditional multimodality is not suitable for mobile terminals, the technical solution of the embodiment of the present application is fully suitable for the face identity authentication scenario of mobile terminals.

[0169] Compared with the existing technology, the adaptive adjustment method of the dynamic strategy of liveness detection in the technical solution of the embodiment of the present application realizes the risk level score and dynamic threshold calculated according to the multimodal abnormal perception matrix of the cloud network end, objectively reflects the risk status of the current user, and thus dynamically adjusts the strategy and detection complexity of multiple factors. That is, high-risk status corresponds to a complex multi-factor fusion detection strategy, the detection is more stringent, and the attack threshold is increased; while general risks adopt a relatively lightweight mode to reduce detection complexity, improve detection efficiency, and overall ensure that the average detection efficiency is maintained at a reasonable response time of the production line.

[0170] Furthermore, the step of determining the attack detection result based on the attack probability corresponding to each of the face attack detection models and the dynamic threshold comprises:

[0171] Step S51: If the attack probabilities corresponding to the face attack detection models are all lower than the dynamic threshold, the attack detection result is determined to be normal;

[0172] Step S52: Otherwise, determine that the attack detection result is abnormal.

[0173] In the embodiment of the present application, the attack probability detected by each face attack detection model participating in the face attack detection is compared with the dynamic threshold to determine whether the attack detection result is normal. The attack detection result is determined to be normal only when the attack probabilities detected by all face attack detection models participating in the face attack detection are all lower than the dynamic threshold. This ensures the accuracy of the attack detection result, avoids the risk of malicious attacks in face recognition scenarios due to missed detection, and further improves the security of face recognition scenarios in mobile scenarios.

[0174] The embodiment of the present application provides a multimodal face attack detection method. First, the initial matrices corresponding to the multimodal data of the mobile Internet user's cloud-network behavior data and facial image data are input into a preset abnormality perception matrix generation deep network to obtain a multimodal abnormality perception data matrix. The technical solution of the embodiment of the present application combines the user's cloud-network behavior data and the facial image data collected by the image sensor to form multimodal data as a new data discrimination dimension, thereby realizing that the corresponding cloud-network behavior data in the mobile scene is used as a broader data field outside the existing biometric dimension, becoming a superimposed new data discrimination dimension, overcoming the technical defect that it is difficult to collect multi-dimensional data due to insufficient sensors on mobile devices. The multimodal abnormality perception data matrix is ​​then input into a preset multimodal abnormality risk measurement deep network to generate a risk measurement vector. The risk measurement vector is used to characterize the risk factors corresponding to each modal data. Then, according to the preset fuzzy calculation rule, the risk measurement vector and the preset benchmark threshold, a dynamic threshold is determined, and then multiple face attack detection models are selected according to the dynamic threshold to perform face attack detection on the facial image data. Detection, obtain the attack probability corresponding to each of the face attack detection models. The technical solution of the embodiment of the present application can flexibly select different numbers of face attack detection models according to different dynamic thresholds to detect the face image data, which can not only ensure accurate detection under different risk levels, but also ensure computational efficiency. Finally, based on the attack probability corresponding to each of the face attack detection models and the dynamic threshold, the attack detection result is determined. In summary, the technical solution of the embodiment of the present application solves the technical problem of multi-dimensional data collection difficulties in multimodal face attack detection schemes in mobile scenarios by using the cloud network behavior data of mobile Internet users as a component of multimodal data. At the same time, it overcomes the defect of insufficient single visual feature information, realizes the creation of a complete chain of evidence for face attack detection, and combines the dynamic threshold adaptive calculation method to improve the credibility of face attack detection. At the same time, due to the proposed strategy adaptive adjustment scheme for selecting different numbers of face attack models for face attack detection based on multimodal fusion dynamic thresholds, a balance is achieved between the accuracy and complexity of face attack detection.

[0175] Example 2

[0176] The embodiment of the present application also provides a multimodal face attack detection device, which is applied to a multimodal face attack detection device. Figure 7 , the multimodal face attack detection device comprises:

[0177] The abnormality perception matrix generation module 101 is used to input the initial matrices corresponding to the multimodal data of the mobile Internet user's cloud network terminal behavior data and facial image data into a preset abnormality perception matrix generation deep network to obtain a multimodal abnormality perception data matrix;

[0178] The risk measurement vector determination module 102 is configured to input the multimodal anomaly perception data matrix into a preset multimodal anomaly risk measurement deep network to generate a risk measurement vector;

[0179] A dynamic threshold determination module 103 is configured to determine a dynamic threshold based on a preset fuzzy calculation rule, the risk measurement vector, and a preset reference threshold;

[0180] a multi-model detection module 104 for selecting a plurality of face attack detection models according to the dynamic threshold to detect the face image data and obtaining attack probabilities corresponding to the respective face attack detection models;

[0181] The detection result determination module 105 is configured to determine an attack detection result based on the attack probability corresponding to each of the face attack detection models and the dynamic threshold.

[0182] Optionally, the multimodal data includes at least terminal data, location data, communication data, network data, and face image data; the abnormality perception matrix generation deep network includes at least a first subnetwork, a second subnetwork, a third subnetwork, a fourth subnetwork, and a fifth subnetwork; and the multimodal abnormality perception data matrix includes at least a terminal abnormality perception vector, a location abnormality perception vector, a communication abnormality perception vector, a network abnormality perception vector, and a face abnormality perception vector;

[0183] The abnormality perception matrix generation module 101 is further used for:

[0184] Preprocessing the collected multimodal data to generate initial matrices corresponding to the respective modal data, wherein the initial matrices include at least a first initial matrix, a second initial matrix, a third initial matrix, a fourth initial matrix, and a fifth initial matrix;

[0185] Comparing the historical state and the current state of the first initial matrix corresponding to the terminal data and the second initial matrix corresponding to the position data by the first sub-network and the second sub-network, respectively, to obtain the terminal anomaly perception vector and the position anomaly perception vector corresponding to the terminal data and the position data, respectively;

[0186] Calculating, by a third sub-network, the correlation between the time series and the absolute value in the third initial matrix corresponding to the communication data and the abnormal behavior to obtain a communication anomaly perception vector corresponding to the communication data;

[0187] Calculating the correlation between each application and the number of accesses and abnormal behavior in the fourth initial matrix corresponding to the network data through the fourth sub-network to obtain a network anomaly perception vector for the network data pair;

[0188] The image features in the fifth initial matrix corresponding to the facial image data are extracted through the fifth sub-network and the image features are calculated to obtain the facial anomaly perception vector corresponding to the facial image data.

[0189] Optionally, the abnormality perception matrix generating module 101 is further configured to:

[0190] Perform word vectorization on the first initial matrix and the second initial matrix respectively to obtain corresponding two-dimensional matrices;

[0191] Inputting each of the two-dimensional matrices into a preset low-level feature calculation network to obtain a plurality of feature maps corresponding to the first initial matrix and the second initial matrix respectively;

[0192] Each of the feature maps is input into a preset fully connected network to obtain a terminal anomaly perception vector and a position anomaly perception vector.

[0193] Optionally, the multimodal anomaly risk measurement deep network includes at least a forward discriminant network, a reverse discriminant network, and a forward and reverse competitive decision network, and the risk measurement vector includes at least a forward discriminant score, a reverse discriminant score, and an anomaly risk probability, wherein the forward discriminant network and the reverse discriminant network are trained using samples consisting of normal users and samples consisting of abnormal users, respectively;

[0194] The risk metric vector determination module 102 is further configured to:

[0195] Inputting the multimodal anomaly perception data matrix into the forward discriminant network and the reverse discriminant network respectively to obtain a forward feature map, a reverse feature map, a forward discriminant score, and a reverse discriminant score;

[0196] Inputting the forward feature map and the reverse feature map into a forward and reverse competitive decision network to obtain an abnormal risk probability;

[0197] A risk measurement vector is constructed according to the positive discriminant score, the negative discriminant score, and the abnormal risk probability.

[0198] Optionally, the risk metric vector determination module 102 is further configured to:

[0199] The multimodal anomaly perception data matrix corresponding to the training sample set is divided into blocks and converted into a corresponding linear embedding sequence;

[0200] Inputting the linear embedding sequence into three preset linear transformation matrices respectively to obtain corresponding target matrices;

[0201] Calculating a representation vector of the multimodal anomaly perception data matrix according to each of the target matrices;

[0202] Optimizing the classification loss of the representation vector according to a preset target optimization network to obtain parameters corresponding to the forward discriminant network and the reverse discriminant network respectively;

[0203] Based on the parameters corresponding to the forward discriminant network and the reverse discriminant network, a forward discriminant network, a reverse discriminant network and a forward and reverse competitive decision network are constructed.

[0204] Optionally, the dynamic threshold determination module 103 is further configured to:

[0205] Performing a grade quantification calculation on the risk measurement vector to obtain the risk grades corresponding to the positive discriminant score, the negative discriminant score, and the abnormal risk probability in the risk measurement vector;

[0206] Calculating a fuzzy function calculation result of the risk measurement vector according to a preset fuzzy calculation rule;

[0207] Determining a risk degree score based on the risk measurement vector and a corresponding fuzzy function calculation result;

[0208] A dynamic threshold is calculated based on the risk level score and a preset baseline threshold.

[0209] Optionally, the multi-model detection module 104 is further configured to:

[0210] The step of selecting multiple face attack detection models according to the dynamic threshold to detect the face image data and obtaining the attack probability corresponding to each face attack detection model comprises:

[0211] Determining a risk level according to the dynamic threshold and the preset threshold, wherein the risk level includes at least a first risk level, a second risk level, and a third risk level, the second risk level being higher than the first risk level and lower than the third risk level;

[0212] If the risk level is the first risk level, selecting a first preset number of face attack detection models to detect the face image data in the multimodal data, and obtaining the attack probability corresponding to each of the face attack detection models;

[0213] If the risk level is the second risk level, selecting a second preset number of face attack detection models to detect the face image data in the multimodal data to obtain the attack probability corresponding to each of the face attack detection models;

[0214] If the risk level is the third risk level, a third preset number of face attack detection models are selected to detect the facial image data in the multimodal data to obtain the attack probability corresponding to each of the face attack detection models, wherein the second preset number is greater than the first preset number and less than the third preset number.

[0215] Optionally, the detection result determination module 105 is further configured to:

[0216] If the attack probabilities corresponding to the face attack detection models are all lower than the dynamic threshold, the attack detection result is determined to be normal;

[0217] Otherwise, the attack detection result is determined to be abnormal.

[0218] The multimodal face attack detection device provided in this application utilizes the multimodal face attack detection method described in the aforementioned embodiment, resolving the technical issue of difficulty in collecting multidimensional data for multimodal face attack detection solutions in mobile scenarios. Compared to the prior art, the multimodal face attack detection device provided in this embodiment achieves the same beneficial effects as the multimodal face attack detection method described in the aforementioned embodiment. Other technical features of this multimodal face attack detection device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0219] Example 3

[0220] An embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively linked to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multimodal face attack detection method of the first embodiment above.

[0221] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable media players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0222] like Figure 8As shown, electronic devices may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from a storage device into random access memory (RAM). RAM also stores various programs and data required for the operation of the electronic device. The processing device, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0223] Typically, the following systems can be linked to an I / O interface: input devices such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices such as a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices such as magnetic tape, hard disk, etc.; and communication devices. Communication devices can allow electronic devices to communicate with other devices wirelessly or by wire to exchange data. While the figures illustrate electronic devices with various systems, it should be understood that not all of the illustrated systems are required to be implemented or present. More or fewer systems may alternatively be implemented or present.

[0224] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0225] The electronic device provided in this application utilizes the multimodal face attack detection method of the aforementioned embodiment to address the technical issue of difficulty in multi-dimensional data collection in multimodal face attack detection schemes in mobile scenarios. Compared to the prior art, the beneficial effects of the electronic device provided in this embodiment are the same as those of the multimodal face attack detection method provided in the first embodiment above. Other technical features of this electronic device are the same as those disclosed in the method of the aforementioned embodiment and are not further elaborated here.

[0226] It should be understood that various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in an appropriate manner.

[0227] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0228] Example 4

[0229] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the multimodal face attack detection method in the above-mentioned embodiment 1.

[0230] The computer-readable storage medium provided in the embodiments of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical link having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, Erasable Programmable Read-Only Memory or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0231] The computer-readable storage medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0232] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device: inputs the initial matrices corresponding to the multimodal data of the mobile Internet user's cloud-network behavior data and facial image data into a preset abnormality perception matrix generation deep network to obtain a multimodal abnormality perception data matrix; inputs the multimodal abnormality perception data matrix into a preset multimodal abnormality risk measurement deep network to generate a risk measurement vector; determines a dynamic threshold based on a preset fuzzy calculation rule, the risk measurement vector and a preset benchmark threshold; selects multiple facial attack detection models based on the dynamic threshold to detect the facial image data and obtain the attack probability corresponding to each of the facial attack detection models; and determines the attack detection result based on the attack probability corresponding to each of the facial attack detection models and the dynamic threshold.

[0233] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be linked to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be linked to an external computer (e.g., via the Internet using an Internet service provider).

[0234] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0235] The modules involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0236] The computer-readable storage medium provided in this application stores computer-readable program instructions for executing the aforementioned multimodal face attack detection method, resolving the technical issue of difficulty in collecting multidimensional data for multimodal face attack detection solutions in mobile scenarios. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this embodiment of the application are similar to those of the multimodal face attack detection method provided in the aforementioned embodiment, and are not further elaborated here.

[0237] Example 5

[0238] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multimodal face attack detection method as described above.

[0239] The computer program product provided in this application solves the technical problem of difficulty in collecting multi-dimensional data for multimodal face attack detection solutions in mobile scenarios. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of this application are the same as those of the multimodal face attack detection method provided in the above embodiments, and will not be elaborated here.

[0240] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A multimodal face attack detection method, characterized in that: The multimodal face attack detection method includes: Input the initial matrices corresponding to the multimodal data of the mobile Internet user's cloud network terminal behavior data and facial image data into the preset abnormality perception matrix generation deep network to obtain the multimodal abnormality perception data matrix; Inputting the multimodal anomaly perception data matrix into a preset multimodal anomaly risk measurement deep network to generate a risk measurement vector; Determining a dynamic threshold value according to a preset fuzzy calculation rule, the risk measurement vector, and a preset benchmark threshold value; Selecting multiple face attack detection models according to the dynamic threshold to detect the face image data, and obtaining attack probabilities corresponding to the respective face attack detection models; Determining an attack detection result based on the attack probability corresponding to each of the face attack detection models and the dynamic threshold; The multimodal data includes at least terminal data, location data, communication data, network data, and face image data; the abnormality perception matrix generation deep network includes at least a first subnetwork, a second subnetwork, a third subnetwork, a fourth subnetwork, and a fifth subnetwork; and the multimodal abnormality perception data matrix includes at least a terminal abnormality perception vector, a location abnormality perception vector, a communication abnormality perception vector, a network abnormality perception vector, and a face abnormality perception vector. The step of inputting the initial matrices corresponding to the multimodal data of the mobile internet user's cloud network terminal behavior data and facial image data into a preset abnormality perception matrix generation deep network to obtain the multimodal abnormality perception data matrix includes: Preprocessing the collected multimodal data to generate initial matrices corresponding to the respective modal data, wherein the initial matrices include at least a first initial matrix, a second initial matrix, a third initial matrix, a fourth initial matrix, and a fifth initial matrix; Comparing the historical state and the current state of the first initial matrix corresponding to the terminal data and the second initial matrix corresponding to the position data by the first sub-network and the second sub-network, respectively, to obtain the terminal anomaly perception vector and the position anomaly perception vector corresponding to the terminal data and the position data, respectively; Calculating, by a third sub-network, the correlation between the time series and the absolute value in the third initial matrix corresponding to the communication data and the abnormal behavior to obtain a communication anomaly perception vector corresponding to the communication data; Calculating the correlation between each application and the number of accesses and abnormal behavior in the fourth initial matrix corresponding to the network data through the fourth sub-network to obtain a network anomaly perception vector for the network data pair; The image features in the fifth initial matrix corresponding to the facial image data are extracted through the fifth sub-network and the image features are calculated to obtain the facial anomaly perception vector corresponding to the facial image data.

2. The multimodal face attack detection method according to claim 1, wherein: The step of comparing the historical state and the current state of the first initial matrix corresponding to the terminal data and the second initial matrix corresponding to the position data by the first sub-network and the second sub-network, and obtaining the terminal anomaly perception vector and the position anomaly perception vector corresponding to the terminal data and the position data respectively includes: Perform word vectorization on the first initial matrix and the second initial matrix respectively to obtain corresponding two-dimensional matrices; Inputting each of the two-dimensional matrices into a preset low-level feature calculation network to obtain a plurality of feature maps corresponding to the first initial matrix and the second initial matrix respectively; Each of the feature maps is input into a preset fully connected network to obtain a terminal anomaly perception vector and a position anomaly perception vector.

3. The multimodal face attack detection method according to claim 1, wherein: The multimodal anomaly risk measurement deep network includes at least a forward discriminant network, a reverse discriminant network, and a forward and reverse competitive decision network. The risk measurement vector includes at least a forward discriminant score, a reverse discriminant score, and an anomaly risk probability. The forward discriminant network and the reverse discriminant network are trained using samples of normal users and samples of abnormal users, respectively. The step of inputting the multimodal anomaly perception data matrix into a preset multimodal anomaly risk measurement deep network to generate a risk measurement vector includes: Inputting the multimodal anomaly perception data matrix into the forward discriminant network and the reverse discriminant network respectively to obtain a forward feature map, a reverse feature map, a forward discriminant score, and a reverse discriminant score; Inputting the forward feature map and the reverse feature map into a forward and reverse competitive decision network to obtain an abnormal risk probability; A risk measurement vector is constructed according to the positive discriminant score, the negative discriminant score, and the abnormal risk probability.

4. The multimodal face attack detection method according to claim 3, wherein: Before the step of inputting the multimodal abnormality perception data matrix into the forward discriminant network and the reverse discriminant network respectively to obtain the forward feature map, the reverse feature map, the forward discriminant score and the reverse discriminant score, the method further includes: The multimodal anomaly perception data matrix corresponding to the training sample set is divided into blocks and converted into a corresponding linear embedding sequence; Inputting the linear embedding sequence into three preset linear transformation matrices respectively to obtain corresponding target matrices; Calculating a representation vector of the multimodal anomaly perception data matrix according to each of the target matrices; Optimizing the classification loss of the representation vector according to a preset target optimization network to obtain parameters corresponding to the forward discriminant network and the reverse discriminant network respectively; Based on the parameters corresponding to the forward discriminant network and the reverse discriminant network, a forward discriminant network, a reverse discriminant network and a forward and reverse competitive decision network are constructed.

5. The multimodal face attack detection method according to claim 3, wherein: The step of determining the dynamic threshold according to the preset fuzzy calculation rule, the risk measurement vector and the preset reference threshold comprises: Performing a grade quantification calculation on the risk measurement vector to obtain the risk grades corresponding to the positive discriminant score, the negative discriminant score, and the abnormal risk probability in the risk measurement vector; Calculating a fuzzy function calculation result of the risk measurement vector according to a preset fuzzy calculation rule; Determining a risk degree score based on the risk measurement vector and a corresponding fuzzy function calculation result; A dynamic threshold is calculated based on the risk level score and a preset baseline threshold.

6. The multimodal face attack detection method according to any one of claims 1 to 5, wherein: The face attack detection model includes at least a visible light silent biopsy model, a dazzling color biopsy model, a dazzling pupil biopsy model, a moiré detection model, a screen border detection model, and a photo detection model; The step of selecting multiple face attack detection models according to the dynamic threshold to detect the face image data and obtaining the attack probability corresponding to each face attack detection model comprises: Determining a risk level according to the dynamic threshold and the preset threshold, wherein the risk level includes at least a first risk level, a second risk level, and a third risk level, the second risk level being higher than the first risk level and lower than the third risk level; If the risk level is the first risk level, selecting a first preset number of face attack detection models to detect the face image data in the multimodal data, and obtaining the attack probability corresponding to each of the face attack detection models; If the risk level is the second risk level, selecting a second preset number of face attack detection models to detect the face image data in the multimodal data to obtain the attack probability corresponding to each of the face attack detection models; If the risk level is the third risk level, a third preset number of face attack detection models are selected to detect the facial image data in the multimodal data to obtain the attack probability corresponding to each of the face attack detection models, wherein the second preset number is greater than the first preset number and less than the third preset number.

7. The multimodal face attack detection method according to any one of claims 1 to 5, characterized in that: The step of determining the attack detection result based on the attack probability and the dynamic threshold corresponding to each of the face attack detection models includes: If the attack probabilities corresponding to the face attack detection models are all lower than the dynamic threshold, the attack detection result is determined to be normal; Otherwise, the attack detection result is determined to be abnormal.

8. A multimodal face attack detection device, characterized in that: The multimodal face attack detection device comprises: The abnormality perception matrix generation module is used to input the initial matrices corresponding to the multimodal data of the mobile Internet user's cloud network terminal behavior data and facial image data into the preset abnormality perception matrix generation deep network to obtain the multimodal abnormality perception data matrix; A risk measurement vector determination module is used to input the multimodal anomaly perception data matrix into a preset multimodal anomaly risk measurement deep network to generate a risk measurement vector; A dynamic threshold determination module, configured to determine a dynamic threshold based on a preset fuzzy calculation rule, the risk measurement vector, and a preset reference threshold; a multi-model detection module, configured to select a plurality of face attack detection models according to the dynamic threshold to detect the face image data, and obtain attack probabilities corresponding to the respective face attack detection models; a detection result determination module, configured to determine an attack detection result based on the attack probability corresponding to each of the face attack detection models and the dynamic threshold; Wherein, the multimodal data includes at least terminal data, location data, communication data, network data and face image data, the abnormality perception matrix generation deep network includes at least a first sub-network, a second sub-network, a third sub-network, a fourth sub-network and a fifth sub-network, and the multimodal abnormality perception data matrix includes at least a terminal abnormality perception vector, a location abnormality perception vector, a communication abnormality perception vector, a network abnormality perception vector and a face abnormality perception vector; the abnormality perception matrix generation module is also used to: pre-process the collected multimodal data to generate an initial matrix corresponding to each modal data, wherein the initial matrix includes at least a first initial matrix, a second initial matrix, a third initial matrix, a fourth initial matrix and a fifth initial matrix; the first sub-network and the second sub-network are used to pre-process the first and second modal data corresponding to the terminal data respectively. An initial matrix and a second initial matrix corresponding to the position data are compared with each other in historical state and current state to obtain a terminal anomaly perception vector and a position anomaly perception vector corresponding to the terminal data and the position data respectively; the correlation between the time series and absolute value in the third initial matrix corresponding to the communication data and the abnormal behavior is calculated by a third sub-network to obtain a communication anomaly perception vector corresponding to the communication data; the correlation between each application and the number of accesses in the fourth initial matrix corresponding to the network data and the abnormal behavior is calculated by a fourth sub-network to obtain a network anomaly perception vector of the network data pair; the image features in the fifth initial matrix corresponding to the facial image data are extracted by a fifth sub-network and the image features are calculated to obtain a facial anomaly perception vector corresponding to the facial image data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively linked to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the multimodal face attack detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program for implementing a multimodal face attack detection method, and the program for implementing the multimodal face attack detection method is executed by a processor to implement the steps of the multimodal face attack detection method according to any one of claims 1 to 7.

Images