Method and system for detecting adversarial attacks

Abstract

The present specification provides a detection method and system for detecting adversarial attacks. After obtaining a plurality of original modality images of the biological features of a target user 100, the plurality of original modality images are input into a multi-modality abnormal region perception model to obtain a multi-modality abnormal region detection result corresponding to the plurality of original modality images. Based on the multi-modality abnormal region detection result, it is determined whether the target user 100 is an adversarial attack object. The method and system use multi-modality information to detect adversarial attacks, which can improve the detection accuracy of adversarial attacks.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method and system for detecting adversarial attacks. Background Technology

[0002] Adversarial attacks are one of the main security threats to facial recognition systems. By deliberately adding subtle, imperceptible interference to the input samples, the model can be made to output an incorrect classification result. Because the attack patterns in adversarial attacks are small and difficult to detect, defending against them is quite challenging.

[0003] Currently, common methods for defending against adversarial attacks mainly include: in the pre-detection / preprocessing stage of face recognition, detecting or neutralizing adversarial attacks by training a classifier or introducing intensive preprocessing. This method involves relatively simple models, resulting in poor adversarial attack detection performance. Alternatively, adversarial attack detection can be achieved through robust training during the recognition stage. This method introduces adversarial examples during training and requires the model to provide correct predictions for these examples, thus making the model robust against adversarial attacks. However, this method cannot effectively defend against adversarial example types not encountered during training, therefore, its adversarial attack detection performance is also poor.

[0004] In summary, there is a need to provide a way to improve the performance of adversarial attack detection. Summary of the Invention

[0005] This specification provides a method and system for detecting adversarial attacks, which can improve the performance of adversarial attack detection.

[0006] Firstly, this specification provides a method for detecting adversarial attacks, comprising: acquiring multiple original modal images of the biometric features of a target user 100, wherein the multiple original modal images correspond to images actually acquired in multiple modalities of an image acquisition module; inputting the multiple original modal images into a multimodal abnormal region perception model to obtain multimodal abnormal region detection results corresponding to the multiple original modal images; and determining whether the target user 100 is an adversarial attack target based on the multimodal abnormal region detection results.

[0007] In some embodiments, the multimodal anomaly region detection results include at least one of the following: multimodal adversarial attack classification results, multimodal anomaly region segmentation results, and cross-modal mapping results.

[0008] In some embodiments, the multimodal anomaly region perception model includes a multimodal feature encoder, and further includes at least one of a multimodal adversarial attack classifier, a multimodal anomaly region segmentation module, and a cross-modal mapping relationship learning module; and inputting multiple original modal images into the multimodal anomaly region perception model to obtain multimodal anomaly region detection results corresponding to the multiple original modal images includes: inputting the multiple original modal images into the multimodal feature encoder 5 for feature extraction to obtain multiple original modal features and multimodal fusion features corresponding to the multiple original modal images; and at least one of the following: inputting the multimodal fusion features into the multimodal adversarial attack classifier to obtain the multimodal adversarial attack classification result; inputting the multimodal fusion features into the multimodal anomaly region segmentation module to obtain the multimodal anomaly region segmentation result; and inputting the multiple original modal features into the cross-modal mapping relationship learning module to obtain the cross-modal mapping result.

[0009] 0 In some embodiments, the multimodal anomaly region perception model is trained using the following steps: The training data is...

[0010] Multiple original modal image samples of a user's biometrics are input to a preset multimodal feature encoder to obtain multiple original modal feature samples and multimodal fused feature samples corresponding to the multiple original modal image samples; and at least one of the following:

[0011] The multiple original modal feature samples and the multimodal fused feature samples are input into a preset multimodal anomaly region segmentation.

[0012] The segmentation module obtains the training abnormal region segmentation results and corresponding abnormal region segmentation loss information corresponding to the multiple original modal image samples; the multiple original modal feature samples and the multimodal fusion feature samples are input into a preset multimodal...

[0013] The system employs a modal adversarial attack classifier to obtain training adversarial attack classification results and corresponding adversarial attack classification loss information; it inputs the multiple original modal feature samples into a preset cross-modal mapping relationship learning module to obtain training cross-modal mapping results and corresponding cross-modal feature prediction loss information; and it uses the abnormal region segmentation loss information and the adversarial attack...

[0014] At least one of the attack classification loss information and the cross-modal feature prediction loss information is iteratively trained until the training ends, thus obtaining the multimodal anomaly region perception model.

[0015] In some embodiments, the training abnormal region segmentation results include multiple single-modal segmentation results and multimodal segmentation results corresponding to multiple modalities, and the abnormal region segmentation loss information includes at least one of multiple single-modal segmentation loss information corresponding to multiple modalities, multimodal segmentation loss information, and consistency loss information of multiple single-modal segmentation results;

[0016] The multiple single-modal segmentation loss information is determined based on the differences between the multiple single-modal segmentation results corresponding to the multiple modalities and the labeled abnormal regions; the multi-modal segmentation loss information is based on the multi-modal segmentation results and the labeled abnormal regions.

[0017] The differences between regions are determined; the consistency loss information of the multiple single-modal segmentation results is determined based on the differences between the multiple single-modal segmentation results.

[0018] In some embodiments, the training cross-modal mapping result includes multiple original modal image samples corresponding to multiple modalities.

[0019] Each training modal feature includes features of other modalities obtained by cross-modal feature transformation based on the original modal features of its corresponding original modal image sample; the cross-modal feature prediction loss information is determined based on the difference between the original modal feature sample and the transformed modal feature sample corresponding to each original modal image sample in the plurality of original modal image samples.

[0020] In some embodiments, the multimodal adversarial attack classification result includes the multimodal adversarial attack probabilities corresponding to the multiple original modal images, the multimodal abnormal region segmentation result includes the multimodal abnormal image regions corresponding to the multiple original modal images, and the cross-modal mapping result includes multiple cross-modal feature sets corresponding to the multiple original modal images. Each cross-modal feature set includes at least one modality-transformed feature of another modality obtained by cross-modal feature transformation based on its corresponding original modal features.

[0021] In some embodiments, determining whether the target user 100 is an adversarial attack target based on the multimodal anomaly region detection results includes: determining detection data based on the anomaly region detection results, the detection data including at least one of the multimodal adversarial attack probability, the area ratio of anomaly image regions, and comprehensive feature differences, wherein the area ratio of anomaly image regions includes the area ratio of the anomaly image regions in the multimodal segmented images corresponding to the multiple original modal images, and the comprehensive feature differences include the fusion of the differences between the original modal features of the multiple original modal images and at least one corresponding transmodal feature; and determining whether the target user 100 is an adversarial attack target based on the detection data.

[0022] In some embodiments, determining whether the target user 100 is an adversarial attack target based on the detection data includes one of the following: determining that at least one data point in the detection data is greater than a corresponding preset first threshold, thus determining the target user 100 as an adversarial attack target, wherein the first threshold includes at least one of a first probability threshold, a first area proportion threshold, and a first feature difference threshold, wherein the first probability threshold corresponds to the multimodal adversarial attack probability, the first area proportion threshold corresponds to the abnormal image area proportion, and the first feature difference threshold corresponds to the comprehensive feature difference; determining that all data points in the detection data are less than a corresponding preset second threshold, thus determining the target user 100 as a legitimate user, wherein the second threshold includes at least one of a second probability threshold, a second area proportion threshold, and a second feature difference threshold, wherein the second probability threshold corresponds to the multimodal adversarial attack probability, the second area proportion threshold corresponds to the abnormal image area proportion, and the second feature difference threshold corresponds to the comprehensive feature difference; and otherwise, determining the target user 100 as a pending user.

[0023] In some embodiments, the comprehensive feature difference is determined by the following steps: each of the plurality of modalities is designated as the target modality, and the weighted sum of the differences between the original modal features corresponding to the target modality and at least one transition modal feature is determined to obtain the plurality of feature differences corresponding to the plurality of modalities; the weighted sum of the plurality of feature differences is determined to obtain the comprehensive feature difference.

[0024] In some embodiments, after determining that the target user 100 is the pending user, the method further includes: re-detecting the pending user.

[0025] In some embodiments, the re-detection of the candidate user includes: determining a first comparison score and a second comparison score of the original modal image of the target modality among multiple original modal images of the candidate user before preprocessing, wherein the preprocessing includes detecting the target part in the original modal image of the target modality to obtain a target part image; determining whether the candidate user is an adversarial attack target based on the comparison score difference between the first comparison score and the second comparison score, including: if the comparison score difference is greater than a preset threshold, then the candidate user is identified as an adversarial attack target; or, if the comparison score difference is less than the threshold, then the candidate user is identified as a legitimate user.

[0026] In some embodiments, determining the first alignment score includes: acquiring the original image of the target modality before preprocessing, and the user-retained image in an adversarial attack detection scenario; and determining the first alignment score based on the difference between the original image and the user-retained image.

[0027] In some embodiments, determining the second comparison score includes: setting the pixels of the abnormal region in the single-modal abnormal region segmentation result corresponding to the target modality to 0 to obtain a restored image; and obtaining the second comparison score based on the difference between the restored image and the user-retained image.

[0028] Secondly, this specification also provides a detection system for counter-attacks, comprising: at least one storage medium storing at least one instruction set for counter-attack detection; and at least one processor communicatively connected to the at least one storage medium, wherein, when the counter-attack detection system is running, the at least one processor reads the at least one instruction set and executes the method described in any one of the first aspects according to the instructions of the at least one instruction set.

[0029] As can be seen from the above technical solutions, the adversarial attack detection method and system provided in this specification acquire multiple original modal images of the biometric features of the target user 100, input these multiple original modal images into a multimodal anomaly region perception model, obtain multimodal anomaly region detection results corresponding to the multiple original modal images, and determine whether the target user 100 is an adversarial attack target based on the multimodal anomaly region detection results. Because multimodal information is used for anomaly region detection, the detection accuracy of adversarial attacks can be improved.

[0030] Other functions of the methods and systems for detecting adversarial attacks provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the methods and systems for detecting adversarial attacks provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A schematic diagram illustrating an application scenario of an anti-attack detection system provided according to an embodiment of this specification is shown.

[0033] Figure 2 A hardware structure diagram of a computing device provided according to an embodiment of this specification is shown;

[0034] Figure 3 A flowchart of a method for detecting adversarial attacks provided according to an embodiment of this specification is shown;

[0035] Figure 4 A data flow diagram for adversarial attack detection provided according to an embodiment of this specification is shown. Detailed Implementation

[0036] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0037] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0038] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0039] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0040] For ease of description, the terms used in this specification are explained as follows:

[0041] Multimodal: In this scheme, it refers to images with multiple modalities, such as RGB, NIR, and Depth. Images with multiple modalities can provide information from multiple angles (dimensions) for adversarial attack detection.

[0042] Abnormal area detection: In this solution, it refers to detecting areas on the face that are different from biological materials, such as glasses, jewelry, and anti-corrosion stickers.

[0043] Two-stage verification: In this scheme, it refers to performing a second-stage verification after the first-stage abnormal region detection to avoid erroneous interception of normal samples.

[0044] Adversarial attacks refer to attacks that involve pasting adversarial stickers (small in area, generally less than 50% of the face) onto the face area, causing the face recognition system to misjudge the user (for example, user A is mistakenly identified as user B after pasting an adversarial sticker).

[0045] Adversarial attack detection: In this solution, it refers to various methods for detecting adversarial attacks on faces.

[0046] Before describing the specific embodiments in this specification, the application scenarios of this specification will be introduced as follows:

[0047] The adversarial attack detection method provided in this specification can be applied to any adversarial attack detection scenario in the biometric recognition process. For example, in scenarios such as face payment or face recognition, the adversarial attack detection method can be used to detect adversarial attacks on multiple original modal images of the biometric features of the target user 100 to be paid or identified. In identity verification scenarios, the adversarial attack detection method can be used to detect adversarial attacks on multiple original modal images of the biometric features of the target user 100. It can also be applied to any other adversarial attack detection scenario, which will not be elaborated here. The biometric features may include, but are not limited to, one or more of the following: facial image, iris, sclera, fingerprint, palm print, voiceprint, and skeletal projection. For ease of description, this application will use the application of the adversarial attack detection method in a face recognition scenario as an example.

[0048] Those skilled in the art should understand that the detection methods and systems for counter-attacks described in this specification are also within the scope of protection of this specification when applied to other use cases.

[0049] Figure 1This diagram illustrates an application scenario of an adversarial attack detection system 001 provided according to an embodiment of this specification. The adversarial attack detection system 001 (hereinafter referred to as System 001) can be applied to adversarial attack detection in any scenario, such as adversarial attack detection in scenarios like facial recognition payment, identity verification, access control, and information query. Figure 1 As shown, the scenario in which system 001 is applied may include target user 100 and client 200. In some embodiments, system 001 may also include server 300 and network 400.

[0050] The target user 100 can be a user who needs to perform biometric identification, or a user who is currently performing biometric identification. The target user 100 can be the object detected by system 001. The target user 100 can initiate an identification process, thereby triggering adversarial attack detection on multiple original modal images of the target user 100. In some embodiments, the target user 100 can trigger adversarial attack detection through client 200.

[0051] Client 200 can acquire multiple original modal images of target user 100 in response to an attack detection operation by target user 100 and input them to the attack detection device (the attack detection device is a computing device, which will be described later). In some embodiments, the attack detection method can be executed on client 200. In this case, client 200 can store data or instructions for executing the attack detection method described in this specification, and can execute or be used to execute the data or instructions. In some embodiments, client 200 may include a hardware device with data information processing capabilities and the necessary programs to drive the hardware device. Figure 1As shown, client 200 can communicate with server 300. In some embodiments, server 300 can communicate with multiple clients 200. In some embodiments, client 200 can interact with server 300 through network 400 to receive or send messages, such as receiving or sending multiple original modal images. In some embodiments, client 200 may include mobile devices, tablets, laptops, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, the mobile device may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, the smart home device may include smart TVs, desktop computers, etc., or any combination thereof. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, gaming devices, navigation devices, etc., or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include virtual reality headsets, virtual reality glasses, virtual reality controllers, augmented reality headsets, augmented reality glasses, augmented reality controllers, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, head-mounted displays, VR, etc. In some embodiments, the built-in device in the motor vehicle may include an in-vehicle computer, an in-vehicle TV, etc. In some embodiments, the client 200 may include an image acquisition device for acquiring multiple original modal images of the target user 100. In some embodiments, the image acquisition device may be a two-dimensional image acquisition device (e.g., an RGB camera, an IR camera, etc.), or a combination of a two-dimensional image acquisition device (e.g., an RGB camera, an IR camera, etc.) and a depth image acquisition device (e.g., a 3D structured light camera, a laser detector, etc.). In some embodiments, the client 200 may be a device with positioning technology for locating the position of the client 200.

[0052] In some embodiments, the client 200 may have one or more applications (APPs) installed. The APPs provide the target user 100 with the ability and interface to interact with the outside world via the network 400. The APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, video APPs, financial management APPs, instant messaging tools, email clients, social media platform software, etc. In some embodiments, the client 200 may have a target APP installed. The target APP can collect multiple native modal images of the target user 100 for the client 200. In some embodiments, the target user 100 can also trigger an adversarial attack detection request through the target APP, such as triggering a biometric recognition program through the target APP, thereby triggering the adversarial attack detection request. The target APP can respond to the adversarial attack detection request and execute the adversarial attack detection method described in this specification. The adversarial attack detection method will be described in detail later.

[0053] In some embodiments, system 001 may further include server 300. Server 300 may be a backend server providing various services, such as a backend server supporting adversarial attack detection of multiple original modal images of target user 100 collected on client 200. In some embodiments, the adversarial attack detection method may be executed on server 300. In this case, server 300 may store data or instructions for executing the adversarial attack detection method described herein, and may execute or be used to execute the data or instructions. In some embodiments, server 300 may include hardware devices with data information processing capabilities and the necessary programs required to drive the hardware devices. Server 300 may communicate with multiple clients 200 and receive data sent by clients 200.

[0054] Network 400 serves as a medium to provide a communication connection between client 200 and server 300. Network 400 facilitates the exchange of information or data. For example... Figure 1 As shown, client 200 and server 300 can connect to network 400 and transmit information or data to each other through network 400. In some embodiments, network 400 can be any type of wired or wireless network, or a combination thereof. For example, network 400 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), and Bluetooth. TM Network, ZigBee TMA network, a near-field communication (NFC) network, or a similar network. In some embodiments, network 400 may include one or more network access points. For example, network 400 may include wired or wireless network access points, such as base stations or internet exchange points, through which one or more components of client 200 and server 300 can connect to network 400 to exchange data or information.

[0055] It should be understood that Figure 1 The number of clients 200, servers 300, and networks 400 shown is merely illustrative. Depending on implementation needs, there can be any number of clients 200, servers 300, and networks 400.

[0056] It should be noted that the method for detecting the anti-attack can be executed entirely on the client 200, entirely on the server 300, or partially on the client 200 and partially on the server 300.

[0057] Figure 2 A hardware structure diagram of a computing device 600 provided according to an embodiment of this specification is shown. The computing device 600 can execute the anti-attack detection method described in this specification. The anti-attack detection method is described in other parts of this specification. When the anti-attack detection method is executed on a client 200, the computing device 600 can be the client 200. When the anti-attack detection method is executed on a server 300, the computing device 600 can be the server 300. When the anti-attack detection method can be executed partly on the client 200 and partly on the server 300, the computing device 600 can be both the client 200 and the server 300.

[0058] like Figure 2 As shown, the computing device 600 may include at least one storage medium 630 and at least one processor 620. In some embodiments, the computing device 600 may also include a communication port 650 and an internal communication bus 610. Additionally, the computing device 600 may include I / O components 660.

[0059] The internal communication bus 610 can connect different system components, including storage medium 630, processor 620 and communication port 650.

[0060] I / O component 660 supports input / output between computing device 600 and other components.

[0061] Communication port 650 is used for data communication between computing device 600 and external sources. For example, communication port 650 can be used for data communication between computing device 600 and network 400. Communication port 650 can be a wired communication port or a wireless communication port.

[0062] Storage medium 630 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 632, a read-only storage medium (ROM) 634, or a random access storage medium (RAM) 636. Storage medium 630 also includes at least one instruction set stored in the data storage device. The instructions are computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the detection methods against attacks provided in this specification.

[0063] At least one processor 620 can be communicatively connected to at least one storage medium 630 and a communication port 650 via an internal communication bus 610. At least one processor 620 is used to execute at least one instruction set. When the computing device 600 is running, at least one processor 620 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the anti-attack detection method provided in this specification. Processor 620 can execute all steps included in the anti-attack detection method. Processor 620 can be in the form of one or more processors. In some embodiments, processor 620 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof. For illustrative purposes only, only one processor 620 is described in this specification for the computing device 600. However, it should be noted that the computing device 600 may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described herein, or they may be executed jointly by multiple processors. For example, if processor 620 of the computing device 600 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 620 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0064] Figure 3A flowchart of a counter-attack detection method P100 provided according to an embodiment of this specification is shown. As previously described, the computing device 600 can execute the counter-attack detection method P100 of this specification. Specifically, the processor 620 can read an instruction set stored in its local storage medium and then execute the counter-attack detection method P100 of this specification according to the provisions of the instruction set. Figure 3 As shown, method P100 may include:

[0065] S110: Acquire multiple original modal images of the biometrics of target user 100.

[0066] The target users 100 are those to be detected for adversarial attacks. Biometric features can be inherent physiological characteristics of the human body, such as facial images, irises, sclera, fingerprints, palm prints, voiceprints, and skeletal projections. For ease of description, this application will use facial images as an example of biometric features.

[0067] Multiple original modal images correspond to images actually acquired in multiple modalities by the image acquisition module. For example, they could be images including all or part of the target user 100's body parts. When biometric recognition of the target user 100 is triggered, the processor 620 can directly acquire multiple initial original modal images of the target user 100's biometric features through the image acquisition module. For example, the image acquisition module integrates a camera module with multiple modalities. By acquiring images of the target user 100's areas to be detected through the multiple modal camera modules, multiple initial original modal images corresponding to multiple modalities can be obtained. The imaging characteristics and points affected by external interference differ among the multiple modalities. The original modality can be understood as the original modality of the image acquisition module, and the original modal image is the same as the original modality of the image acquisition module, with no change in modality type. For example, the image acquisition module includes at least two of an RGB camera module, a NIR camera module, and a Depth camera module. The initial original modal images of the RGB camera module, the NIR camera module, and the Depth camera module are the initial RGB image, the initial NIR image, and the initial Depth image, respectively. When biometric identification of the target user 100 is triggered, the processor 620 can also receive multiple initial original modal images of the target user 100 uploaded by the target user 100 through the client 200 or the terminal device. Alternatively, it can also obtain multiple initial original modal images from the storage address based on an adversarial attack detection request received for the storage address of multiple initial original modal images carrying the biometric features of the target user 100.

[0068] Figure 4 This is a data flow diagram of the anti-attack detection method provided in the embodiments of this specification. (See diagram for example.) Figure 4As illustrated, RGB modality, NIR modality, and Depth modality are used as examples of multimodal processing. Processor 620 can directly receive multiple initial protomodal images of target user 100 uploaded by target user 100 through client 200 or terminal device. Alternatively, it can obtain multiple initial protomodal images from the storage address based on an adversarial attack detection request that receives the storage address of multiple initial protomodal images carrying the biometric features of target user 100. After obtaining the multiple initial protomodal images of target user 100, processor 620 can perform preprocessing on the multiple initial protomodal images to obtain multiple protomodal images. Preprocessing can be face detection, for example, performing face detection on the multiple initial protomodal images to obtain face regions (face regions in RGB images, NIR images, and Depth images) in the multiple initial protomodal images. Of course, the processor 620 can also directly receive multiple original modal images of the target user 100 uploaded by the target user 100 through the client 200 or terminal device, or it can obtain multiple original modal images from the storage address based on the adversarial attack detection request received with the storage address of multiple original modal images carrying the biometrics of the target user 100.

[0069] It should be understood that the explanation of RGB mode, NIR mode and Depth mode as multiple modes is for illustrative purposes only. Those skilled in the art can select other numbers and other modes as multiple modes according to actual needs, and this specification does not limit this.

[0070] S120: Input multiple original modal images into the multimodal anomaly region perception model to obtain the multimodal anomaly region detection results corresponding to the multiple original modal images.

[0071] The multimodal anomaly detection result is the perception result of anomaly regions in multiple original modal images corresponding to multiple modalities. The multimodal anomaly detection result includes at least one of the following: multimodal adversarial attack classification result, multimodal anomaly region segmentation result, and cross-modal mapping result. That is, the multimodal anomaly detection result can include multimodal adversarial attack classification result, multimodal anomaly region segmentation result, or cross-modal mapping result. Alternatively, the multimodal anomaly detection result includes multimodal adversarial attack classification result and multimodal anomaly region segmentation result. Alternatively, the multimodal anomaly detection result includes multimodal adversarial attack classification result and cross-modal mapping result. Alternatively, the multimodal anomaly detection result includes multimodal anomaly region segmentation result and cross-modal mapping result. Alternatively, the multimodal anomaly detection result simultaneously includes multimodal adversarial attack classification result, multimodal anomaly region segmentation result, and cross-modal mapping result.

[0072] The multimodal anomaly region perception model includes a multimodal feature encoder, and further includes at least one of an adversarial attack classifier, a multimodal anomaly region segmentation module, and a cross-modal mapping relationship learning module. That is, the multimodal anomaly region perception model includes a multimodal feature encoder and a multimodal adversarial attack classifier. Alternatively, the multimodal anomaly region perception model includes a multimodal feature encoder and a multimodal anomaly region segmentation module. Alternatively, the multimodal anomaly region perception model includes a multimodal feature encoder and a cross-modal mapping relationship learning module. Alternatively, the multimodal anomaly region perception model includes a multimodal feature encoder, a multimodal adversarial attack classifier, and a multimodal anomaly region segmentation module. Alternatively, the multimodal anomaly region perception model includes a multimodal feature encoder, a multimodal adversarial attack classifier, and a cross-modal mapping relationship learning module. Alternatively, the multimodal anomaly region perception model includes a multimodal feature encoder, a multimodal anomaly region segmentation module, and a cross-modal mapping relationship learning module. Alternatively, the multimodal anomaly region perception model includes a multimodal feature encoder, a multimodal adversarial attack classifier, a multimodal anomaly region segmentation module, and a cross-modal mapping relationship learning module.

[0073] Figure 4 The image shows a multimodal anomaly region perception model that includes a multimodal adversarial attack classifier, a multimodal anomaly region segmentation module, and a cross-modal mapping relationship learning module. Please refer to the following section. Figure 4 A multimodal feature encoder can be a network such as ResNet used for feature extraction. By inputting multiple original modal images into the multimodal feature encoder for feature extraction, multiple original modal features and multimodal fusion features corresponding to the original modal images can be obtained. There are several ways to achieve multimodal fusion features based on feature extraction from multiple original modal images, specifically as follows: For example, the multimodal feature encoder extracts features from each of the multiple original modal images separately, obtaining the original modal features corresponding to each modality. Then, the multiple original modal features corresponding to multiple modalities are combined to obtain the multimodal fusion features. The combination can be done in several ways, such as concatenating multiple original modal features to obtain the multimodal fusion features, or weighted summing of multiple original modal features to obtain the multimodal fusion features. Figure 4 As shown, feature extraction is performed on the original modal images corresponding to the RGB, NIR, and Depth modes respectively, yielding the original modal features corresponding to the RGB, NIR, and Depth modes. Combining the original modal features corresponding to the RGB, NIR, and Depth modes also yields the multimodal fusion features corresponding to the RGB, NIR, and Depth modes.

[0074] After obtaining multiple original modal features and multimodal fusion features, processor 620 can perform the following operations: inputting the multimodal fusion features into a multimodal adversarial attack classifier to obtain a multimodal adversarial attack classification result; inputting the multimodal fusion features into a multimodal anomaly region segmentation module to obtain a multimodal anomaly region segmentation result; and inputting multiple original modal fusion features into a cross-modal mapping relationship learning module to obtain at least one of the cross-modal mapping results. In other words, processor 620 can perform any of the following seven operations:

[0075] (1) Input the multimodal fusion features into the multimodal adversarial attack classifier to obtain the multimodal adversarial attack classification results.

[0076] (2) The processor 620 inputs the multimodal fusion features into the multimodal abnormal region segmentation module to obtain the multimodal abnormal region segmentation result.

[0077] (3) The processor 620 inputs multiple original modal fusion features into the cross-modal mapping relationship learning module to obtain the cross-modal mapping result.

[0078] (4) The processor 620 can input the multimodal fusion features into the multimodal adversarial attack classifier to obtain the multimodal adversarial attack classification result, and input the multimodal fusion features into the multimodal abnormal region segmentation module to obtain the multimodal abnormal region segmentation result.

[0079] (5) The processor 620 can input the multimodal fusion features into the multimodal adversarial attack classifier to obtain the multimodal adversarial attack classification result, and input multiple original modal fusion features into the cross-modal mapping relationship learning module to obtain the cross-modal mapping result.

[0080] (6) The processor 620 can input the multimodal fusion features into the multimodal abnormal region segmentation module to obtain the multimodal abnormal region segmentation result, and input multiple original modal fusion features into the cross-modal mapping relationship learning module to obtain the cross-modal mapping result.

[0081] (7) The processor 620 can input the multimodal fusion features into the multimodal adversarial attack classifier to obtain the multimodal adversarial attack classification result, input the multimodal fusion features into the multimodal abnormal region segmentation module to obtain the multimodal abnormal region segmentation result, and input multiple original modal fusion features into the cross-modal mapping relationship learning module to obtain the cross-modal mapping result.

[0082] The following describes the implementation process of the processor 620 in performing adversarial attack classification, anomaly region segmentation, and cross-modal mapping relationship learning based on multiple original modal features and multimodal fusion features:

[0083] (1) After obtaining the multimodal fusion features, the processor 620 can input the multimodal fusion features into the multimodal adversarial attack classifier for adversarial attack classification, thereby obtaining the multimodal adversarial attack classification result. The multimodal adversarial attack classification result includes the multimodal adversarial attack probabilities corresponding to multiple original modal images. For example, the adversarial attack classifier classifies adversarial attacks based on the multimodal fusion features, obtaining a multimodal adversarial attack classification result where the target user is either an adversarial attack target or a normal user. The adversarial attack classifier can be a network structure with fully connected layers + Softmax layers. Figure 4 As shown, adversarial attack classification is performed based on the original modal features and multimodal fusion features corresponding to the RGB modality, NIR modality, and Depth modality, respectively. This yields the multimodal adversarial attack classification results for the RGB modality, NIR modality, and Depth modality, as well as the single-modal adversarial attack classification results for each of the RGB modality, NIR modality, and Depth modality.

[0084] (2) After obtaining the multimodal fusion features, the processor 620 can also input the multimodal fusion features into the multimodal anomaly region segmentation module to perform anomaly region segmentation, thereby obtaining the multimodal anomaly region segmentation result. The multimodal anomaly region segmentation result includes multimodal anomaly image regions corresponding to multiple original modal images. The multimodal anomaly region segmentation module can segment anomaly regions based on the multimodal fusion features. Anomaly regions can be understood as non-biological material regions in the multimodal segmented image corresponding to the multimodal fusion features of the target user 100 mapped to pixel space, such as adversarial attacks, occlusions, jewelry, and glasses. The multimodal anomaly region segmentation module can be a network used for image segmentation, such as the Unet network. Figure 4 As shown, abnormal region segmentation is performed based on the original modal features and multimodal fusion features corresponding to RGB modality, NIR modality and Depth modality, respectively. This yields the single-modal abnormal region segmentation results and multimodal abnormal region segmentation results for RGB modality, NIR modality and Depth modality.

[0085] (3) After obtaining multiple original modal features, the processor 620 can also input the multiple original modal features into the cross-modal mapping relationship learning module to perform cross-modal feature transformation and obtain cross-modal mapping results. The cross-modal mapping results include multiple cross-modal feature sets corresponding to multiple original modal images. Each cross-modal feature set includes at least one modality-transformed feature of another modality obtained by cross-modal feature transformation based on its corresponding original modal features. In detail, it can be understood that each modality among multiple modalities is denoted as the target modality, and the original modal features corresponding to the target modality are input into the cross-modal mapping relationship learning module to obtain at least one modality-transformed feature corresponding to the target modality.

[0086] This section only describes the implementation process of the processor 620 for adversarial attack classification, anomaly region segmentation, and cross-modal mapping relationship learning based on multiple original modal features and multimodal fusion features. For the pairwise or triadic combinations of adversarial attack classification, anomaly region segmentation, and cross-modal mapping relationship learning, the implementation processes of 1), 2), and 3) above can be referred to for combination, which will not be elaborated here.

[0087] After obtaining the cross-modal mapping results, the comprehensive feature differences corresponding to multiple modalities can be determined based on these results. For example, each modality among the multiple modalities can be designated as the target modality, and the weighted sum of the differences between the original modal features and at least one transformed modal feature corresponding to the target modality can be determined to obtain multiple feature differences corresponding to multiple modalities. Furthermore, based on the weighted sum of the multiple feature differences corresponding to multiple modalities, the comprehensive feature differences corresponding to multiple modalities can be obtained.

[0088] Please continue reading. Figure 4 For example, for the original RGB modality, original NIR modality, and original Depth modality, by inputting the original modality image of the original RGB modality into the multimodal anomaly region perception model, we can obtain the original modality features corresponding to the original RGB modality and the corresponding NIR-converted modality features and Depth-converted modality features after modality conversion. Similarly, we can obtain the RGB-converted modality features and Depth-converted modality features corresponding to the original NIR modality, as well as the RGB features and NIR features corresponding to the original Depth modality. Subsequently, for the RGB original mode, the comprehensive feature difference can be determined based on the cosine similarity 1 between the original modal features corresponding to the RGB mode and the NIR-converted modal features, the cosine similarity 2 between the original modal features corresponding to the RGB mode and the Depth-converted modal features, and for the NIR original mode, based on the cosine similarity 3 between the original modal features corresponding to the NIR mode and the RGB-converted modal features, the cosine similarity 4 between the original modal features corresponding to the NIR mode and the Depth-converted modal features, and for the Depth original mode, based on the cosine similarity 5 between the original modal features corresponding to the Depth mode and the RGB-converted modal features, the cosine similarity 6 between the original modal features corresponding to the Depth mode and the NIR-converted modal features, and the weighted sum of the above cosine similarity 1, cosine similarity 2, cosine similarity 3, cosine similarity 4, cosine similarity 5 and cosine similarity 6, and the weighted average of the above cosine similarity 1, cosine similarity 2, cosine similarity 3, cosine similarity 4, cosine similarity 5 and cosine similarity 6. Alternatively, first perform a weighted summation of cosine similarity 1 and cosine similarity 2, then perform a weighted summation of cosine similarity 3 and cosine similarity 4, and finally perform a weighted summation of cosine similarity 5 and cosine similarity 6. Then, combine the three weighted summations and perform a weighted summation again to obtain the comprehensive feature difference.

[0089] It should be noted that the steps of obtaining cross-modal mapping results and obtaining comprehensive feature differences can both be completed by the cross-modal mapping relationship learning module. Of course, it is also possible that some of these steps can be completed by the cross-modal mapping relationship learning module and others by other separate modalities outside the cross-modal mapping relationship learning module. This specification does not impose any restrictions on this.

[0090] The above describes the application process of the multimodal anomaly region perception model. In practice, before applying the multimodal anomaly region perception model for multimodal anomaly region detection, it is necessary to train the model. The training process of the multimodal anomaly region perception model will be described below. The multimodal anomaly region perception model can be trained using the following steps: acquire multiple original modal image samples of the biometric features of the training user, iteratively train the preset multimodal anomaly region perception model based on the multiple original modal image samples until the training is completed, and obtain the multimodal anomaly region perception model.

[0091] First, the network structure of the preset multimodal anomaly region perception model is introduced. Corresponding to the structure of the multimodal anomaly region perception model, the preset multimodal anomaly region perception model may include a preset multimodal feature encoder, and at least one of the following: a preset multimodal anomaly region segmentation module, a preset adversarial attack classifier, and a preset cross-modal mapping relationship learning module. That is, the preset multimodal anomaly region perception model can be any of the following seven network structures:

[0092] (1) The preset multimodal anomaly region perception model includes a preset multimodal feature encoder and a preset multimodal adversarial attack classifier.

[0093] (2) The preset multimodal abnormal region perception model includes a preset multimodal feature encoder and a preset multimodal abnormal region segmentation module.

[0094] (3) The preset multimodal anomaly region perception model includes a preset multimodal feature encoder and a preset cross-modal mapping relationship learning module.

[0095] (4) The preset multimodal abnormal region perception model includes a preset multimodal feature encoder, a preset multimodal adversarial attack classifier, and a preset multimodal abnormal region segmentation module.

[0096] (5) The preset multimodal anomaly region perception model includes a preset multimodal feature encoder, a preset multimodal adversarial attack classifier, and a preset cross-modal mapping relationship learning module.

[0097] (6) The preset multimodal abnormal region perception model includes a preset multimodal feature encoder, a preset multimodal abnormal region segmentation module, and a preset cross-modal mapping relationship learning module.

[0098] (7) The preset multimodal abnormal region perception model includes a preset multimodal feature encoder, a preset multimodal adversarial attack classifier, a preset multimodal abnormal region segmentation module, and a preset cross-modal mapping relationship learning module.

[0099] The iterative training of a pre-defined multimodal anomaly region perception model can be implemented as follows: For example, multiple original modal image samples are input into a pre-defined multimodal feature encoder to obtain multiple original modal feature samples and multimodal fusion feature samples corresponding to the multiple original modal image samples. There are several ways for the pre-defined multimodal feature encoder to obtain multiple original modal feature samples and multimodal fusion feature samples based on multiple original modal image samples, specifically as follows: For example, the pre-defined multimodal feature encoder performs feature extraction on each original modal image sample in the multiple original modal image samples separately to obtain the original modal feature sample corresponding to each original modal image sample in the multiple original modal image samples. Then, the pre-defined multimodal feature encoder can also combine multiple original modal feature samples to obtain multimodal fusion feature samples. There are several ways to combine multiple original modal feature samples, such as concatenating multiple original modal feature samples to obtain multimodal fusion feature samples, or performing a weighted summation of multiple original modal feature samples to obtain multimodal fusion feature samples.

[0100] After obtaining multiple original modal feature samples and multimodal fusion feature samples, the processor 620 can perform at least one of the following: anomaly region segmentation, multimodal adversarial attack classification, and cross-modal mapping relationship learning. Specifically, it can be done in any of the following seven ways:

[0101] (1) Input multiple original modal feature samples and multimodal fusion feature samples into the preset multimodal abnormal region segmentation module to obtain the training abnormal region segmentation results and the corresponding abnormal region segmentation loss information corresponding to multiple original modal image samples.

[0102] (2) The processor 620 inputs multiple original modal feature samples and multimodal fusion feature samples into a preset multimodal adversarial attack classifier to obtain the training adversarial attack classification results and the corresponding adversarial attack classification loss information.

[0103] (3) The processor 620 inputs multiple original modal feature samples into the preset cross-modal mapping relationship learning module to obtain the training cross-modal mapping results and the corresponding cross-modal feature prediction loss information.

[0104] (4) The processor 620 inputs multiple original modal feature samples and multimodal fusion feature samples into a preset multimodal abnormal region segmentation module to obtain the training abnormal region segmentation results and corresponding abnormal region segmentation loss information corresponding to multiple original modal image samples. In addition, it inputs multiple original modal feature samples and multimodal fusion feature samples into a preset multimodal adversarial attack classifier to obtain the training adversarial attack classification results and corresponding adversarial attack classification loss information.

[0105] (5) The processor 620 inputs the multiple original modal feature samples and the multimodal fusion feature samples to a preset multimodal abnormal region segmentation module to obtain the training abnormal region segmentation results and the corresponding abnormal region segmentation loss information corresponding to the multiple original modal image samples. In addition, the processor 620 inputs the multiple original modal feature samples to a preset cross-modal mapping relationship learning module to obtain the training cross-modal mapping results and the corresponding cross-modal feature prediction loss information.

[0106] (6) The processor 620 inputs multiple original modal feature samples and multimodal fusion feature samples into a preset multimodal adversarial attack classifier to obtain training adversarial attack classification results and corresponding adversarial attack classification loss information. In addition, the processor 620 inputs the multiple original modal feature samples into a preset cross-modal mapping relationship learning module to obtain training cross-modal mapping results and corresponding cross-modal feature prediction loss information.

[0107] (7) The processor 620 inputs multiple original modal feature samples and multimodal fusion feature samples into a preset multimodal abnormal region segmentation module to obtain training abnormal region segmentation results and corresponding abnormal region segmentation loss information corresponding to multiple original modal image samples. It also inputs multiple original modal feature samples and the multimodal fusion feature samples into a preset multimodal adversarial attack classifier to obtain training adversarial attack classification results and corresponding adversarial attack classification loss information. Furthermore, it inputs the multiple original modal feature samples into a preset cross-modal mapping relationship learning module to obtain training cross-modal mapping results and corresponding cross-modal feature prediction loss information.

[0108] The following sections will describe the implementation process of anomaly region segmentation, adversarial attack classification, and cross-modal mapping relationship learning, along with their respective loss information:

[0109] (1) Input multiple original modal feature samples and multimodal fusion feature samples into a preset multimodal anomaly region segmentation module to obtain training anomaly region segmentation results and corresponding anomaly region segmentation loss information corresponding to multiple original modal image samples. The training anomaly region segmentation results include multiple single-modal anomaly region segmentation results and multimodal anomaly region segmentation results. The anomaly region segmentation loss information includes at least one of the following: multiple single-modal segmentation loss information corresponding to multiple single-modal anomaly region segmentation results, consistency loss information of multiple single-modal segmentation results, and multimodal segmentation loss information corresponding to multimodal anomaly region segmentation results.

[0110] The anomaly region segmentation loss information can be determined using the following steps: For example, the processor 620 inputs multiple original modal feature samples into a preset multimodal anomaly region segmentation module, so that the preset multimodal anomaly region segmentation module segments the anomaly regions in the corresponding original modal image samples based on each original modal feature sample, obtaining the single-modal anomaly region segmentation result corresponding to each original modal image sample. Furthermore, based on the difference between each single-modal anomaly region segmentation result and the labeled anomaly region, the single-modal segmentation loss information corresponding to each modality is determined. After performing the above anomaly region segmentation steps for each modality in multiple modalities, multiple single-modal segmentation loss information corresponding to multiple modalities can be obtained.

[0111] Here, the pre-defined multimodal anomaly region segmentation module can be viewed as a network structure comprising multiple single-modal segmenter branches and a multimodal segmenter branch. Each modality among the multiple modalities can be designated as the target modality. The original modal feature samples corresponding to the target modality are input into the corresponding single-modal segmenter branch for anomaly region segmentation, yielding the single-modal anomaly region segmentation result for the target modality. Furthermore, based on the difference between the single-modal anomaly region segmentation result and the labeled anomaly region, the single-modal segmentation loss information corresponding to the target modality is determined. Similarly, the multimodal fusion feature samples are input into the multimodal segmenter branch for anomaly region segmentation, yielding the multimodal anomaly region segmentation result. Based on the difference between the multimodal anomaly region segmentation result and the labeled anomaly region, the multimodal segmentation loss information is determined. The multiple single-modal segmentation loss information aims to constrain each of the multiple original modal images to perform correct anomaly region segmentation, ensuring the accuracy of anomaly region segmentation. The pre-defined multimodal anomaly region segmentation module performs anomaly region segmentation on multimodal image samples (images mapped from multimodal fusion feature samples to pixel space) based on multimodal fusion feature samples, obtaining the multimodal anomaly region segmentation results corresponding to the multimodal image samples. Furthermore, based on the difference between the multimodal anomaly region segmentation results and the labeled anomaly regions, multimodal segmentation loss information is determined. Multimodal fusion feature samples are features that integrate information from multiple modalities, resulting in a more accurate representation of image features and richer semantic information. Therefore, anomaly region segmentation based on multimodal fusion feature samples can yield more accurate anomaly region segmentation results.

[0112] After obtaining the segmentation results of multiple single-modal anomalous regions corresponding to multiple modalities, the consistency loss information of the multiple single-modal segmentation results can be determined based on the differences between them. For example, for the single-modal segmentation results corresponding to RGB, NIR, and Depth modes, the consistency loss information between the RGB and NIR modes can be determined based on the similarity between the RGB and NIR modes; the consistency loss information between the RGB and Depth modes can be determined based on the similarity between the RGB and Depth modes; and the consistency loss information between the NIR and Depth modes can be determined based on the similarity between the NIR and Depth modes. Then, the consistency loss information of segmentation results between RGB and NIR modes, between RGB and Depth modes, and between NIR and Depth modes is calculated by weighted summation to obtain the consistency loss information of multiple single-modal segmentation results. Here, the similarity between each pairwise segmentation result of the RGB, NIR, and Depth modes can be calculated in various ways. For example, cosine similarity, Euclidean distance, or L2 distance can be used to determine the pairwise similarity between multiple modes. The consistency loss information of multiple modal segmentation results is used to constrain the segmentation results of multiple modalities to remain consistent.

[0113] After obtaining multiple single-modal segmentation loss information, multi-modal segmentation loss information, and multiple single-modal segmentation result consistency loss information, these information can be accumulated to obtain the abnormal region segmentation loss information.

[0114] (2) Input multiple original modal feature samples and multimodal fusion feature samples into a preset adversarial attack classifier to obtain adversarial attack classification results and corresponding adversarial attack classification loss information. The adversarial attack classification results include multiple single-modal adversarial attack classification results corresponding to multiple modalities and their corresponding single-modal adversarial attack classification loss information, as well as multimodal adversarial attack classification results and their corresponding multimodal adversarial attack classification loss information. There are multiple ways to implement the preset adversarial attack classifier to perform adversarial attack classification based on multiple original modal feature samples and multimodal fusion feature samples to obtain adversarial attack classification results and corresponding adversarial attack classification loss information. Specifically, it can be as follows: For example, the preset adversarial attack classifier can be regarded as a network structure including multiple single-modal classifier branches and multimodal classifier branches. Each modality in the multiple modalities is denoted as the target modality, and the original modal feature samples corresponding to the target modality are input into the corresponding single-modal classifier branch for adversarial attack classification to obtain the single-modal adversarial attack classification result corresponding to the target modality. Furthermore, based on the difference between the single-modal adversarial attack classification result and the labeled adversarial attack classification result, the single-modal adversarial attack classification loss information corresponding to the target modality is determined. Multimodal fusion feature samples are input into the multimodal classifier branch for adversarial attack classification, resulting in multimodal adversarial attack classification results. Based on the difference between the multimodal adversarial attack classification results and the labeled adversarial attack classification results, the multimodal adversarial attack classification loss information can be determined.

[0115] (3) Input multiple original modal feature samples into the cross-modal mapping relationship learning module to obtain the training cross-modal mapping results and the corresponding cross-modal feature prediction loss information. The training cross-modal mapping results include multiple training-to-modal features corresponding to multiple original modal image samples. Each training-to-modal feature includes features of other modalities obtained by cross-modal feature transformation based on the original modal features of its corresponding original modal image sample. The cross-modal feature prediction loss information is determined based on the difference between the original modal feature sample and the transformed modal feature sample corresponding to each original modal image sample in the multiple original modal image samples of normal training users.

[0116] The cross-modal mapping relationship learning module performs cross-modal feature transformation based on the original modal feature samples of multiple modalities to obtain at least one transformed modal feature sample corresponding to each modality in multiple modalities. For the specific implementation method of performing cross-modal feature transformation on the original modal features to obtain at least one transformed modal feature corresponding to each modality in multiple modalities, please refer to the above implementation method of performing cross-modal feature transformation on the original modal features to obtain at least one transformed modal feature corresponding to each modality in multiple modalities, which will not be repeated here.

[0117] Here, the cross-modal feature prediction loss information can include cross-modal feature prediction loss sub-information corresponding to each modality in multiple modalities. The cross-modal feature prediction loss sub-information corresponding to each modality can be obtained based on the weighted sum of the differences between the original modal feature samples corresponding to normal training users and at least one corresponding transmodal feature sample.

[0118] It's important to note that the cross-modal feature prediction loss information here refers to the cross-modal feature prediction loss information corresponding to the original modal image samples from normal training users. During training, each original modal image sample is labeled with either the category of a normal training user or the adversarial attack target. By identifying the labeled categories, the cross-modal feature prediction loss information of the original modal image sample can be determined.

[0119] To determine if a user is a normal training user, the module can then input the original modal features of normal training users into the cross-modal mapping relationship learning module. Alternatively, the original modal features of all training users can be input into the cross-modal mapping relationship learning module.

[0120] Obtain the transmodal feature samples of all training users, and use the transmodal feature samples of normal training users to determine the cross-modal feature prediction loss information for normal training users. Alternatively, other methods can be used to determine the cross-modal feature prediction loss information for normal training users. Regardless of the method chosen, it is essential to ensure that the cross-modal feature prediction loss information for normal training users is used.

[0121] The loss information from the state feature prediction is backpropagated to update the network parameters of the cross-modal mapping relationship learning module. This ensures that the cross-modal mapping relationship learning module is trained using multimodal image samples from normal training users.

[0122] The cross-modal mapping learning module can ensure that it learns the mapping relationship between the multimodal image samples of normal training users and the transformed modal features. Since the cross-modal feature mapping relationship of adversarial attack targets is significantly different from that of normal users, when the multimodal image samples of adversarial attack targets are input into the cross-modal mapping learning module,

[0123] This mapping relationship will not be applicable to adversarial attack targets, thus providing additional detection information for adversarial attack detection, which is conducive to effectively detecting adversarial attack targets.

[0124] After obtaining at least one of the anomaly region segmentation loss information, adversarial attack classification loss information, and cross-modal feature prediction loss information, the pre-defined multimodal anomaly region perception model can be iteratively trained based on at least one of these loss information until the model is fully trained.

[0125] The process ends, and a multimodal anomaly region perception model is obtained. Specifically, it can include any one of the following seven cases: 0(1) Iteratively train the preset multimodal anomaly region perception model based on the anomaly region segmentation loss information.

[0126] During iterative training, the following methods can be used: For example, backpropagation can be performed on the corresponding single-modal segmenter branch based on the loss information of each single-modal segmentation and the consistency loss information of multiple modal segmentation results, in order to update the network parameters of the single-modal segmenter branch and the single-modal feature encoder branch; or, based on the loss information of each single-modal segmentation and the consistency loss information of multiple modal segmentation results, the network parameters of the single-modal segmenter branch and the single-modal feature encoder branch can be updated.

[0127] The weighted sum of the consistency loss information of each modal segmentation result is backpropagated to the five branches of the single-modal segmenter branch and the single-modal feature encoder branch to update the network parameters of the single-modal segmenter branch and the single-modal feature encoder branch. And,

[0128] Backpropagation is performed on the multimodal segmenter branch based on the multimodal segmentation loss information to update the network parameters of the multimodal segmenter branch. The total loss information during iterative training can be expressed as the following formula (1):

[0129] Loss_total1=Loss_seg+Loss_consistency; (1)

[0130] In equation (1), Loss_total1 is the total loss information during iterative training, Loss_seg includes multiple single-modal segmentation loss information and multimodal segmentation loss information, and Loss_consistency is the consistency loss information of multiple modal segmentation results. When Loss_total1 is minimized or the preset number of training iterations is reached, the training ends, and the multimodal anomaly region perception model is obtained.

[0131] (2) Iteratively train the preset multimodal anomaly region perception model based on the adversarial attack classification loss information. During iterative training, the following methods can be used: for example, backpropagation is performed on the corresponding single-modal classifier branch based on multiple single-modal adversarial attack classification loss information to update the network parameters of the single-modal classifier branch, and backpropagation is performed on the corresponding multimodal classifier branch based on the multimodal adversarial attack classification loss information to update the network parameters of the multimodal classifier branch. The total loss information during iterative training can be expressed as the following formula (2):

[0132] Loss_total2 = Loss_cls; (2)

[0133] In equation (2), Loss_total2 represents the total loss information during iterative training, and Loss_cls represents the adversarial attack classification loss information, including multiple unimodal adversarial attack classification loss information and multimodal adversarial attack classification loss information. When Loss_total2 is minimized or reaches the preset number of training iterations, training ends, and the multimodal anomaly region perception model is obtained.

[0134] (3) Iteratively train the preset multimodal anomaly region perception model based on cross-modal feature prediction loss information. During iterative training, the following method can be used: For example, the preset cross-modal mapping relationship learning module can be understood as including multiple single-modal mapping relationship learning branches. Based on the cross-modal feature prediction loss information of the single modality, backpropagation is performed on the corresponding single-modal mapping relationship learning branch to update the network parameters of the single-modal mapping relationship learning branch. The total loss information during iterative training can be expressed as the following formula (3):

[0135] Loss_total3 = Loss_feat; (3)

[0136] In equation (3), Loss_total3 represents the total loss information, and Loss_feat represents the cross-modal mapping relationship learning loss information. Training ends when Loss_total3 is minimized or the preset number of training iterations is reached, resulting in a multimodal anomaly region perception model.

[0137] (4) The pre-defined multimodal anomaly region perception model is iteratively trained based on the anomaly region segmentation loss information and the adversarial attack classification loss information. The total loss information during iterative training can be expressed as the following formula (4):

[0138] Loss_total4=Loss_seg+Loss_consistency+Loss_cls; (4)

[0139] In equation (4), Loss_total4 represents the total loss information, Loss_seg includes multiple unimodal segmentation loss information and multimodal segmentation loss information, Loss_consistency represents the consistency loss information of multiple modal segmentation results, and Loss_cls represents the adversarial attack classification loss information. Training ends when Loss_total4 is minimized or the preset number of training iterations is reached, resulting in a multimodal anomaly region perception model. Backpropagation is performed based on Loss_seg and Loss_consistency, and also based on Loss_cls. For details on the backpropagation process, please refer to the descriptions in the first and second cases above.

[0140] (5) The pre-defined multimodal anomaly region perception model is iteratively trained based on the anomaly region segmentation loss information and the cross-modal feature prediction loss information. The total loss information during iterative training can be expressed as the following formula (5):

[0141] Loss_total5=Loss_seg+Loss_consistency+Loss_feat; (5)

[0142] In equation (5), Loss_total5 represents the total loss information, Loss_seg includes multiple single-modal segmentation loss information and multimodal segmentation loss information, Loss_consistency represents the consistency loss information of multiple modal segmentation results, and Loss_feat represents the cross-modal mapping relationship learning loss information. Training ends when Loss_total5 is minimized or the preset number of training iterations is reached, resulting in a multimodal anomaly region perception model. Backpropagation is performed based on Loss_seg and Loss_consistency, and also based on Loss_feat. For details on the backpropagation process, please refer to the descriptions in the first and third cases above.

[0143] (6) The pre-defined multimodal anomaly region perception model is iteratively trained based on the adversarial attack classification loss information and the cross-modal feature prediction loss information. The total loss information during iterative training can be expressed as the following formula (6):

[0144] Loss_total6=Loss_cls+Loss_feat; (6)

[0145] In equation (6), Loss_total6 represents the total loss information, Loss_cls represents the adversarial attack classification loss information, and Loss_feat represents the cross-modal mapping relationship learning loss information. Training ends when Loss_total6 is minimized or the preset number of training iterations is reached, resulting in a multimodal anomaly region perception model. Backpropagation is performed based on both Loss_cls and Loss_feat. For details on the backpropagation process, please refer to the descriptions in the second and third cases above.

[0146] (7) The pre-defined multimodal anomaly region perception model is iteratively trained based on the anomaly region segmentation loss information, adversarial attack classification loss information, and cross-modal feature prediction loss information. The total loss information during iterative training can be expressed as the following formula (7):

[0147] Loss_total7=Loss_seg+Loss_consistency+Loss_cls+Loss_feat; (7)

[0148] In equation (7), Loss_total7 represents the total loss information, Loss_seg includes multiple single-modal segmentation loss information and multimodal segmentation loss information, Loss_consistency represents the consistency loss information of multiple modal segmentation results, Loss_cls represents the adversarial attack classification loss information, and Loss_feat represents the cross-modal mapping relationship learning loss information. Training ends when Loss_total7 is minimized or the preset number of training iterations is reached, resulting in a multimodal anomaly region perception model. Backpropagation is performed based on Loss_seg and Loss_consistency, based on Loss_cls, and based on Loss_feat. For details on the backpropagation process, please refer to the descriptions in the first, second, and third cases above.

[0149] S130: Based on the multimodal anomaly region detection results, determine whether the target user 100 is a target of adversarial attack.

[0150] Step S130 can be implemented as follows: For example, based on the abnormal region detection results, detection data is determined. The detection data includes at least one of the following: multimodal adversarial attack probability, abnormal image region area ratio, and comprehensive feature difference. Specifically, the detection data includes the multimodal adversarial attack probability, abnormal image region area ratio, or comprehensive feature difference. Alternatively, the detection data includes the multimodal adversarial attack probability and the abnormal image region area ratio. Alternatively, the detection data includes the multimodal adversarial attack probability and comprehensive feature difference. Alternatively, the detection data includes the abnormal image region area ratio and comprehensive feature difference. Alternatively, the detection data includes the multimodal adversarial attack probability, abnormal image region area ratio, and comprehensive feature difference.

[0151] The multimodal adversarial attack probability is the adversarial attack probability obtained by classifying adversarial attacks based on multimodal fusion features, and is used to characterize the probability that target user 100 belongs to the adversarial attack object.

[0152] The percentage of abnormal image region area includes the proportion of the area of ​​the abnormal image region in the multimodal segmentation image corresponding to multiple original modal images. For example, multimodal fusion features can be mapped to pixel space to obtain a multimodal segmentation image. The multimodal segmentation image may include one or more abnormal image regions. The area of ​​the one or more abnormal image regions is added together and divided by the area of ​​the multimodal segmentation image to obtain the percentage of abnormal image region area.

[0153] The comprehensive feature difference includes the fusion of the differences between the original modal features of multiple original modal images and their corresponding at least one cross-modal feature. For the specific implementation process of the comprehensive feature difference, reference can be made to the introduction in the foregoing content, which will not be elaborated here. After determining the detection data, it is possible to determine whether the target user 100 is an object of adversarial attack based on the detection data. There are various ways to determine whether the target user 100 is an object of adversarial attack based on the detection data, specifically as follows: For example, determining that at least one of the data in the detection data is greater than its corresponding preset first threshold, and determining that the target user 100 is an object of adversarial attack, where the first threshold includes at least one of the first probability threshold, the first area occupancy threshold, and the first feature difference threshold. Among them, the first probability threshold corresponds to the multi-modal adversarial attack probability, the first area occupancy threshold corresponds to the abnormal image area occupancy, and the first feature difference threshold corresponds to the comprehensive feature difference; determining that all of the data in the detection data are less than their corresponding preset second threshold, and determining that the target user 100 is a legitimate user, where the second threshold includes at least one of the second probability threshold, the second area occupancy threshold, and the second feature difference threshold. Among them, the second probability threshold corresponds to the multi-modal adversarial attack probability, the second area occupancy threshold corresponds to the abnormal image area occupancy, and the second feature difference threshold corresponds to the comprehensive feature difference; and otherwise, determining that the target user 100 is a pending user.

[0154] Denote the first probability threshold as Tp1, the second probability threshold as Tp2, the first area occupancy threshold as Tr1, the second area occupancy threshold as Tr2, the first feature difference threshold as Td1, and the second feature difference threshold as Td2. When at least one of P > Tp1, r > Tr1, and d > Td1 is satisfied, the target user 100 is identified as an object of adversarial attack. That is, when P > Tp1, the target user 100 is determined to be an object of adversarial attack. Or, when r > Tr1, the target user 100 is determined to be an object of adversarial attack. Or, when d > Td1, the target user 100 is determined to be an object of adversarial attack. Or, when P > Tp1 and r > Tr1, the target user 100 is determined to be an object of adversarial attack. Or, when P > Tp1 and d > Td1, the target user 100 is determined to be an object of adversarial attack. Or, when r > Tr1 and d > Td1, the target user 100 is determined to be an object of adversarial attack. Or, when P > Tp1, r > Tr1, and d > Td1 are all satisfied simultaneously, the target user 100 is determined to be an object of adversarial attack. And when P < Tp2, r < Tr2, and d < Td2 are all satisfied simultaneously, the target user 100 is identified as a legitimate user. For other cases except the above, for example, when Tp2 < p < Tp1, Tr2 < r < Tr1, and Td2 < d < Td1 are all satisfied simultaneously, the target user 100 is identified as a pending user.

[0155] When at least one of P = Tp1, r = Tr1, and d = Td1 is satisfied, the target user 100 can be identified as an adversarial attack target. Or, when Tp2 < p ≤ Tp1, Tr2 < r ≤ Tr1, and Td2 < d ≤ Td1 are all satisfied simultaneously, the target user 100 is identified as a pending user. Or, when Tp2 ≤ p < Tp1, Tr2 ≤ r < Tr1, and Td2 ≤ d < Td1 are all satisfied simultaneously, the target user 100 is identified as a pending user. Or, when Tp2 ≤ p ≤ Tp1, Tr2 ≤ r ≤ Tr1, and Td2 ≤ d ≤ Td1 are all satisfied simultaneously, the target user 100 is identified as a pending user.

[0156] After determining that the target user 100 is a pending user, the pending user can be re-detected to determine whether the user is an adversarial attack target. There are multiple ways to implement the re-detection of the pending user, which can be specifically as follows: For example, determine the first comparison score before preprocessing and the second comparison score after preprocessing of the original modal image of the target modality among multiple original modal images of the target user 100, and based on the comparison score difference between the first comparison score and the second comparison score, determine whether the pending user is an adversarial attack target. Based on the comparison score difference between the first comparison score and the second comparison score, determining whether the pending user is an adversarial attack target includes: determining that the comparison score difference is greater than a preset threshold, then identifying the target user 100 as an adversarial attack target. Or, determining that the comparison score difference is less than the preset threshold, then identifying the target user 100 as a legitimate user. The target modality can be any modality among multiple modalities.

[0157] For the case where the comparison score difference is equal to the preset threshold, the target user 100 can be identified as an adversarial attack target or a legitimate user.

[0158] The first comparison score can be determined based on the following method: For example, obtain the original image before preprocessing of the original modal image of the target modality and the user's archived image in the adversarial attack detection scenario, and based on the difference between the original image and the user's archived image, determine the first comparison score. The second comparison score can be determined based on the following method: For example, set the pixels of the abnormal region in the single-modal abnormal region segmentation result corresponding to the target modality to 0 to obtain a restored image, and based on the difference between the restored image and the user's archived image, obtain the second comparison score. The user's archived image is a pre-recorded user image. For example, in the access control scenario, a pre-recorded user's face image for face comparison, in the face payment scenario, a pre-recorded user's face image for face comparison, and so on.

[0159] For example, when a user registers an account on website A, they input a facial image as their Face ID. This Face ID image is the user's backup image. Let's denote the unprocessed original image as x, and the user's backup image as x0. By calculating the cosine similarity between x and x0, we can obtain the first comparison score s. Similarly, by calculating the cosine similarity between x1 and x0, we can obtain the second comparison score s1. Then, by calculating the difference ds = abs(s-s1), we can obtain the comparison score difference. We compare ds with a pre-set threshold T. If ds is greater than the pre-set threshold T, the user is identified as an adversarial attack target; otherwise, they are considered a normal user. This two-stage comparison score verification is based on the characteristic that there are significant differences between the multimodal images of normal users and adversarial attack targets before and after preprocessing—that is, the significant difference in the comparison results between the multimodal images and the backup images before and after the adversarial sticker is added. This further refines the user identification process, thereby improving the accuracy of adversarial attack detection.

[0160] Here, the first step is to classify adversarial attacks based on the multimodal anomaly region detection results to determine whether target user 100 is a target of the adversarial attack. If the first-stage judgment cannot determine whether target user 100 is a target of the adversarial attack, that is, if the first-stage judgment result is that target user 100 is a pending user, then a second-stage comparison and verification method is used to classify the adversarial attack. Through two stages of adversarial attack detection, the ability to detect adversarial attacks can be further improved.

[0161] To improve the detection capability of adversarial attacks, this solution proposes a two-stage verification adversarial attack detection scheme based on multimodal anomaly region perception. This scheme mainly consists of four parts: data acquisition and preprocessing, multimodal anomaly region perception, two-stage perception using a comparison model, and adversarial attack detection, as detailed below:

[0162] (1) Data acquisition and preprocessing: After the target user 100 starts face recognition, multiple original modal images corresponding to multiple modalities are acquired and data preprocessed. Data preprocessing can include face detection to obtain the face region.

[0163] (2) Multimodal anomaly region perception: Anomaly region detection and perception are performed by designing a multimodal feature fusion network. The trained multimodal anomaly region perception model can perceive anomaly regions that may be adversarial stickers before face recognition.

[0164] (3) Compare the two-stage perception of the comparison model and compare the differences in multimodal comparison scores of abnormal regions before and after preprocessing in order to detect adversarial attacks.

[0165] (4) Adversarial attack detection: Adversarial attacks are detected by analyzing the differences in scores.

[0166] In summary, the adversarial attack detection method P100 and system 001 provided in this specification acquire multiple original modal images of the biometric features of the target user 100, input these original modal images into a multimodal anomaly region perception model, obtain multimodal anomaly region detection results corresponding to the multiple original modal images, and determine whether the target user 100 is an adversarial attack target based on the multimodal anomaly region detection results. Since the anomaly region detection results fully utilize multimodal information, the detection accuracy of adversarial attacks can be improved. Regarding adversarial attack classification results, the use of multimodal information improves the accuracy of adversarial attack classification; regarding anomaly region segmentation, the use of multimodal information improves the accuracy of segmentation results; and regarding the cross-modal mapping relationship learning module, it uses original modal image samples from normal training users for training, enabling it to learn the mapping relationship between the original modal image samples of normal training users and predicted features, thereby significantly distinguishing it from the mapping relationship between the original modal image samples and predicted features of adversarial attack users, effectively detecting adversarial attack targets. Furthermore, the abnormal image region segmentation results can provide a perception effect of abnormal regions that may be adversarial stickers before face recognition. Subsequently, adversarial attack detection can be performed based on the multimodal abnormal region segmentation results, which can further improve the detection accuracy of adversarial attacks. This scheme performs abnormal region perception in the pre-face recognition stage and performs adversarial attack detection based on the perceived abnormal regions during the face recognition stage.

[0167] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for performing anti-attack detection. When the executable instructions are executed by a processor, they instruct the processor to implement the steps of the anti-attack detection method P100 described in this specification. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on a computing device 600, the program code causes the computing device 600 to perform the steps of the anti-attack detection method P100 described in this specification. The program product for implementing the above method may employ a portable compact disk read-only memory (CD-ROM) containing program code and may run on the computing device 600. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on computing device 600, partially on computing device 600, as a standalone software package, partially on computing device 600 and partially on a remote computing device, or entirely on a remote computing device.

[0168] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0169] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0170] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0171] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0172] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. All contents used for all purposes, except for any history of prosecution documents relating to it, that may be inconsistent with or conflict with this document, or any such history of prosecution documents that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0173] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for detecting adversarial attacks, comprising: Acquire multiple original modal images of the target user's biometrics, wherein the multiple original modal images correspond to the images actually acquired in multiple modalities of the image acquisition module; The multiple original modal images are input into a multimodal anomaly region perception model to obtain multimodal anomaly region detection results and single-modal anomaly region segmentation results corresponding to the multiple original modal images in multiple modalities; Based on the multimodal anomaly region detection results, the target user is determined to be an adversarial attack target, a legitimate user, or a user to be identified. as well as In response to determining that the target user is the pending user, the pending user is re-detected, including: The process involves acquiring the original image of the target modality before preprocessing and the user-recorded image under the adversarial attack detection scenario, and determining a first comparison score based on the difference between the original image and the user-recorded image; the target modality is at least one of the multiple modalities, and the preprocessing includes detecting the target part in the original modality image of the target modality to obtain the target part image; The pixels of the abnormal region in the single-modal abnormal region segmentation result corresponding to the target modality are set to 0 to obtain the restored image. Based on the difference between the restored image and the user-retained image, a second comparison score is obtained. Determine the comparison difference between the first comparison score and the second comparison score; If the comparison difference value is greater than a preset threshold, the user to be identified as a target of adversarial attack; or if the comparison difference value is less than the threshold, the user to be identified as a legitimate user.

2. The method of claim 1, wherein, The multimodal anomaly region detection results include at least one of the following: multimodal adversarial attack classification results, multimodal anomaly region segmentation results, and cross-modal mapping results.

3. The method according to claim 2, wherein, The multimodal anomaly region perception model includes a multimodal feature encoder, and also includes at least one of a multimodal adversarial attack classifier, a multimodal anomaly region segmentation module, and a cross-modal mapping relationship learning module; as well as The step of inputting the multiple original modal images into a multimodal anomaly region perception model to obtain multimodal anomaly region detection results corresponding to the multiple original modal images includes: The multiple original modal images are input into the multimodal feature encoder for feature extraction, resulting in multiple original modal features and multimodal fusion features corresponding to the multiple original modal images; and At least one of the following: The multimodal fusion features are input into the multimodal adversarial attack classifier to obtain the multimodal adversarial attack classification result; The multimodal fusion features are input into the multimodal anomaly region segmentation module to obtain the multimodal anomaly region segmentation result; The multiple original modal features are input into the cross-modal mapping relationship learning module to obtain the cross-modal mapping result.

4. The method according to claim 1, wherein, The multimodal anomaly region perception model is trained using the following steps: Multiple original modal image samples of the biometrics of the training user are input into a preset multimodal feature encoder to obtain multiple original modal feature samples and multimodal fusion feature samples corresponding to the multiple original modal image samples; And, at least one of the following: The multiple original modal feature samples and the multimodal fusion feature samples are input into a preset multimodal abnormal region segmentation module to obtain the training abnormal region segmentation results and the corresponding abnormal region segmentation loss information corresponding to the multiple original modal image samples. The original modal feature samples and the multimodal fusion feature samples are input into a preset multimodal adversarial attack classifier to obtain the training adversarial attack classification result and the corresponding adversarial attack classification loss information. The multiple original modal feature samples are input into a preset cross-modal mapping relationship learning module to obtain the training cross-modal mapping results and the corresponding cross-modal feature prediction loss information; as well as The multimodal anomaly region perception model is obtained by iteratively training based on at least one of the anomaly region segmentation loss information, the adversarial attack classification loss information, and the cross-modal feature prediction loss information until the training ends.

5. The method according to claim 4, wherein, The training abnormal region segmentation results include multiple single-modal segmentation results and multimodal segmentation results corresponding to multiple modalities. The abnormal region segmentation loss information includes at least one of the following: multiple single-modal segmentation loss information corresponding to multiple modalities, multimodal segmentation loss information, and consistency loss information of multiple single-modal segmentation results. The multiple single-modal segmentation loss information is determined based on the differences between the multiple single-modal segmentation results corresponding to the multiple modalities and the labeled abnormal regions; The multimodal segmentation loss information is determined based on the difference between the multimodal segmentation result and the labeled abnormal regions; The consistency loss information of the multiple single-modal segmentation results is determined based on the differences between the multiple single-modal segmentation results.

6. The method according to claim 4, wherein, The training cross-modal mapping result includes multiple training-to-modal features corresponding to the multiple original modal image samples. Each training-to-modal feature includes features of other modalities obtained by cross-modal feature transformation based on the original modal features of its corresponding original modal image sample. The cross-modal feature prediction loss information is determined based on the difference between the original modal feature sample and the converted modal feature sample corresponding to each of the multiple original modal image samples.

7. The method according to claim 2, wherein, The multimodal adversarial attack classification result includes the multimodal adversarial attack probabilities corresponding to the multiple original modal images. The multimodal anomaly region segmentation result includes the multimodal anomaly image regions corresponding to the multiple original modal images. The cross-modal mapping result includes multiple cross-modal feature sets corresponding to the multiple original modal images. Each cross-modal feature set includes at least one modality-transformed feature of another modality obtained by cross-modal feature transformation based on its corresponding original modal features.

8. The method according to claim 7, wherein, The step of determining whether the target user is an adversarial attack target, a legitimate user, or a pending user based on the multimodal anomaly region detection results includes: Based on the abnormal region detection results, detection data is determined. This detection data includes at least one of the multimodal adversarial attack probability, abnormal image region area ratio, and comprehensive feature differences. The abnormal image region area ratio includes the area ratio of the abnormal image region in the multimodal segmentation images corresponding to the multiple original modal images. The comprehensive feature differences include the fusion of differences between the original modal features of the multiple original modal images and at least one corresponding transmodal feature. Based on the detection data, it is determined whether the target user is a target of adversarial attacks.

9. The method according to claim 8, wherein, The determination of whether the target user is an adversarial attack target, a legitimate user, or a pending user based on the detection data includes one of the following: If at least one data point in the detection data is greater than a corresponding preset first threshold, the target user is determined to be the adversarial attack target. The first threshold includes at least one of a first probability threshold, a first area proportion threshold, and a first feature difference threshold. The first probability threshold corresponds to the multimodal adversarial attack probability, the first area proportion threshold corresponds to the abnormal image area proportion, and the first feature difference threshold corresponds to the comprehensive feature difference. Determine that all data in the detected data are less than their corresponding preset second threshold, and determine that the target user is the legitimate user. The second threshold includes at least one of a second probability threshold, a second area proportion threshold, and a second feature difference threshold. The second probability threshold corresponds to the probability of a multimodal adversarial attack, the second area proportion threshold corresponds to the area proportion of the abnormal image, and the second feature difference threshold corresponds to the comprehensive feature difference. Otherwise, the target user is determined as the pending user.

10. The method according to claim 9, wherein, The differences in the comprehensive features are determined using the following methods and steps: Each of the multiple modalities is designated as the target modality, and a weighted sum of the differences between the original modal features corresponding to the target modality and at least one transformed modal feature is determined to obtain multiple feature differences corresponding to the multiple modalities; The weighted sum of the multiple feature differences is determined to obtain the comprehensive feature difference.

11. A detection system against attacks, comprising: At least one storage medium storing at least one instruction set for detecting adversarial attacks; as well as At least one processor is communicatively connected to the at least one storage medium. When the anti-attack detection system is running, the at least one processor reads the at least one instruction set and executes the method according to any one of claims 1-10 according to the instructions of the at least one instruction set.

Images