Method and system for detecting adversarial attacks

Abstract

The method and system for detecting adversarial attacks provided in the specification, after obtaining the face video of the target user under multiple colors of light, the pixel of each face image frame in the multiple face image frames of the face video is classified by using an adversarial attack detection model to obtain the pixel material category of each pixel, and based on the pixel material category, the adversarial attack detection result of the target user is determined, and the adversarial attack detection result is output; the scheme can improve the detection accuracy of adversarial attacks.

Description

Technical Field

[0001] This specification relates to the field of image recognition, and in particular to a method and system for detecting adversarial attacks. Background Technology

[0002] In recent years, with the rapid development of internet technology, the application of facial recognition has become increasingly widespread. In the facial recognition process, adversarial attacks, due to their stealth, are among the most dangerous attack methods. Taking facial recognition as an example, adversarial attacks refer to attacks such as attaching adversarial stickers or wearing adversarial glasses to the face area, causing the facial recognition system to misjudge (for example, user A is incorrectly identified as user B after attaching an adversarial sticker). Existing methods for detecting adversarial attacks are often based on techniques for discovering and detecting adversarial attacks or on techniques for neutralizing adversarial attacks.

[0003] In the process of researching and practicing existing technologies, the inventors of this application found that adversarial attack-based detection methods are difficult to accurately detect attacks with small facial features. While adversarial attack invalidation methods invalidate adversarial elements, they also preprocess normal samples, thus affecting the success rate of normal samples in subsequent facial recognition. Therefore, the detection accuracy of adversarial attacks is low. Summary of the Invention

[0004] This specification provides a more accurate method and system for detecting adversarial attacks.

[0005] In a first aspect, this specification provides a method for detecting adversarial attacks, comprising: acquiring a facial video of a target user under multiple colors of light, the facial video comprising multiple facial image frames; employing an adversarial attack detection model to classify the pixels of each facial image frame in the multiple facial image frames into a pixel material category; and determining the adversarial attack detection result of the target user based on the pixel material category, and outputting the adversarial attack detection result.

[0006] In some embodiments, acquiring a facial video of a target user under multiple colors of light includes: selecting multiple colors from a preset color set and determining the target emission order corresponding to the multiple colors; displaying the color light corresponding to the multiple colors based on the target emission order and a preset display time; and acquiring a facial video of the target user's face under the reflection of the color light.

[0007] In some embodiments, the plurality of colors includes at least two of red, blue, yellow, or green.

[0008] In some embodiments, the color light corresponding to each of the multiple colors emits light individually, and the continuous emission time exceeds a preset time threshold.

[0009] In some embodiments, acquiring a facial video of the target user's face under the reflection of the colored light includes: performing face detection on a preset number of consecutive video frames in the acquired real-time facial video of the target user; when the target user's face is not detected in the consecutive video frames, stopping the display of the colored lights corresponding to the multiple colors and displaying a prompt message so that the target user can adjust the acquisition position based on the prompt message; and returning to the step of selecting multiple colors from the preset color set until the target user's face is detected in the consecutive video frames, thereby obtaining a facial video of the target user's face under the reflection of the colored light.

[0010] In some embodiments, the adversarial attack detection model includes a pixel material classification network; and the step of using the adversarial attack detection model to classify the material of pixels in each of the multi-frame facial image frames to obtain the pixel material category of each pixel includes: using the pixel material network to extract features from each of the multi-frame facial image frames to obtain a feature map corresponding to each facial image frame; classifying the material of each pixel in the corresponding facial image frame based on the feature map to obtain the pixel material classification probability of each pixel; and determining the pixel material category of each pixel based on the pixel material classification probability.

[0011] In some embodiments, the training process of the pixel material classification network includes the following steps: acquiring facial image samples, the facial image samples including facial images labeled with material categories; using a preset pixel material classification network to predict the prediction region boundary and the predicted material category of each pixel in the facial image samples, the prediction region boundary being the boundary between the predicted live area and the attack area in the facial image samples; and determining the target classification loss information of the facial image samples based on the prediction region boundary, the predicted material category and the labeled material category, and converging the preset pixel material classification network based on the target classification loss information to obtain the pixel material classification network.

[0012] In some embodiments, the preset pixel material classification network includes a feature map encoding subnetwork, a pixel material classification subnetwork, and a boundary awareness subnetwork; and the step of using the preset pixel material classification network to predict the boundary of the prediction region and the predicted material category of each pixel in the facial image sample includes: using the feature map encoding subnetwork to extract features from the facial image sample to obtain a sample feature map; based on the sample feature map, using the pixel material classification subnetwork to classify the material of each pixel in the facial image sample to obtain the predicted material category of each pixel in the facial image sample; and based on the sample feature map, using the boundary awareness subnetwork to identify the boundary between the live area and the attack area in the facial image sample to obtain the predicted region boundary.

[0013] In some embodiments, determining the target classification loss information of the facial image sample based on the predicted region boundary, the predicted material category, and the labeled material category includes: comparing the predicted material category and the labeled material category to obtain pixel classification loss information of pixels in the facial image sample; selecting a preset number of adjacent predicted material categories of pixels in the facial image sample from the predicted material categories, and comparing the adjacent predicted material categories with the predicted material categories of the corresponding pixels to obtain continuity classification loss information; obtaining the labeled target region boundary in the facial image sample, and comparing the target region boundary with the predicted region boundary to obtain boundary classification loss information; and accumulating the pixel classification loss information, the continuity classification loss information, and the boundary classification loss information to obtain the target classification loss information.

[0014] In some embodiments, the pixel material category includes either a live object category or a non-live object category; and determining the adversarial attack detection result of the target user based on the pixel material category includes: selecting at least one target pixel corresponding to the non-live object category in the facial image frame, identifying the target attack region corresponding to the at least one target pixel in the facial image frame, and determining the adversarial attack detection result of the target user based on the target attack region.

[0015] In some embodiments, determining the adversarial attack detection result of the target user based on the target attack region includes: determining the area ratio of the target attack region in the corresponding facial image frame to obtain the target area ratio; and when the target area ratio is greater than a preset area ratio threshold, determining the target user as an adversarial attack user and using the adversarial attack user as the adversarial attack detection result.

[0016] In some embodiments, the adversarial attack detection model further includes an optical sequence verification network; and after acquiring the facial video of the target user under multiple colors of light, it further includes: inputting the multiple facial image frames into the optical sequence verification network to obtain predicted image features and the current emission order of the colors of light corresponding to the multiple colors, wherein the predicted image features include the image features of the next facial image frame of each of the predicted multiple facial image frames, and based on the predicted image features and the current emission order, determining the adversarial attack detection result of the target user, and outputting the adversarial attack detection result.

[0017] In some embodiments, the optical sequence verification network includes an optical sequence feature encoding subnetwork, an optical sequence prediction subnetwork, and a feature prediction subnetwork; and the step of inputting the multiple facial image frames into the optical sequence verification network to obtain predicted image features and the current emission order of the color light corresponding to the multiple colors includes: using the optical sequence feature encoding subnetwork to extract features from the multiple facial image frames to obtain image features of each facial image frame; based on the image features, using the optical sequence prediction subnetwork to predict the current emission order of the color light corresponding to the multiple colors; and based on the image features, using the feature prediction subnetwork to predict the predicted image features of the next facial image frame of each facial image frame.

[0018] In some embodiments, determining the adversarial attack detection result of the target user based on the current emission order and the predicted image features includes: obtaining the feature difference between the image features of each facial image frame and the corresponding predicted image features; and determining the target user as an adversarial attack user when the current emission order prediction is incorrect or the feature difference is greater than a preset feature difference threshold, and using the adversarial attack user as the adversarial attack detection result.

[0019] In some embodiments, the training process of the optical sequence verification network includes the following steps: acquiring facial video samples of user samples under multiple colors of illumination, the facial video samples including multiple frames of facial image frames; extracting features from the multiple frames of facial image frames using a preset optical sequence verification network to obtain sample image features of each facial image frame sample in the multiple frames of facial image frames; determining the predicted emission order and predicted sample image features of the multiple colors of illumination based on the sample image features, the predicted sample image features including the predicted sample image features of the next frame of facial image frames of each facial image frame sample in the multiple frames of facial image frames; and determining the target verification loss information of the user samples based on the predicted emission order and the predicted sample image features, and converging the preset optical sequence verification network based on the target verification loss information to obtain the trained optical sequence verification network.

[0020] In some embodiments, the user sample is a live user sample.

[0021] In some embodiments, determining the target verification loss information of the user sample based on the predicted emission order and the predicted sample image features includes: obtaining the labeled emission order corresponding to the facial video sample, and comparing the labeled emission order with the predicted emission order to obtain the light sequence loss information; determining the sample feature difference between the sample image features of each facial image frame sample and the corresponding predicted sample image features to obtain the feature loss information; and fusing the light sequence loss information and the feature loss information to obtain the target verification loss information of the user sample.

[0022] Secondly, this specification also provides a detection system for counter-attacks, comprising: at least one storage medium storing at least one instruction set for detecting counter-attacks; and at least one processor communicatively connected to the at least one storage medium, wherein, when the counter-attack detection system is running, the at least one processor reads the at least one instruction set and executes the counter-attack detection method described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0023] As can be seen from the above technical solutions, the adversarial attack detection method and system provided in this specification, after acquiring the facial video of the target user under multiple colored lighting conditions, uses an adversarial attack detection model to classify the pixel material of each facial image frame in the multi-frame facial image video, obtains the pixel material category of each pixel, and determines the adversarial attack detection result of the target user based on the pixel material category, and outputs the adversarial attack detection result. Since this solution introduces random color saturation (multi-colored lighting) in the interactive stage of adversarial attack detection, more facial information is introduced. Then, the pixel material category of each facial image frame in the acquired facial video is predicted, and the adversarial attack detection result is determined based on the pixel material category. This can cope with attacks with small attack element areas, and does not affect the success rate of subsequent facial recognition of normal samples. Therefore, it can improve the detection accuracy of adversarial attacks.

[0024] Other functions of the methods and systems for detecting adversarial attacks provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the methods and systems for detecting adversarial attacks provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram illustrating an application scenario of an anti-attack detection system provided according to an embodiment of this specification is shown.

[0027] Figure 2 A hardware structure diagram of a computing device provided according to an embodiment of this specification is shown;

[0028] Figure 3 A flowchart illustrating a method for detecting adversarial attacks according to embodiments of this specification is shown; and

[0029] Figure 4 A schematic diagram of the overall process for detecting adversarial attacks in a face recognition scenario, according to an embodiment of this specification, is shown. Detailed Implementation

[0030] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0031] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0032] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0033] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0034] For ease of description, the terms that will appear in the following descriptions will be explained as follows:

[0035] Adversarial attacks: Taking facial recognition as an example, this can involve attaching adversarial stickers or wearing adversarial glasses (covering a small area, generally less than 50% of the face) to the facial recognition system, causing it to misidentify the user. For instance, user A might be incorrectly identified as user B after attaching an adversarial sticker or wearing adversarial glasses.

[0036] Random color rendering: Taking a face scene as an example, random lighting can be applied to the interactive screen during the face recognition stage (for example, changing colors in the order of red, yellow and blue). Different colors of light will exhibit different reflection characteristics on different materials, and these characteristics can be used to detect attacks.

[0037] Material Classification: Taking a face scene as an example, the material of the face region can be classified to determine what material different areas of the face belong to. Adversarial attacks can be detected through material classification. This is mainly because adversarial stickers (or adversarial glasses, etc.) often use materials such as paper, which are significantly different from normal faces. Therefore, adversarial attacks can be detected by leveraging this difference.

[0038] Before describing the specific embodiments in this specification, the application scenarios of this specification will be introduced as follows:

[0039] The adversarial attack detection method provided in this manual can be applied to any adversarial attack detection scenario. For example, in scenarios such as face payment or face recognition, the adversarial attack detection method in this manual can be used to detect adversarial attacks on the facial videos of the target user to be paid or identified under random color rendering. In identity verification scenarios, the adversarial attack detection method in this manual can be used to detect adversarial attacks on the collected facial videos. It can also be applied to any other adversarial attack detection scenario, which will not be elaborated on here.

[0040] Those skilled in the art should understand that the liveness detection methods and systems described in this specification are also within the scope of protection of this specification when applied to other application scenarios.

[0041] Figure 1 This diagram illustrates an application scenario of an adversarial attack detection system 001 provided according to an embodiment of this specification. The adversarial attack detection system 001 (hereinafter referred to as System 001) can be applied to the detection of adversarial attacks in any scenario, such as the detection of adversarial attacks in face payment scenarios, face recognition scenarios, identity verification scenarios, etc. Figure 1 As shown, system 001 may include user 100, client 200, server 300, and network 400.

[0042] User 100 can be the user who triggers the detection of adversarial attacks on the facial video of the target user. User 100 can perform the adversarial attack detection operation on client 200.

[0043] Client 200 can be a device that acquires facial video of a target user under various colors of lighting and performs adversarial attack detection in response to an adversarial attack detection operation by user 100. In some embodiments, the adversarial attack detection method can be executed on client 200. In this case, client 200 may store data or instructions for executing the adversarial attack detection method described herein, and may execute or be used to execute said data or instructions. In some embodiments, client 200 may include a hardware device with data processing capabilities and the necessary programs required to drive the hardware device. Figure 1 As shown, client 200 can communicate with server 300. In some embodiments, server 300 can communicate with multiple clients 200. In some embodiments, client 200 can interact with server 300 via network 400 to receive or send messages, such as receiving or sending facial video. In some embodiments, client 200 may include mobile devices, tablets, laptops, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, the mobile device may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, the smart home device may include smart TVs, desktop computers, etc., or any combination thereof. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, gaming devices, navigation devices, etc., or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include virtual reality headsets, virtual reality glasses, virtual reality controllers, augmented reality headsets, augmented reality glasses, augmented reality controllers, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, head-mounted displays, VR, etc. In some embodiments, the built-in device in the motor vehicle may include an in-vehicle computer, an in-vehicle TV, etc. In some embodiments, the client 200 may include an image acquisition device for acquiring facial video of a target user under various colors of light. In some embodiments, the image acquisition device may be a two-dimensional image acquisition device (e.g., an RGB camera), or a combination of a two-dimensional image acquisition device (e.g., an RGB camera) and a depth image acquisition device (e.g., a 3D structured light camera, a laser detector, etc.). In some embodiments, the client 200 may be a device with positioning technology for locating the position of the client 200.

[0044] In some embodiments, the client 200 may have one or more applications (APPs) installed. The APPs provide the user 100 with the ability and interface to interact with the outside world via the network 400. The APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, video APPs, financial management APPs, instant messaging tools, email clients, social media platform software, etc. In some embodiments, a target APP may be installed on the client 200. The target APP can capture facial videos of the target user under various colors of lighting for the client 200. In some embodiments, the user 100 can also trigger a counter-attack detection request through the target APP. The target APP can respond to the counter-attack detection request and execute the counter-attack detection method described in this specification. The counter-attack detection method will be described in detail later.

[0045] Server 300 may be a server providing various services, such as a backend server supporting adversarial attack detection of facial videos captured on client 200. In some embodiments, the adversarial attack detection method may be executed on server 300. In this case, server 300 may store data or instructions for executing the adversarial attack detection method described herein, and may execute or be used to execute said data or instructions. In some embodiments, server 300 may include hardware devices with data processing capabilities and necessary programs to drive the hardware devices. Server 300 may communicate with multiple clients 200 and receive data sent by clients 200.

[0046] Network 400 serves as a medium to provide a communication connection between client 200 and server 300. Network 400 facilitates the exchange of information or data. For example... Figure 1 As shown, client 200 and server 300 can connect to network 400 and transmit information or data to each other through network 400. In some embodiments, network 400 can be any type of wired or wireless network, or a combination thereof. For example, network 400 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), and Bluetooth. TM Network, ZigBee TMA network, a near-field communication (NFC) network, or a similar network. In some embodiments, network 400 may include one or more network access points. For example, network 400 may include wired or wireless network access points, such as base stations or internet exchange points, through which one or more components of client 200 and server 300 can connect to network 400 to exchange data or information.

[0047] It should be understood that Figure 1 The number of clients 200, servers 300, and networks 400 shown is merely illustrative. Depending on implementation needs, there can be any number of clients 200, servers 300, and networks 400.

[0048] It should be noted that the method for detecting the anti-attack can be executed entirely on the client 200, entirely on the server 300, or partially on the client 200 and partially on the server 300.

[0049] Figure 2 A hardware structure diagram of a computing device 600 provided according to an embodiment of this specification is shown. The computing device 600 can execute the anti-attack detection method described in this specification. The anti-attack detection method is described in other parts of this specification. When the anti-attack detection method is executed on a client 200, the computing device 600 can be the client 200. When the anti-attack detection method is executed on a server 300, the computing device 600 can be the server 300. When the anti-attack detection method can be executed partly on the client 200 and partly on the server 300, the computing device 600 can be both the client 200 and the server 300.

[0050] like Figure 2 As shown, the computing device 600 may include at least one storage medium 630 and at least one processor 620. In some embodiments, the computing device 600 may also include a communication port 650 and an internal communication bus 610. Additionally, the computing device 600 may also include I / O components 660.

[0051] The internal communication bus 610 can connect different system components, including storage medium 630, processor 620 and communication port 650.

[0052] I / O component 660 supports input / output between computing device 600 and other components.

[0053] Communication port 650 is used for data communication between computing device 600 and external sources. For example, communication port 650 can be used for data communication between computing device 600 and network 400. Communication port 650 can be a wired communication port or a wireless communication port.

[0054] Storage medium 630 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 632, a read-only storage medium (ROM) 634, or a random access storage medium (RAM) 636. Storage medium 630 also includes at least one instruction set stored in the data storage device. The instructions are computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the detection methods against attacks provided in this specification.

[0055] At least one processor 620 can be communicatively connected to at least one storage medium 630 and a communication port 650 via an internal communication bus 610. At least one processor 620 is used to execute at least one instruction set. When the computing device 600 is running, at least one processor 620 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the anti-attack detection method provided in this specification. Processor 620 can execute all steps included in the anti-attack detection method. Processor 620 can be in the form of one or more processors. In some embodiments, processor 620 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof. For illustrative purposes only, only one processor 620 is described in this specification for the computing device 600. However, it should be noted that the computing device 600 may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described herein, or they may be executed jointly by multiple processors. For example, if processor 620 of the computing device 600 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 620 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0056] Figure 3A flowchart of a counter-attack detection method P100 provided according to an embodiment of this specification is shown. As previously described, the computing device 600 can execute the counter-attack detection method P100 of this specification. Specifically, the processor 620 can read an instruction set stored in its local storage medium and then execute the counter-attack detection method P100 of this specification according to the provisions of the instruction set. Figure 3 As shown, method P100 may include:

[0057] S110: Acquire facial videos of the target user under various colored lighting conditions.

[0058] The facial video can be video information captured from the target user's face under illumination of multiple colors of light. The facial video includes multiple facial image frames. The number of facial image frames in the facial video can be two or more. Each facial image frame can be a video frame in the facial video that includes the target user's face. The facial video can be video information captured from the target user's face under illumination of multiple colors of light.

[0059] The multiple colors can include at least two of red, blue, yellow, or green. In addition to the colors mentioned above, any other arbitrary colors can be included. Each color in the multiple colors emits light independently, and the duration of this emission exceeds a preset time threshold. For example, in a scenario where multiple colors include red, yellow, and blue, only one color of light is emitted in each emission. For instance, when emitting infrared light, neither yellow nor blue light is emitted. Furthermore, the duration of each color's emission must exceed the preset time threshold. For example, with a preset time threshold of 1 second, when emitting red light, the duration of the red light must exceed 1 second. The purpose of exceeding the preset time threshold is to acquire sufficient facial video footage under that color of light, thereby improving the accuracy of facial pixel material classification.

[0060] There are several ways to obtain facial videos of the target user under various colored lighting conditions, including the following:

[0061] For example, the processor 620 can directly acquire facial videos of the target user under various colors of light, sent by the user 100 through the client 200 or terminal; or it can acquire facial videos of the target user under various colors of light through an image acquisition device, and so on.

[0062] In this context, user 100 and the target user can be the same user or different users. Furthermore, when acquiring facial video of the target user under various colored lights, sent by user 100 through client 200 or a terminal, the emission order of the various colored lights in the facial video can also be obtained to determine the target emission order.

[0063] The processor 620 can acquire facial video of the target user under multiple colors of light through an image acquisition device in various ways. For example, the processor 620 can select multiple colors from a preset color set, determine the target emission order corresponding to the multiple colors, display the color light corresponding to the multiple colors based on the target emission order and a preset display time, and acquire facial video of the target user's face under the reflection of the color light.

[0064] There are several ways to select multiple colors from the preset color set. For example, the processor 620 can randomly select multiple colors from the preset color set, or it can obtain the acquisition distance between the target user and the image acquisition device and select multiple colors corresponding to that acquisition distance from the preset color set, and so on.

[0065] The acquisition distance can be the distance between the target user and the image acquisition device. Different colors of light have different penetrating or reflecting properties. Therefore, the longer the acquisition distance, the more diverse the colors of light with stronger penetrating or reflecting properties are needed to illuminate the target user's face.

[0066] After selecting multiple colors, the processor 620 can determine the target emission sequence corresponding to the multiple colors. The target emission sequence can be the timing information of each color emitting light individually. There are several ways to determine the target emission sequence. For example, the processor 620 can randomly determine the emission sequence of each color among the multiple colors to obtain the target emission sequence. Alternatively, it can obtain the historical emission sequences corresponding to the multiple colors and randomly determine the target emission sequence based on the historical emission sequences. The target emission sequence can be different from the historical emission sequences or partially the same as the historical emission sequences. In addition, it should be noted that the target emission sequence can include the emission sequence of one round of emission among the multiple colors. For example, taking the target emission sequence as yellow, red, and green, the emission process of the multiple colors can be that yellow light emits light for a preset time, then red light emits light for a preset time, then green light emits light for a preset time. If no request, instruction, or condition for stopping emission is received, the emission sequence can continue with yellow light emitting light - red light emitting light - green light emitting light until a request, instruction, or condition for stopping emission is received. The target emission sequence can also include a complete emission sequence. For example, if the target emission sequence is yellow, red, yellow, green, and red, then the entire emission process could include a preset duration of yellow light emission followed by a preset duration of red light emission followed by a preset duration of yellow light emission followed by a preset duration of green light emission followed by a preset duration of red light emission, and finally, each color light stops emitting light. The duration of each color light emission can be the same or different, but it must always exceed a preset time threshold.

[0067] After selecting multiple colors and determining the target emission order corresponding to these colors, the processor 620 can display the corresponding colored light based on the target emission order and a preset display time. The preset display time can be the total emission time of the multiple colors or the emission time of each individual color (equivalent to the preset emission duration mentioned above). Display here can be understood as emitting the colored light corresponding to that color. There are several ways to display the corresponding colored light of multiple colors. For example, if the preset display time is the total emission time of the multiple colors, the processor 620 can display the corresponding colored light in the display area based on the target emission order until the preset display time is reached. Alternatively, if the preset display time is the continuous emission time of each colored light, the processor 620 can display the corresponding colored light in the display area based on the target emission order. When the continuous emission time of that colored light reaches the preset display time, it switches to another colored light until all the colored lights of multiple colors have been displayed. Furthermore, it should be noted that a timer can be used when displaying the corresponding colored light of multiple colors so that the emission can stop or switch to another colored light after the preset display time is reached.

[0068] When displaying multiple colors of light, the processor 620 can capture facial video of the target user under the emitted color light. There are several ways to capture facial video. For example, the processor 620 can perform face detection on a preset number of consecutive video frames in the captured real-time facial video of the target user. If the target user's face is not detected in the consecutive video frames, the processor stops displaying the multiple colors of light and displays a prompt message so that the target user can adjust the capture position based on the prompt message and return to the step of selecting multiple colors from the preset color set until the target user's face is detected in the consecutive video frames, thus obtaining the facial video of the target user under the emitted color light.

[0069] The real-time facial video can be a video of the target user's face captured in real time while displaying multiple colors of light. The consecutive video frames can be a preset number of video frames in a sequential order that are adjacent to each other. There are various ways to perform face detection on the preset number of consecutive video frames in the captured real-time facial video of the target user. For example, the processor 620 can detect whether the target user's face is present in the preset number of consecutive frames in the real-time facial video. Taking a preset number of 3 as an example, it can detect whether the target user's face (human face) is present in 3 consecutive frames of the real-time facial video, and so on.

[0070] When the processor 620 detects the target user's face in consecutive video frames, it can use the captured real-time facial video as the target user's facial video under colored light emission. When the processor 620 does not detect the target user's face in consecutive video frames, it stops displaying the colored light corresponding to various colors and displays a prompt message so that the target user can adjust their acquisition position based on the prompt message. Stopping the display of the colored light corresponding to various colors can be understood as the colored light stopping emission. The prompt message could be a prompt for the user to keep their face (human face) within a preset acquisition area (e.g., the center of the illuminated screen or a preset area), etc. The target user can then adjust their position according to the prompt message so that the image acquisition device or the emitting device can capture a real-time facial video containing the target user's face.

[0071] After the processor 620 stops displaying the color light corresponding to the multiple colors and the explicit prompt information, it can return to the step of selecting multiple colors from the preset color set until the target user's face is detected in the continuous video frames. In this way, the captured real-time facial video can be used as the facial video of the target user's face under the reflection of the color light.

[0072] It should be noted that after stopping the display of multiple colors corresponding to the color lights and the display prompts, the random colorful light emission / display can be restarted (selecting multiple colors corresponding to the color lights and randomly emitting light). While the randomly selected multiple colors are emitting light / displaying light, the timer can continue, but the timer starts from 0. That is to say, every time multiple colors corresponding to the color lights are selected again and emitted, the timer needs to start from 0 again, so that the facial video corresponding to the preset display time can be obtained.

[0073] S120: An adversarial attack detection model is used to classify the pixel material of each facial image frame in the multi-frame facial image to obtain the pixel material category of each pixel.

[0074] The pixel material category can be the material category corresponding to the pixels of the facial image frame. This pixel material category can include either a liveness category or a non-liveness category. The liveness category can be the material of a normal face. The non-liveness category can be an anti-attack material, such as paper, glasses, or a screen.

[0075] The adversarial attack detection model can be a model that detects adversarial attacks targeting a user's facial video. This model can include a pixel material classification network, which can classify the material properties of pixels in a facial image frame.

[0076] Specifically, an adversarial attack detection model is used to classify the pixel material of each facial image frame in the multi-frame facial image dataset. There are several ways to obtain the pixel material category of each pixel, such as as follows:

[0077] For example, the processor 620 can use a pixel material classification network to extract features from each facial image frame in multiple facial image frames, obtain a feature map corresponding to each facial image frame, classify the material of each pixel in the corresponding facial image frame based on the feature map, obtain the pixel material classification probability of each pixel, and determine the pixel material category of each pixel based on the pixel material classification probability.

[0078] The pixel material classification network can further include a pixel-level material classification sub-network. The pixel material classification probability can be [missing information]. Based on the feature map, there are multiple ways to classify the material of each pixel in the corresponding facial image frame. For example, the processor 620 can use the material classification sub-network to extract the material classification features of each pixel in the feature map, and determine the pixel material classification probability based on these features. Alternatively, the processor 620 can extract the pixel image features of each pixel in the feature map, and classify the pixel image features using the material classification sub-network to obtain the pixel material classification probability.

[0079] After obtaining the pixel material classification probability for each pixel, the processor 620 can determine the pixel material category for each pixel based on the pixel material classification probability. Pixel material classification can include the probability corresponding to each pixel material category, or it can include the probability corresponding to a specific pixel material category. There are multiple ways to determine the pixel material category for each pixel. For example, the processor 620 can extract the classification probabilities corresponding to candidate pixel material categories from the pixel material classification probability, and select the candidate pixel material category with the highest classification probability as the pixel material category for the corresponding pixel. Alternatively, it can extract the classification probabilities corresponding to candidate pixel material categories from the pixel material classification probability, and select the candidate pixel material category with a classification probability exceeding a preset probability threshold as the pixel material category for the corresponding pixel.

[0080] The pixel material classification network can be obtained directly or by training a preset pixel material classification network. Therefore, the training process of the pixel material classification network can include the following steps: the processor 620 can acquire facial image samples, which include facial images labeled with material categories; the preset pixel material classification network is used to predict the predicted boundary regions and the predicted material category of each pixel in the facial image samples; based on the predicted region boundaries, predicted material categories, and labeled material categories, the target classification loss information of the facial image samples is determined; and the preset pixel material classification network is converged based on the target classification loss information to obtain the pixel material classification network.

[0081] The predicted region boundary can be the boundary between the live area and the attack area in the predicted facial image sample. The live area can be a region on the face with a live material. The attack area can be a region on the face with a non-live material. Taking adversarial attacks as an example, the attack area can also be the area where the adversarial sticker is located; therefore, it can also be called the adversarial region. The preset pixel material classification network can include a feature map encoding sub-network, a pixel material classification sub-network, and a boundary awareness sub-network. The feature map encoding sub-network is a network that encodes the facial image sample using feature maps. The pixel material classification network is a network that classifies the material of each pixel in the facial image sample. The boundary awareness sub-network is a network that identifies the boundary between the live area and the attack area in the facial image sample. There are several ways to use a pixel material classification network to predict the boundary of the predicted region and the predicted material category of each pixel in a facial image sample. For example, the processor 620 can use a feature map encoding subnetwork to extract features from the facial image sample to obtain a sample feature map. Based on the sample feature map, the pixel material classification subnetwork can be used to classify the material of each pixel in the facial image sample to obtain the predicted material category of each pixel in the facial image sample. Based on the sample feature map, a boundary awareness subnetwork can be used to identify the boundary between the live area and the attack area in the facial image sample to obtain the predicted region boundary.

[0082] The processor 620 uses a pixel material classification subnetwork based on sample feature maps to predict the predicted material category of each pixel in a facial image sample in a similar way to predict the predicted material category of each pixel in a facial video frame, as described above, and will not be repeated here.

[0083] There are several ways to identify the boundary between the live region and the attack region in a facial image sample based on sample features using a boundary-aware subnetwork. For example, the processor 620 can use the boundary-aware subnetwork to identify the region boundary classification probability corresponding to each pixel in the facial image sample. Based on the region boundary classification probability, the target pixel corresponding to the region boundary is determined in the facial image sample. Based on the pixel position of the target pixel, the boundary between the live region and the attack region in the facial image sample is determined, and the predicted region boundary is obtained. Alternatively, the boundary-aware subnetwork can be used to identify the live region and the attack region in the facial image sample, thereby obtaining the region boundary between the live region and the attack region, and the predicted region boundary is obtained.

[0084] After obtaining the predicted region boundary and the predicted material category of each pixel of the facial image sample, the processor 620 can determine the target classification loss information of the facial image sample based on the predicted region boundary, the predicted material category, and the labeled material category. The target classification loss information can be the loss information generated during pixel material classification in the facial image sample. There are several ways to determine the target classification loss information. For example, the processor 620 can compare the predicted material category and the labeled material category to obtain the pixel classification loss information of the pixels in the facial image sample; select a preset number of neighboring pixels in the predicted material category and compare the neighboring predicted material categories with the corresponding pixel's predicted material category to obtain the continuity classification loss information; obtain the labeled target boundary region in the facial image sample and compare the target boundary region with the predicted region boundary to obtain the boundary classification loss information; and accumulate the pixel classification loss information, the continuity classification loss information, and the boundary classification loss information to obtain the target classification loss information.

[0085] The pixel classification loss information can be the loss information generated by the difference between the predicted material category of each pixel in the facial image sample and the material category labeled in the facial image sample. There are various ways to compare the predicted pixel material category with the labeled material category to obtain the pixel classification loss information of each pixel in the facial image sample. For example, the processor 620 can use a first classification loss function to compare the predicted pixel material category with the labeled material category to obtain the pixel classification loss information of each pixel in the facial image sample. Alternatively, other loss functions can be used to compare the predicted pixel material category with the labeled material category to obtain the pixel classification loss information of each pixel in the facial image sample, and so on.

[0086] The first classification loss function can be of various types, such as cross-entropy loss function or other types of loss functions, etc.

[0087] The continuous classification loss information can be the loss information generated by the difference between the predicted material category of each pixel in the facial image sample and the predicted material category of its neighboring pixels. The constraint condition for the continuous classification loss information is that the pixel material classification results of neighboring pixels should be as consistent as possible. The neighboring pixels can be pixels adjacent to a pixel in the facial image sample. Taking a preset number of 4 as an example, the neighboring pixels can be the pixels in front, behind, left, and right of a pixel. Taking a preset number of 8 as an example, the neighboring pixels can be pixels in eight directions: front, left-front, right-front, back, left-back, right-back, left, and right. There are multiple ways to select the neighboring predicted material categories of the preset number of neighboring pixels in the facial image sample from the predicted material categories. For example, the processor 620 selects a preset number of neighboring pixels corresponding to each pixel in the facial image sample and extracts the predicted material categories corresponding to the neighboring pixels from the predicted material categories to obtain the neighboring predicted material categories.

[0088] After selecting the neighboring predicted material categories of each pixel's adjacent pixels, the processor 620 can compare these neighboring predicted material categories with the predicted material category of the corresponding pixel to obtain continuous classification loss information. There are several ways to compare neighboring predicted material categories with the predicted material category of the corresponding pixel. For example, the processor 620 compares the neighboring predicted material categories with the predicted material category of the corresponding pixel, selects target neighboring pixels with the same material category from among the neighboring pixels based on the comparison results, and determines the continuous classification loss information based on the number of target neighboring pixels. Alternatively, a second classification loss function can be used to compare the neighboring predicted material categories with the predicted material category of the corresponding pixel to obtain continuous classification loss information, and so on.

[0089] The second classification loss function can be the cross-entropy loss function or other classification loss functions, etc. It should also be noted that the first and second classification loss functions can be the same or different.

[0090] The boundary classification loss information can be the loss information generated by the difference between the predicted boundary region and the target region boundary predicted by the boundary-aware sub-network. There are several ways to compare the target region boundary with the predicted region boundary. For example, the processor 620 can obtain the positional error between the target region boundary and the predicted region boundary, and determine the boundary classification loss information based on the positional error. Alternatively, a third classification loss function can be used to compare the target region boundary with the predicted region boundary to obtain the boundary classification loss information, and so on.

[0091] The third classification loss function can be of various types, such as the cross-entropy loss function or other types of classification loss functions. It should also be noted that the first, second, and third classification loss functions can be the same or different.

[0092] After determining the pixel classification loss information, the continuity classification loss information, and the boundary classification loss information, the processor 620 can accumulate the pixel classification loss information, the continuity classification loss information, and the boundary classification loss information to obtain the target classification loss information, as shown in formula (1):

[0093] Loss total1 =Loss pixel-cls +Loss cls-consistency +Loss edge (1)

[0094] Among them, Loss total1 Loss information for target classification pixel-cls For pixel classification loss information, Loss cls-consistency For continuous classification loss information, Loss edge Classify loss information for the boundary.

[0095] In some embodiments, the pixel classification loss information, continuity classification loss information, and boundary classification loss information are weighted and then accumulated. For example, the processor 620 can obtain the classification loss weights and, based on the classification loss weights, weight the pixel classification loss information, continuity classification loss information, and boundary classification loss information respectively, and then accumulate the weighted pixel classification loss information, the weighted continuity classification loss information, and the weighted boundary classification loss information to obtain the target classification loss information.

[0096] After determining the target classification loss information, the processor 620 can converge the preset pixel material classification network based on the target classification loss information to obtain the pixel material classification network. There are several ways to converge the preset material classification network. For example, the processor 620 can update the network parameters of the preset pixel material classification network using a gradient descent algorithm based on the target classification loss information, and then return to execute the step of acquiring facial image samples until the preset pixel material classification network converges, thus obtaining the trained pixel material classification network. Alternatively, it can update the network parameters of the preset pixel material classification network using other network parameter update algorithms based on the target classification loss information, and then return to execute the step of acquiring facial image samples until the preset pixel material classification network converges, thus obtaining the trained pixel material classification network, and so on.

[0097] S130: Based on the pixel material category, determine the adversarial attack detection result of the target user and output the adversarial attack detection result.

[0098] For example, the processor 620 can select at least one target pixel corresponding to the non-living category in the facial image frame, identify the non-living region corresponding to at least one target pixel in the facial image frame, and determine the adversarial attack detection result of the target user based on the non-living region.

[0099] The target attack region can be the area occupied by target pixels corresponding to the non-liveness category in a facial image frame. Based on the target attack region, there are multiple ways to determine the adversarial attack detection result for the target user. For example, the processor 620 can determine the area proportion of the target attack region in the corresponding facial image frame to obtain the target area proportion, and when the target area proportion is greater than a preset area proportion threshold, determine the target user as an adversarial attack user and use this adversarial attack user as the adversarial attack detection result.

[0100] The target area ratio can be the ratio between the area of ​​the target attack region and the area of ​​the facial image frame. When the target area ratio is greater than a preset area ratio threshold, it can be understood that the area of ​​the predicted attack region (adversarial region) on the target user's face exceeds the preset area threshold. In this case, it can be determined that an attack region exists on the target user's face. This attack region could be the area occupied by adversarial stickers or adversarial glasses. Therefore, the target user can be identified as an adversarial attack user. Conversely, when the target facial ratio is less than a preset facial ratio threshold, it can be understood that the area of ​​the predicted attack region (adversarial region) on the target user's face does not exceed the preset area threshold. In this case, it can be determined that an attack region may not exist on the target user's face. Therefore, the target user can be identified as a live user, and the live user is used as the adversarial attack detection result for the target user.

[0101] After determining the detection result of the adversarial attack on the target user, the processor 620 can output the adversarial attack detection result. There are several ways to output the adversarial attack detection result. For example, the processor 620 can directly send the adversarial attack detection result to the client 200, terminal or server corresponding to user 100, so that the client 200, terminal or server can respond to the target user or the business request corresponding to the target user based on the adversarial attack detection result. Alternatively, the adversarial attack detection result can be directly visualized, and so on.

[0102] There are several ways to visualize the results of adversarial attack detection. For example, the processor 620 can directly display the results, or it can display them through sound and light (e.g., by broadcasting the results, by displaying different colored lights to show different types of adversarial attack detection results, or by displaying them through sound and light linkage). Alternatively, it can display the results for specific types of adversarial attacks (e.g., only displaying the results for adversarial attacks on users, or only displaying the results for adversarial attacks on normal users, etc.).

[0103] In some embodiments, after determining or outputting the anti-attack detection result of the target user, the processor 620 may respond to the target user or the corresponding business request based on the anti-attack detection result. There may be various response methods. For example, the processor 620 may directly intercept the target user or the corresponding business request, or the processor 620 may directly perform secondary verification on the target user and, based on the secondary verification result, provide a final response to the target user or the corresponding business request, and so on.

[0104] In some embodiments, the adversarial attack detection model may further include an optical sequence verification network, which verifies the emission timing of colored lights corresponding to multiple colors, thereby enabling adversarial attack detection against the target user. There are various ways to use an optical sequence verification network for adversarial attack detection. For example, after acquiring facial video of the target user under multiple colors of illumination, the processor 620 may input multiple frames of facial images into the optical sequence verification network to obtain predicted image features and the current emission order of the colored lights corresponding to multiple colors. Based on the predicted image features and the current emission order, the adversarial attack detection result for the target user is determined and output.

[0105] The predicted image features can include image features of the next facial image frame from each of the predicted multi-frame facial image data. These image features can be feature vectors representing image information within the facial image frames, and these feature vectors can be multi-dimensional, such as 256-dimensional or other dimensions. The optical sequence verification network can include an optical sequence feature encoding sub-network, an optical sequence prediction sub-network, and a feature prediction sub-network. The optical sequence feature encoding sub-network is a network that encodes features for the facial image frames. The optical sequence prediction sub-network is a network that predicts the emission order of light corresponding to various colors. The feature prediction sub-network is a network that predicts the image features of the next facial image frame from the predicted facial image frames. There are several ways to input multiple facial image frames into an optical sequence verification network to obtain predicted image features and the current emission order of the color light corresponding to multiple colors. For example, the processor 620 can use an optical sequence feature encoding subnetwork to extract features from multiple facial image frames to obtain the image features of each facial image frame. Based on the image features, an optical sequence prediction subnetwork is used to predict the current emission order of the color light corresponding to multiple colors. Based on the image features, a feature prediction subnetwork is used to predict the predicted image features of the next facial image frame of each facial image frame.

[0106] After determining the predicted image features and the current emission order of the colored lights corresponding to various colors, the processor 620 can determine the adversarial attack detection result for the target user based on the predicted image features and the current emission order. There are several ways to determine the adversarial attack detection result. For example, the processor 620 can obtain the feature difference between the image features of each facial image frame and the corresponding predicted image features. If the current emission order is predicted incorrectly or the feature difference exceeds a preset feature difference threshold, the target user is identified as an adversarial attacker, and the adversarial attacker is used as the adversarial attack detection result.

[0107] There are multiple ways to obtain the feature difference between the image features of each facial image frame and the corresponding predicted image features. For example, the facial image features of the first frame can predict the predicted image features of the facial image features of the second frame, and so on. Therefore, the processor 620 can compare the image features of the facial image frame of the second frame with the predicted image features of the facial image frame of the second frame predicted by the image features of the first facial image frame, so as to obtain the feature difference between the actual image features of the second facial image frame and the corresponding predicted image features. By analogy, the feature difference corresponding to each facial image frame (excluding the first facial image frame) can be obtained.

[0108] It can also obtain the target emission order of colored light corresponding to multiple colors, compare the target emission order with the predicted current emission order, and if the target emission order is the same as the current emission order, it can be determined that the current emission prediction is correct. If the target emission order is different from the current emission order, it can be determined that the current emission prediction is wrong.

[0109] When the processor 620 determines that the target user is an adversarial attacker if the current emission order prediction is incorrect or the feature difference exceeds a preset feature difference threshold, it will use the adversarial attacker as the target user's adversarial attack detection result. An incorrect current emission order prediction can occur when the current emission order differs from the actual emission order of the corresponding colors. The condition for the feature difference to exceed the preset feature difference threshold can be varied, such as: the feature difference of each facial image frame is greater than the preset feature difference threshold; the extreme values ​​(maximum and / or minimum values) of the feature differences in each facial image frame are greater than the preset feature difference threshold; or the mean of the feature differences in each facial image frame is greater than the preset feature difference threshold, and so on.

[0110] The processor 620 can also determine that the target user is a live user when the current emission order prediction is correct or the feature difference is less than a preset feature difference threshold, and will then display the adversarial attack detection results for using a live user as the target user. A correct prediction of the current emission order means that the current emission order is the same as the actual emission order of the color light corresponding to multiple colors. The condition that the feature difference is less than the preset feature difference threshold can be varied, such as: the feature difference of each facial image frame is less than the preset feature difference threshold; or, the extreme values ​​(maximum and / or minimum values) of the feature differences in each facial image frame are less than the preset feature difference threshold; or, the mean of the feature differences in each facial image frame is less than the preset feature difference threshold, and so on.

[0111] After determining the detection result of the adversarial attack on the target user, the processor 620 can output the adversarial attack detection result. The process of outputting the adversarial attack detection result is detailed above and will not be repeated here.

[0112] In some embodiments, before the processor 620 uses the optical sequence verification network to detect adversarial attacks against the target user, it can directly acquire the optical sequence verification network or train a preset optical sequence verification network to obtain a trained optical sequence verification network.

[0113] The training process of the optical sequence verification network may include the following steps: The processor 620 can acquire facial video samples of user samples under multiple colors of light. The facial video samples include multiple frames of facial image samples. The preset optical sequence verification network is used to extract features from the multiple frames of facial image samples to obtain the sample image features of each facial image frame sample in the multiple frames of facial image samples. Based on the sample image features, the predicted emission order and predicted sample image features of the multiple colors of light are determined. The predicted sample image features include the sample image features of the next frame of facial image samples of each facial image frame sample in the predicted multiple frames of facial image samples. Based on the predicted emission order and predicted sample image features, the target verification order of the user samples is determined. The preset optical sequence verification network is converged based on the target verification loss information to obtain the trained optical sequence verification network.

[0114] The user samples are live user samples, which can be understood as training samples from live users whose faces do not include adversarial or attack regions. Alternatively, it can be understood as the pixel texture of the user sample's facial image being entirely live. When training the preset optical sequence verification network using facial video samples from live user samples, the preset optical sequence verification network can learn the emission order (optical sequence information) and predicted sample image features in the facial video samples corresponding to live user samples (normal samples or non-adversarial attack samples), thereby improving the detection accuracy of adversarial attacks. Furthermore, the method by which the processor 620 acquires facial video samples is similar to the method for acquiring facial videos, as detailed above, and will not be repeated here. Additionally, the method by which the processor 620 extracts sample image features and predicts the predicted emission order and predicted sample image features is similar to the method for extracting image features, predicting the current emission order, and predicting image features, as detailed above, and will not be repeated here.

[0115] After predicting the predicted emission order and the predicted sample image features, the processor 620 can determine the target verification loss information of the user sample based on the predicted emission order and the predicted sample image features. The target verification loss information can be the loss information generated after predicting the emission order and sample image features corresponding to the facial video of the user sample using an optical sequence. There are several ways to determine the target verification loss information of the user sample. For example, the processor 620 can obtain the labeled emission order corresponding to the facial video sample and compare it with the predicted emission order to obtain optical sequence loss information; determine the sample feature difference between the sample image features of each facial image frame and the corresponding predicted sample image features to obtain feature loss information; and fuse the optical sequence loss information and the feature loss information to obtain the target verification loss information of the user sample.

[0116] The optical sequence loss information can be the loss information generated by the difference between the emission order predicted by the optical sequence verification network and the actual emission order corresponding to the facial video sample. There are several ways to compare the labeled emission order with the predicted emission order. For example, the processor 620 can use a contrastive loss function to compare the labeled emission order with the predicted emission order to obtain the optical sequence loss information. Alternatively, it can compare the labeled emission order with the predicted emission order and determine the optical sequence loss information based on the comparison result, and so on.

[0117] The feature loss information can be the loss information generated by the difference between the predicted sample image features of the facial image frame samples predicted by the optical sequence verification network and the actual extracted sample image features. The method for determining the sample feature difference between the sample image features of each facial image frame sample and the corresponding predicted sample image features is similar to the method for determining the feature difference, as detailed above, and will not be repeated here. After determining the sample feature difference, the processor 620 can determine the feature loss information based on this sample feature difference. For example, the processor 620 can directly use the sample feature difference as the feature loss information, or it can obtain the mean or extreme value (maximum and / or minimum value) of the sample feature difference of each facial image frame sample and use this mean or extreme value as the feature loss information, and so on.

[0118] After determining the optical sequence loss information and the feature loss information, the processor 620 can fuse the optical sequence loss information and the feature loss information to obtain the target verification loss information. There are several ways to fuse the optical sequence loss information and the feature loss information. For example, the processor 620 can directly add the optical sequence loss information and the feature loss information to obtain the target verification loss information, as shown in formula (2):

[0119] Loss total2 =Loss light +Loss pred (2)

[0120] Among them, Loss total2 To validate the loss information for the target, Loss light For optical sequence loss information, Loss pred This is the feature loss information.

[0121] In some embodiments, the method by which the processor 620 fuses the optical sequence loss information and the feature loss information may further include: the processor 620 obtaining a preset verification loss weight, and weighting the optical sequence loss information and the feature loss information respectively based on the preset verification weight, and adding the weighted optical sequence loss information and the weighted feature loss information together to obtain the target verification loss information.

[0122] After determining the target verification loss information, the processor 620 can converge the preset optical sequence verification network based on the target verification loss information. The convergence method for the preset optical sequence verification network is similar to the convergence method for the preset pixel material classification network, as detailed above, and will not be repeated here.

[0123] It should be noted that the adversarial attack detection model can include at least one of a pixel material classification network or an optical sequence verification network. That is, in the adversarial attack detection method, the processor 620 can simultaneously use both the pixel material classification network and the optical sequence verification network to detect adversarial attacks on the target user. If either network detects an adversarial attack, the final adversarial attack detection result for the target user can be determined as an adversarial attack user. Alternatively, either the pixel material classification network or the optical sequence verification network can be used for adversarial attack detection, and the detection result can be taken as the final adversarial attack detection result for the target user. For example, when the processor 620 uses the pixel material classification network for adversarial attack detection, if the adversarial attack detection result is an adversarial attack user, the adversarial attack user can be taken as the final adversarial attack detection result for the target user; if the adversarial attack detection result is a live user, the live user can be taken as the final adversarial attack detection result for the target user.

[0124] After determining the detection result of the adversarial attack on the target user, the processor 620 can output the adversarial attack detection result. There are several ways to output the adversarial attack detection result. For example, the processor 620 can directly send the adversarial attack detection result to the client 200, terminal or server corresponding to user 100, so that the client 200, terminal or server can respond to the target user or the business request corresponding to the target user based on the adversarial attack detection result. Alternatively, the adversarial attack detection result can be directly visualized, and so on.

[0125] There are several ways to visualize the results of adversarial attack detection. For example, the processor 620 can directly display the results, or it can display them through sound and light (e.g., by broadcasting the results, by displaying different colored lights to show different types of adversarial attack detection results, or by displaying them through sound and light linkage). Alternatively, it can display the results for specific types of adversarial attacks (e.g., only displaying the results for adversarial attacks on users, or only displaying the results for adversarial attacks on normal users, etc.).

[0126] In some embodiments, after determining or outputting the anti-attack detection result of the target user, the processor 620 may respond to the target user or the corresponding business request based on the anti-attack detection result. There may be various response methods. For example, the processor 620 may directly intercept the target user or the corresponding business request, or the processor 620 may directly perform secondary verification on the target user and, based on the secondary verification result, provide a final response to the target user or the corresponding business request, and so on.

[0127] In the context of face recognition, this solution employs an adversarial attack detection model to detect adversarial attacks on the target user. Taking an adversarial attack detection model comprising a pixel material classification network and an optical sequence verification network as an example, the overall process of adversarial attack detection can be as follows: Figure 4 As shown, it can specifically include four parts: data acquisition and preprocessing, pixel-level material classification, optical sequence verification, and adversarial attack detection, as detailed below:

[0128] (1) Data Acquisition and Preprocessing: Traditional adversarial attack detection methods typically employ silent acquisition, meaning no active information is introduced during the acquisition process. This aims to improve user experience, but the lack of additional interactive information significantly reduces the detection capability for certain adversarial attacks. Therefore, this solution introduces random lighting information during the data acquisition phase and utilizes the reflectivity of different materials for different colors of light to perform adversarial attack detection. The data acquisition process can include randomly selecting three colors from {red, blue, yellow, green} and applying lighting in a random order for 3 seconds, with each selected color lasting 1 second. During the 3 seconds of random lighting, the user's facial data is continuously acquired. The data preprocessing process can include face detection on the acquired data. If no face is detected for 3 consecutive frames, the lighting is stopped, and the user is prompted to keep their face centered on the screen. Simultaneously, the random, colorful lighting and timing restart (starting from 0) are restarted, resulting in a facial video of the target user under various colors of light (light sequence).

[0129] (2) Pixel-level Material Classification: Material classification is used in liveness detection algorithms, such as classifying the material of a face image as live, screen, paper, etc. However, traditional methods classify the entire face region as one material, which is difficult to apply to adversarial attacks (adversarial stickers only occupy a part of the face region, so in reality, there is part live and part non-live (e.g., paper)). Therefore, this solution trains a pixel-level material classification network. The specific training process is described above. The trained pixel material classification network classifies the pixels in the face image frame into materials. The specific classification process is described above, thus enabling the solution to adversarial attacks that only occupy a part of the face area. In addition, due to the introduction of multiple lighting information, the material classification results are more accurate than those under ordinary lighting conditions (equivalent to multispectral material judgment), thereby improving the detection accuracy of adversarial attacks.

[0130] (3) Optical Sequence Verification: To further explore the useful information of optical information for adversarial attack detection, this step mines temporal information and uses optical sequence prediction as a proxy task to train the optical sequence verification network, resulting in the trained optical sequence verification network. The specific training process can be found above. The optical sequence verification network and the pixel material classification network are used as adversarial attack detection models.

[0131] (4) Adversarial Attack Detection: This can be divided into adversarial attack detection based on material classification and adversarial attack detection based on optical sequence verification. The process of adversarial attack detection based on material classification can include inputting the acquired illuminated face image into a pixel-level material classification model to obtain the corresponding material classification probability. If the area ratio of the material classification as a non-live region (attack region or adversarial region) is higher than a pre-set threshold T1, it is judged as an adversarial attack; otherwise, it is judged as a live target. The process of adversarial attack detection based on optical sequence verification can include inputting the acquired illuminated face image into an optical sequence verification model to obtain the optical sequence prediction result (predicted emission order) and the corresponding feature prediction result. The difference between the predicted features and the true features is calculated. If the optical sequence prediction result is incorrect or the feature difference is greater than a pre-set threshold T2, it is judged as an adversarial attack; otherwise, it is judged as a live target. For both methods, if either method judges it as an adversarial attack, it is judged as an adversarial attack; otherwise, it is judged as a live target.

[0132] This scheme first introduces random color effects (such as light of various colors) during the interaction phase, and performs corresponding acquisition and preprocessing to incorporate more information. Then, a pixel-level material classification model is trained to handle adversarial attacks that only occupy a portion of the face area. Additionally, color prediction of the acquired sequences can assist in adversarial attack detection. Finally, based on the pixel-level material classification results and color sequence analysis, the final adversarial attack detection is performed. This scheme significantly improves the accuracy of adversarial attack detection by introducing random actions, ensuring that normal samples are not affected.

[0133] In summary, the adversarial attack detection method P100 and system 001 provided in this specification, after acquiring facial videos of the target user under multiple colored lighting conditions, employ an adversarial attack detection model to classify the pixel material of each facial image frame in the multi-frame facial image video, obtaining the pixel material category of each pixel, and determining the adversarial attack detection result of the target user based on the pixel material category, and outputting the adversarial attack detection result. Because this scheme introduces random color saturation (multi-colored lighting) in the interactive stage of adversarial attack detection, it introduces more facial information. Then, it predicts the pixel material category of each facial image frame in the acquired facial video, and determines the adversarial attack detection result based on the pixel material category. This can cope with attacks with small attack element areas, and does not affect the success rate of subsequent facial recognition of normal samples. Therefore, it can improve the detection accuracy of adversarial attacks.

[0134] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for detecting counter-attacks. When the executable instructions are executed by a processor, they instruct the processor to perform the steps of the counter-attack detection method P100 described in this specification. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on a computing device 600, the program code causes the computing device 600 to perform the steps of the counter-attack detection method P100 described in this specification. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the computing device 600. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on computing device 600, partially on computing device 600, as a standalone software package, partially on computing device 600 and partially on a remote computing device, or entirely on a remote computing device.

[0135] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0136] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0137] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0138] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0139] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. All contents used for all purposes, except for any history of prosecution documents relating to it, that may be inconsistent with or conflict with this document, or any such history of prosecution documents that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0140] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for detecting adversarial attacks, comprising: Acquire facial videos of a target user under lighting of various colors, the facial videos including multiple facial image frames; An adversarial attack detection model is used to classify the pixel material of each facial image frame in the multi-frame facial image to obtain the pixel material category of each pixel. The multi-frame facial image is input into the optical sequence verification network in the adversarial attack detection model to obtain the predicted image features and the current emission order of the color light corresponding to the multiple colors. The predicted image features include the image features of the next facial image frame of each of the multi-frame facial image frames. Based on the pixel material category, the target attack area corresponding to the non-living object category is identified in the facial image frame; Obtain the feature difference between the image features of each facial image frame and the corresponding predicted image features; as well as Based on the target attack area, the current emission order, and the feature difference, the adversarial attack detection result of the target user is determined and the adversarial attack detection result is output.

2. The method for detecting adversarial attacks according to claim 1, wherein, The process of acquiring facial videos of the target user under various colors of lighting includes: Select multiple colors from a preset color set and determine the target emission order corresponding to the multiple colors; Based on the target emission sequence and preset display time, display the color light corresponding to the various colors; and The target user's face is captured as a video of the reflected color light.

3. The method for detecting adversarial attacks according to claim 2, wherein, The multiple colors include at least two of red, blue, yellow, or green.

4. The method for detecting adversarial attacks according to claim 2, wherein, Each of the multiple colors emits its own color light, and the duration of this emission exceeds a preset time threshold.

5. The method for detecting adversarial attacks according to claim 4, wherein, The process of acquiring facial video of the target user under the reflection of the colored light includes: Face detection is performed on a preset number of consecutive video frames in the real-time facial video of the target user; When the target user's face is not detected in the consecutive video frames, the display of the color lights corresponding to the various colors stops, and a prompt message is displayed so that the target user can adjust the acquisition position based on the prompt message; and Return to the step of selecting multiple colors from the preset color set until the target user's face is detected in the continuous video frames, and obtain the facial video of the target user's face under the reflection of the colored light.

6. The method for detecting adversarial attacks according to claim 1, wherein, The adversarial attack detection model includes a pixel material classification network; as well as The adversarial attack detection model is used to classify the pixel material of each facial image frame in the multi-frame facial image, obtaining the pixel material category of each pixel, including: The pixel material network is used to extract features from each of the multiple facial image frames to obtain a feature map corresponding to each facial image frame. Based on the feature map, material classification is performed on each pixel of the corresponding facial image frame to obtain the pixel material classification probability of each pixel, and The pixel material category of each pixel is determined based on the pixel material classification probability.

7. The method for detecting adversarial attacks according to claim 6, wherein, The training process of the pixel material classification network includes the following steps: Obtain facial image samples, including facial images labeled with material categories; A preset pixel material classification network is used to predict the boundary of the predicted region and the predicted material category of each pixel in the facial image sample. The boundary of the predicted region is the boundary between the predicted live area and the attack area in the facial image sample; and Based on the predicted region boundary, predicted material category, and labeled material category, the target classification loss information of the facial image sample is determined, and the preset pixel material classification network is converged based on the target classification loss information to obtain the pixel material classification network.

8. The method for detecting adversarial attacks according to claim 7, wherein, The preset pixel material classification network includes a feature map encoding sub-network, a pixel material classification sub-network, and a boundary awareness sub-network; as well as The method of using a preset pixel material classification network to predict the boundary of the predicted region and the predicted material category of each pixel in the facial image sample includes: The feature map encoding subnetwork is used to extract features from the facial image samples to obtain sample feature maps. Based on the sample feature map, the pixel material classification sub-network is used to classify the material of each pixel of the facial image sample, resulting in the predicted material category of each pixel of the facial image sample. Based on the sample feature map, the boundary-aware subnetwork is used to identify the boundary between the live area and the attack area in the facial image sample, and the predicted region boundary is obtained.

9. The method for detecting adversarial attacks according to claim 7, wherein, The step of determining the target classification loss information of the facial image sample based on the predicted region boundary, predicted material category, and labeled material category includes: The predicted material category and the labeled material category are compared to obtain the pixel classification loss information of the pixels in the facial image sample; In the predicted material category, a preset number of neighboring pixels in the facial image sample are selected as neighboring predicted material categories, and the neighboring predicted material categories are compared with the predicted material category of the corresponding pixel to obtain continuous classification loss information. Obtain the labeled target region boundary in the facial image sample, and compare the target region boundary with the predicted region boundary to obtain boundary classification loss information; and The pixel classification loss information, the continuity classification loss information, and the boundary classification loss information are summed to obtain the target classification loss information.

10. The method for detecting adversarial attacks according to claim 1, wherein, The pixel material category includes either a living category or a non-living category; as well as The step of identifying the target attack region corresponding to the non-living category in the facial image frame based on the pixel material category includes: Select at least one target pixel corresponding to the non-living category in the facial image frame; as well as The target attack region corresponding to at least one target pixel is identified in the facial image frame.

11. The method for detecting adversarial attacks according to claim 10, wherein, The determination of the adversarial attack detection result of the target user based on the target attack area includes: Determine the area proportion of the target attack region in the corresponding facial image frame to obtain the target area proportion; and When the target area ratio is greater than a preset area ratio threshold, the target user is determined to be an adversarial attack user, and the adversarial attack user is used as the adversarial attack detection result.

12. The method for detecting adversarial attacks according to claim 11, wherein, The optical sequence verification network includes an optical sequence feature encoding subnetwork, an optical sequence prediction subnetwork, and a feature prediction subnetwork; as well as The step of inputting the multiple facial image frames into the optical sequence verification network to obtain the predicted image features and the current emission order of the color light corresponding to the multiple colors includes: The optical sequence feature coding subnetwork is used to extract features from the multiple facial image frames to obtain the image features of each facial image frame. Based on the image features, the light sequence prediction subnetwork is used to predict the current emission order of the color light corresponding to the multiple colors, and Based on the image features, the feature prediction subnetwork is used to predict the image features of the next facial image frame for each facial image frame.

13. The method for detecting adversarial attacks according to claim 12, wherein, The determination of the adversarial attack detection result for the target user based on the target attack region, the current emission order, and the feature difference includes: When the current emission order prediction is incorrect or the feature difference is greater than a preset feature difference threshold, the target user is determined to be an adversarial attack user, and the adversarial attack user is used as the adversarial attack detection result.

14. The method for detecting adversarial attacks according to claim 1, wherein, The training process of the optical sequence verification network includes the following steps: Acquire facial video samples of user samples under lighting of various colors, wherein the facial video samples include multiple frames of facial image frames; A preset optical sequence verification network is used to extract features from the multi-frame facial image samples to obtain the sample image features of each facial image frame sample in the multi-frame facial image samples. Based on the sample image features, the predicted emission order and predicted sample image features of the multiple colors of illumination are determined. The predicted sample image features include the sample image features of the next facial image frame sample for each of the predicted multi-frame facial image samples; and Based on the predicted emission order and the predicted sample image features, the target verification loss information of the user sample is determined, and the preset optical sequence verification network is converged based on the target verification loss information to obtain the trained optical sequence verification network.

15. The method for detecting adversarial attacks according to claim 14, wherein, The user samples are live user samples.

16. The method for detecting adversarial attacks according to claim 14, wherein, The step of determining the target verification loss information of the user sample based on the predicted emission order and the predicted sample image features includes: Obtain the labeled emission order corresponding to the facial video sample, and compare the labeled emission order with the predicted emission order to obtain light sequence loss information; Determine the sample feature difference between the sample image features of each facial image frame sample and the corresponding predicted sample image features to obtain feature loss information; and The optical sequence loss information and feature loss information are fused to obtain the target verification loss information of the user sample.

17. A detection system against attacks, comprising: At least one storage medium storing at least one instruction set for detecting adversarial attacks; as well as At least one processor is communicatively connected to the at least one storage medium. When the anti-attack detection system is running, the at least one processor reads the at least one instruction set and executes the anti-attack detection method according to any one of claims 1-16.

Images