Liveness detection method and system

Abstract

The live body detection method and system provided in the specification, after obtaining a user data set corresponding to a plurality of accounts and account login information, and extracting background features corresponding to each account in the plurality of accounts from the user data set, constructing a graph network with each account in the plurality of accounts as a node based on the account login information, and propagating the background features in the graph network to obtain live body features corresponding to each account, and based on the live body features, performing live body detection on the plurality of accounts and outputting a live body detection result, the live body detection result including at least one of an attack cluster or a live body cluster; the scheme can improve the accuracy of live body detection for black production aggregation attacks.

Description

Technical Field

[0001] This specification relates to the field of image recognition, and in particular to a liveness detection method and system. Background Technology

[0002] In recent years, with the rapid development of internet technology, liveness detection has become an indispensable part of facial recognition systems, effectively blocking non-liveness attack samples. Existing liveness detection methods often employ either deep learning-based silent liveness detection or interactive action-based liveness detection methods.

[0003] In the process of researching and practicing existing technologies, the inventors of this application discovered that silent liveness detection and liveness detection based on interactive actions are often used to detect single or a small number of attack samples. However, in real-world liveness attacks, cybercriminals often launch batch or clustered attacks, which makes the liveness detection performance for such cybercriminals poor, thus reducing the accuracy of liveness detection. Summary of the Invention

[0004] This manual provides a more accurate method and system for liveness detection.

[0005] In a first aspect, this specification provides a liveness detection method, comprising: acquiring a user data set and account login information corresponding to multiple accounts, and extracting background features corresponding to each of the multiple accounts in the user data set; constructing a graph network with each of the multiple accounts as a node based on the account login information, and propagating the background features in the graph network to obtain liveness features corresponding to each account; and performing liveness detection on the multiple accounts based on the liveness features, and outputting liveness detection results, wherein the liveness detection results include at least one of attack clusters or liveness clusters.

[0006] In some embodiments, the user data set includes a user image corresponding to each account and a voice signal corresponding to the user image; and the step of extracting background features corresponding to each of the plurality of accounts from the facial data set includes: extracting voice background features from the voice signal, extracting image background features from the user image, and using the image background features and the voice background features as background features corresponding to each account.

[0007] In some embodiments, extracting speech background features from the speech signal includes: extracting a background signal from the speech signal and converting the background signal into a background speech map; and using a background noise extraction network to extract speech background features from the background speech map.

[0008] In some embodiments, the training process of the background noise extraction network includes the following steps: acquiring background speech map samples; dividing the background speech map samples into regions using a preset background noise extraction network to obtain multiple region maps; performing liveness detection on the multiple region maps to obtain a first predicted liveness category of the background speech map samples and a second predicted liveness category corresponding to each region map in the multiple region maps; and determining the speech liveness loss information of the background speech map samples based on the first predicted liveness category and the second predicted liveness category, and converging the preset background noise extraction network based on the speech liveness loss information to obtain the trained background noise extraction network.

[0009] In some embodiments, determining the speech liveness loss information of the background speech map sample based on the first predicted liveness category and the second predicted liveness category includes: comparing the first predicted liveness category with the labeled liveness category of the background speech map sample to obtain global classification loss information; comparing the second predicted liveness categories corresponding to different region maps to obtain local consistency loss information, wherein the constraint condition of the local consistency loss information is that the second predicted liveness category of each region map is consistent; comparing the second predicted liveness category with the labeled liveness category of the background speech map sample to obtain local classification loss information; and accumulating the global classification loss information, the local consistency loss information, and the local classification loss information to obtain the speech liveness loss information of the background speech map sample.

[0010] In some embodiments, extracting image background features from the user image includes: segmenting a human body region in the user image to obtain a user background image; and using an image background extraction network to extract image background features from the user background image.

[0011] In some embodiments, segmenting the human body region in the user image to obtain a user background image includes: segmenting the region image corresponding to the human body region in the user image, and filling the human body region with all zero pixels to obtain an initial user background image; and obtaining a preset filled background corresponding to the human body region, and filling the human body region in the initial user background image with the preset filled background to obtain the user background image.

[0012] In some embodiments, the image background extraction network includes a background feature encoding subnetwork and a salient object detection subnetwork; and the extraction of image background features from the user background image using the image background extraction network includes: using the salient object detection subnetwork to detect at least one salient region in the user background image to obtain a salientity map, and using the background feature encoding subnetwork to perform feature encoding on the user background image based on the salientity map to obtain the image background features.

[0013] In some embodiments, the training process of the image background extraction network includes the following steps: acquiring user background image samples, the user background image samples including saliency map samples; performing liveness detection on the user background image samples using a preset image background extraction network to obtain a third predicted liveness category corresponding to each pixel; determining image liveness loss information of the user background image samples based on the saliency map samples, the third predicted liveness category, and the labeled liveness category of the user background image samples; and converging the preset image background extraction network based on the image liveness loss information to obtain the trained image background extraction network.

[0014] In some embodiments, determining the image liveness loss information of the user background image sample based on the saliency map sample, the third predicted liveness category, and the labeled liveness category of the user background image sample includes: comparing the third predicted liveness category and the labeled liveness category of the user background image sample to obtain the initial image liveness loss information corresponding to each pixel; identifying the pixel value at each pixel position in the saliency map sample to obtain the liveness loss weight corresponding to each pixel; and performing weighted fusion of the initial image liveness loss information based on the liveness loss weight to obtain the image liveness loss information of the user background image sample.

[0015] In some embodiments, constructing a graph network by using each of the plurality of accounts as a node based on the account login information includes: determining the edge information of an initial graph network based on the account login information; constructing the initial graph network by using each of the plurality of accounts as a node based on the edge information; and training the initial graph network using a fusion classification loss function to obtain a trained graph network.

[0016] In some embodiments, determining the edge information of the initial graph network based on the account login information includes: extracting the device identifier of the verification device logged in by each of the plurality of accounts from the account login information; selecting two accounts logged in to the same verification device from the plurality of accounts based on the device identifier to obtain at least one account pair; and determining the edge information of the initial graph network based on the at least one account pair.

[0017] In some embodiments, the graph network includes a speech graph network layer, an image graph network layer, and a fused feature graph network layer, wherein nodes corresponding to the same account in the speech graph network layer, the image graph network layer, and the fused feature graph network layer are respectively connected.

[0018] In some embodiments, propagating the background features in the graph network to obtain the liveness feature corresponding to each account includes: propagating the speech background features in the speech graph network layer to obtain the propagated target speech background features; propagating the image background features in the image graph network layer to obtain the propagated target image background features; and propagating the target speech background features and the target image background features in the fused feature graph network layer to obtain the liveness feature corresponding to each account.

[0019] In some embodiments, propagating the speech background features in the speech graph network layer to obtain the propagated target speech background features includes: selecting a target node and a preset number of neighboring nodes corresponding to the target node in the speech graph network layer; aggregating the speech background features of the neighboring nodes with the speech background features of the target node to obtain the target speech background features corresponding to the target node; and returning to the step of selecting the target node and the preset number of neighboring nodes corresponding to the target node in the speech graph network layer until all nodes in the speech graph network layer are target nodes, thereby obtaining the target speech background features corresponding to each account.

[0020] In some embodiments, the step of aggregating the speech background features of the neighboring nodes with the speech background features of the target node to obtain the target speech background features corresponding to the target node includes: accumulating the speech background features of the neighboring nodes and calculating the average value of the accumulated speech background features; adding the average value to the feature value of the speech background features of the target node to obtain the current speech background features corresponding to the target node; and inputting the current speech background features into a preset fully connected layer to obtain the target speech background features corresponding to the target node.

[0021] In some embodiments, the step of propagating the target speech background features and the target image background features in the fusion feature map network layer to obtain the liveness features corresponding to each account includes: calculating the average features of the target speech background features and the target image background features to obtain fused features; propagating the fused features in the fusion feature map network layer to obtain the propagated target fused features; and using the target fused features as the liveness features corresponding to each account.

[0022] In some embodiments, the step of performing liveness detection on the plurality of accounts based on the liveness features includes: selecting at least one attacking account from the plurality of accounts based on the liveness features to obtain a first attack cluster; performing anomaly detection on candidate accounts from the plurality of accounts based on the liveness features to obtain a second attack cluster and a liveness cluster, wherein the candidate accounts are accounts from the plurality of accounts other than the first attack cluster; and using the first attack cluster and the second attack cluster as the attack cluster, and using the attack cluster and the liveness cluster as the liveness detection results of the plurality of accounts.

[0023] In some embodiments, selecting at least one attacking account from the plurality of accounts based on the liveness characteristics to obtain a first attack cluster includes: classifying each of the plurality of accounts as liveness based on the liveness characteristics; and selecting at least one attacking account from the plurality of accounts based on the liveness classification results to obtain the first attack cluster.

[0024] In some embodiments, the step of performing anomaly detection on candidate accounts among the plurality of accounts based on the liveness features to obtain a second attack cluster and a liveness cluster includes: reducing the dimensionality of the liveness features to a preset dimension to obtain a dimensionality-reduced target liveness feature; performing distribution estimation on the candidate accounts based on the target liveness feature to obtain the normal distribution probability of the candidate accounts; and selecting at least one account among the candidate accounts whose normal distribution probability is less than a preset probability threshold to obtain a second attack cluster, and designating at least one account among the candidate accounts other than the second attack cluster as a liveness cluster.

[0025] Secondly, this specification also provides a liveness detection system, comprising: at least one storage medium storing at least one instruction set for performing liveness detection; and at least one processor communicatively connected to the at least one storage medium, wherein, when the liveness detection system is running, the at least one processor reads the at least one instruction set and executes the liveness detection method described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0026] As can be seen from the above technical solutions, the liveness detection method and system provided in this specification, after acquiring user data sets and account login information corresponding to multiple accounts, and extracting the background features corresponding to each account in the user data set, constructs a graph network based on the account login information, using each account as a node, and propagates the background features in the graph network to obtain the liveness features corresponding to each account. Based on the liveness features, liveness detection is performed on multiple accounts, and the liveness detection results are output. The liveness detection results include at least one of attack clusters or liveness clusters. Since this solution can extract and analyze background information (voice background noise and image noise) ignored in general liveness detection algorithms, and uses graph networks to propagate features and risks, it can fully explore risky users, thereby achieving liveness detection against black market attacks involving clusters (aggregation), thus improving the accuracy of liveness detection.

[0027] Other functions of the liveness detection methods and systems provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the liveness detection methods and systems provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A schematic diagram illustrating an application scenario of a liveness detection system provided according to an embodiment of this specification is shown.

[0030] Figure 2 A hardware structure diagram of a computing device provided according to an embodiment of this specification is shown;

[0031] Figure 3 A flowchart of a liveness detection method according to embodiments of this specification is shown; and

[0032] Figure 4 A schematic diagram of the overall process of a liveness detection method for cyberattacks targeting organized crime groups, provided according to an embodiment of this specification, is shown. Detailed Implementation

[0033] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0034] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0035] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0036] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0037] For ease of description, the terms that will appear in the following descriptions will be explained as follows:

[0038] Background noise refers to the background frequency bands of speech and background information in images (excluding human and facial regions). Background noise is often ignored in liveness detection.

[0039] Liveness detection: Algorithm technology used in facial recognition or other face-based verification systems to detect and intercept liveness attacks (including attacks using mobile phone photos, paper photos, and masks).

[0040] Cluster attacks: also known as swarm attacks, refer to organized and large-scale mass attacks launched by cybercriminals after discovering system vulnerabilities; compared to trial attacks by ordinary users, these attacks are more dangerous.

[0041] Before describing the specific embodiments in this specification, the application scenarios of this specification will be introduced as follows:

[0042] The liveness detection method provided in this manual can be applied to any liveness detection scenario. For example, in a face payment scenario, the liveness detection method in this manual can be used to perform liveness detection on the collected face image of the user to be paid; in an identity verification scenario, the liveness detection method in this manual can be used to perform liveness detection on the collected identity image of the user; it can also be applied to any face recognition scenario, which will not be elaborated here.

[0043] Those skilled in the art should understand that the liveness detection methods and systems described in this specification are also within the scope of protection of this specification when applied to other application scenarios.

[0044] Figure 1 This diagram illustrates an application scenario of a liveness detection system 001 provided according to an embodiment of this specification. The liveness detection system 001 (hereinafter referred to as System 001) can be applied to liveness detection in any scenario, such as liveness detection in face payment scenarios, liveness detection in identity verification scenarios, liveness detection in other face recognition scenarios, etc. Figure 1 As shown, system 001 may include target user 100, client 200, server 300 and network 400.

[0045] The target user 100 can be the user who triggers the liveness detection of the target facial image, and the target user 100 can perform the liveness detection operation on the client 200.

[0046] Client 200 can be a device for performing liveness detection on a target facial image in response to a liveness detection operation by target user 100. In some embodiments, the liveness detection method can be executed on client 200. In this case, client 200 may store data or instructions for executing the liveness detection method described herein, and may execute or be used to execute said data or instructions. In some embodiments, client 200 may include a hardware device with data information processing capabilities and the necessary programs required to drive the hardware device to operate. Figure 1As shown, client 200 can communicate with server 300. In some embodiments, server 300 can communicate with multiple clients 200. In some embodiments, client 200 can interact with server 300 via network 400 to receive or send messages, etc. In some embodiments, client 200 may include mobile devices, tablets, laptops, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, the mobile device may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, the smart home device may include smart TVs, desktop computers, etc., or any combination thereof. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, gaming devices, navigation devices, etc., or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include virtual reality headsets, virtual reality glasses, virtual reality patches, augmented reality headsets, augmented reality glasses, augmented reality patches, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, head-mounted displays, VR, etc. In some embodiments, the built-in device in the motor vehicle may include an in-vehicle computer, an in-vehicle TV, etc. In some embodiments, the client 200 may include an image acquisition device and an audio acquisition device for collecting user data of the account. In some embodiments, the image acquisition device may be a two-dimensional image acquisition device (such as an RGB camera), or a two-dimensional image acquisition device (such as an RGB camera) and a depth image acquisition device (such as a 3D structured light camera, a laser detector, etc.). In some embodiments, the client 200 may be a device with positioning technology for locating the position of the client 200.

[0047] In some embodiments, the client 200 may have one or more applications (APPs) installed. The APPs provide the target user 100 with the ability and interface to interact with the outside world via the network 400. The APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, video APPs, financial management APPs, instant messaging tools, email clients, social media platform software, etc. In some embodiments, the client 200 may have a target APP installed. The target APP can collect facial images and corresponding audio information of users corresponding to multiple accounts for the client 200, thereby creating a user data set. In some embodiments, the target object 100 can also trigger a liveness detection request through the target APP. The target APP can respond to the liveness detection request by executing the liveness detection method described in this specification. The liveness detection method will be described in detail later.

[0048] Server 300 may be a server providing various services, such as a backend server that supports liveness detection for multiple accounts collected from client 200, including user data sets and account login information. In some embodiments, the liveness detection method can be executed on server 300. In this case, server 300 may store data or instructions for executing the liveness detection method described herein, and may execute or be used to execute the data or instructions. In some embodiments, server 300 may include a hardware device with data processing capabilities and the necessary programs to drive the hardware device. Server 300 can communicate with multiple clients 200 and receive data sent by clients 200.

[0049] Network 400 serves as a medium to provide a communication connection between client 200 and server 300. Network 400 facilitates the exchange of information or data. For example... Figure 1 As shown, client 200 and server 300 can connect to network 400 and transmit information or data to each other through network 400. In some embodiments, network 400 can be any type of wired or wireless network, or a combination thereof. For example, network 400 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), and Bluetooth networks. TM ZigBee TM A network, a near-field communication (NFC) network, or a similar network. In some embodiments, network 400 may include one or more network access points. For example, network 400 may include wired or wireless network access points, such as base stations or internet exchange points, through which one or more components of client 200 and server 300 can connect to network 400 to exchange data or information.

[0050] It should be understood that Figure 1 The number of clients 200, servers 300, and networks 400 shown is merely illustrative. Depending on implementation needs, there can be any number of clients 200, servers 300, and networks 400.

[0051] It should be noted that the liveness detection method can be executed entirely on the client 200, entirely on the server 300, or partially on the client 200 and partially on the server 300.

[0052] Figure 2A hardware structure diagram of a computing device 600 provided according to an embodiment of this specification is shown. The computing device 600 can execute the liveness detection method described in this specification. The liveness detection method is described in other parts of this specification. When the liveness detection method is executed on a client 200, the computing device 600 can be the client 200. When the liveness detection method is executed on a server 300, the computing device 600 can be the server 300. When the liveness detection method can be executed partly on the client 200 and partly on the server 300, the computing device 600 can be both the client 200 and the server 300.

[0053] like Figure 2 As shown, the computing device 600 may include at least one storage medium 630 and at least one processor 620. In some embodiments, the computing device 600 may also include a communication port 650 and an internal communication bus 610. Additionally, the computing device 600 may include I / O components 660.

[0054] The internal communication bus 610 can connect different system components, including storage medium 630, processor 620 and communication port 650.

[0055] I / O component 660 supports input / output between computing device 600 and other components.

[0056] Communication port 650 is used for data communication between computing device 600 and external sources. For example, communication port 650 can be used for data communication between computing device 600 and network 400. Communication port 650 can be a wired communication port or a wireless communication port.

[0057] Storage medium 630 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 632, a read-only storage medium (ROM) 634, or a random access storage medium (RAM) 636. Storage medium 630 also includes at least one instruction set stored in the data storage device. The instructions are computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the liveness detection method provided in this specification.

[0058] At least one processor 620 can be communicatively connected to at least one storage medium 630 and a communication port 650 via an internal communication bus 610. At least one processor 620 is used to execute the at least one instruction set described above. When the computing device 600 is running, at least one processor 620 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the liveness detection method provided in this specification. Processor 620 can execute all the steps included in the liveness detection method. Processor 620 can be in the form of one or more processors. In some embodiments, processor 620 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof. For illustrative purposes only, only one processor 620 is described in this specification for the computing device 600. However, it should be noted that the computing device 600 may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described herein, or they may be executed jointly by multiple processors. For example, if processor 620 of the computing device 600 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 620 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0059] Figure 3 A flowchart of a liveness detection method P100 according to an embodiment of this specification is shown. As previously described, computing device 600 can execute the liveness detection method P100 of this specification. Specifically, processor 620 can read an instruction set stored in its local storage medium and then execute the liveness detection method P100 of this specification according to the instructions in the instruction set. Figure 3 As shown, method P100 may include:

[0060] S110: Obtain user data sets and account login information corresponding to multiple accounts, and extract the background features corresponding to each account in the user data set.

[0061] The user dataset may include user data corresponding to each of multiple accounts. The user data may include a user image and a corresponding voice signal, etc. The voice signal may include the audio signal corresponding to the audio information captured when the user image was taken.

[0062] The account login information can be information about the account logging in on a verification device. The verification device can be a device performing liveness verification or facial verification. The device can include a terminal, client, or platform, etc. The account login information can include the device ID of the target device, login time, login type, or login history, etc.

[0063] The background features can be features of background information contained in the user data. These background features can include speech background features and image background features. The speech background features can be features corresponding to background noise in the speech signal. The image background features can be feature information of areas in the user image other than the user's face and body.

[0064] This involves obtaining user data sets and account login information corresponding to multiple accounts, and extracting the background features corresponding to each account from the user data sets. Specifically, this can be done as follows:

[0065] S111: Obtain user data sets and account login information corresponding to multiple accounts.

[0066] For example, the processor 620 can directly receive user data sets and account login information corresponding to multiple accounts uploaded by the target user 100 through the client 200 or terminal; or, it can receive user data sets corresponding to multiple accounts uploaded by the target user 100 through the client 200 or terminal, and obtain the account login information of each account based on the account identifier of each account among the multiple accounts; or, it can acquire user video corresponding to each account among the multiple accounts through a video acquisition device, extract the user image corresponding to each account and the voice signal corresponding to the user image from the user video, thereby obtaining user data corresponding to each account, and then, based on the account identifier of each account, obtain the user data of each account. The system can obtain user data sets for multiple accounts by acquiring user images of each account through an image acquisition device and audio signals of each account through an audio acquisition device. Then, based on the account identifier of each account, the system can obtain the user data sets and login information of each account. Alternatively, when there are many accounts or the memory of the user data set is large, the system can also receive liveness detection requests sent by the target user 100 through the client 200 or the terminal. These liveness detection requests carry the storage address of the user data set and the account login information, and based on this storage address, the system can obtain the user data set and account login information of multiple accounts.

[0067] S112: Extract the background features corresponding to each account from multiple accounts in the user dataset.

[0068] For example, processor 620 can extract voice background features from voice signals, extract image background features from user images, and use the image background features and voice background features as background features for each account, as follows:

[0069] (1) Extract speech background features from speech signals

[0070] For example, the processor 620 can extract background signals from speech signals and convert the background signals into background speech maps, and use a background noise extraction network to extract speech background features from the background speech maps.

[0071] The background signal can be the signal from the edge region of the speech signal, or it can be understood as the speech signal from the region other than the foreground region. The processor 620 can extract the background signal from the speech signal in various ways. For example, the processor 620 can add an inverse Hamming window to the speech signal to extract the background signal, or it can use other algorithms to identify the signal from the edge region of the speech signal to obtain the background signal, and so on.

[0072] After extracting the background signal from the speech signal, the processor 620 can convert the background signal into a background speech map. The background speech map can be a feature map corresponding to the features in the background signal, and this feature map can be a two-dimensional map. The processor 620 can convert the background signal into a background speech map in various ways. For example, the processor 620 can use convolutional layers or other network layers to extract two-dimensional feature information from the background signal to obtain the background speech map.

[0073] After converting the background signal into a background speech map, the processor 620 can use a background noise extraction network to extract speech background features from the speech map. The network structure of the background noise extraction network can be a commonly used classification network, such as any one of ResNet (residual network), VGGNet (a deep convolutional network), or InceptionV3 (asymmetric convolutional network). The input of the background noise extraction network can be a two-dimensional background speech map, and the output can be the corresponding speech background features, the liveness classification result, and the predicted probability corresponding to the liveness classification result.

[0074] The training process of the background noise extraction network may include the following steps: the processor 620 can acquire background speech map samples, divide the background speech map samples into regions using a preset background noise extraction network to obtain multiple region maps, perform liveness detection on the multiple region maps to obtain the first predicted liveness category of the background speech map samples and the second predicted liveness category corresponding to each region map in the multiple region maps, and determine the speech liveness loss information of the background speech map samples based on the first predicted liveness category and the second predicted liveness category, and converge the preset background noise extraction network based on the speech liveness loss information to obtain the trained background noise extraction network.

[0075] There are several ways to obtain background speech map samples. For example, the processor 620 can obtain speech signal samples, extract background signal samples from the speech signal samples, and then convert the background signal samples into background speech map samples.

[0076] After acquiring background speech map samples, the processor 620 can use a preset background noise extraction network to divide the background speech map samples into regions, obtaining multiple region maps. The region maps can be the maps corresponding to the regions divided from the background speech map. There are various ways to divide the background speech map into regions. For example, the processor 620 can use a preset background noise extraction network to divide the background speech map samples into a preset number of regions, thereby obtaining multiple region maps. Alternatively, it can acquire the size information of the background speech map samples, determine the number of regions to be divided based on the size information, and then use the preset background noise extraction network to divide the background speech map samples into regions based on the number of regions, thereby obtaining multiple region maps, and so on.

[0077] After dividing the background speech map samples into regions, the processor 620 can perform liveness detection on multiple region maps, obtaining a first predicted liveness category for the background speech map samples and a second predicted liveness category for each region map. The first predicted liveness category can be the overall liveness classification result for the background speech map samples, and the second predicted liveness category can be the local liveness classification result for each region map. It should be noted that the first and second predicted liveness categories can be the same or different. Furthermore, the second predicted liveness categories for different region maps can be the same or different.

[0078] After performing liveness detection on multiple region maps, the processor 620 can determine the speech liveness loss information of the background speech map samples based on the first predicted liveness category and the second predicted liveness category. The speech liveness loss information can be the loss information generated by a preset background noise extraction network after performing liveness prediction on the background speech map samples. The processor 620 can determine the speech liveness loss information of the background speech map samples in several ways. For example, the processor 620 can compare the first predicted liveness category with the labeled liveness category of the background speech map samples to obtain global classification loss information; compare the second predicted liveness categories corresponding to different region maps to obtain local consistency loss information; compare the second predicted liveness category with the labeled liveness category of the background speech map samples to obtain local classification loss information; and accumulate the global classification loss information, local consistency loss information, and local classification loss information to obtain the speech liveness loss information of the background speech map samples.

[0079] The global classification loss information can be the classification loss information between the overall predicted liveness category of the background speech map and the labeled liveness category. The local consistency loss information can be the loss information generated by the difference between the second predicted liveness categories corresponding to different regions of the map. The constraint of this local consistency loss information is that the second predicted liveness category of each region map is consistent, that is, the background noise distribution is relatively uniform, so the prediction results obtained from different regions should be consistent. The local classification loss information can be the loss information generated by the difference between the second predicted liveness category corresponding to each region map and the labeled liveness category. This can be understood as whether to use only local features for black market attack classification when performing liveness classification. During the learning process, it is hoped that the features corresponding to each region map can accurately predict the liveness category, thereby achieving the classification of black market attacks. The loss functions corresponding to the global classification loss information, local consistency loss information, and local classification loss information can be the same, such as the cross-entropy loss function (CE) or other classification loss functions. The loss functions corresponding to the global classification loss information, local consistency loss information, and local classification loss information can also be set differently according to the actual application.

[0080] After determining the global classification loss information, local consistency loss information, and local classification loss information, the processor 620 can accumulate these three information to obtain the speech liveness loss information of the background speech map sample. Taking the cross-entropy loss function as the loss function corresponding to the classification loss information as an example, the speech liveness loss information can be calculated as shown in formula (1):

[0081] Loss voice =Loss CE +Loss consistency +Loss local-CE (1)

[0082] Among them, LOSS vooce For speech liveness loss information, Loss CE For global classification loss information, Loss cobsistency For local consistency loss information, Loss local-CE This is information on local classification loss.

[0083] In some embodiments, after determining the global classification loss information, the local consistency loss information, and the local classification loss information, the processor 620 can also obtain the speech loss weights, and weight the global classification loss information, the local consistency loss information, and the local classification loss information based on the speech loss weights, and accumulate the weighted global classification loss information, the weighted local consistency loss information, and the weighted local classification loss information to obtain the speech liveness loss information of the background speech map sample.

[0084] After determining the speech liveness loss information of the background speech map samples, the processor 620 can converge the preset background noise extraction network based on the speech liveness loss information to obtain the trained background noise extraction network. There are several ways to converge the preset background noise extraction network based on the speech liveness loss information. For example, the processor 620 can use a gradient descent algorithm to update the network parameters of the preset background noise extraction network based on the speech liveness loss information, and then return to the step of obtaining background speech map samples until the preset background noise extraction network converges, thus obtaining the trained background noise extraction network. Alternatively, other network parameter update algorithms can be used to update the network parameters of the preset background noise extraction network based on the speech liveness loss information to obtain the trained background noise extraction network, and so on.

[0085] (2) Extracting background features from user images

[0086] For example, the processor 620 can segment human body regions in a user image to obtain a user background image, and use an image background extraction network to extract image background features from the user background image.

[0087] The human body region can be any area in the user image that includes all parts of the human body, or it can be any area enclosed by the outline of the human body. The user background image can be any background area in the user image excluding the human body region. The processor 620 can segment the human body region from the user image in several ways. For example, the processor 620 can segment the region image corresponding to the human body region from the user image and fill the human body region with all zero pixels to obtain the initial user background image; it can also obtain a preset filled background corresponding to the human body region and fill the human body region in the initial user background image with the preset filled background to obtain the user background image.

[0088] The processor 620 can segment the region image corresponding to the human body region in the user image in various ways. For example, the processor 620 can input the user image into a pre-trained human body segmentation model, so that the region image corresponding to the human body region can be segmented directly in the user image.

[0089] The preset fill background can be a pre-set fill background. The fill background can be the background information used to fill the segmented human body region. There are several ways to obtain the preset fill background corresponding to the human body region. For example, the processor 620 can directly generate the fill background corresponding to the human body region using a pre-trained generative network and use this fill background as the preset fill background; alternatively, it can directly select the preset fill background corresponding to the human body region from a set of fill backgrounds, and so on. After obtaining the preset fill background, the processor 620 can fill the human body region in the initial user background image with the preset fill background, thereby obtaining the user background image.

[0090] After obtaining the user's background image, the processor 620 can use an image background extraction network to extract image background features from the user's background image. The image background extraction network may include a background feature encoding subnetwork and a salient object detection subnetwork. The processor 620 can extract image background features from the user's background image using the image background extraction network in various ways. For example, the processor 620 can use the salient object detection subnetwork to detect at least one salient region in the user's background image, obtaining a salientity map, and based on the salientity map, use the background feature encoding subnetwork to encode features in the user's background image to obtain image background features.

[0091] The saliency map can be the image corresponding to at least one salient object or region in the user's background image. Based on the saliency map, there are various ways to encode features of the user's background image using a background feature encoding subnetwork. For example, the processor 620 can identify at least one corresponding salient object or region in the user's background image based on the saliency map using the background feature encoding subnetwork, and then perform feature extraction encoding on at least one salient object or region to obtain the image background features. Since the background feature encoding subnetwork is a classification network, after outputting the image background features, it can also perform liveness classification on the user's background image based on the image background features to obtain the predicted liveness category, and thus determine the predicted probability of a cyberattack.

[0092] The background feature encoding subnetwork can be a common classification network, and the network structure can be of various types, such as ResNet, VGGNet, or InceptionV3, etc.

[0093] The training process of the image background extraction network may include the following steps: the processor 620 can acquire user background image samples, which include saliency map samples; a preset image background extraction network is used to perform liveness detection on the background image samples to obtain the third predicted liveness category corresponding to each pixel; based on the saliency map samples, the third predicted liveness category and the labeled liveness category of the user background image, the image liveness loss information of the user background image samples is determined; and the preset image background extraction network is converged based on the image liveness loss information to obtain the trained image background extraction network.

[0094] There are several ways to perform liveness detection on user background image samples using a preset image background extraction network. For example, the processor 620 can use a preset image background extraction network to extract pixel-level image background features from user background image samples, obtain sample image background features corresponding to each pixel, and perform liveness detection on the corresponding pixel region in the user background image based on the sample image background features, thereby obtaining the third predicted liveness category corresponding to each pixel.

[0095] After performing liveness detection on the user background image sample, the processor 620 can determine the image liveness loss information of the user background image sample based on the saliency map sample, the third predicted liveness category, and the labeled liveness category of the user background image sample. The processor 620 can determine the image liveness loss information of the user background image sample in several ways. For example, the processor 620 can compare the third predicted liveness category with the labeled liveness category of the user background image sample to obtain the initial image liveness loss information corresponding to each pixel; identify the pixel value at each pixel position in the saliency map sample to obtain the liveness loss weight corresponding to each pixel; and perform weighted fusion of the initial liveness loss information based on the liveness loss weight to obtain the image liveness loss information of the user background image sample.

[0096] The initial image liveness loss information can be understood as the loss information generated after performing liveness classification at the pixel level. The loss function used to determine the initial liveness loss information can be a classification loss function, such as the cross-entropy loss function or other loss functions.

[0097] In this process, after identifying the pixel value at each pixel location in the salient image sample, the pixel value can be directly used as the liveness loss information of the corresponding pixel or pixel location. The processor 620 performs weighted fusion of the initial image liveness loss information based on the liveness loss weight to obtain the image liveness loss information of the user background image sample, as shown in formula (2):

[0098]

[0099] Among them, Loss image The image liveness loss information for the user background image sample is given, where H / W is the size of the user background image sample, i is the corresponding pixel, and S... i L is the liveness loss weight for the corresponding pixel. CE-i This provides the initial image liveness loss information for the corresponding pixels.

[0100] After determining the image liveness loss information of the user's background image sample, the processor 620 can converge the preset image background extraction network based on the image liveness loss information. The convergence method for the preset image background extraction network is similar to the convergence process for the preset background noise extraction network, as described above, and will not be repeated here.

[0101] (3) Use image background features and voice background features as the background features corresponding to each account.

[0102] For example, the processor 620 can use the image background features and voice background features of each account as the corresponding background features, or it can combine the image background features and voice background features of each account according to a preset combination method to obtain the background features of the corresponding account.

[0103] S120: Based on account login information, construct a graph network by treating each of the multiple accounts as a node, and propagate the background features in the graph network to obtain the liveness features corresponding to each account.

[0104] Among them, a graph network (GCN) can be a network that represents the relationships between nodes. A graph network can include nodes and the edges connecting them.

[0105] Among them, liveness features can be characteristic information that indicates whether an account is a live user.

[0106] Specifically, based on account login information, a graph network is constructed using each of the multiple accounts as a node, and background features are propagated within the graph network to obtain the liveness features corresponding to each account. The details are as follows:

[0107] S121: Based on account login information, construct a graph network by treating each of the multiple accounts as a node.

[0108] For example, the processor 620 can determine the edge information of the initial graph network based on the account login information, construct the initial image network by taking each account as a node based on the edge information, and train the initial image network using a fusion classification loss function to obtain the trained graph network.

[0109] The edge information can be the information about the edges formed by connecting nodes. Edge information can include the identifiers or positions of the two connected nodes corresponding to each edge. There are multiple ways to determine the edge information of the initial graph network based on account login information. For example, the processor 620 can extract the device identifier of the verification device for each account from the account login information, select two accounts that log in to the same verification device from the multiple accounts based on the device identifier, obtain at least one account pair, and determine the edge information of the initial graph network based on at least one account pair.

[0110] There are several ways to determine the edge information of the initial graph network based on at least one account pair. For example, the processor 620 obtains the account identifier of each account pair in at least one account, and determines the node edge information corresponding to each account pair based on the account identifier, and uses the node edge information corresponding to each account pair as the edge information of the initial graph network.

[0111] After determining the edge information of the initial network, the processor 620 can construct an initial graph network based on the edge information, treating each account among multiple accounts as a node. The processor 620 can construct the initial graph network in several ways. For example, it can construct a three-layer GCN, treating each account among multiple accounts as a node. In each layer, the nodes are connected based on the edge information. Then, nodes corresponding to the same account in different layers of the GCN are connected to obtain the initial graph network.

[0112] After constructing the initial graph network, the processor 620 can train the initial graph network to obtain the final graph network. The graph network includes a speech graph network layer, an image graph network layer, and a fusion feature graph network layer. Nodes corresponding to the same account in the speech graph network layer, image graph network layer, and feature fusion graph network layer are interconnected. Nodes in the speech graph network layer represent the speech background features of the account; that is, the features propagated in the speech graph network layer are speech background features. Nodes in the image graph network layer represent the image background features of the account; that is, the features propagated in the image graph network layer are image background features. Nodes in the feature fusion graph network layer represent the fused features of the account (initialized using the average features of the speech background features and image background features); that is, the features propagated in the feature fusion graph network layer are fused features.

[0113] The processor 620 can train the initial graph network in several ways. For example, the processor 620 can use a classification loss function that fuses features to train the initial graph network, or it can use other types of classification loss functions to train the initial graph network, thereby obtaining the trained graph network.

[0114] The processor 620 can train the initial graph network using a classification loss function based on fused features in several ways. For example, the processor 620 can acquire user data samples and extract sample speech background features and sample image background features from them. The speech background feature samples are input into the speech graph network layer to obtain the propagated target sample speech background features, and the sample image background features are input into the image graph network layer to obtain the propagated target sample image background features. Then, the target sample speech background features and target sample image background features are fused to obtain sample fused features. These fused features are then input into the fused feature graph network layer to obtain sample liveness features. Based on these liveness features, the predicted liveness category corresponding to the user data sample is determined, and the predicted liveness category is compared with the labeled liveness category of the user data to obtain the fused classification loss information of the fused features. The initial graph network is then converged based on this fused classification loss information to obtain the trained graph network.

[0115] After training the initial graph network, the processor 620 can propagate the background features in the graph network to obtain the liveness features corresponding to each account.

[0116] S122: Propagate the background features in the graph network to obtain the liveness features corresponding to each account.

[0117] The propagation of background features within a graph network can be viewed as a feature extraction process that extracts liveness features from the background features. There are several ways to propagate background features within a graph network, including the following:

[0118] For example, the processor 620 can propagate the speech background features in the speech graph network layer to obtain the propagated target speech background features, propagate the image background features in the image graph network layer to obtain the propagated target image background features, and propagate the target speech background features and target image background features in the fusion feature graph network layer to obtain the liveness features corresponding to each account.

[0119] The processor 620 can propagate speech background features in the speech graph network layer in various ways. For example, the processor 620 can use the aggregation of first-order neighbor attention as the propagation function to propagate the speech background features in the speech graph network layer to obtain the propagated target speech background features. Alternatively, other propagation functions can be used to propagate the speech background features in the speech graph network layer to obtain the propagated target speech background features, and so on.

[0120] The processor 620 uses first-order neighbor attention aggregation as the propagation function. There are multiple ways to propagate speech background features in the speech graph network layer. For example, the processor 620 can select the target node and a preset number of neighboring nodes in the speech graph network layer, aggregate the speech background features of the neighboring nodes with the speech background features of the target node to obtain the target speech background features corresponding to the target node, and return to execute the step of selecting the target node and a preset number of neighboring nodes in the speech graph network layer until all nodes in the speech graph network layer are target nodes, thus obtaining the target speech background features corresponding to each account.

[0121] The processor 620 can aggregate the speech background features of neighboring nodes with the speech background features of the target node in various ways. For example, the processor 620 can accumulate the speech background features of neighboring nodes and calculate the average value of the accumulated speech background features. Then, it can add the average value to the feature value of the speech background features of the target node to obtain the current speech background features corresponding to the target node. Finally, it can input the current speech background features into a preset fully connected layer to obtain the target speech background features corresponding to the target node.

[0122] The processor 620 propagates image background features in the image graph network layer in a similar way to propagating speech background features in the speech image network layer, as described above, and will not be repeated here.

[0123] After propagating the speech background features and image background features separately, the processor 620 can propagate the propagated target speech background features and target image background features in the fusion feature map network layer to obtain the liveness feature corresponding to each account. The processor 620 can propagate the propagated target speech background features and target image background features in the fusion feature map network layer in several ways. For example, the processor 620 can calculate the average feature of the target speech background features and target image background features to obtain the fusion feature of the corresponding account, propagate the fusion feature in the fusion feature map network layer to obtain the propagated target fusion feature, and use the target fusion feature as the liveness feature corresponding to each account.

[0124] The processor 620 propagates the fused features in the fused feature graph network layer in a similar manner to propagating the speech background features in the speech graph network layer, as detailed above, and will not be repeated here. After obtaining the target fused feature of each account after propagating the fused features in the feature fusion graph network layer, the processor can use the target fused feature as the liveness feature of the corresponding account.

[0125] S130: Based on liveness features, perform liveness detection on multiple accounts and output the liveness detection results.

[0126] The liveness detection result includes at least one of an attack cluster or a liveness cluster. An attack cluster can be a cluster of multiple attack accounts, where each attack account is an account that attacks facial recognition or facial verification. A liveness cluster can be a cluster of multiple live accounts, where each live account is an account that indicates the user is alive during facial recognition or facial verification.

[0127] There are several ways to perform liveness detection on multiple accounts based on liveness features, as follows:

[0128] For example, the processor 620 can select at least one attack account from multiple accounts based on liveness features to obtain a first attack cluster. Based on liveness features, it can perform anomaly detection on candidate accounts from multiple accounts to obtain a second attack cluster and a liveness cluster. The candidate accounts are accounts from multiple accounts other than the first attack cluster. The processor 620 can also use the first attack cluster and the second attack cluster as the attack cluster and use the attack cluster and the liveness cluster as the liveness detection results of multiple accounts.

[0129] The first attack cluster can be a cluster composed of attack group leaders identified based on liveness characteristics. There are multiple ways to select at least one attack account from multiple accounts based on liveness characteristics. For example, the processor 620 can classify each account in the multiple accounts as liveness based on liveness characteristics, and select at least one attack account from the multiple accounts based on the liveness classification results to obtain the first attack cluster.

[0130] There are various ways to classify each account among multiple accounts based on liveness features. For example, the graph network can also include a classification subnetwork. The processor 620 inputs the liveness features of each account into the classification subnetwork to obtain the liveness classification result for each account. This liveness classification result can include either a live account or an attacking account.

[0131] After classifying each account based on liveness characteristics, the processor 620 can select at least one attack account from multiple accounts based on the liveness classification results. The processor 620 can select at least one attack account from multiple accounts in several ways; for example, it can select at least one account whose liveness classification result is an attack account, thus obtaining the first attack cluster.

[0132] Graph networks have already revealed many clustered attacks by malicious actors, but further detection is needed for sparse, outlier attacks. Therefore, accounts other than the primary attack cluster can be selected from multiple accounts to obtain candidate accounts.

[0133] After selecting candidate accounts, the processor 620 can perform anomaly detection on the candidate accounts among multiple accounts based on liveness features to obtain a second attack cluster and a liveness cluster. The processor 620 can perform anomaly detection on candidate accounts in several ways. For example, the processor 620 can reduce the dimensionality of the liveness features to a preset dimension to obtain the dimensionality-reduced target liveness features. Based on the target liveness features, the processor 620 can estimate the distribution of the candidate accounts to obtain the normal distribution probability of the candidate accounts, and select at least one account among the candidate accounts whose normal distribution probability is less than a preset probability threshold to obtain the second attack cluster. At least one account among the candidate accounts other than the second attack cluster is then considered as the liveness cluster.

[0134] There are several ways to reduce the dimensionality of liveness features to a preset dimension. For example, the processor 620 can use an autoencoder to reduce the dimensionality of liveness features to a preset dimension. The preset dimension can be any dimension, including 128 dimensions or less, and so on.

[0135] The processor 620 can train the autoencoder using liveness features and the loss function is reconstruction loss, thus obtaining the trained autoencoder.

[0136] After reducing the dimensionality of the liveness features, the processor 620 can estimate the distribution of candidate accounts based on the reduced target liveness features, thus obtaining the normal distribution probability of the candidate accounts. Distribution estimation is primarily used to estimate the probability that an account belongs to the normal account distribution and the probability that it belongs to the attack account distribution. Therefore, the processor 620 can perform distribution estimation for candidate accounts in various ways. For example, the processor 620 can use a Gaussian mixture model (GMM) based on the target liveness features to estimate the distribution of each account, thereby obtaining the normal distribution probability of the candidate accounts.

[0137] After estimating the distribution of candidate accounts, the processor 620 can select at least one account from the candidate accounts whose normal distribution probability is less than a preset probability threshold, thereby obtaining a second attack cluster. Then, at least one account from the candidate accounts other than the second attack cluster is taken as a live cluster.

[0138] After identifying the first attack cluster, the second attack cluster, and the liveness cluster, the processor 620 can use the first attack cluster and the second attack cluster as the attack cluster, and use the attack cluster and the liveness cluster as the liveness detection results of multiple accounts.

[0139] After determining the liveness detection results for multiple accounts, the processor 620 can output the liveness detection results. There are several ways to output the liveness detection results. For example, the processor 620 can directly return the liveness detection results to the client 200 or terminal of the target user 100; alternatively, it can return the liveness detection results to the verification device that needs to perform facial recognition or verification, so that the verification device can block or allow each of the multiple accounts based on the liveness detection results; or, the liveness detection results can be visualized.

[0140] There are several ways to visualize the liveness detection results. For example, the processor 620 can display the liveness detection results through a display device. During the display process, it can also display the attack clusters and liveness clusters in the liveness detection results in different styles. Alternatively, it can provide sound and light prompts when attack clusters are present in the liveness detection results, and so on.

[0141] After outputting the liveness detection result, the processor 620 can also intercept the attack account corresponding to the attack cluster or perform other preset processing steps, etc.

[0142] It should be noted that when performing liveness detection or attack judgment on multiple accounts, the results of graph network and anomaly detection should be used as the standard. For example, for a face recognition transaction, if the judgment result of one or more accounts output by the graph network is a black market attack or the anomaly detection is a black market attack, then it is judged as a black market attack; otherwise, it is judged as a liveness detection.

[0143] This solution primarily relies on background noise to identify clustered attacks by malicious actors. Therefore, the overall flowchart of the liveness detection method targeting clustered attacks can be illustrated as follows: Figure 4 As shown, it can include four parts: speech background noise analysis, image background information analysis, GCN tag propagation, and anomaly detection and attack judgment, specifically as follows:

[0144] (1) Speech background noise analysis

[0145] This solution focuses on detecting cyberattacks using speech signals. Typically, the foreground information (human voice) in a cyberattack is indistinguishable from that of a normal user, while the background information is relatively quiet, closer to white noise. Therefore, background noise analysis can be used as a means of detecting cyberattacks. Background noise analysis mainly involves training a background noise extraction network and inputting the preprocessed background speech map into the network to obtain speech features and the predicted probability of cyberattacks. The training process of the background noise extraction network is detailed above and will not be repeated here.

[0146] (2) Image background information analysis

[0147] Traditional liveness detection methods often focus only on the face, body, or areas closely related to both, which is effective for detecting general liveness attacks. However, for cybercrime attacks, these areas are usually pre-tuned to eliminate any traces of the attack. The background area, on the other hand, is often where cybercriminals tend to overlook certain elements (e.g., multiple mobile devices or attack tools). Therefore, background information in an image can be used to identify cybercrime attacks. Image background information analysis primarily involves segmenting the user image into a human body to obtain the user's background image. A salient object detector within an image background extraction network then identifies salient spectra in the user's background image. These salient spectra and the user's background image are input into the feature encoder of the image background extraction network to obtain the corresponding image background features and the predicted probability of a cybercrime attack. The training process of the image background extraction network can be found above and will not be repeated here.

[0148] (3) GCN tag propagation

[0149] By extracting voice and image background features from user data as background features, the probability of malicious attacks corresponding to these background features can be detected. However, not all malicious attacks exhibit obvious background anomalies. To detect more malicious attacks, a GCN (Graphical Network Array) at the account level can be constructed to propagate background features, thereby uncovering batch malicious attacks targeting related users and improving the accuracy of liveness detection against clustered malicious attacks. The label propagation of the GCN can involve constructing a three-layer GCN as an initial graph network. In this initial graph network, the nodes in the first layer are voice features, the nodes in the second layer are image features, and the nodes in the third layer are fused features (initialized using the average features of voice and image). Each node represents an account or a feature of that account. If two accounts have logged into the same device simultaneously, the two nodes are connected. The same device account node in the third layer is connected to the corresponding nodes in the first and second layers. Then, a malicious attack classification loss based on fused features is used to train the initial graph network, resulting in a trained graph network. First-order neighbor attention aggregation is used as the propagation function to propagate speech background features in the first layer and image background features in the second layer. Then, the propagated target speech background features and target image background features are fused, and the fused features are propagated in the third layer to obtain the liveness features for each account. Based on these liveness features, multiple accounts are classified for liveness detection, thereby identifying the first attack cluster.

[0150] (4) Anomaly detection and attack judgment

[0151] While GCN tag propagation can uncover many clustered illicit attacks, it often performs poorly in detecting sparse outlier attacks. Therefore, anomaly detection can also be performed using Gaussian Mixture Model (GMM) based on autoencoders. The anomaly detection process can involve reducing the dimensionality of liveness features to below 128 dimensions using an autoencoder, and then applying a GMM to all samples for distributed attacks. When the probability of a sample being a normal sample is below a threshold, the sample (account) can be identified as an anomalous illicit attack.

[0152] To determine if a face recognition transaction is a malicious attack, if the GCN judgment in the third step is a malicious attack or the anomaly detection in the fourth step is a malicious attack, then the account or user associated with the face recognition transaction is considered to be a malicious attacker; otherwise, it can be considered a liveness detection.

[0153] In summary, the liveness detection method P100 and system 001 provided in this specification, after acquiring user data sets and account login information corresponding to multiple accounts, and extracting the background features corresponding to each account in the user data set, construct a graph network based on the account login information, using each account as a node, and propagating the background features in the graph network to obtain the liveness features corresponding to each account. Based on the liveness features, liveness detection is performed on multiple accounts, and the liveness detection results are output. The liveness detection results include at least one of attack clusters or liveness clusters. Since this scheme can extract and analyze background information (voice background noise and image noise) ignored in general liveness detection algorithms, and use graph networks to propagate features and risks, it can fully explore risky users, thereby achieving liveness detection against black market activities that target cluster attacks. Therefore, it can improve the accuracy of liveness detection.

[0154] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for performing liveness detection. When the executable instructions are executed by a processor, they instruct the processor to implement the steps of the liveness detection method P100 described herein. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on a computing device 600, the program code causes the computing device 600 to perform the steps of the liveness detection method P100 described herein. The program product for implementing the above method may employ a portable compact disk read-only memory (CD-ROM) containing program code and may run on the computing device 600. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on computing device 600, partially on computing device 600, as a standalone software package, partially on computing device 600 and partially on a remote computing device, or entirely on a remote computing device.

[0155] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0156] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0157] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0158] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0159] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. All contents used for all purposes, except for any history of prosecution documents relating to it, that may be inconsistent with or conflict with this document, or any such history of prosecution documents that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0160] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for detecting liveness, comprising: Obtain user data sets and account login information corresponding to multiple accounts, and extract background features corresponding to each of the multiple accounts from the user data sets, wherein the background features include voice background features and image background features; Based on the account login information, a graph network is constructed using each of the multiple accounts as a node, and the background features are propagated within the graph network to obtain the liveness features corresponding to each account; and Based on the liveness characteristics, liveness detection is performed on the multiple accounts, and liveness detection results are output. The liveness detection results include at least one of attack clusters or liveness clusters.

2. The liveness detection method according to claim 1, wherein, The user data set includes the user image corresponding to each account and the voice signal corresponding to the user image; as well as The step of extracting the background features corresponding to each of the multiple accounts in the user data set includes: Extract speech background features from the speech signal. Extract the image background features from the user image, and The image background features and the voice background features are used as the background features corresponding to each account.

3. The liveness detection method according to claim 2, wherein, Extracting speech background features from the speech signal includes: Extracting the background signal from the speech signal and converting the background signal into a background speech map; and A background noise extraction network is used to extract speech background features from the background speech graph.

4. The live detection method according to claim 3, wherein, The training process of the background noise extraction network includes the following steps: Background speech map samples are obtained, and a preset background noise extraction network is used to divide the background speech map samples into regions to obtain multiple region maps. Liveness detection is performed on the multiple region maps to obtain a first predicted liveness category for the background speech map samples and a second predicted liveness category for each region map in the multiple region maps; and Based on the first predicted liveness category and the second predicted liveness category, the speech liveness loss information of the background speech map sample is determined, and the preset background noise extraction network is converged based on the speech liveness loss information to obtain the trained background noise extraction network.

5. The liveness detection method according to claim 4, wherein, The step of determining the speech liveness loss information of the background speech map samples based on the first predicted liveness category and the second predicted liveness category includes: The first predicted liveness category is compared with the labeled liveness category of the background speech map sample to obtain global classification loss information; The second predicted liveness category corresponding to different region maps is compared to obtain local consistency loss information. The constraint condition of the local consistency loss information is that the second predicted liveness category of each region map is consistent. The second predicted liveness category is compared with the labeled liveness category of the background speech map sample to obtain local classification loss information; and The global classification loss information, the local consistency loss information, and the local classification loss information are summed to obtain the speech liveness loss information of the background speech map sample.

6. The liveness detection method according to claim 2, wherein, The step of extracting image background features from the user image includes: The human body region is segmented from the user image to obtain the user background image; and An image background extraction network is used to extract image background features from the user's background image.

7. The liveness detection method according to claim 6, wherein, The step of segmenting the human body region from the user image to obtain the user background image includes: Segment the human body region from the user image, and fill the human body region with all zero pixels to obtain the initial user background image; and Obtain the preset fill background corresponding to the human body area, and fill the human body area in the initial user background image with the preset fill background to obtain the user background image.

8. The liveness detection method according to claim 6, wherein, The image background extraction network includes a background feature encoding subnetwork and a salient object detection subnetwork; as well as The method of using an image background extraction network to extract image background features from the user's background image includes: The salient object detection subnetwork is used to detect at least one salient region in the user's background image, resulting in a salient map. Based on the saliency map, the background feature encoding subnetwork is used to encode the user's background image to obtain the image background features.

9. The liveness detection method according to claim 6, wherein, The training process of the image background extraction network includes the following steps: Obtain user background image samples, which include saliency map samples; A preset image background extraction network is used to perform liveness detection on the user background image samples to obtain the third predicted liveness category for each pixel; Based on the saliency map samples, the third predicted liveness category, and the labeled liveness category of the user background image samples, determine the image liveness loss information of the user background image samples; and The preset image background extraction network is converged based on the image liveness loss information to obtain the trained image background extraction network.

10. The liveness detection method according to claim 9, wherein, The step of determining the image liveness loss information of the user background image sample based on the saliency map sample, the third predicted liveness category, and the labeled liveness category of the user background image sample includes: The third predicted liveness category is compared with the labeled liveness category of the user background image sample to obtain the initial image liveness loss information corresponding to each pixel; The pixel value at each pixel location is identified in the saliency map sample to obtain the liveness loss weight corresponding to each pixel; and The initial image liveness loss information is weighted and fused based on the liveness loss weight to obtain the image liveness loss information of the user background image sample.

11. The liveness detection method according to claim 1, wherein, The step of constructing a graph network based on the account login information, using each of the multiple accounts as a node, includes: Based on the account login information, determine the edge information of the initial graph network; Based on the edge information, the initial graph network is constructed by treating each of the multiple accounts as a node; and The initial graph network is trained using a fusion classification loss function to obtain the trained graph network.

12. The liveness detection method according to claim 11, wherein, The step of determining the edge information of the initial graph network based on the account login information includes: Extract the device identifier of the verification device for each of the multiple accounts from the account login information; Based on the device identifier, two accounts that log in to the same verification device are selected from the plurality of accounts to obtain at least one account pair; and Based on the at least one account pair, the edge information of the initial graph network is determined.

13. The liveness detection method according to claim 1, wherein, The graph network includes a speech graph network layer, an image graph network layer, and a fusion feature graph network layer, with nodes corresponding to the same account in the speech graph network layer, the image graph network layer, and the fusion feature graph network layer being interconnected.

14. The liveness detection method according to claim 13, wherein, The process of propagating the background features in the graph network to obtain the liveness features corresponding to each account includes: The speech background features are propagated in the speech graph network layer to obtain the propagated target speech background features. The image background features are propagated in the image graph network layer to obtain the propagated target image background features; and The target speech background features and target image background features are propagated in the fusion feature map network layer to obtain the liveness features corresponding to each account.

15. The liveness detection method according to claim 14, wherein, The step of propagating the speech background features in the speech graph network layer to obtain the propagated target speech background features includes: In the speech graph network layer, a target node and a preset number of adjacent nodes corresponding to the target node are selected; The speech background features of the adjacent nodes are aggregated with the speech background features of the target node to obtain the target speech background features corresponding to the target node; and Return to the step of selecting the target node and a preset number of adjacent nodes in the speech graph network layer until all nodes in the speech graph network layer are target nodes, and obtain the target speech background features corresponding to each account.

16. The liveness detection method according to claim 15, wherein, The step of aggregating the speech background features of the adjacent nodes with the speech background features of the target node to obtain the target speech background features corresponding to the target node includes: The speech background features of the adjacent nodes are accumulated, and the average value of the accumulated speech background features is calculated. The average value is added to the feature value of the speech background feature of the target node to obtain the current speech background feature corresponding to the target node; and The current speech background features are input into a preset fully connected layer to obtain the target speech background features corresponding to the target node.

17. The liveness detection method according to claim 14, wherein, The step of propagating the target speech background features and target image background features in the fused feature map network layer to obtain the liveness features corresponding to each account includes: Calculate the average features of the target speech background features and the target image background features to obtain the fused features; The fused features are propagated in the fused feature map network layer to obtain the propagated target fused features; and The target fusion feature is used as the liveness feature corresponding to each account.

18. The liveness detection method according to claim 1, wherein, The liveness detection of the multiple accounts based on the liveness features includes: Based on the liveness characteristics, at least one attacking account is selected from the plurality of accounts to obtain the first attack cluster; Based on the liveness characteristics, anomaly detection is performed on candidate accounts among the multiple accounts to obtain a second attack cluster and a liveness cluster, wherein the candidate accounts are accounts other than those in the first attack cluster among the multiple accounts; and The first attack cluster and the second attack cluster are used as the attack cluster, and the attack cluster and the liveness cluster are used as the liveness detection results of the multiple accounts.

19. The liveness detection method according to claim 18, wherein, The step of selecting at least one attacking account from the plurality of accounts based on the liveness characteristics to obtain a first attack cluster includes: Based on the aforementioned liveness characteristics, each of the multiple accounts is classified as a liveness entity; and Based on the liveness classification results, at least one attacking account is selected from the multiple accounts to obtain the first attack cluster.

20. The liveness detection method according to claim 18, wherein, The step of performing anomaly detection on candidate accounts among the multiple accounts based on the liveness characteristics to obtain the second attack cluster and the liveness cluster includes: The liveness features are reduced to a preset dimension to obtain the target liveness features after dimensionality reduction. Based on the target liveness features, the distribution of the candidate accounts is estimated to obtain the normal distribution probability of the candidate accounts; and At least one account with a normal distribution probability less than a preset probability threshold is selected from the candidate accounts to obtain a second attack cluster, and at least one account other than the second attack cluster in the candidate accounts is taken as a live cluster.

21. A liveness detection system, comprising: At least one storage medium storing at least one instruction set for performing liveness detection; as well as At least one processor is communicatively connected to the at least one storage medium. When the liveness detection system is running, the at least one processor reads the at least one instruction set and executes the liveness detection method according to any one of claims 1-20.

Images