Face image anomaly detection method, device, medium and equipment

Abstract

This application provides a method, apparatus, medium, and device for anomaly detection of facial images. The method includes: acquiring multiple key facial images of the same target user recorded within a preset time period; constructing a connected graph data with related relationships based on the multiple key facial images; inputting the connected graph data into a feature extraction model to aggregate features and obtain a first output result after data information transfer and aggregation calculation among all neighboring nodes in the connected graph data; inputting the first output result into an image classifier to perform classification calculation to obtain a second output result; and determining whether there are abnormal facial images among the multiple key facial images based on the second output result. Using this application, it is possible to accurately screen, classify, and identify facial recognition post-event records that are part of a "re-photographed" black market facial recognition attack.

Description

Technical Field

[0001] This application relates to the field of electronic communication technology, and more particularly to a data update technology field, specifically to a method, apparatus, medium, and device for anomaly detection of facial images. Background Technology

[0002] With the continuous development of technology, facial recognition is being applied in increasingly widespread scenarios. However, examples of cybercriminals using fake faces to conduct online facial recognition, causing losses to customers, are commonplace. Therefore, the authenticity, security, and credibility of facial recognition in various business scenarios have become crucial issues. While many liveness detection products exist in the market, they currently have significant shortcomings. Primarily, they only utilize two-dimensional facial information, allowing individuals intending to attack facial recognition systems to use authorized user photos to spoof and deceive the system, thereby carrying out illegal and criminal activities. Summary of the Invention

[0003] This application provides an anomaly detection method, apparatus, medium, and device for facial images. The anomaly detection method for facial images provided in this application constructs a "multiple facial recognition post-record similarity connected graph data" based on multiple facial recognition post-record data for each customer. This data is then input into a pre-constructed image classifier. Utilizing the significant differences between the features of the facial recognition record "node" data used in black market attacks and the features of most of the customer's other normal facial recognition record "node" data, the output results can accurately screen, classify, and identify facial recognition post-records used in black market facial recognition attacks that are "re-photographed."

[0004] This application provides an anomaly detection method for face images, the anomaly detection method for face images including:

[0005] Acquire multiple key facial images of the same target user recorded within a preset time period;

[0006] A connected graph data with related relationships is constructed based on the multiple key facial images;

[0007] The connected graph data is input into the trained feature extraction model to perform aggregate feature extraction operations, and the first output result is obtained after data information is transferred and aggregated between all neighboring nodes in the connected graph data.

[0008] The first output result is input into an image classifier for classification calculation to obtain the second output result;

[0009] Based on the second output result, determine whether there are any abnormal face images among the multiple key face images.

[0010] In the anomaly detection method for face images described in this application embodiment, the connected graph data includes node data and edge data, wherein the node data is used to store user information and key face images, and the edge data is used to store the similarity values ​​between key face images corresponding to neighboring node data.

[0011] In the anomaly detection method for facial images described in this application embodiment, the step of acquiring multiple key facial images of the same target user recorded within a preset time period includes:

[0012] Acquire multiple original facial images of the same target user recorded within a preset time period;

[0013] The central region of each of the original face images is obtained, and each of the original face images is cropped according to the central region to obtain a cropped face image that retains the central region.

[0014] An angle correction operation is performed on the face image of the preserved central region to obtain the corrected key face image.

[0015] In the anomaly detection method for face images described in this application embodiment, the step of inputting the first output result into an image classifier for classification calculation to obtain a second output result includes:

[0016] The first output result is input into the trained node classification model, and the image classifier of the node classification model performs classification calculations to obtain a second output result for determining whether the face image stored in each node of the connected graph data is an abnormal face image.

[0017] In the anomaly detection method for face images described in this application embodiment, the classification calculation operation performed by the image classifier using the node classification model is based on a preset first calculation formula, wherein the first calculation formula is:

[0018] rv i =Θ cls (LeakyReLU(v. i ))

[0019] Among them, rv i This indicates whether the face images stored in each node of the connected graph data are abnormal face images; Θ cls This represents a fully connected affine transformation network; LeakyReLU represents the activation function; v .i This represents the data feature corresponding to any node data in the connected graph data, i.e., the first output result, v. i This represents any node data in the connected graph data.

[0020] In the anomaly detection method for face images described in this application embodiment, the step of inputting the first output result into an image classifier for classification calculation to obtain a second output result includes:

[0021] The first output result is input into the trained node classification model, and the first image classifier of the node classification model performs classification calculation operation, outputting a first probability value corresponding to each node data in the connected graph data. The first probability value is used to indicate whether the face image stored in each node data in the connected graph data is an abnormal face image.

[0022] The first output result is input into the trained graph classification model, and the second image classifier of the graph classification model performs classification calculation operation, outputting a second probability value corresponding to the first output result. The second probability value is used to indicate whether there is node data storing abnormal face images in the connected graph data.

[0023] The combined probability value is obtained by weighting the second probability value with the first probability value.

[0024] The second output result is obtained based on the difference between the comprehensive probability value and the preset threshold.

[0025] In the anomaly detection method for face images described in this application embodiment, the classification calculation operation performed by the second image classifier of the graph classification model is based on a preset second calculation formula, the second calculation formula being:

[0026] rg = W cls (LeakyReLU(sv g ))

[0027]

[0028] Wherein, rg represents whether there is node data in the connected graph data that stores abnormal face images; W cls Represents a fully connected layer transformation computation network; LeakyReLU represents the activation function; sv g This represents the fused feature data obtained by fusing the feature data corresponding to all nodes in the connected graph data;

[0029] G represents the fusion process of feature data; simi This represents the connected graph data.

[0030] Accordingly, another aspect of this application embodiment also provides an anomaly detection device for face images, the anomaly detection device for face images comprising:

[0031] The acquisition module is used to acquire multiple key facial images of the same target user within a preset time period;

[0032] A construction module is used to construct a connected graph data with related relationships based on the multiple key face images;

[0033] The extraction module is used to input the connected graph data into the trained feature extraction model to perform aggregate feature extraction operations, and obtain the first output result after data information transmission and aggregation calculation between all neighbor nodes in the connected graph data;

[0034] The calculation module is used to input the first output result into the image classifier for classification calculation to obtain the second output result;

[0035] The judgment module is used to determine whether there are abnormal face images among the multiple key face images based on the second output result.

[0036] Accordingly, in another aspect, this application also provides a storage medium storing a plurality of instructions adapted for loading by a processor to execute the face image anomaly detection method as described above.

[0037] Accordingly, another aspect of this application embodiment also provides a terminal device, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the face image anomaly detection method as described above.

[0038] This application provides a method, apparatus, medium, and device for anomaly detection of facial images. The method involves acquiring multiple key facial images of the same target user recorded within a preset time period; constructing a connected graph with related relationships based on the multiple key facial images; inputting the connected graph data into a feature extraction model to aggregate features and obtain a first output result after data information transfer and aggregation calculations among all neighboring nodes in the connected graph data; inputting the first output result into an image classifier for classification calculation to obtain a second output result; and determining whether abnormal facial images exist among the multiple key facial images based on the second output result. Using this application, it is possible to accurately screen, classify, and identify post-event facial recognition records that are part of a "re-photographed" black market facial recognition attack. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the anomaly detection method for face images provided in an embodiment of this application.

[0041] Figure 2 This is a schematic diagram of the connected graph data in the face image anomaly detection method provided in the embodiments of this application.

[0042] Figure 3 This is a schematic diagram of the structure of the anomaly detection device for human face images provided in the embodiments of this application.

[0043] Figure 4 Another structural schematic diagram of the face image anomaly detection device provided in the embodiments of this application.

[0044] Figure 5 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0046] It should be noted that the following is a brief introduction to the background of this solution:

[0047] This solution primarily addresses the technical issue of "the existence of individuals in the black market illegally obtaining customers' personal account information and photos to forge customer identities for facial recognition operations, attempting to bypass facial recognition systems and commit illegal activities." Understandably, facial recognition is currently required in many business scenarios, such as logging into bank apps, transferring funds, applying for loan products, and purchasing wealth management products. Simultaneously, some remote and online intelligent services also require facial recognition to verify customer identity. Facial recognition can verify that the customer currently applying for a service matches the person in the retained photo, preventing unauthorized applications. Furthermore, facial recognition can verify that the customer is a living individual, preventing the use of photos or pre-recorded videos for facial recognition during application processes.

[0048] However, in reality, many individuals involved in the black market illegally obtain personal account information and photos of bank customers. They then attempt to bypass the bank's facial recognition system using either the customer's photo directly or through photo-enhancing techniques. Alternatively, they may use masks, facial stickers, or other methods to impersonate the person in the photo. Once they succeed in breaching the bank's facial recognition system, they can potentially infiltrate the customer's bank account and steal funds.

[0049] To address the aforementioned technical problems, this application provides an anomaly detection method for facial images. Using this method, a "multiple facial recognition post-record similarity connected graph data" is constructed based on multiple facial recognition post-record data for each customer. This graph is then input into a pre-constructed image classifier. By exploiting the significant differences between the features of the facial recognition record "node" data used in black market attacks and the features of the majority of normal facial recognition record "node" data for the same customer, the output results can accurately screen, classify, and identify facial recognition post-records used in black market facial recognition attacks that are "re-photographed."

[0050] Please see Figure 1 , Figure 1 This is a flowchart illustrating the anomaly detection method for facial images provided in an embodiment of this application. The anomaly detection method for facial images is applied to a terminal device. Optionally, the terminal device can be a terminal or a server. Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, or in-vehicle terminal, but is not limited to these.

[0051] In one embodiment, the method may include the following steps:

[0052] Step 101: Obtain multiple key facial images of the same target user recorded within a preset time period.

[0053] Here, the key face image refers to the image obtained after preprocessing. Since the main purpose of this scheme is to extract facial information from face images, it is necessary to remove as much interference information as possible, such as background information. Therefore, in order to further improve the accuracy of face recognition, the center region of the face in the key video frame image can be selected. Specifically, multiple original face images recorded by the same target user within a preset time period can be obtained, and the center region of each original face image can be obtained. The original face images can be cropped according to the center region to obtain the cropped face image with the center region retained. An angle correction operation is performed on the face image with the center region retained to obtain the corrected key face image.

[0054] It should be explained that the central region of the original face image can be obtained by using the dlib algorithm to obtain the face and facial landmarks;

[0055] Then, based on the facial landmarks, obtain the top left corner of the smallest rectangle of the face: left_corner.x = min(landmark.x), left_corner.y = min(landmark.y);

[0056] The width is w = max(landmark.x)min(landmark.x);

[0057] The height is: h=max(landmark.y)min(landmark.y).

[0058] The center region of the face is obtained by scaling it according to a certain ratio.

[0059] The coordinates of the top left corner of the face center are:

[0060] face_center.x=left_corner.x+0.15*left_corner.x;

[0061] face_center.y=left_corner.y;

[0062] Width: face_center_w = 0.8w;

[0063] The height is: face_center_h = 0.8h.

[0064] In this embodiment, the key face image is an image that includes a central region and removes redundant images (e.g., an image that removes redundant background on both sides and below the face), which can further improve the detection efficiency while ensuring accuracy.

[0065] Step 102: Construct a connected graph data with related relationships based on the multiple key face images.

[0066] Among them, such as Figure 2 As shown, the connected graph data includes node data and edge data. The node data is used to store user information and key face images, while the edge data is used to store the similarity values ​​between the key face images corresponding to neighboring node data.

[0067] It should be noted that the similarity value in this scheme can be obtained by inputting the key face image to be calculated into a trained similarity value calculation model.

[0068] The similarity value calculation model used here is shown in the following formula:

[0069]

[0070]

[0071] In the formula above, i0 / i1 represent two paired key face images; CNN represents a multi-layer convolutional neural network used to extract data features from the central region; F i0 / F i1 This represents the data features extracted from the central region of two paired key face images; similarity dual This represents the final calculated similarity value between two key facial images.

[0072] The "similarity connected graph data of multiple face recognition records after the event" for a single customer is shown in the figure above. It includes two parts of data: "nodes" and "edges", as shown in the following formula:

[0073]

[0074]

[0075]

[0076] In the formula above, G simi The "similarity connected graph data of multiple face recognition records after the fact" represents a single customer; n represents the number of "node" data in the graph; m represents the number of "edge" data in the graph.

[0077] V nFor this similarity connected graph data, all node data, namely "post-facial recognition record data", each node data contains sub-data such as "customer's mobile phone model" P, "customer's business scenario code when performing facial recognition at the bank" S, and "similarity score between the customer's current facial recognition snapshot and the photo kept by the bank or public security" B. dual For this similarity connected graph data, all edge data contains a sub-data of "pairwise similarity values ​​between images after face recognition".

[0078] In the constructed "similarity connected graph data of multiple face recognition post-recordings" for a single customer, if there are n "node" data, then each "node" data will be a "neighbor node" data with the other n-1 "node" data, and the pairs of "neighbor node" data will be connected by "edge" data E. m Connected.

[0079] Based on the above process, the "similarity connectivity graph data of multiple face recognition post-recordings" for a single customer is constructed.

[0080] Step 103: Input the connected graph data into the trained feature extraction model to perform aggregate feature extraction operation, and obtain the first output result after data information is transferred and aggregated between all neighbor nodes in the connected graph data.

[0081] In this embodiment, after the "similarity connected graph data of multiple face recognition post-records" for a single customer is constructed, it can be input into the constructed "deep graph neural network model of graph point joint classification" to perform face recognition attack post-screening and classification calculation.

[0082] In the feature extraction model, the framework used here is the "Oversmoothing Suppression Dynamic Graph Attention Network" model, as shown in the following formula:

[0083]

[0084] In the formula of the model above, For: The "Deep Graph Neural Network" model uses a dynamic graph attention mechanism. i,i / a i,j In a single customer's "similarity connectivity graph data recorded after multiple facial recognition events" G simi In this process, data information is transferred and aggregated among all neighboring nodes along the "edge".

[0085] Among them, v i With v j Let represent any two pairs of "neighbor nodes"; n and m represent the number of "nodes" and "edges" in the array, respectively; Θs Let Θ be the affine transformation operation matrix. Here, the affine transformation operation matrix Θ needs to undergo an s-th power self-product operation. This self-product operation can effectively alleviate and suppress the "oversmoothing phenomenon" in the convolution operation of traditional graph neural networks. s is the oversmoothing suppression hyperparameter. .i This means that in G simi In this process, each "node" data is used to exchange and aggregate data information with all "neighboring nodes" along the "edge" direction, and the output result is calculated.

[0086] In the "Oversmoothing Inhibition Dynamic Graph Attention Network", the dynamic graph attention mechanism a i,j The specific calculation process is shown in the following formula:

[0087]

[0088] Attention mechanism in the above animated diagram i,j In the calculation formula, v i Indicates the current "node" v i Any pair of "neighbor node" data, and v k This represents the data of all pairs of n-1 neighboring nodes of the current "node"; E i,j / E i,k / represents the feature E used to connect the "edge" data between pairs of "neighboring nodes". T Both θ and θ represent affine transformation operations in fully connected layers; LeakyReLU is the activation function; and exp represents exponential operations with the natural number e as the base.

[0089] Step 104: Input the first output result into the image classifier for classification calculation to obtain the second output result.

[0090] The first output result is input into the trained node classification model, and the image classifier of the node classification model performs classification calculations to obtain the second output result used to determine whether the face image stored in each node of the connected graph data is an abnormal face image.

[0091] The image classifier using the node classification model performs classification calculations based on a preset first calculation formula:

[0092] rv i =Θ cls (LeakyReLU(v. i ))

[0093] Among them, rv i This indicates whether the face images stored in each node of the connected graph data are abnormal face images; Θ clsThis represents a fully connected affine transformation network; LeakyReLU represents the activation function; v .i This represents the data feature corresponding to any node data in the connected graph data, i.e., the first output result, v. i This represents any node data in the connected graph data.

[0094] Since neural network models cannot be 100% accurate, in order to reduce the error rate of the model, in another embodiment, the first output result is input into a trained node classification model, and the first image classifier of the node classification model performs classification calculation operations, outputting a first probability value corresponding to each node data in the connected graph data. The first probability value is used to indicate whether the face image stored in each node data in the connected graph data is an abnormal face image. The first output result is input into a trained graph classification model, and the second image classifier of the graph classification model performs classification calculation operations, outputting a second probability value corresponding to the first output result. The second probability value is used to indicate whether there is node data in the connected graph data that stores abnormal face images. The second probability value and the first probability value are weighted to obtain a comprehensive probability value. The second output result is obtained according to the difference between the comprehensive probability value and a preset threshold.

[0095] The classification calculation operation of the second image classifier in the graph classification model is performed based on a preset second calculation formula, which is:

[0096] rg = W cls (LeakyReLU(sv g ))

[0097]

[0098] Wherein, rg represents whether there is node data in the connected graph data that stores abnormal face images; W cls Represents a fully connected layer transformation computation network; LeakyReLU represents the activation function; sv g This represents the fused feature data obtained by fusing the feature data corresponding to all nodes in the connected graph data; G represents the fusion process of feature data; simi This represents the connected graph data.

[0099] In each customer's "similarity connectivity graph data of multiple face recognition post-records", the features of most or all of the normal face recognition record "node" data are similar or convergent among the multiple face recognition record "node" data of each customer, because they all come from the face recognition records of the same customer.

[0100] If a single customer's multiple facial recognition record "nodes" contain "nodes" indicating a black market attack on their bank account, the characteristics of these black market attack "nodes" will differ significantly from the characteristics of the majority of the customer's other legitimate facial recognition record "nodes." Therefore, when the graph classification model performs "connected subgraph screening classification," it can accurately determine whether a single customer's "multiple facial recognition post-record similarity connected graph data" contains "nodes" indicating a black market attack. Similarly, when the node classification model performs "fine-grained subgraph node screening classification," it can accurately determine whether each facial recognition record "node" in a single customer's "multiple facial recognition post-record similarity connected graph data" is a "node" indicating a black market attack.

[0101] By fusing the outputs of both node classification and graph classification models, a more reasonable second output result can be derived to determine whether abnormal face images exist among the multiple key face images. This multi-task modeling approach of graph-node joint learning can mutually enhance the screening and classification effect of face recognition attacks, accurately screening, classifying, and identifying which face recognition record "node" data are "re-photographed" black market face recognition attack records.

[0102] Step 105: Determine whether there are any abnormal face images among the multiple key face images based on the second output result.

[0103] By constructing a "multiple face recognition post-record similarity connected graph data" based on multiple face recognition post-record data for each customer, and then inputting it into a pre-constructed image classifier, the characteristics of the face recognition record "node" data attacked by the black market industry are significantly different from the characteristics of the "node" data of most of the other normal face recognition records of the same customer. Based on the second output result, the face recognition post-record data of the black market industry attack can be accurately screened, classified, and identified as "re-photographed" face recognition attack data.

[0104] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0105] In practice, this application is not limited by the execution order of the described steps. Without causing conflicts, some steps may be performed in other orders or simultaneously.

[0106] As can be seen from the above, the face image anomaly detection method provided in this application acquires multiple key face images of the same target user within a preset time period; constructs connected graph data with correlation based on the multiple key face images; inputs the connected graph data into a trained feature extraction model for aggregate feature extraction operation to obtain a first output result after data information transmission and aggregation calculation between all neighboring nodes in the connected graph data; inputs the first output result into an image classifier for classification calculation operation to obtain a second output result; and determines whether there are abnormal face images among the multiple key face images based on the second output result. The anomaly detection method for facial images provided in this application constructs a "multiple facial recognition post-record similarity connected graph data" based on multiple facial recognition post-record data for each customer. This data is then input into a pre-constructed image classifier. By utilizing the significant differences between the features of the facial recognition record "node" data used in black market attacks and the features of the "node" data of most other normal facial recognition records for the same customer, the output results can accurately screen, classify, and identify facial recognition post-records used in black market facial recognition attacks that are "re-shot".

[0107] This application also provides an anomaly detection device for facial images, which can be integrated into a terminal device.

[0108] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of the face image anomaly detection device provided in an embodiment of this application. The face image anomaly detection device 30 may include:

[0109] The acquisition module 31 is used to acquire multiple key facial images of the same target user within a preset time period;

[0110] The construction module 32 is used to construct connected graph data with related relationships based on the multiple key face images;

[0111] Extraction module 33 is used to input the connected graph data into the trained feature extraction model to perform aggregate feature extraction operation, and obtain the first output result after data information transmission and aggregation calculation between all neighbor nodes in the connected graph data;

[0112] The calculation module 34 is used to input the first output result into the image classifier for classification calculation to obtain the second output result;

[0113] The judgment module 35 is used to determine whether there are abnormal face images among the multiple key face images based on the second output result.

[0114] In some embodiments, the connected graph data includes node data and edge data, wherein the node data is used to store user information and key face images, and the edge data is used to store similarity values ​​between key face images corresponding to neighboring node data.

[0115] In some embodiments, the acquisition module 31 is used to acquire multiple original face images recorded by the same target user within a preset time period; acquire the central region of each original face image respectively; crop each original face image according to the central region to obtain a cropped face image retaining the central region; and perform an angle correction operation on the face image retaining the central region to obtain a corrected key face image.

[0116] In some embodiments, the calculation module 34 is used to input the first output result into a trained node classification model, and perform classification calculation operations through the image classifier of the node classification model to obtain a second output result for determining whether the face image stored in each node of the connected graph data is an abnormal face image.

[0117] In some embodiments, the classification calculation operation performed by the image classifier through the node classification model is based on a preset first calculation formula, wherein the first calculation formula is:

[0118] rv i =Θ cls (LeakyReLU(v. i ))

[0119] Among them, rv i This indicates whether the face images stored in each node of the connected graph data are abnormal face images; Θ cls This represents a fully connected affine transformation network; LeakyReLU represents the activation function; v .i This represents the data feature corresponding to any node data in the connected graph data, i.e., the first output result, v. i This represents any node data in the connected graph data.

[0120] In some embodiments, the calculation module 34 is configured to: input the first output result into a trained node classification model; perform classification calculations using a first image classifier of the node classification model; output a first probability value corresponding to each node data in the connected graph data; the first probability value indicates whether the face image stored in each node data in the connected graph data is an abnormal face image; input the first output result into a trained graph classification model; perform classification calculations using a second image classifier of the graph classification model; output a second probability value corresponding to the first output result; the second probability value indicates whether there is node data storing abnormal face images in the connected graph data; perform a weighted calculation of the second probability value and the first probability value to obtain a comprehensive probability value; and obtain a second output result based on the difference between the comprehensive probability value and a preset threshold.

[0121] In some embodiments, the classification calculation operation performed by the second image classifier of the graph classification model is based on a preset second calculation formula, wherein the second calculation formula is:

[0122] rg = W cls (LeakyReLU(sv g ))

[0123]

[0124] Wherein, rg represents whether there is node data in the connected graph data that stores abnormal face images; W cls Represents a fully connected layer transformation computation network; LeakyReLU represents the activation function; sv g This represents the fused feature data obtained by fusing the feature data corresponding to all nodes in the connected graph data; G represents the fusion process of feature data; simi This represents the connected graph data.

[0125] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities.

[0126] As can be seen from the above, the face image anomaly detection device 30 provided in this application embodiment includes an acquisition module 31 for acquiring multiple key face images recorded by the same target user within a preset time period; a construction module 32 for constructing connected graph data with correlation based on the multiple key face images; an extraction module 33 for inputting the connected graph data into a trained feature extraction model to perform aggregate feature extraction operations, obtaining a first output result after data information transmission and aggregation calculation among all neighboring nodes in the connected graph data; a calculation module 34 for inputting the first output result into an image classifier to perform classification calculation operations, obtaining a second output result; and a judgment module 35 for judging whether there are abnormal face images among the multiple key face images based on the second output result.

[0127] Please see Figure 4 , Figure 4 This is another schematic diagram of the face image anomaly detection device provided in this application embodiment. The face image anomaly detection device 30 includes a memory 120, one or more processors 180, and one or more application programs, wherein the one or more application programs are stored in the memory 120 and configured to be executed by the processors 180; the processors 180 may include an acquisition module 31, a construction module 32, an extraction module 33, a calculation module 34, and a judgment module 35. For example, the structure and connection relationship of the above components can be as follows:

[0128] Memory 120 can be used to store applications and data. The applications stored in memory 120 contain executable code. Applications can be organized into various functional modules. Processor 180 executes various functional applications and anomaly detection of facial images by running the applications stored in memory 120. Furthermore, memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory 120 may also include a memory controller to provide processor 180 with access to memory 120.

[0129] The processor 180 is the control center of the device, connecting various parts of the terminal through various interfaces and lines. It performs various functions and processes data by running or executing applications stored in the memory 120 and calling data stored in the memory 120, thereby providing overall monitoring of the device. Optionally, the processor 180 may include one or more processing cores; preferably, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications.

[0130] Specifically, in this embodiment, the processor 180 loads the executable code corresponding to the processes of one or more applications into the memory 120 according to the following instructions, and the processor 180 runs the applications stored in the memory 120 to achieve various functions:

[0131] The acquisition command is used to acquire multiple key facial images of the same target user within a preset time period;

[0132] Construction instructions are used to construct a connected graph data with related relationships based on the multiple key face images;

[0133] The extraction instruction is used to input the connected graph data into the trained feature extraction model to perform aggregate feature extraction operations, and obtain the first output result after data information is transferred and aggregated between all neighbor nodes in the connected graph data;

[0134] The calculation instruction is used to input the first output result into the image classifier for classification calculation to obtain the second output result;

[0135] The judgment instruction is used to determine whether there are abnormal face images among the multiple key face images based on the second output result.

[0136] This application also provides a terminal device. The terminal device may be a server, smartphone, computer, tablet computer, or other similar device.

[0137] Please see Figure 5 , Figure 5 A schematic diagram of the structure of a terminal device provided in an embodiment of this application is shown. This terminal device can be used to implement the anomaly detection method for face images provided in the above embodiments. The terminal device 1200 can be a smartphone or a tablet computer.

[0138] like Figure 5 As shown, the terminal device 1200 may include an RF (Radio Frequency) circuit 110, a memory 120 including one or more (only one is shown in the figure) computer-readable storage media, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a transmission module 170, a processor 180 including one or more (only one is shown in the figure) processing cores, and a power supply 190, etc. Those skilled in the art will understand that... Figure 5 The structure of the terminal device 1200 shown does not constitute a limitation on the terminal device 1200, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0139] RF circuit 110 is used to receive and transmit electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, thereby enabling communication with communication networks or other devices. RF circuit 110 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 110 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks.

[0140] The memory 120 can be used to store software programs and modules, such as the program instructions / modules corresponding to the face image anomaly detection method in the above embodiment. The processor 180 executes various functional applications and face image anomaly detection by running the software programs and modules stored in the memory 120. It can automatically select a vibration reminder mode to update data according to the current scene of the terminal device, ensuring that it is not disturbed in scenarios such as meetings, while also ensuring that users can sense incoming calls, thus improving the intelligence of the terminal device. The memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 120 may further include memory remotely located relative to the processor 180, and these remote memories can be connected to the terminal device 1200 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0141] Input unit 130 can be used to receive input numerical or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, input unit 130 may include touch-sensitive surface 131 and other input devices 132. Touch-sensitive surface 131, also known as a touch display screen or touchpad, can collect user touch operations on or near it (such as user operations using fingers, styluses, or any suitable object or accessory on or near touch-sensitive surface 131), and drive corresponding connection devices according to a pre-set program. Optionally, touch-sensitive surface 131 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to processor 180, and can receive and execute commands from processor 180. In addition, the touch-sensitive surface 131 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. Besides the touch-sensitive surface 131, the input unit 130 may also include other input devices 132. Specifically, other input devices 132 may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0142] Display unit 140 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of terminal device 1200. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 140 may include display panel 141, optionally configured as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, touch-sensitive surface 131 may cover display panel 141. When touch-sensitive surface 131 detects a touch operation on or near it, it transmits the information to processor 180 to determine the type of touch event. Subsequently, processor 180 provides corresponding visual output on display panel 141 according to the type of touch event. Although in Figure 5 In this embodiment, the touch-sensitive surface 131 and the display panel 141 are implemented as two separate components to realize input and output functions. However, in some embodiments, the touch-sensitive surface 131 and the display panel 141 can be integrated to realize input and output functions.

[0143] The terminal device 1200 may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 141 according to the ambient light level, and the proximity sensor can turn off the display panel 141 and / or backlight when the terminal device 1200 is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that the terminal device 1200 may also be configured with, such as a gyroscope, barometer, hygrometer, thermometer, and infrared sensor, will not be described in detail here.

[0144] Audio circuitry 160, speaker 161, and microphone 162 provide an audio interface between the user and terminal device 1200. Audio circuitry 160 converts received audio data into electrical signals, which are then transmitted to speaker 161, where they are converted into sound signals for output. Conversely, microphone 162 converts collected sound signals into electrical signals, which are received by audio circuitry 160, converted back into audio data, and then processed by processor 180 before being transmitted via RF circuitry 110 to, for example, another terminal, or output to memory 120 for further processing. Audio circuitry 160 may also include an earphone jack to facilitate communication between peripheral headphones and terminal device 1200.

[0145] Terminal device 1200, through transmission module 170 (e.g., Wi-Fi module), can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 5 The transmission module 170 is shown, but it is understood that it is not a necessary component of the terminal device 1200 and can be omitted as needed without changing the nature of the invention.

[0146] The processor 180 is the control center of the terminal device 1200. It connects to various parts of the mobile phone via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 120, and by calling data stored in the memory 120, it performs various functions of the terminal device 1200 and processes data, thereby providing overall monitoring of the mobile phone. Optionally, the processor 180 may include one or more processing cores; in some embodiments, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 180.

[0147] The terminal device 1200 also includes a power supply 190 that supplies power to the various components. In some embodiments, the power supply can be logically connected to the processor 180 through a power management system, thereby enabling functions such as discharge management and power consumption management through the power management system. The power supply 190 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0148] Although not shown, the terminal device 1200 may also include a camera (such as a front-facing camera and a rear-facing camera), a Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit 140 of the terminal device 1200 is a touch screen display, and the terminal device 1200 also includes a memory 120 and one or more programs, one or more of which are stored in the memory 120 and configured to be executed by one or more processors 180. One or more programs contain instructions for performing the following operations:

[0149] The acquisition command is used to acquire multiple key facial images of the same target user within a preset time period;

[0150] Construction instructions are used to construct a connected graph data with related relationships based on the multiple key face images;

[0151] The extraction instruction is used to input the connected graph data into the trained feature extraction model to perform aggregate feature extraction operations, and obtain the first output result after data information is transferred and aggregated between all neighbor nodes in the connected graph data;

[0152] The calculation instruction is used to input the first output result into the image classifier for classification calculation to obtain the second output result;

[0153] The judgment instruction is used to determine whether there are abnormal face images among the multiple key face images based on the second output result.

[0154] This application also provides a storage medium storing a computer program. When the computer program is run on a computer, the computer executes the anomaly detection method for face images described in any of the above embodiments.

[0155] It should be noted that, regarding the anomaly detection method for face images described in this application, those skilled in the art will understand that all or part of the process of implementing the anomaly detection method for face images described in the embodiments of this application can be accomplished by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium, such as in the memory of a terminal device, and executed by at least one processor within the terminal device. During execution, it can include the process of the embodiments of the anomaly detection method for face images. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), etc.

[0156] For the anomaly detection device for facial images described in this application embodiment, its functional modules can be integrated into a single processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0157] The above provides a detailed description of the anomaly detection method, apparatus, medium, and device for facial images provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas; furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for anomaly detection in facial images, characterized in that, include: Acquire multiple key facial images of the same target user recorded within a preset time period; A connected graph data with related relationships is constructed based on the multiple key facial images; The connected graph data is input into the trained feature extraction model to perform aggregate feature extraction operations, and the first output result is obtained after data information is transferred and aggregated between all neighboring nodes in the connected graph data. The first output result is input into the trained node classification model, and the first image classifier of the node classification model performs classification calculation operation, outputting a first probability value corresponding to each node data in the connected graph data. The first probability value is used to indicate whether the face image stored in each node data in the connected graph data is an abnormal face image. The first output result is input into the trained graph classification model, and the second image classifier of the graph classification model performs classification calculations, outputting a second probability value corresponding to the first output result. The second probability value is used to indicate whether there are node data in the connected graph data that store abnormal face images. The graph classification model fuses the feature data corresponding to all node data in the connected graph data through a long short-term memory network to obtain fused feature data, and performs classification calculations based on the fused feature data. The combined probability value is obtained by weighting the second probability value with the first probability value. The second output result is obtained based on the difference between the comprehensive probability value and the preset threshold. Based on the second output result, determine whether there are any abnormal face images among the multiple key face images.

2. The anomaly detection method as described in claim 1, characterized in that, The connected graph data includes node data and edge data. The node data is used to store user information and key facial images, and the edge data is used to store the similarity values ​​between key facial images corresponding to neighboring node data.

3. The anomaly detection method as described in claim 1, characterized in that, The acquisition of multiple key facial images of the same target user recorded within a preset time period includes: Acquire multiple original facial images of the same target user recorded within a preset time period; The central region of each of the original face images is obtained, and each of the original face images is cropped according to the central region to obtain a cropped face image that retains the central region. An angle correction operation is performed on the face image of the preserved central region to obtain the corrected key face image.

4. The anomaly detection method as described in claim 1, characterized in that, The step of inputting the first output result into an image classifier for classification calculation to obtain a second output result includes: The first output result is input into the trained node classification model, and the image classifier of the node classification model performs classification calculations to obtain a second output result for determining whether the face image stored in each node of the connected graph data is an abnormal face image.

5. The anomaly detection method as described in claim 4, characterized in that, The classification calculation operation performed by the image classifier using the node classification model is based on a preset first calculation formula, wherein the first calculation formula is: in, This indicates whether the face images stored in each node of the connected graph data are abnormal face images; This represents a fully connected affine transformation network. Indicates the activation function; This represents the data characteristics corresponding to any node data in the connected graph data, i.e., the first output result. This represents any node data in the connected graph data.

6. The anomaly detection method as described in claim 1, characterized in that, The classification calculation operation performed by the second image classifier of the graph classification model is based on a preset second calculation formula, wherein the second calculation formula is: in, This indicates whether there are node data in the connected graph data that store abnormal face images; This represents a fully connected layer transform computation network; Indicates the activation function; This represents the fused feature data obtained by fusing the feature data corresponding to all nodes in the connected graph data; This represents the fusion process of the fusion feature data; This represents the connected graph data.

7. An anomaly detection device for human face images, characterized in that, The anomaly detection device for the face image includes: The acquisition module is used to acquire multiple key facial images of the same target user within a preset time period; A construction module is used to construct a connected graph data with related relationships based on the multiple key face images; The extraction module is used to input the connected graph data into the trained feature extraction model to perform aggregate feature extraction operations, and obtain the first output result after data information transmission and aggregation calculation between all neighbor nodes in the connected graph data; The calculation module is used to input the first output result into a trained node classification model, perform classification calculations through the first image classifier of the node classification model, and output a first probability value corresponding to each node data in the connected graph data. The first probability value is used to indicate whether the face image stored in each node data in the connected graph data is an abnormal face image. The module also inputs the first output result into a trained graph classification model, performs classification calculations through the second image classifier of the graph classification model, and outputs a second probability value corresponding to the first output result. The second probability value is used to indicate whether there is node data in the connected graph data that stores abnormal face images. A weighted average of the second probability value and the first probability value is calculated to obtain a comprehensive probability value. Finally, a second output result is obtained based on the difference between the comprehensive probability value and a preset threshold. The judgment module is used to determine whether there are abnormal face images among the multiple key face images based on the second output result.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the anomaly detection method for a face image as described in any one of claims 1 to 6.

9. A terminal device, characterized in that, The device includes a processor and a memory, the memory storing multiple instructions, and the processor loading the instructions to execute the anomaly detection method for a face image as described in any one of claims 1 to 6.

Images