Abnormal recognition method and device, equipment, storage medium and program product

Abstract

The invention provides an anomaly recognition method and device, equipment, a storage medium and a program product. The method comprises the steps of collecting a target face image under an illumination condition corresponding to an initial color light sequence; for each collected target face image, dividing the face area of the target face image to obtain at least two subareas, determining the light color of each subarea, and determining the light color of the target face image according to the light colors of the at least two subareas; determining a target color light sequence according to the light color of the collected target face image; and according to the matching condition of the initial color light sequence and the target color light sequence, determining whether the face recognition process is abnormal or not. According to the embodiment of the invention, the abnormity in the face recognition process can be effectively recognized.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an abnormality identification method and device, equipment, storage medium, and program product. Background Art

[0002] At present, face recognition technology is developing rapidly and is widely used in many fields. It is an important means of information security in these fields. It can be seen that the accuracy of face recognition is an important guarantee of information security. In the related applications of face recognition, how to determine whether there is an abnormality in the face recognition process is a hot research issue. Summary of the invention

[0003] The present invention provides an abnormality identification method and apparatus, equipment, storage medium, and program product.

[0004] In a first aspect, the present disclosure provides an abnormality identification method, the method comprising:

[0005] Acquire the target face image under the illumination conditions corresponding to the initial color light sequence;

[0006] For each collected target facial image, dividing the facial area of ​​the target facial image into at least two partitions, determining the light color of each partition, and determining the light color of the target facial image based on the light colors of the at least two partitions;

[0007] Determine the target color light sequence according to the light color of the collected target face image;

[0008] According to the matching condition between the initial color light sequence and the target color light sequence, it is determined whether there is an abnormality in the face recognition process.

[0009] In a second aspect, the present disclosure provides an abnormality identification device, the device comprising:

[0010] An acquisition module, used for acquiring a target face image under the illumination conditions corresponding to the initial color light sequence;

[0011] A processing module, for dividing the face area of ​​each target face image collected into at least two partitions, and determining the light color of each partition, and determining the light color of the target face image according to the light colors of the at least two partitions;

[0012] A determination module, used to determine a target color light sequence according to the light color of the collected target face image;

[0013] The judgment module is used to determine whether there is an abnormality in the face recognition process according to the matching situation of the initial color light sequence and the target color light sequence.

[0014] In a third aspect, the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the above-mentioned abnormality identification method.

[0015] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned abnormality identification method when executed by a processor.

[0016] In a fifth aspect, the present disclosure provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned abnormality identification method.

[0017] The abnormality recognition method provided by the embodiment of the present disclosure collects the target face image under the lighting conditions of the initial color light sequence, divides the face area of ​​each target face image collected to obtain at least two partitions, determines the light color of each partition respectively, and then determines the light color of the entire target face image based on the light colors of the at least two partitions. This method can overcome the interference of non-face areas on light color recognition, and can perform light color recognition in combination with the reflective characteristics of each partition, thereby improving the accuracy of light color recognition, and further improving the accuracy of the target color light sequence determined according to each target face image collected, thereby ensuring the matching accuracy of the initial color light sequence and the target color light sequence, so as to effectively determine whether there is an abnormality in the face recognition process.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram of an application scenario of an abnormality identification method and device provided in an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of an abnormality identification method provided by an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of an optional light color design provided in an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of facial key point distribution provided by an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of a human face effective area provided in an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of each partition in a face area provided by an embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of a mask image and corresponding partitions provided in an embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of the structure of a convolutional neural network model provided in an embodiment of the present disclosure;

[0028] Fig. 9 A schematic diagram of an exemplary processing process of abnormality identification provided by an embodiment of the present disclosure;

[0029] Fig.10 A block diagram of an abnormality identification device provided in an embodiment of the present disclosure;

[0030] Fig.11 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0032] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0033] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0034] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0035] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0036] In the technical solution disclosed in this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution complies with relevant national laws and regulations (for example, the "Information Security Technology Personal Information Security Specification", etc.). For example: corresponding prescribed measures are taken for personal information access control; prescribed restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; when using personal information, clear identity orientation is eliminated to avoid precise positioning of specific individuals.

[0037] The inventors discovered during the process of implementing the present disclosure that:

[0038] In face recognition applications, it is particularly important to ensure the authenticity of face data. Common attacks on face recognition systems include screen reshoots, deep forgeries, three-dimensional (3D) molds, adversarial attacks, and link hijacking attacks.

[0039] As for link hijacking attacks, there is currently no particularly effective means to prevent them, given the great uncertainty of the time and nodes at which they occur. The data uploaded to the system by link hijacking attacks is the real data of the target user, and the identity characteristics are completely consistent with those of the target user. How to effectively identify link hijacking attacks is the difficulty in intercepting link hijacking attacks.

[0040] As for link hijacking attacks, there is currently no particularly effective means to prevent them, given the great uncertainty of the time and nodes at which they occur. Link hijacking attacks generally include three stages. First, the attacker maliciously collects data from the target user. The collected data mainly includes the target user's common facial expressions such as blinking, opening the mouth, nodding and shaking the head. This collection is usually completed without the user's knowledge. Subsequently, the attacker will simulate the target user for identity verification, and during the verification process, hijack any link in the data transmission link. Usually, the hijacking point is the camera data, that is, the real-time data stream of the camera is replaced with maliciously collected data. Finally, the maliciously collected data successfully passes the identity verification and then commits fraud or crime. Among them, since the data uploaded to the system is the real data of the target user, its identity characteristics are completely consistent with the reserved characteristics. Therefore, the security algorithm of such data system cannot intercept it at all, which is also the difficulty of intercepting link hijacking attacks.

[0041] In related technologies, the core idea of ​​intercepting link hijacking attacks is that the system randomly generates a key each time it verifies the identity. The key needs to have a large random space and be sensed by the camera. A common solution is that the system randomly initializes a series of actions, such as opening the mouth, nodding, etc., and then checks in real time whether the user has completed the action as required. The biggest drawback of this solution is that the random space is not large enough, because if the actions are designed to be too many and too complicated, it may be difficult for real users to complete them as required.

[0042] Based on this, an abnormality identification method is designed in an embodiment of the present disclosure.

[0043] It should be noted that the collection of facial images described in the following embodiments is performed after obtaining authorization and consent.

[0044] Figure 1 The application scenario diagram of the abnormality identification method and device provided by the embodiment of the present disclosure is schematically shown.

[0045] like Figure 1 As shown, the application scenario of the embodiment of the present disclosure may include a terminal device 101, a network 103 and a server 102. The network 103 is used to provide a medium for a communication link between the terminal device 101 and the server 102. The network 103 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0046] The user can use the terminal device 101 to interact with the server 102 through the network 103 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0047] The terminal device 101 may be any electronic device having a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, and the like.

[0048] The server 102 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal device 101. The background management server may analyze and process received data such as user requests, and feed back processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0049] It should be noted that the abnormality identification method and apparatus provided in the embodiments of the present disclosure may be executed by the server 102. Accordingly, the abnormality identification method and apparatus provided in the embodiments of the present disclosure may be arranged in the server 102. The abnormality identification method and apparatus provided in the embodiments of the present disclosure may also be executed by a server or server cluster that is different from the server 102 and that can communicate with the terminal device 101 and / or the server 102. Accordingly, the abnormality identification method and apparatus provided in the embodiments of the present disclosure may also be arranged in a server or server cluster that is different from the server 102 and that can communicate with the terminal device 101 and / or the server 102.

[0050] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0051] Figure 2 A flowchart of an abnormality identification method provided by an embodiment of the present disclosure. Figure 2 , the method comprising:

[0052] Step 201 : capturing a target face image under lighting conditions corresponding to an initial color light sequence.

[0053] In some embodiments, the initial colored light sequence is randomly generated.

[0054] The initial color light sequence includes S light colors, where S is an integer greater than 1. Each light color corresponds to a certain duration, and the durations corresponding to different light colors may be the same or different.

[0055] In an exemplary embodiment, a random initial color light sequence C5 is generated, and C5 can specify a sequence of light color and duration, such as C5=(1 250 , 2 250 , 3 250 , 4 250 ,5 250) means that the phone screen will then display light color 1 for 250ms, light color 2 for 250ms...

[0056] In the exemplary embodiment, it is assumed that the system design uses 6 light colors, such as Figure 3 The figure shows a schematic diagram of the designed 6 light colors. Assume that the system randomly selects 5 light colors from the 6 light colors to prevent link hijacking attacks, and the initial color light sequence is C5 = (4-2-1-3-6). The numbers represent the light color sequence numbers. The duration of each light color here is 250ms. For the working condition with a frame rate of 25 frames, the system can collect 10 frames of valid data during the duration of each light color. Under ideal conditions, the data sequence collected by the system should be [light color 4 / 10 frame, light color 2 / 10 frame, light color 1 / 10 frame, light color 3 / 10 frame, light color 6 / 10 frame]. The color light sequence verification method proposed in the embodiment of the present disclosure has a probability of correct random simulation of light color coupling of: 1 / 6**5≈0.0001, which is highly secure. With the initial color light sequence C5, the system drives the screen to display colorful light on the one hand, and drives the front camera to collect face data on the other hand.

[0057] In some embodiments, collecting a target face image under the illumination conditions corresponding to the initial color light sequence includes:

[0058] During the continuous illumination time of each light color of the initial color light sequence, collecting a face image;

[0059] In each case where a face image is collected, determining whether the quality of the collected face image meets a preset condition;

[0060] If the preset condition is met, it is determined that the collected face image is used as the target face image.

[0061] By judging in real time whether the quality of the collected face image meets the preset conditions, a target face image that meets the preset conditions is obtained, thereby ensuring the quality of the face image used for light and color recognition and improving the accuracy of light and color recognition.

[0062] In some embodiments, the method further comprises:

[0063] If the quality of the collected face image does not meet the preset condition, discard the face image and extend the illumination time of the light color corresponding to the collection time of the face image;

[0064] The continuous illumination time of the corresponding light color is updated according to the extended illumination time, and the facial image acquisition and subsequent steps are repeated.

[0065] In this embodiment, while ensuring the quality of the face image, the time slice of the problematic light color is backtracked and extended to avoid the loss of a certain light color in the detection result.

[0066] In an exemplary embodiment, a synchronous dual thread is started, one thread drives the mobile phone screen (or other devices capable of providing lighting conditions) to provide lighting conditions according to the initial color light sequence, and the other thread drives the front camera of the mobile phone to collect face data, and the collected data frames are placed in a double-ended queue Q. In the process of face data collection, face images are stored in the double-ended queue Q in real time, and face images are read from the double-ended queue Q in real time to determine whether the quality of the read face images meets the preset conditions. When the quality of the face image does not meet the preset conditions, the user is reminded, and the duration of the current light color is added, and the face images that do not meet the preset conditions are discarded.

[0067] In an exemplary embodiment, the illumination time of the light color is extended each time by one time slice unit (eg, 25 ms).

[0068] In some embodiments, the determining whether the quality of the collected facial image meets a preset condition includes:

[0069] For the facial key points of the collected face image, the extraction result is obtained;

[0070] Determining whether the extraction result contains facial key points;

[0071] If the extraction result does not include facial key points, determining that the quality of the facial image does not meet a preset condition;

[0072] If the extraction result includes facial key points, it is determined whether the quality of the facial image meets a preset condition based on the facial key points.

[0073] By judging whether the facial image contains facial key points, if the facial key points are not included, it means that the facial image does not contain a face, and it can be directly judged that the quality of the facial image does not meet the preset conditions. If the facial key points are included, the quality of the facial image can be further judged based on the facial key points.

[0074] In an exemplary embodiment, Mediapipe or other facial key point detection models are used to detect facial key points on a facial image. Figure 4The figure shows the distribution diagram of facial key points extracted by Mediapipe. Using Mediapipe or other facial key point detection models to detect facial key points, we can get the coordinates of 468 key points in the face of the input face image. Key points can be indexed by serial number. The numbers in the key point sequence mentioned later represent the serial number of the key point, such as 62 represents the 62nd key point. Each key point coordinate usually contains at least one point (x, y) coordinate, where x represents the horizontal offset in the image pixel coordinate system, and y represents the vertical offset. In the pixel coordinate system of the image, the origin is the upper left corner of the image, so the values ​​of x and y are non-negative, that is, x and y ≥ 0. As a facial key point detection model, Mediapipe has the advantages of stability, fast speed, low memory consumption, and rich key points.

[0075] In an exemplary embodiment, assuming that it is detected that the current frame does not contain facial key points, the user is reminded that “the face has disappeared”.

[0076] In some embodiments, judging whether the quality of the face image meets a preset condition according to the facial key points includes:

[0077] Determining a quality judgment parameter according to the facial key points, the quality judgment parameter including one or more of the following: a proportion of the facial area in the facial image, a positional relationship between the facial area in the facial image and a facial effective area in the facial image;

[0078] According to the quality judgment parameter, it is judged whether the quality of the face image meets a preset condition.

[0079] The facial area ratio can be used to determine the distance of the face from the camera, thus avoiding quality problems such as unclear facial images caused by the face being too far or too close.

[0080] The positional relationship between the facial area and the effective face area is used to determine whether the face is within the effective face area, thereby avoiding quality problems such as incomplete facial images caused by the face exceeding the boundary of the effective face area.

[0081] In some embodiments, judging whether the quality of the face image meets a preset condition according to the quality judgment parameter includes:

[0082] Determine whether the proportion value in the quality judgment parameter is greater than or equal to a first preset threshold and less than or equal to a second preset threshold, and obtain a first judgment result, wherein the first preset threshold is less than the second preset threshold;

[0083] Determine whether the positional relationship in the quality judgment parameter satisfies that the face area of ​​the face image does not exceed the boundary of the face valid area, and obtain a second judgment result;

[0084] When the first judgment result is yes, and / or the second judgment result is yes, it is determined that the quality of the facial image meets a preset condition.

[0085] It should be noted that one or more of the distance of the facial image and the integrity of the facial image can be used to judge the quality of the facial image as needed.

[0086] In the exemplary embodiment, the collected face data is assumed to be image I, and face key point detection is performed using Mediapipe. The model will output the key point information of all faces in image I. Each face contains 468 key points, and each key point has pixel coordinates (x, y) to indicate the position of the key point in the image. Using this key point information, the following processing is performed:

[0087] Face disappearance judgment: If the result returned by Mediapipe is an empty list, it means that there is no face in the current image, that is, the face has disappeared. At this time, it is necessary to remind the user to keep his face facing the camera and extend the time slice of the current light color.

[0088] Face distance judgment: According to the facial key point information in image I, find out the left, top, right and bottom boundary information of the face, so as to obtain the width information of the face, calculate the ratio of the face width to the image width and compare it with the first preset threshold β1. If it is less than β1, remind the user that the face is too far away and extend the current glare time slice. Because when the face is too far away from the screen, the color light of the screen cannot effectively illuminate the face, which will cause the glare to be invalid. At the same time, if it is greater than the second preset threshold β2, it reminds the user that the face is too close, because a face that is too close is likely to be incomplete and there is a risk of fraud. The first preset threshold is generally set according to the security level, for example, β1=0.4, β2=0.95.

[0089] Face integrity judgment: Assume that a face has a valid area, such as Figure 5 As shown, the face boundary information is obtained according to the face key points in image I, and it is judged whether it exceeds Figure 5 If the effective area is exceeded, the user will be reminded that "the face is beyond the boundary box" and the current glare time slice will be extended. The effective area is usually divided so that the distance from the image boundary cannot be less than 0.1 times the image side length.

[0090] Step 202, for the collected target facial image, divide the facial area of ​​the target facial image into at least two partitions, determine the light color corresponding to each partition, and determine the light color of the target facial image according to the light colors corresponding to the at least two partitions.

[0091] In some embodiments, the dividing the facial area of ​​the facial image into at least two partitions includes:

[0092] Extracting key points in the face area of ​​the face image, and obtaining the sequence number and position coordinates of each key point;

[0093] Determine a mask image of a preset block area in the face area according to the sequence number and position coordinates of each key point;

[0094] The face area of ​​the face image is divided using a mask image of a preset block area to obtain at least two partitions.

[0095] In the exemplary embodiment, see Figure 6 The figure shows a schematic diagram of each partition in the face area, that is, a total of 5 mask images, namely, the left atrium, the upper atrium, the middle atrium, the lower atrium and the right atrium. These 5 preset block areas, each preset block area corresponds to a mask image. Figure 7 Shown is a schematic diagram of the mask image and the corresponding partitions.

[0096] In some embodiments, determining the mask image of each preset block area in the face area according to the sequence number and position coordinates of each key point includes:

[0097] For each preset block area, obtaining a key point sequence number set corresponding to the preset block area in the face area, wherein the key point sequence number set includes sequence numbers of key points required to enclose the preset block area;

[0098] Obtaining the position coordinates corresponding to the sequence numbers in the key point sequence number set, to obtain a position coordinate set corresponding to the key point sequence number set;

[0099] Two key points in the position coordinate set that are adjacent to each other are connected to obtain a closed polygonal area, and a mask image of the preset block area is determined based on the closed polygonal area.

[0100] In the exemplary embodiment, the key point information output by Mediapipe is used to perform mask partitioning on the face area. Figure 6 As shown, it is divided into 5 regions: D1, D2, D3, D4, and D5. The D1 region is connected in sequence by the key point sequence [62, 57, 214, 138, 172, 136, 150, 149, 176, 148, 152, 377, 400, 378, 379, 365, 397, 367, 434, 287, 293] to form a closed polygonal region. Similar:

[0101] The D2 area is connected in sequence by the key point sequence [62, 293, 426, 330, 449, 359, 463, 243, 130, 229, 101, 206] to form a closed polygonal area;

[0102] The D3 region consists of the following sequence of key points:

[0103] [130,243,463,359,353,301,251,284,332,297,338,10,109,67,103,54,21,71,124,130];

[0104] The D3 region is connected by the key points of the D3 region to form a closed polygonal region;

[0105] The D4 region consists of the following key point sequence:

[0106] [172,138,214,57,62,206,101,229,130,124,71,21,162,127,234,93,132,58];

[0107] The D4 region is connected by the key points of the D4 region to form a closed polygonal region;

[0108] The D5 area consists of the following key point sequence:

[0109] [293,287,434,367,397,288,361,323,454,356,389,251,301,353,359,449,330,426];

[0110] The D5 region is connected by the key points of the D5 region to form a closed polygonal region. The face region corresponding to the partition D1-D5 is shown as follows: Figure 6 shown.

[0111] In some embodiments, determining the mask image of the preset block area based on the closed polygonal area includes:

[0112] Based on the closed polygonal area, the width of the mask area formed by the expanded area and the polygonal area is N times the width of the polygonal area, and the height of the mask area is M times the height of the polygonal area;

[0113] The pixel values ​​of the pixels inside the closed polygonal area are set to 1, and the pixel values ​​of the pixels in the extended area are set to 0, so as to obtain a mask image corresponding to the closed polygonal area as the mask image of the preset block area.

[0114] Wherein, N and M are numbers greater than 1 respectively.

[0115] In an exemplary embodiment, based on the key point information of the face in image I, the mask shapes of areas D1, D2, D3, D4, and D5 are extracted, thereby cropping the mask images of each area from image I. When cropping, the mask shape is expanded outward, such as expanding the width to 1.1 times the original, and expanding the height to 1.1 times the original, so as to ensure the boundary integrity of the masked part of the face image.

[0116] In some embodiments, determining the light color of each partition and determining the light color of the target facial image according to the light colors of at least two partitions includes:

[0117] For each partition, input the partition into a light color prediction model to perform light color prediction, and obtain an output vector of the partition, wherein the output vector includes a probability value of each light color in the initial color light sequence;

[0118] Performing a preset operation on the probability values ​​of the same light color in the output vectors of at least two partitions to obtain a result vector corresponding to the target face image, wherein the result vector includes a preset operation result of the probability value corresponding to each light color in the initial color light sequence;

[0119] The maximum value of the preset operation results of the probability values ​​in the result vector is obtained, and the light color corresponding to the obtained maximum value is determined as the light color of the target face image.

[0120] In an exemplary embodiment, the preset operation may be a weighted sum, and the preset operation result may be a weighted sum result.

[0121] In an exemplary embodiment, the light color prediction module may be a deep learning module such as a convolutional neural network module. For example, the convolutional neural network model adopts the open source resnet50, and the structure is as follows: Figure 8 shown.

[0122] It should be noted that the convolutional neural network model can also be a network model such as resnet18, mobilenet series, etc., and the present disclosure does not limit the specific network model used.

[0123] In an exemplary embodiment, the size of each of the five partitions of the face image is adjusted to 224*224*3, combined into a batch, and input into the convolutional neural network model. By forming a batch, it is convenient for the subsequent convolutional neural network model to be accelerated in parallel during reasoning. Each partition will output the category confidence of S categories. For example, the result vector of partition D1 includes S components, each of which represents the probability that D1 belongs to the light color of the corresponding category.

[0124] Combination Figure 8, assuming that the input data is a three-channel image (batch inference is essentially the same, here for the sake of convenience, we use one image inference as an example) with a size of 224x 224x 3.

[0125] After the first Conv2d, 7x7 convolution processing, stride = 2, the data is downsampled by 2 times, the number of channels is expanded to 64, and the feature map M1 is obtained. The size of M1 is 112x 112x 64;

[0126] M1 is processed by Max pool with stride 2 to achieve 2x downsampling, while the number of channels remains unchanged, and the feature map M2 is obtained with a size of 56x 56x 64;

[0127] M2 is processed by the Conv2_x module group, which is stacked 3 times. The convolution stride in the first two times is 1, the number of channels remains unchanged, and residual connections are added. The stride of the last time is 2, the number of channels is increased, and downsampling is achieved. After processing by the Conv2_x module group, feature M3 is obtained, whose size is 28x 28x 256;

[0128] M3 is processed by the Conv3_x module group to obtain M4, with a size of 14x 14x 512. M4 is processed by the Conv4_x module group to obtain M5, with a size of 7x 7x 1024, and M5 is processed by the Conv5_x module group to obtain M6, with a size of 7x 7x 2048 (no downsampling is performed in this group of processing);

[0129] Finally, the feature adaptive global average pooling of M6 is performed to obtain a 2048x 1 vector, which is finally sent to the fully connected layer for classification, with an output category of 6 and processed using the activation function (softmax).

[0130] In the exemplary embodiment, the output vectors of each partition D1-D5 are weighted summed to simulate the voting process. However, considering that the reflectivity and emphasis of each partition D1-D5 are different, the voting weights are different. The result of the weighted summation is softmax processed to obtain the light color category of the current target face image. The detailed process includes:

[0131] Assume that the output vector α of D1 is D1 =[α 1-1 , α 1-2 , α 1-3 , α 1-4 , α 1-5 , α 1-6 ], the output vector α of D2 D2 =[α 2-1 , α 2-2 , α 2-3 , α2-4 , α 2-5 , α 2-6 ], the output vector α of D3 D3 =[α 3-1 , α 3-2 , α 3-3 , α 3-4 , α 3-5 , α 3-6 ], the output vector α of D4 D4 =[α 4-1 , α 4-2 , α 4-3 , α 4-4 , α 4-5 , α 4-6 ], the output vector α of D5 D5 =[α 5-1 , α 5-2 , α 5-3 , α 5-4 , α 5-5 , α 5-6 ], perform weighted summation of D1-D5 to obtain D:

[0132] D=β1*α D1 +β2*α D2 +β3*α D3 +β4*α D4 +β5*α D5 , where β 1～5 It represents the weight of each block. Because the partition positions are different, their reflective characteristics are different, and their contributions to the final classification are also different. However, under normal conditions, the reflective imaging of the middle part of the face will be better and have a greater impact. Therefore, a slightly larger weight is given to the middle part, specifically: β2=0.3, β1=β3=0.2, β4=β5=0.15.

[0133] After D is calculated, softmax is used for processing to obtain the light color category Cls to which the current frame belongs.

[0134] Step 203, determining a target color light sequence according to the light color of the collected target face image.

[0135] In some embodiments, determining a target color light sequence according to the light color of the collected target face image includes:

[0136] According to the light colors of the target face image collected each time, a light color sequence is formed;

[0137] Using a sliding window to filter out abnormal values ​​in the light color sequence to obtain a filtered light color sequence;

[0138] The continuously repeated light colors in the filtered light color sequence are removed to obtain a target color light sequence.

[0139] By filtering out outliers through a sliding window, the robustness of the anomaly identification method is effectively improved, reducing the possibility of anomalies.

[0140] In the exemplary embodiment, for each frame of the target face image collected, a light color category Cls is obtained after prediction using the convolutional neural network model and partition voting. Ideally, the predicted category is completely consistent with the category in the label. However, due to the complexity of the real world and the existence of various interference factors, the classification accuracy of the convolutional neural network model for data cannot be guaranteed to be 100%, that is, misclassification may occur. Suppose there is such a situation that the color light sequence should be [light color 4 / 10 frame, light color 2 / 10 frame, light color 1 / 10 frame, light color 3 / 10 frame, light color 6 / 10 frame], but due to imaging and other reasons, the convolutional neural network model parses the first frame into light color 2 when parsing. The light color in the actual label is light color 4, and the parsed sequence becomes [2-4-2-1-3-6], which is different from the label [4-2-1-3-6]. In practical applications, it is easy to have unstable imaging when the light color changes, which leads to classification errors of the convolutional neural network model. In order to solve this problem, the embodiment of the present disclosure proposes to use a sliding window with a window size of 5 to scan the predicted light color sequence. If the number of category frames in the center of the window is less than 3, the sequence is eliminated and considered to be a misjudgment. For example, for the frame-by-frame predicted sequence [2,4,4,4,4,4,2,4,4,4,4,2,2,2,2,1,2,2,2,2,2,2…], where the first 2 2s and the last 1 are false detections, they can be filtered out using the sliding window technology. It should be noted that this sliding window screening method cannot cope with the interference of strong external light, such as colored lights illuminating the face, because in this case, the predicted category will have continuous false detections. However, coping with the problem of color light restoration with extremely complex ambient light is a relatively complex problem, and users are generally reminded to avoid this environment. Discussion of this issue is beyond the scope of the present disclosure and will not be expanded here.

[0141] Step 204: Determine whether there is an abnormality in the face recognition process according to the matching status of the initial color light sequence and the target color light sequence.

[0142] In some embodiments, determining whether there is an abnormality in the face recognition process according to the matching of the initial color light sequence and the target color light sequence includes:

[0143] If the initial color light sequence and the target color light sequence match, it is determined that there is no abnormality in the face recognition process;

[0144] If the matching situation of the initial color light sequence and the target color light sequence is not matching, it is determined that there is an abnormality in the face recognition process.

[0145] In an exemplary embodiment, whether there is a link hijacking attack in the face recognition process is determined based on the matching of the initial color light sequence and the target color light sequence. If the matching of the initial color light sequence and the target color light sequence is a match, it is determined that there is no link hijacking attack in the face recognition process; if the matching of the initial color light sequence and the target color light sequence is a mismatch, it is determined that there is a link hijacking attack in the face recognition process.

[0146] In some embodiments, the method further comprises:

[0147] Acquire the longest common subsequence between the initial color light sequence and the target color light sequence;

[0148] Determine the ratio of the length value of the longest common subsequence to the length value of the initial color light sequence;

[0149] If the ratio is greater than a preset threshold, determining that the match between the initial color light sequence and the target color light sequence is a match;

[0150] If the ratio is less than or equal to a preset threshold, it is determined that the match between the initial color light sequence and the target color light sequence is a mismatch.

[0151] In an exemplary embodiment, the matching degree between the initial color light sequence C5 and the target color light sequence C is calculated by first finding the longest common subsequence C between C5 and C. L , assuming its length is L, the matching degree is: L / 5. The system can set different thresholds β according to the security requirement level, that is, the verification is passed only when the matching degree is greater than β, for example, β is set to 0.8.

[0152] In an exemplary embodiment, Fig. 9 The figure is a schematic diagram of an exemplary process of abnormality identification provided in an embodiment of the present disclosure, which mainly includes the following process:

[0153] Step 901, start verification;

[0154] Step 902, generating an initial color light sequence C5;

[0155] Step 903, the mobile phone screen displays according to C5, and at the same time, the front camera and the screen work synchronously;

[0156] Step 904, the front camera collects facial image data;

[0157] Step 905, using Mediapipe to detect key points of the face image;

[0158] Step 906, perform a quality check on the face image to determine whether the quality of the face image meets the requirements; if the quality meets the requirements, execute step 907; if the quality does not meet the requirements, return to execute step 903;

[0159] Step 907, performing mask partitioning on the face area of ​​the face image;

[0160] Step 908, analyzing the face image data by partition voting to obtain a target color light sequence C;

[0161] Step 909, determine whether C5 matches C. If so, the verification is successful and the verification process ends. If not, the verification fails and the process returns to step 901 to perform verification again.

[0162] The abnormality recognition method provided by the embodiment of the present disclosure collects the target face image under the lighting conditions of the initial color light sequence, divides the face area of ​​each target face image collected to obtain at least two partitions, determines the light color of each partition respectively, and then determines the light color of the entire target face image based on the light colors of the at least two partitions. This method can overcome the interference of non-face areas on light color recognition, and can perform light color recognition in combination with the reflective characteristics of each partition, thereby improving the accuracy of light color recognition, and further improving the accuracy of the target color light sequence determined according to each target face image collected, thereby ensuring the matching accuracy of the initial color light sequence and the target color light sequence, so as to effectively determine whether there is an abnormality in the face recognition process.

[0163] In addition, while ensuring the quality of facial images, the time slices when there are quality problems are extended backtracking to avoid the loss of colorful light; when classifying, the mask information of key points is used to perform partition voting on the faces, and a sliding window method is used to filter out false detection frames, which effectively improves the robustness of the algorithm and reduces the possibility of abnormalities in the face recognition process.

[0164] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0165] In addition, the present disclosure also provides an abnormality identification device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any abnormality identification method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0166] Fig.10 A block diagram of an abnormality identification device provided in an embodiment of the present disclosure.

[0167] Reference Fig.10 , the embodiment of the present disclosure provides an abnormality identification device, the abnormality identification device comprising:

[0168] The acquisition module 1001 is used to acquire the target face image under the illumination conditions corresponding to the initial color light sequence;

[0169] The processing module 1002 is used to divide the face area of ​​each target face image collected into at least two partitions, determine the light color of each partition, and determine the light color of the target face image according to the light colors of the at least two partitions;

[0170] A determination module 1003 is used to determine a target color light sequence according to the light color of the collected target face image;

[0171] The judgment module 1004 is used to determine whether there is an abnormality in the face recognition process according to the matching situation between the initial color light sequence and the target color light sequence.

[0172] Each module in the above-mentioned abnormality identification device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0173] Fig.11 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0174] Reference Fig.11 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 1101; at least one memory 1102, and one or more I / O interfaces 1103, connected between the processor 1101 and the memory 1102; wherein the memory 1102 stores one or more computer programs that can be executed by the at least one processor 1101, and the one or more computer programs are executed by the at least one processor 1101, so that the at least one processor 1101 can execute the above-mentioned abnormality identification method.

[0175] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0176] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned abnormality identification method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0177] The embodiment of the present disclosure also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned abnormality identification method.

[0178] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0179] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0180] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0181] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0182] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0183] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0184] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0185] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0186] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0187] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. An abnormality identification method, It is characterized in that include: Acquire the target face image under the illumination conditions corresponding to the initial color light sequence; For each collected target facial image, dividing the facial area of ​​the target facial image into at least two partitions, determining the light color of each partition, and determining the light color of the target facial image based on the light colors of the at least two partitions; Determine the target color light sequence according to the light color of the collected target face image; According to the matching condition between the initial color light sequence and the target color light sequence, it is determined whether there is an abnormality in the face recognition process.

2. The method according to claim 1, It is characterized in that The target face image is collected under the illumination conditions corresponding to the initial color light sequence, including: During the continuous illumination time of each light color in the initial color light sequence, collecting a face image; In each case where a face image is collected, determining whether the quality of the collected face image meets a preset condition; If the preset condition is met, the collected face image is used as the target face image.

3. The method according to claim 2, It is characterized in that The method further comprises: If the quality of the collected face image does not meet the preset condition, the face image is discarded, and the illumination time of the light color corresponding to the collection time of the face image is extended; The continuous illumination time of the corresponding light color is updated according to the extended illumination time, and the facial image acquisition and subsequent steps are repeated.

4. The method according to claim 2, It is characterized in that The determining whether the quality of the collected face image meets a preset condition includes: Extract facial key points from the collected face image to obtain extraction results; Determining whether the extraction result contains facial key points; If the extraction result does not include facial key points, determining that the quality of the facial image does not meet a preset condition; If the extraction result includes facial key points, it is determined whether the quality of the facial image meets a preset condition according to the facial key points.

5. The method according to claim 4, It is characterized in that The step of judging whether the quality of the face image meets a preset condition according to the face key points includes: Determining a quality judgment parameter according to the facial key points, the quality judgment parameter including one or more of the following: a proportion of the facial area in the facial image, a positional relationship between the facial area in the facial image and a facial effective area in the facial image; According to the quality judgment parameter, it is judged whether the quality of the face image meets a preset condition.

6. The method according to claim 5, It is characterized in that The step of judging whether the quality of the face image meets a preset condition according to the quality judgment parameter includes: Determine whether the proportion value in the quality judgment parameter is greater than or equal to a first preset threshold and less than or equal to a second preset threshold, and obtain a first judgment result, wherein the first preset threshold is less than the second preset threshold; Determine whether the positional relationship in the quality judgment parameter satisfies that the face area of ​​the face image does not exceed the boundary of the face valid area, and obtain a second judgment result; When the first judgment result is yes, and / or the second judgment result is yes, it is determined that the quality of the facial image meets a preset condition.

7. The method according to claim 1, It is characterized in that The dividing the facial area of ​​the facial image into at least two partitions includes: Extracting key points in the face area of ​​the face image, and obtaining the sequence number and position coordinates of each key point; Determine a mask image of a preset block area in the face area according to the sequence number and position coordinates of each key point; The face area of ​​the face image is divided using a mask image of a preset block area to obtain at least two partitions.

8. The method according to claim 7, It is characterized in that The step of determining the mask image of the preset block area in the face area according to the sequence number and position coordinates of each key point includes: For each preset block area, obtaining a key point sequence number set corresponding to the preset block area in the face area, wherein the key point sequence number set includes sequence numbers of key points required to enclose the preset block area; Obtaining the position coordinates of the numbers in the key point sequence number set, and obtaining the position coordinate set corresponding to the key point sequence number set; Two key points in the position coordinate set that are adjacent to each other are connected to obtain a closed polygonal area, and a mask image of the preset block area is determined based on the closed polygonal area.

9. The method according to claim 1, It is characterized in that The step of determining the light color of each partition and determining the light color of the target face image according to the light colors of at least two partitions includes: For each partition, input the partition into a light color prediction model to perform light color prediction, and obtain an output vector of the partition, wherein the output vector includes a probability value of each light color in the initial color light sequence; Performing a preset operation on the probability values ​​of the same light color in the output vectors of at least two partitions to obtain a result vector corresponding to the target face image, wherein the result vector includes a preset operation result of the probability value corresponding to each light color in the initial color light sequence; The maximum value of the preset operation results of the probability values ​​in the result vector is obtained, and the light color corresponding to the obtained maximum value is determined as the light color of the target face image.

10. The method according to claim 1, It is characterized in that Determining a target color light sequence according to the collected light color of the target face image includes: According to the light color of the target face image collected each time, a light color sequence is formed; Using a sliding window to filter out abnormal values ​​in the light color sequence to obtain a filtered light color sequence; The continuously repeated light colors in the filtered light color sequence are removed to obtain a target color light sequence.

11. The method according to claim 1, It is characterized in that The determining whether there is an abnormality in the face recognition process according to the matching of the initial color light sequence and the target color light sequence includes: If the initial color light sequence and the target color light sequence match, it is determined that there is no abnormality in the face recognition process; If the matching situation of the initial color light sequence and the target color light sequence is not matching, it is determined that there is an abnormality in the face recognition process.

12. The method according to claim 11, It is characterized in that The method further comprises: Acquire the longest common subsequence between the initial color light sequence and the target color light sequence; Determine the ratio of the length value of the longest common subsequence to the length value of the initial color light sequence; If the ratio is greater than a preset threshold, determining that the match between the initial color light sequence and the target color light sequence is a match; If the ratio is less than or equal to a preset threshold, it is determined that the match between the initial color light sequence and the target color light sequence is a mismatch.

13. An abnormality identification device, It is characterized in that include: An acquisition module, used for acquiring a target face image under the illumination conditions corresponding to the initial color light sequence; A processing module, for dividing the face area of ​​each target face image collected into at least two partitions, and determining the light color of each partition, and determining the light color of the target face image according to the light colors of the at least two partitions; A determination module, used to determine a target color light sequence according to the light color of the collected target face image; The judgment module is used to determine whether there is an abnormality in the face recognition process according to the matching situation of the initial color light sequence and the target color light sequence.

14. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the abnormality identification method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the computer program implements the abnormality identification method according to any one of claims 1 to 12.

16. A computer program product, It is characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the abnormality identification method as described in any one of claims 1 to 12.

Images