Multi-modal face anti-counterfeiting detection method and device, computer device and storage medium

Abstract

The application discloses a multi-modal face anti-counterfeiting detection method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining a clock-in picture and preprocessing the clock-in picture to obtain a first sample set and a second sample set; the first sample set and the second sample set are respectively divided into a training set and a verification set according to a proportion to obtain a first training set and a first verification set, a second training set and a second verification set; the first training set is input into an improved YOLOv7 model for training to obtain a first weight model and the first weight model is verified by using the first verification set; the second training set is input into an original YOLOv7 model for training to obtain a second weight model and the second weight model is verified by using the second verification set; a to-be-detected picture is input into the first weight model for inference to determine whether a face in the to-be-detected picture is a false face; if the face in the to-be-detected picture is the false face, the to-be-detected picture is input into the second weight model for inference, and an inference result of the second weight model is used as a final detection result. The method improves the detection performance of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face detection, and in particular to a multi-modal face anti-forgery detection method and device, computer equipment and a storage medium. BACKGROUND

[0002] Currently, some work requires staff to perform attendance check-in at the corresponding attendance location and upload pictures of personnel present check-in as attendance basis. In order to complete check-in, some staff use mobile phone pictures or portrait pictures to let others check in at the corresponding attendance location, thereby forging check-in records. The authenticity of check-in is difficult to determine, and the attendance record personnel need to identify and judge the uploaded check-in pictures to confirm the real face pictures and false face pictures. SUMMARY

[0003] The embodiments of the present application provide a multi-modal face anti-forgery detection method, device, computer equipment and storage medium, aiming to solve the technical problem that it is difficult to determine the authenticity of check-in pictures in the prior art.

[0004] In a first aspect, the embodiments of the present application provide a multi-modal face anti-forgery detection method, which includes:

[0005] Obtaining a check-in picture, preprocessing the check-in picture to obtain a first sample set and a second sample set, and dividing the first sample set and the second sample set into a training set and a validation set in proportion to obtain a first training set and a first validation set, a second training set and a second validation set;

[0006] Inputting the first training set into an improved YOLOv7 model for training to obtain a first weight model and verifying the first weight model using data in the first validation set, the improved YOLOv7 model including adding an external attention module in the backbone network in the original YOLOv7 model, and adding a branch of a frequency spectrum module in the REP module of the head network of the original YOLOv7 model;

[0007] Inputting the second training set into the original YOLOv7 model for training to obtain a second weight model and verifying the second weight model using data in the second validation set;

[0008] Inputting a to-be-detected picture into the first weight model for inference to determine whether the face in the to-be-detected picture is a false face, and if so, inputting the to-be-detected picture into the second weight model for inference, and taking the inference result of the second weight model as the final detection result.

[0009] In a second aspect, the embodiments of the present application provide a multi-modal face anti-forgery detection device, which includes:

[0010] The acquisition module is configured to acquire a check-in picture, pre-process the check-in picture, obtain a first sample set and a second sample set, and divide the first sample set and the second sample set into a training set and a verification set in a proportion, respectively, to obtain a first training set and a first verification set, a second training set and a second verification set;

[0011] The first training module is configured to input the first training set into an improved YOLOv7 model for training, obtain a first weight model, and verify the first weight model by using data in the first verification set, wherein the improved YOLOv7 model comprises an external attention module added in a backbone network in an original YOLOv7 model, and a branch of a frequency spectrum module added in a REP module of a head network of the original YOLOv7 model;

[0012] The second training module is configured to input the second training set into an original YOLOv7 model for training, obtain a second weight model, and verify the second weight model by using data in the second verification set.

[0013] The inference module is configured to input a to-be-detected picture into the first weight model for inference, determine whether a face in the to-be-detected picture is a non-living body, if yes, input the to-be-detected picture into the second weight model for inference, and take an inference result of the second weight model as a final detection result.

[0014] In a third aspect, an embodiment of the present application further provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-modal face anti-fake detection method of the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program causes a processor to execute the multi-modal face anti-fake detection method of the first aspect when the processor executes the computer program.

[0016] The embodiment of the present application provides a multi-modal face anti-counterfeiting detection method, device, computer equipment and storage medium. The method comprises the following steps: obtaining a clock-in picture, preprocessing the clock-in picture to obtain a first sample set and a second sample set, and dividing the first sample set and the second sample set into a training set and a verification set according to a proportion to obtain a first training set and a first verification set, a second training set and a second verification set; inputting the first training set into an improved YOLOv7 model for training to obtain a first weight model and verifying the first weight model by using data in the first verification set, wherein the improved YOLOv7 model comprises adding an external attention module in a backbone network in an original YOLOv7 model and adding a branch of a frequency spectrum module in a REP module of a head network of the original YOLOv7 model; inputting the second training set into the original YOLOv7 model for training to obtain a second weight model and verifying the second weight model by using data in the second verification set; inputting a to-be-detected picture into the first weight model for inference to determine whether the face in the to-be-detected picture is a false face, and if yes, inputting the to-be-detected picture into the second weight model for inference, and taking an inference result of the second weight model as a final detection result. The method performs face anti-counterfeiting detection on the clock-in picture based on the improved YOLOv7 model, adds the external attention module between the second CBS module and the third CBS module of the backbone network in the original YOLOv7 model to enhance the semantic representation of the shallow feature map, and adds the frequency spectrum module in the REP module of the head network in the original YOLOv7 model to facilitate obtaining the frequency spectrum image of the to-be-detected picture for frequency spectrum analysis, improve the detection performance of the model, and strengthen the robustness and generalization performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 The flowchart of the multi-modal face anti-counterfeiting detection method provided by the embodiment of the present application is shown.

[0019] Figure 2 The schematic block diagram of the multi-modal face anti-counterfeiting detection device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] Please see Figure 1 This is a flowchart illustrating the multimodal face anti-spoofing detection method provided in an embodiment of the present invention, which includes steps S110 to S140.

[0025] Step S110: Obtain the check-in image, preprocess the check-in image to obtain the first sample set and the second sample set, and divide the first sample set and the second sample set into training set and validation set according to the ratio to obtain the first training set and the first validation set, the second training set and the second validation set.

[0026] In this embodiment, check-in photos from the construction site are collected, and the rules for irregular check-ins are confirmed with the business side to filter out low-quality check-in photos that cannot be identified. The check-in photos are grouped according to the proportion of the face area in the photos: a first sample set consists of check-in photos with a face area greater than or equal to a preset proportion, and a second sample set consists of check-in photos with a face area less than a preset proportion. Then, the faces in the check-in photos of the first and second sample sets are labeled to obtain corresponding tag information, including the center point coordinates, length, width, and tag category of the tag frame.

[0027] For example, the obtained check-in pictures are labeled and divided into an A sample set in which the face area ratio is greater than or equal to a preset proportion and a B sample set in which the face area ratio is less than the preset proportion, 70% of the A sample set is taken as a first training set, and the remaining 30% is taken as a first verification set, and 70% of the B sample set is taken as a second training set, and the remaining 30% is taken as a second verification set.

[0028] It should be noted that the check-in pictures are labeled, that is, the relevant information of the specified object in the check-in picture is labeled. Taking the first sample set as an example, the face of a person wearing a mask or not in the check-in picture is labeled with corresponding label information by a label frame. The label information includes position information and label category of the label frame in the check-in picture. Let the label frame of the check-in picture be K:

[0029] K∈R 5 ={x,y,w,h,c},

[0030] wherein x, y represent the center point coordinates of the label frame, w and h represent the length and height of the label frame respectively, and c represents the category of the label (the label of living body: living, and the label of non-living body: no_lving). Similarly, labeling the check-in pictures of the second sample set is the same as labeling the first sample set, and the labels can be phone, photo, hand_phone, hand_photo, and person.

[0031] It should be noted that, since the quality of the collected check-in pictures is uneven, the following requirements need to be met when making the sample set: 1. The training set and the verification set pictures require clear labels, and pictures with undefined labels cannot appear; 2. The number of pictures of personnel at different distances from the check-in camera needs to be collected in proportion; 3. The number of pictures of personnel facing the check-in camera at different angles needs to be collected in proportion. These adjustments can ensure that the data sources are extensive and more close to the actual scene, thereby ensuring that the trained model has strong generalization.

[0032] In step S120, the first training set is input into an improved YOLOv7 model for training to obtain a first weight model, and the data in the first verification set is used to verify the first weight model. The improved YOLOv7 model includes adding an external attention module to the backbone network in the original YOLOv7 model and adding a branch of a frequency spectrum module to the REP module of the head network of the original YOLOv7 model.

[0033] In step S130, the second training set is input into the original YOLOv7 model for training to obtain a second weight model, and the data in the second verification set is used to verify the second weight model.

[0034] In this embodiment, the backbone network of the original YOLOv7 model is a CNN network, which has translational invariance and locality, lacks the ability of long-distance modeling, and the effective information of the model applied to real and fake face discrimination may not be completely distributed in the face area, but also may be anywhere in the viewfinder screen (such as the frame, moire, etc.), therefore, an external attention module is added in the backbone network (Backbone network) of the original YOLOv7 model. For example, an external attention module is added between the second CBS module and the third CBS module of the backbone network (Backbone network), and the CBS module is an integrated module of Conv+BatchNorm+SiLU. The second CBS module of the backbone network (Backbone network) inputs the external attention module with a feature map of 320*320*64, and the external attention module outputs a feature map of 320*320. The improved backbone network of the YOLOv7 model extracts features from the clock-in pictures in the first training set to enhance the semantic representation of the shallow feature map and obtain more rich feature information in a larger area, thereby further improving the performance of the backbone network. Compared with the original YOLOv7 model, the YOLOv7 model with the external attention module can capture the global feature dependency relationship in space and channel, strengthen the feature information interaction, and effectively improve the extraction ability of the backbone network for shallow semantic features. In addition, after converting the fake clock-in picture and the real clock-in picture into frequency domain graphs, it is found that the high-frequency information of the fake clock-in picture is relatively single, only extending along the horizontal and vertical directions, while the high-frequency information of the real clock-in picture is divergent from the center of the image, and it can be found that the fake clock-in picture has some noise, therefore, a branch is introduced in the REP module of the head network (Head network) of the original YOLOv7 model to increase the frequency spectrum module, and the improved YOLOv7 model is obtained.

[0035] In this embodiment, the improved YOLOv7 model is trained by inputting the first training set into the improved YOLOv7 model to obtain the first prediction result corresponding to the first training set. Then, the first prediction loss of the first prediction result and the corresponding label is calculated according to the preset loss function, the model parameters of the improved YOLOv7 model are iterated based on the first prediction loss, and the first weight model is obtained. Finally, the data in the first verification set is used to verify the first weight model.

[0036] Furthermore, training the original YOLOv7 model involves inputting the second training set into the original YOLOv7 model to obtain the second prediction result corresponding to the second training set; then, calculating the second prediction loss of the second prediction result and the corresponding label based on the preset loss function; iterating the model parameters of the original YOLOv7 model based on the second prediction loss to obtain the second weight model; finally, validating the second weight model using data from the second validation set.

[0037] In one embodiment, both the first and second weight models are trained for 250 rounds. After each round of training, the corresponding validation set data is input into the currently trained weight model for validation to examine its performance after each round of training. The validation test process is as follows: load the validation set data, perform forward inference to obtain the inference result, calculate the validation set error, use non-maximum suppression to select the bounding box with the highest confidence in the image as the predicted value of the current target (delete other predicted boxes whose overlap with the predicted box exceeds a certain threshold), and save the prediction result. Taking the first sample set as an example, the label category (label for live: living, label for non-live: no_lving), i.e., "whether it is live", is used as an example, and the prediction result R is:

[0038] R living R no_living ={x, y, w, h, c, g},

[0039] Where x, y, w, and h represent the coordinates, length, and height of the predicted bounding box obtained through inference, c represents the category of the predicted bounding box (living for live subjects and no_living for non-live subjects), and g represents the confidence level of the predicted bounding box belonging to a certain category.

[0040] In one embodiment, step S120 includes:

[0041] Step S210: Extract features from the check-in images in the first training set using the backbone network of the improved YOLOv7 model, and enhance the semantic representation of the feature extraction process using the external attention module.

[0042] Step S220: Predict the features extracted by the backbone network using the head network of the improved YOLOv7 model, and perform spectral analysis on the prediction results using the spectral module.

[0043] In this embodiment, an external attention module using an external attention mechanism is added. The attention mechanism is external, small, learnable and shared memory, and is easy to implement using only two linear layers and a normalization layer BN, which can replace the existing method of self-attention mechanism. The external attention has a linear complexity and implicitly considers the relationship between different feature maps, further integrates the multi-head mechanism into the external attention, and provides a full MLP architecture for image classification, i.e. external attention MLP (EAMLP), thereby bringing sufficient global information to the YOLOv7 network and improving the capacity of the network. The working process of the external attention module is as follows:

[0044] (1) The attention between the input pixels and the external storage unit M ∈ R S×d is calculated in the following manner, S represents the number of pixels, d represents the feature dimension, and R represents the attention matrix:

[0045] A=a ij =Norm(FM T ),

[0046] F out =AM,

[0047] In the formula, a ij represents the similarity between the i-th pixel and the j-th row of M; M represents a learnable parameter independent of the input, which acts as a memory training data set as a whole; the Norm operation is a softmax function, Fout is the extracted feature; F represents the feature input to the external attention module; A represents the attention feature map obtained from the prior information, which is normalized in a manner similar to self-attention. Finally, the input feature in M is updated through the similarity in A.

[0048] (2) Normalization, the attention map is calculated by matrix multiplication. Unlike cosine similarity, the external attention module is sensitive to the scale of the input feature. In order to avoid this problem, the external attention adopts double normalization, which normalizes the columns and rows respectively. This double normalization is expressed as:

[0049]

[0050]

[0051]

[0052] wherein, represents the i-th pixel j-row normalization, exp represents the exponential function, represents the similarity between the i-th pixel and the j-th row, and k represents the identification of the k-th row, similarity of the kth pixel and the jth row, wherein ai,j represents the column normalization of the ith pixel and the jth row.

[0053] (3) Feature extraction using multi-head external attention:

[0054] h i = ExternalAttention(F i ,M k ,M v ),

[0055] F out = MultiHead(F i ,M k ,M v ) = Concat(h i ,...,h H )w o ,

[0056] wherein h i represents the ith head, H is the total number of heads, w o represents a linear transformation matrix, M k ∈R S×d and M v ∈R S×d represent shared storage units of different heads, F i represents the attention feature of the ith head, and Concat represents a merging function.

[0057] In addition, the features extracted by the backbone network are predicted by the head network of the improved YOLOv7 model, and the predicted results are analyzed by using the frequency spectrum module. The analysis process is as follows: based on Fourier transform, the predicted results are converted into a frequency spectrum graph; the frequency spectrum graph is normalized, and the normalized result is used as a label information during training. In the training, a branch is introduced from the middle of the head network (Head network), and then a new convolution layer is connected to reduce the channel to 1, and then the L2 distance is calculated with the real label.

[0058] In an embodiment, the prediction loss includes a positioning loss, a classification loss, a confidence loss, and a Fourier loss.

[0059] In this embodiment, the prediction results output by the model (i.e., the improved YOLOv7 model and the original YOLOv7 model) are optimized according to the prediction loss, and the prediction results gradually approach the label true value as the training round increases. Among them, the positioning loss is used to calculate the error between the prediction box and the labeled box; the classification loss is used to calculate whether the classification of the prediction box and the labeled box is correct; the confidence loss is used to represent the credibility of the prediction box, and the greater the value, the more likely there is a target in the labeled box; the Fourier loss is used to calculate the L2 distance between the true value and the labeled box.

[0060] Further, the most commonly used calculation index of the positioning loss is the intersection of union (IOU), which represents the intersection of the labeled box and the prediction box, that is, the ratio of the intersection area to the union area of the labeled box and the prediction box. The calculation formula is as follows:

[0061]

[0062] The present application adopts CIOU as the positioning loss. CIOU considers the center distance, overlap rate, size and penalty term between the labeled box and the prediction box, so that the prediction box is more stable and will not have the problem of divergence in the training process like IOU or GIOU. The penalty factor considers the fitting of the length-width ratio of the prediction box to the length-width ratio of the labeled box, and the calculation formula is as follows:

[0063]

[0064]

[0065]

[0066] wherein truth represents the true box, pred represents the prediction box, IOU(truth, pred) represents the area intersection of the true box and the prediction box, a represents the weight coefficient, v represents the similarity of the length ratio of the true box and the prediction box, c represents the diagonal distance of the circumscribed rectangle of the true box and the prediction box, and p(b pred b gt ) represents the Euclidean distance of the center position of the true box and the prediction box, w and h represent the length and height of the true box, respectively, and w gt and h gt represent the length and height of the prediction box, respectively.

[0067] Further, the classification loss and the confidence loss both adopt the binary cross-entropy loss function, and the prediction loss L is calculated according to the following formula:

[0068]

[0069]

[0070] wherein, L box represents a localization loss, L obj represents a classification loss, L cls represents a confidence loss, L FT represents a Fourier loss, N represents the number of detection layers, λ1, λ2, λ3, λ4 respectively represent the weights of the localization loss, the classification loss, the confidence loss and the Fourier loss, i represents the i-th detection layer, S i x S i is the number of networks into which the scale is divided, B i is the number of target labels assigned to the prior box, F p represents the position of the real label; F G represents the position of the predicted label; represents the loss of the j-th coordinate; represents the loss of the j-th classification; represents the loss of the j-th confidence.

[0071] Step S140, input the to-be-detected picture into the first weight model for inference, judge whether the face in the to-be-detected picture is a false face, if yes, input the to-be-detected picture into the second weight model for inference, and take the inference result of the second weight model as the final detection result.

[0072] In this embodiment, the to-be-detected picture is grouped to obtain a first detection set with a face area greater than or equal to a preset proportion and a second detection set with a face area less than the preset proportion; the first detection set and the second detection set are respectively input into the first weight model and the second weight model for inference to obtain a first inference result and a second inference result; if the confidence of the first inference result is less than a preset threshold, it indicates that the face area of the punch-in picture of the corresponding first detection set is less than the preset proportion, and the first detection set corresponding to the first inference result must be input into the second weight model for inference, and the inference result of the second weight model is taken as the final detection result; if the confidence of the first inference result is greater than or equal to the preset threshold, the first inference result is taken as the final detection result.

[0073] For example, the to-be-detected picture is grouped to obtain an A detection set with a face area greater than or equal to a preset proportion and a B detection set with a face area less than the preset proportion; the A detection set and the B detection set are respectively input into the first weight model and the second weight model for inference to obtain an a inference result and a b inference result; if the confidence of the a inference result is less than a preset threshold, the A detection set corresponding to the a inference result is input into the second weight model for inference, and the inference result of the second weight model is taken as the final detection result; if the confidence of the a inference result is greater than or equal to the preset threshold, the a inference result is taken as the final detection result.

[0074] In an embodiment, if the final detection result output by the second weight model is one of the labels of a mobile phone, an A4 paper, a hand holding a mobile phone, and a hand holding an A4 paper, it indicates that the corresponding picture to be detected is a non-living body, and the final detection result is a fake face clock-in picture. Finally, the final detection result is pushed to the anti-counterfeiting platform, and the anti-counterfeiting platform classifies the clock-in pictures of living bodies and non-living bodies.

[0075] The method performs face anti-counterfeiting detection on the clock-in picture based on an improved YOLOv7 model, increases an external attention module between the second CBS module and the third CBS module of the backbone network in the original YOLOv7 model to enhance the semantic representation of the shallow feature map, and increases a frequency spectrum module in the REP module of the head network in the original YOLOv7 model to facilitate obtaining the frequency spectrum image of the picture to be detected for frequency spectrum analysis, improve the detection performance of the model, and strengthen the robustness and generalization performance of the model.

[0076] The embodiment of the present application also provides a multi-modal face anti-counterfeiting detection device, which is used to execute any one of the embodiments of the multi-modal face anti-counterfeiting detection method. Specifically, please refer to Figure 2 , Figure 2 is a schematic block diagram of the multi-modal face anti-counterfeiting detection device provided by the embodiment of the present application. The multi-modal face anti-counterfeiting detection device 100 can be configured in a server.

[0077] As Figure 2 shown, the multi-modal face anti-counterfeiting detection device 100 includes an acquisition module 110, a first training module 120, a second training module 130, and an inference module 140.

[0078] The acquisition module 110 is configured to acquire a clock-in picture, pre-process the clock-in picture, obtain a first sample set and a second sample set, and divide the first sample set and the second sample set into a training set and a validation set in a certain proportion to obtain a first training set and a first validation set, a second training set and a second validation set.

[0079] The first training module 120 is configured to input the first training set into an improved YOLOv7 model for training to obtain a first weight model, and verify the first weight model using data in the first validation set. The improved YOLOv7 model includes an external attention module added between the second CBS module and the third CBS module of the backbone network in the original YOLOv7 model, and a branch of a frequency spectrum module added in the REP module of the head network of the original YOLOv7 model.

[0080] The second training module 130 is configured to input the second training set into the original YOLOv7 model for training, to obtain a second weight model and to verify the second weight model by using data in the second verification set.

[0081] The inference module 140 is configured to input a to-be-detected picture into the first weight model for inference, to determine whether a face in the to-be-detected picture is a non-living body, and if so, to input the to-be-detected picture into the second weight model for inference, and to take an inference result of the second weight model as a final detection result.

[0082] The embodiment of the present application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-modal face anti-counterfeiting detection method as described above when executing the computer program.

[0083] In another embodiment of the present application, a computer readable storage medium is provided. The computer readable storage medium can be a non-volatile computer readable storage medium. The computer readable storage medium stores a computer program, which, when executed by a processor, causes the processor to execute the multi-modal face anti-counterfeiting detection method as described above.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here. Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, each example has been described in the foregoing description in terms of its general functionality, without regard to its specific combination of hardware and software. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0085] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the embodiments of the apparatus described above are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, or a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electric, mechanical or in other forms.

[0086] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0087] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software functional units.

[0088] When the integrated unit is implemented in the form of software functional units and sold or used as an independent product, it can be stored in a storage medium. Based on such an understanding, the technical solutions of the present application essentially, or the part that contributes to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), magnetic disk or optical disk, and various media that can store program codes.

[0089] The above description is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-modal face anti-spoofing detection method, characterized in that, The method comprises the following steps: acquiring a check-in picture, preprocessing the check-in picture to obtain a first sample set and a second sample set, and dividing the first sample set and the second sample set into a training set and a validation set in proportion to obtain a first training set and a first validation set, a second training set and a second validation set; inputting the first training set into an improved YOLOv7 model for training to obtain a first weight model, and verifying the first weight model by using data in the first validation set, wherein the improved YOLOv7 model comprises an external attention module added between a second CBS module and a third CBS module in a backbone network of an original YOLOv7 model, and a branch of a frequency spectrum module added in a REP module of a head network of the original YOLOv7 model; inputting the second training set into the original YOLOv7 model for training to obtain a second weight model, and verifying the second weight model by using data in the second validation set; inputting a to-be-detected picture into the first weight model for inference to determine whether a face in the to-be-detected picture is a fake face, and if so, inputting the to-be-detected picture into the second weight model for inference, and taking an inference result of the second weight model as a final detection result; the step of inputting the first training set into the improved YOLOv7 model for training comprises the following steps: performing feature extraction on the check-in pictures in the first training set by a backbone network of the improved YOLOv7 model, and performing semantic representation enhancement processing on the feature extraction process by using the external attention module; and performing prediction on the features extracted by the backbone network by a head network of the improved YOLOv7 model, and performing frequency spectrum analysis on the predicted result by using the frequency spectrum module; the step of performing frequency spectrum analysis on the predicted result by using the frequency spectrum module comprises the following steps: converting the predicted result into a frequency spectrum graph based on Fourier transform; and performing normalization processing on the frequency spectrum graph, and taking the normalized result as a label information during training.

2. The multi-modal face liveness detection method of claim 1, wherein, the step of preprocessing the check-in picture to obtain the first sample set and the second sample set comprises the following steps: grouping the check-in pictures according to the proportion of the area of the face in the check-in picture, taking the check-in pictures with the area of the face greater than or equal to a preset proportion as the first sample set, and taking the check-in pictures with the area of the face less than the preset proportion as the second sample set; annotating the face in the check-in pictures in the first sample set and the second sample set to obtain corresponding label information, wherein the label information comprises the center point coordinates, the length, the width and the label category of a label box.

3. The multi-modal face liveness detection method of claim 1, wherein, the step of inputting the first training set into the improved YOLOv7 model for training further comprises the following steps: calculating a prediction loss of a first predicted result output by the improved YOLOv7 model and a corresponding label according to a preset loss function, and iteratively updating the model parameters of the improved YOLOv7 model based on the prediction loss, wherein the prediction loss comprises a positioning loss, a classification loss, a confidence loss and a Fourier loss.

4. The multi-modal face liveness detection method of claim 3, wherein, the process of calculating the prediction loss comprises the following steps: calculating the prediction loss L according to the following formula: where L box represents the localization loss, L obj represents the classification loss, L cls represents the confidence loss, L FT represents the Fourier loss, N represents the number of detection layers, λ1, λ2, λ3, λ4 represent the weights of the localization loss, the classification loss, the confidence loss and the Fourier loss respectively, i represents the ith detection layer, S i × S i is the number of networks into which the scale is divided, B i is the number of targets to which the label is assigned to the prior box, F p represents the position of the real label, F G represents the position of the predicted label, represents the loss of the jth coordinate, represents the loss of the jth classification, represents the loss of the jth confidence.

5. The multi-modal face liveness detection method of claim 1, wherein, The first weight model is input into the first weight model for inference, and it is judged whether the face in the to-be-detected picture is a false face. If so, the to-be-detected picture is input into the second weight model for inference, and the inference result of the second weight model is taken as the final detection result, comprising: Grouping the to-be-detected picture to obtain a first detection set with a face area greater than or equal to a preset proportion and a second detection set with a face area less than a preset proportion; The first detection set and the second detection set are input into the first weight model and the second weight model respectively for inference to obtain a first inference result and a second inference result; If the confidence of the first inference result is less than a preset threshold, the first detection set corresponding to the first inference result is input into the second weight model for inference, and the inference result of the second weight model is taken as the final detection result. If the confidence of the first inference result is greater than or equal to a preset threshold, the first inference result is taken as the final detection result.

6. A multi-modal face liveness detection apparatus, characterized in that, Comprising: The acquisition module is used for acquiring a check-in picture, preprocessing the check-in picture to obtain a first sample set and a second sample set, and dividing the first sample set and the second sample set into a training set and a validation set in proportion to obtain a first training set and a first validation set, a second training set and a second validation set; The first training module is used for inputting the first training set into the improved YOLOv7 model for training to obtain a first weight model, and verifying the first weight model by using data in the first validation set, wherein the improved YOLOv7 model comprises an external attention module added in a backbone network in an original YOLOv7 model, and a frequency spectrum module added in a REP module of a head network in the original YOLOv7 model; The second training module is used for inputting the second training set into the original YOLOv7 model for training to obtain a second weight model, and verifying the second weight model by using data in the second validation set; The inference module is used for inputting a to-be-detected picture into the first weight model for inference to judge whether the face in the to-be-detected picture is a non-living body. If so, the to-be-detected picture is input into the second weight model for inference, and the inference result of the second weight model is taken as the final detection result; The first training set is input into the improved YOLOv7 model for training, comprising: feature extraction of the check-in picture in the first training set is performed by the backbone network of the improved YOLOv7 model, and the external attention module is used for semantic representation enhancement processing on the feature extraction process; the features extracted by the backbone network are predicted by the head network of the improved YOLOv7 model, and the frequency spectrum module is used for frequency spectrum analysis on the predicted result; The frequency spectrum module is used for frequency spectrum analysis on the predicted result, comprising: based on Fourier transform, the predicted result is converted into a frequency spectrum graph; the frequency spectrum graph is normalized, and the normalized result is taken as a label information during training.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the multi-modal face anti-counterfeiting detection method according to any one of claims 1-5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program causes the processor to execute the multi-modal face anti-counterfeiting detection method according to any one of claims 1-5 when executed by the processor.

Images