An attack detection method and a retort detection method

Abstract

The application provides an attack detection method and a mouth replacement detection method, which comprises the following steps: obtaining a face video to be detected; performing living body detection on the face video to be detected based on a lip region in the face video to be detected, obtaining a living body detection result, and performing mouth replacement detection on the face video to be detected by using a neural network model to obtain a mouth replacement detection result; wherein the neural network model is obtained by training based on a mouth replacement video sample obtained by synthesis; and performing attack detection on the face video to be detected based on the living body detection result and the mouth replacement detection result to obtain an attack detection result. By combining the living body detection result based on the lip and the mouth replacement detection result, the attack detection is performed on the face video to be detected, so that the problem that the mouth replacement video attack synthesized cannot be recognized when the living body detection is performed without the mouth replacement detection is avoided, and the safety of the living body detection on the target object is effectively improved.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and more specifically, to an attack detection method and a mouth-swapping detection method. Background Technology

[0002] Liveness detection refers to the process of determining the true physiological characteristics of an object in some identity verification scenarios, used to verify whether a user is a real living person.

[0003] Current liveness detection methods typically use lip-reading to verify a user's identity. For example, a four-digit number is sent to the client, which the user then reads aloud. Video and audio files of the user's reading are then obtained. Deep learning methods are used to identify if the user's lip movements match the reading, and speech recognition is used to confirm the correct four-digit number. If the lip movements match and the reading is correct, the user is confirmed to be the actual person. However, in practice, this method has proven difficult to prevent lip-swapping attacks, resulting in low security for liveness detection of target individuals. Summary of the Invention

[0004] The purpose of this application is to provide an attack detection method and a mouth-swapping detection method to improve the low security of liveness detection of target objects.

[0005] Firstly, embodiments of this application provide an attack detection method, comprising: acquiring a face video to be detected; performing liveness detection on the face video to be detected based on the lip region in the face video to be detected, obtaining a liveness detection result; and using a neural network model to perform lip-swapping detection on the face video to be detected, obtaining a lip-swapping detection result; wherein the neural network model is trained based on synthesized lip-swapping video samples; and performing attack detection on the face video to be detected based on the liveness detection result and the lip-swapping detection result, obtaining an attack detection result. In the above implementation process, by combining the liveness detection result based on lip region and the lip-swapping detection result to perform attack detection on the face video to be detected, the problem of difficulty in identifying synthesized lip-swapping video attacks due to the lack of lip-swapping detection during liveness detection is avoided, effectively improving the security of liveness detection of the target object.

[0006] In one optional implementation of the first aspect, a neural network model is used to perform lip-swapping detection on the face video to be detected, and to obtain lip-swapping detection results. This includes: determining multiple target video frames for lip-swapping detection based on the face video to be detected; and performing lip-swapping attack detection based on the multiple target video frames and the neural network model to obtain lip-swapping detection results. In the above implementation process, by determining multiple target video frames for lip-swapping detection based on the face video to be detected, and performing lip-swapping attack detection based on the multiple target video frames and the neural network model to obtain lip-swapping detection results, the problem of not performing lip-swapping detection during liveness detection, which leads to difficulty in identifying synthesized lip-swapping video attacks, is avoided, effectively improving the security of liveness detection of target objects.

[0007] In one optional implementation of the first aspect, determining multiple target video frames for mouth-swapping detection based on the face video to be detected includes: performing keypoint detection on each video frame in the face video; and selecting a target number of video frames in an open-mouth state based on the distance between the upper lip keypoint and the lower lip keypoint as multiple target video frames. In the above implementation, by performing keypoint detection on the video frames and selecting a target number of video frames in an open-mouth state based on the distance between the upper lip keypoint and the lower lip keypoint as multiple target video frames, the problem of performing mouth-swapping detection on multiple closed-mouth video frames is avoided, effectively improving the success rate of mouth-swapping detection.

[0008] In one optional implementation of the first aspect, determining multiple target video frames for lip-swapping detection based on the face video to be detected includes: extracting multiple video frames from the face video to be detected; and determining the difference image between two adjacent video frames in the extracted multiple video frames as the target video frames. In the above implementation, by calculating the difference image between two adjacent video frames, since the neural network model has learned that only the mouth area shows significant changes while other areas remain largely unchanged, resulting in a generated attack video, while facial muscles and mouth muscles move together in a real user video, this neural network model can effectively classify the difference image to obtain a detection result indicating whether the video is an attack, thereby effectively identifying attack videos synthesized using face images and lip-swapping videos.

[0009] In one optional implementation of the first aspect, attack detection is performed on the face video to be detected based on the liveness detection result and the lip-swapping detection result to obtain the attack detection result. This includes: determining whether the liveness detection result meets a first objective condition, and determining whether the lip-swapping detection result meets a second objective condition. The first objective condition includes: the face in the face video to be detected is a live face; the second objective condition includes: the face in the face video to be detected has not been lip-swapped. If the liveness detection result meets the first objective condition and the lip-swapping detection result meets the second objective condition, then it is determined that the face video to be detected is not attacked; otherwise, it is determined that the face video to be detected is attacked. In the above implementation process, by determining that the face video to be detected is not attacked only when the liveness detection result meets the first objective condition and the lip-swapping detection result meets the second objective condition, the lip-swapping attack video synthesized using a face image and a lip-swapping video is effectively identified.

[0010] In one optional implementation of the first aspect, the lip-swapping video sample is synthesized through the following process: determining a first mouth region in a face sample image, and determining a second mouth region in each frame of the target video; generating a mask pattern corresponding to the mouth in the face sample image based on the first mouth region, filling each second mouth region into the mask pattern to generate a corresponding mask image; fusing each mask image with the face sample image to obtain multiple frames of lip-swapping images, and synthesizing the multiple frames of lip-swapping images into a lip-swapping video sample.

[0011] Secondly, embodiments of this application also provide a method for detecting lip-swapping, comprising: acquiring a face video to be detected; determining multiple target video frames for lip-swapping detection within the face video to be detected; performing lip-swapping detection on the face video to be detected based on the multiple target video frames and a neural network model, and obtaining a lip-swapping detection result; wherein the neural network model is trained based on synthesized lip-swapping video samples. In the above implementation process, by training a neural network model based on synthesized lip-swapping video samples and using the neural network model to perform lip-swapping detection on the face video to be detected, the situation where it is difficult to identify synthetic lip-swapping video attacks is avoided, effectively improving the security of liveness detection of target objects.

[0012] In one optional implementation of the second aspect, determining multiple target video frames for mouth-swapping detection in the face video to be detected includes: performing keypoint detection on each video frame in the face video; and selecting a target number of video frames in an open-mouth state based on the distance between the upper lip keypoint and the lower lip keypoint as multiple target video frames. In the above implementation, by performing keypoint detection on the video frames and selecting a target number of video frames in an open-mouth state based on the distance between the upper lip keypoint and the lower lip keypoint as multiple target video frames, the problem of performing mouth-swapping detection on multiple closed-mouth video frames is avoided, effectively improving the success rate of mouth-swapping detection.

[0013] In one optional implementation of the second aspect, determining multiple target video frames for lip-swapping detection in the face video to be detected includes: extracting multiple video frames from the face video to be detected; and determining the difference image between two adjacent video frames in the extracted multiple video frames as the target video frames. In the above implementation, by calculating the difference image between two adjacent video frames, since the neural network model has learned that only the mouth area shows significant changes while other areas remain largely unchanged, resulting in a generated attack video, while facial muscles and mouth muscles move together in a real user video, this neural network model can effectively classify the difference image to obtain a detection result indicating whether the video is an attack, thereby effectively identifying attack videos synthesized using face images and lip-swapping videos.

[0014] Thirdly, embodiments of this application also provide an attack detection device, including: a face video acquisition module for acquiring a face video to be detected; a detection result acquisition module for performing liveness detection on the face video to be detected based on the lip region in the face video to be detected, obtaining a liveness detection result, and performing lip-swapping detection on the face video to be detected using a neural network model, obtaining a lip-swapping detection result; wherein the neural network model is trained based on synthesized lip-swapping video samples; and a video attack detection module for performing attack detection on the face video to be detected based on the liveness detection result and the lip-swapping detection result, obtaining an attack detection result.

[0015] Fourthly, embodiments of this application also provide a mouth-swapping detection device, comprising: a face video acquisition module for acquiring a face video to be detected; a target video frame determination module for determining multiple target video frames for mouth-swapping detection within the face video to be detected; and a video mouth-swapping detection module for performing mouth-swapping detection on the face video to be detected based on the multiple target video frames and a neural network model, thereby obtaining a mouth-swapping detection result; wherein the neural network model is trained based on synthesized mouth-swapping video samples.

[0016] Fifthly, embodiments of this application also provide an electronic device, including: a processor and a memory, the memory storing machine-readable instructions executable by the processor, wherein when the machine-readable instructions are executed by the processor, they perform the methods described in the first aspect or any optional implementation thereof, or, when the machine-readable instructions are executed by the processor, they perform the methods described in the second aspect or any optional implementation thereof.

[0017] Sixthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when run by a processor, executes the methods described in the first aspect or any optional implementation thereof, or, when machine-readable instructions are executed by a processor, executes the methods described in the second aspect or any optional implementation thereof.

[0018] In a seventh aspect, embodiments of this application also provide a computer program product, including computer program instructions. When the computer program instructions are read and run by a processor, they execute the method described in the first aspect or any optional implementation thereof above. Alternatively, when the machine-readable instructions are executed by a processor, they execute the method described in the second aspect or any optional implementation thereof above. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the embodiments of this application, and therefore should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 The diagram shown is a flowchart of the attack detection method provided in an embodiment of this application;

[0021] Figure 2 The diagram shown is a schematic representation of key points of the mouth in a face image provided in an embodiment of this application.

[0022] Figure 3 The diagram shown is a flowchart illustrating the synthesized mouth-swapping video sample provided in an embodiment of this application.

[0023] Figure 4 The diagram shown is a flowchart of the model training method provided in an embodiment of this application;

[0024] Figure 5 The diagram shown is a flowchart of the nozzle-changing detection method provided in an embodiment of this application;

[0025] Figure 6 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of this application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0028] Before introducing the method for generating video with lip-syncing provided in the embodiments of this application, let's first introduce some concepts involved in the embodiments of this application:

[0029] Facial landmark detection is a technique that automatically searches for the locations of predefined facial feature points in any image. These facial landmarks include, for example, the pupils, nose, and corners of the mouth.

[0030] Convolutional Neural Networks (CNNs) are a type of artificial neural network. The artificial neurons in a CNN respond to surrounding units, enabling large-scale image processing. A CNN can include convolutional layers and pooling layers. CNNs are better able to extract features from images, such as color, texture, shape, and spatial relationships.

[0031] It should be noted that the attack detection method and the mouth-swapping detection method provided in this application embodiment can be executed by electronic devices. Here, electronic devices refer to device terminals or servers with the function of executing computer programs. Device terminals include, for example, smartphones, personal computers (PCs), tablet computers, personal digital assistants (PDAs), or mobile internet devices (MIDs). Servers refer to devices that provide computing services through a network. Servers include, for example, x86 servers and non-x86 servers. Non-x86 servers include, for example, mainframes, minicomputers, and UNIX servers.

[0032] Please see Figure 1 The illustrated flowchart shows an attack detection method provided in an embodiment of this application. The main idea of ​​this attack detection method is to combine liveness detection results based on lip detection with mouth-swapping detection results to perform attack detection on the face video to be detected. This avoids the problem of difficulty in identifying synthesized mouth-swapping video attacks due to the lack of mouth-swapping detection during liveness detection, effectively improving the security of liveness detection on the target object. Specifically, the above-mentioned attack detection method may include:

[0033] Step S110: Obtain the video of the face to be detected.

[0034] The above-mentioned step S110 can be implemented in the following ways: In the first embodiment, a camera, video recorder or color camera or other acquisition device is used to capture the target object to obtain a face video to be detected; then the acquisition device sends the face video to be detected to an electronic device, and then the electronic device receives the face video to be detected sent by the acquisition device; In the second embodiment, when the target object is being detected for liveness detection, the electronic device itself captures the face video of the target object as the face video to be detected. Here, the electronic device is, for example, a camera, video recorder or color camera.

[0035] Step S120: Based on the lip region in the face video to be detected, perform liveness detection on the face video to be detected to obtain liveness detection results; and use a neural network model to perform mouth-swapping detection on the face video to be detected to obtain mouth-swapping detection results; wherein, the neural network model is trained based on the synthesized mouth-swapping video samples.

[0036] In practice, liveness detection can be performed first, followed by mouth-swapping detection; alternatively, mouth-swapping detection can be performed first, followed by liveness detection; or both can be performed simultaneously. Therefore, the execution order of liveness detection and mouth-swapping detection should not be construed as a limitation of the embodiments of this application. Step S130: Based on the liveness detection results and mouth-swapping detection results, attack detection is performed on the face video to be detected to obtain the attack detection results.

[0037] As one implementation of the above step S120, which uses a neural network model to detect lip-swapping in the face video to be detected, the video frames that need to be input into the model can be determined first, and then lip-swapping attack detection can be performed on the video frames. This implementation may include:

[0038] Step S121: Determine the multiple target video frames for mouth-swapping detection based on the face video to be detected;

[0039] Step S122: Detect lip-swapping attacks based on multiple target video frames and a neural network model to obtain lip-swapping detection results.

[0040] Please see Figure 2 The illustration shows a schematic diagram of key points of the mouth in a face image provided in an embodiment of this application. In one optional embodiment of step S121, key point detection can be performed on each video frame in the face video to be detected, and target video frames can be determined based on the key points in the mouth region. Target video frames are selected based on the distance between the detected upper lip key points and lower lip key points. Specifically, for example, for each video frame in the face video, a face key point detection network model is used to perform key point detection on the video frames. Based on the distance between the upper lip key points and lower lip key points, a target number of video frames in an open-mouth state are selected as multiple target video frames.

[0041] Specifically, the open-mouth state here can be determined by filtering the distance between the key points of the upper lip and the key points of the lower lip, selecting the maximum distance or one greater than a preset distance threshold. The key points of the upper lip can be selected from... Figure 2 The key points are any one or more of 48, 47, 57, 50, and 51. The key point for the lower lip can be selected. Figure 2The key points are any one or more of 39, 40, 37, 43, and 42. In practice, key points for the upper and lower lip can also be selected from several key points in the middle region. In this case, the average distance between multiple upper lip key points and multiple lower lip key points can be used to determine if the average distance is greater than a preset distance threshold. If the average distance for the video frame is greater than the preset distance threshold, it is determined to be an open mouth state; if the average distance for the video frame is less than the preset distance threshold, it is determined not to be an open mouth state. The facial landmark detection network models that can be used in the embodiments of this application include Dlib or PFLD (Practical Facial Landmark Detector), etc.

[0042] In one of the optional embodiments of step S121, the target video frame can also be determined in the following way: the difference image between two adjacent video frames in the face video to be detected is taken as the target video frame. The main idea of ​​this embodiment is to extract multiple video frames from the face video to be detected and determine the difference image between two adjacent video frames in the extracted multiple video frames as the target video frame.

[0043] One specific implementation of the above-mentioned method for determining the difference image is as follows: Convert multiple video frames (e.g., 4 video frames) from RGB mode to grayscale mode; or, add and concatenate the three RGB channels and divide by the number of channels (i.e., take the arithmetic mean of the three channel values); or, multiply and concatenate the three RGB channels and take their cube root (i.e., take the geometric mean of the three channel values), thereby retaining only the luminance component to reduce the influence of other colors on the attack detection results. Then, subtract adjacent grayscale images and take the absolute value, so that the value of stationary image areas is almost 0, while the value of moving image areas (e.g., the mouth) is very large (e.g., above 100), finally obtaining the difference image between the multiple video frames. Here, RGB refers to the abbreviation of the three color channels: red (R), green (G), and blue (B). It can be understood that if 4 video frames are selected for the difference operation, then 3 difference images are obtained, and these 3 difference images are used as the target video frames.

[0044] In one optional embodiment of step S121, based on the similarity between two adjacent video frames, the image with the lowest similarity and the image with the highest similarity are selected as multiple target video frames. For example, if the running speed is fast enough (i.e., meets the requirements of the application scenario), the similarity between each pair of adjacent video images in the video to be detected can be calculated first, and then multiple video images can be extracted from the video to be detected based on the similarity. For example, a preset number of video images with the highest similarity and a preset number of video images with the lowest similarity are extracted from all video images in the video to be detected as target video frames.

[0045] In step S122 above, mouth-swapping attack detection is performed based on multiple target video frames and a neural network model. The obtained mouth-swapping detection result indicates whether the mouth region in the video to be detected has been replaced.

[0046] As one optional implementation of step S122 above, the neural network model directly predicts the lip-swapping detection result based on the face region in the target video frame. Specifically, this implementation involves determining the face region in multiple target video frames based on the facial key points corresponding to the target video frame. The face regions in the multiple target video frames are then input into the neural network model to obtain the lip-swapping detection result output by the neural network model. It is understood that the face region includes the mouth region and the face region. If it is a synthesized lip-swapping attack video, the neural network model can detect it as a synthesized lip-swapping attack video if there is movement in the mouth region between multiple frames (i.e., the difference is greater than a threshold) and no movement in the face region between multiple frames (i.e., the difference is almost 0, for example, less than a threshold). Similarly, if it is a normal video, the neural network model can detect it as a normal video if there is movement in the mouth region between multiple frames (i.e., the difference is greater than a threshold) and also if there is movement in the face region between multiple frames (i.e., the difference is greater than a threshold).

[0047] The implementation of step S130 above includes, but is not limited to: determining whether the liveness detection result meets the first objective condition, and determining whether the mouth-swapping detection result meets the second objective condition; wherein, the first objective condition includes: the face in the face video to be detected is a live face, and the second objective condition includes: the face in the face video to be detected has not been mouth-swapped. If the liveness detection result meets the first objective condition and the mouth-swapping detection result meets the second objective condition, then it is determined that the face video to be detected is not attacked. If the liveness detection result does not meet the first objective condition, or the mouth-swapping detection result does not meet the second objective condition, then it is determined that the face video to be detected is attacked.

[0048] In the above implementation process, liveness detection is first performed on the face video to be detected based on the lip region to obtain the liveness detection result, and then a neural network model is used to perform lip-swapping detection on the face video to be detected to obtain the lip-swapping detection result. Then, based on the liveness detection result and the lip-swapping detection result, attack detection is performed on the face video to be detected. This avoids the problem that it is difficult to identify synthesized lip-swapping video attacks when lip-swapping is not performed during liveness detection, and effectively improves the security of liveness detection of target objects.

[0049] Please see Figure 3 The diagram illustrates the process of synthesizing lip-swapping video samples according to an embodiment of this application. It is understood that the neural network model described above can be trained using synthesized lip-swapping video samples. The specific synthesis process of the lip-swapping video samples will be described first, followed by the training process of the neural network model. Assuming an attacker needs to conceal their identity (i.e., not display a real face image), they can pre-record their target video and then synthesize a lip-swapping video sample using the face region from another person's face sample image and the mouth region from the target video. The lip-swapping video sample can be synthesized through the following process.

[0050] Step S210: Determine the first mouth region in the face sample image, and determine the second mouth region in each frame of the target video.

[0051] A face sample image refers to an image in which the mouth area in the target video needs to be replaced. Understandably, the mouth in the face image is different from the mouth in the target video.

[0052] There are many ways to obtain the aforementioned face sample images, including but not limited to: the first method is to use a generative adversarial network to generate face images and obtain face sample images; the second method is to perform image enhancement on the generated face sample images to obtain face sample images.

[0053] The target video refers to a video of the mouth area that can be identified through lip-reading liveness detection. It can be a video of the attacker's mouth area or a video of a non-attacker's mouth area, as long as the mouth moves and can be identified through lip-reading liveness detection.

[0054] The implementation of step S210 above includes: using a facial landmark detection network model to detect landmarks in a facial sample image to obtain the mouth landmarks corresponding to the facial sample image. Based on the mouth landmarks, the bounding rectangular region corresponding to the mouth in the facial sample image is determined as the first mouth region. Specifically, for example, according to the position and shape of all mouth landmarks in the facial sample image, a rectangular box suitable for covering the position and shape of all mouth landmarks is selected, and expanded outward by a preset ratio (e.g., 1 / 4) to obtain the enlarged rectangular box shown in the figure. The area where the enlarged rectangular box is located can be understood as the mouth region in the facial sample image.

[0055] Then, a facial landmark detection network model is used to detect landmarks in each frame of the target video, obtaining the mouth landmarks corresponding to each frame. Based on the mouth landmarks, the bounding rectangular region corresponding to the mouth in each frame of the target video is determined as the second mouth region. Specifically, the mouth region in each frame of the target video is determined based on the second mouth landmarks, which involves determining a rectangular frame covering the position and shape of all mouth landmarks in each frame, and expanding it outward by a preset ratio (e.g., 1 / 4) to obtain the enlarged rectangular frame shown in the figure. Here, the enlarged rectangular frame serves as the mouth region in each frame. Of course, in practice, other shapes can also be used as the mouth region, such as rhombuses, pentagons, hexagons, or other polygonal frames.

[0056] Step S220: Generate a mask pattern corresponding to the mouth in the face sample image based on the first mouth region, and fill each second mouth region into the mask pattern to generate the corresponding mask image.

[0057] An example implementation of step S220 above is as follows: Assume an enlarged rectangular frame is used as the first mouth region corresponding to the face image, and a mask pattern is constructed based on the first mouth region to obtain the mask pattern corresponding to the face image. Then, a Gaussian function is used to smooth the mask pattern corresponding to the face image to obtain a smoothed mask pattern. Finally, the mouth region of each frame in the target video is filled into the smoothed mask pattern to generate a mask image corresponding to each frame.

[0058] Step S230: Fuse each mask image with the face sample image to obtain multiple frames of lip-swapping images, and then combine the multiple frames of lip-swapping images into a lip-swapping video sample.

[0059] For example, the implementation of step S230 above involves using a Poisson fusion algorithm to fuse the mask image corresponding to each frame of the image with the mouth region in the face image to obtain multiple frames of mouth-swapping images. These multiple frames of mouth-swapping images are then combined to form mouth-swapping video samples, which are used to train a neural network model for attack detection.

[0060] In the above implementation process, a mask image corresponding to each frame of the target video is generated by filling the mouth region of each frame into a mask graphic; the mask image corresponding to each frame is fused with the mouth region in the face image to obtain multiple frames of mouth-swapping images; and the multiple frames of mouth-swapping images are synthesized into a mouth-swapping video, thereby simulating various mouth-swapping situations in face images. This allows the neural network model trained on the mouth-swapping video to effectively identify attack videos synthesized from face images and mouth-shape videos, avoiding the problem of difficulty in identifying attack videos synthesized from face images and mouth-shape videos, thus increasing the security of liveness detection.

[0061] Please see Figure 4 The diagram illustrates a flowchart of the model training method provided in this application embodiment. The specific synthesis process of the lip-switching video samples has been described above; the specific process of training a neural network model using the synthesized lip-switching video samples will be described below. This application embodiment provides a model training method, including:

[0062] Step S310: Obtain multiple sample videos and multiple sample labels. The sample labels are category labels indicating whether there is a mouth region replacement in the sample videos. The sample videos include mouth-swapping videos obtained using methods such as mouth-swapping video generation.

[0063] It is understandable that the sample videos above include not only lip-swapping videos obtained by the lip-swapping video generation method, but also real videos without lip-swapping.

[0064] There are many ways to obtain the sample labels in step S310 above, including but not limited to: The first method involves manually watching the sample video and observing whether the lip movements match and whether there are significant changes in the facial muscles around the mouth during vocalization. If the lip movements do not match and there are no significant changes in the facial muscles around the mouth during vocalization, then it can be determined that the category label of the sample video involves lip region replacement; otherwise, it can be determined that the sample video does not have local region replacement. The second method involves using a sufficient number of sample videos and sample labels to train a neural network model. Then, the trained neural network model can be used to label the remaining sample videos, obtaining the corresponding sample labels for the remaining sample videos.

[0065] Step S320: Train the neural network using multiple sample videos as training data and multiple sample labels as training labels to obtain a neural network model.

[0066] A neural network model refers to a neural network used to classify videos and determine whether a video belongs to an attack detection category. This neural network is obtained after training. It's understood that the neural network used here is not limited; specific neural networks that can be used include: VGG networks, ResNet networks, Wide ResNet networks, and Inception networks, etc. Specifically, VGG networks include, for example, VGG16 or VGG19; ResNet networks include, for example, ResNet12, ResNet18, ResNet50, or ResNet101; Wide ResNet networks include, for example, Wide ResNet-28-10 (sometimes abbreviated as WRN-28-10); and Inception networks include, for example, Inception v1, Inception v2, or Inception v3.

[0067] Of course, in practical applications, the neural network described above can also be a custom-built network from scratch. A typical network structure for this is as follows: The input image first passes through a 7x7 convolutional layer and downsampling to obtain a 56x56x16 feature map. Then, it passes through two bottleneck modules and a downsampling layer to obtain a 28x28x64 feature map. Each bottleneck module contains three convolutional layers and one shortcut layer. After two more convolutional layers and a global pooling layer, a 64-dimensional feature map is obtained. Finally, the output is obtained through two fully connected layers and a softmax layer, and the second value is recorded as the classification score of the image.

[0068] Understandably, preprocessing of the sample videos is necessary before training. Preprocessing steps include: extracting multiple video frames from the sample videos and calculating the difference images between these frames. Specific details can be found in the attack detection method section below. If four video frames are selected for interpolation, three difference images are obtained. These three difference images are then concatenated (e.g., weighted average concatenation and geometric average concatenation) to obtain a three-channel input image. Common data augmentation operations such as random cropping, random rotation, random noise addition, random translation, and flipping are then applied to the input image. Finally, the image is scaled to the network input size.

[0069] The implementation of step S320 above may include: using a neural network to predict the predicted category of the lip-syncing video; using a cross-entropy loss function to calculate the loss value between the predicted category and the sample label; training the neural network based on the loss value; specifically, for example: updating the network weight parameters of the neural network based on the loss value until the loss value is less than a preset proportion or the number of iterations (epochs) is greater than a preset threshold, thus obtaining the trained neural network model. The preset proportion can be set according to specific circumstances, such as 5% or 10%; the preset threshold can also be set according to specific circumstances, such as 100 or 1000, etc.

[0070] In the above implementation process, by simulating various mouth-swapping scenarios in face images, the neural network model trained on the mouth-swapping video can effectively identify attack videos synthesized from face images and mouth-swapping videos, thereby avoiding the problem of difficulty in identifying attack videos synthesized from face images and mouth-swapping videos.

[0071] Please see Figure 5 The illustration shows a flowchart of a nozzle-changing detection method provided in an embodiment of this application; this application provides a nozzle-changing detection method, including:

[0072] Step S410: Obtain the video of the face to be detected.

[0073] The above step S410 can be implemented in the following ways: In the first implementation, during mouth-swapping detection, a camera, video recorder, or color camera is used to capture an image of the target object, obtaining a video of the face to be detected; then, the capture device sends the video of the face to be detected to an electronic device, and the electronic device receives the video of the face to be detected sent by the capture device; In the second implementation, during mouth-swapping detection, an electronic device is used to capture the video of the face to be detected, where the electronic device can be a camera, video recorder, or color camera, etc.

[0074] Step S420: Determine the target video frames for mouth-swapping detection in the face video to be detected.

[0075] Step S430: Perform mouth-swapping detection on the target video video based on multiple target video frames and a neural network model to obtain mouth-swapping detection results; wherein, the neural network model is trained based on synthesized mouth-swapping video samples.

[0076] As one optional implementation of step S420 above, target video frames are selected based on the distance between the detected upper lip key points and lower lip key points. Specifically, for each frame in a face video, key point detection is performed on the video frame to obtain multiple key points. Based on the distance between the upper lip key points and lower lip key points among these multiple key points, a target number of video frames in an open-mouth state are selected as the multiple target video frames.

[0077] The implementation principle of step S420 is similar to that of selecting target video frames in step S121 above, so it will not be repeated here. If there is any ambiguity, please refer to the description of step S121. In the above implementation process, by calculating the difference image between two adjacent video frames, since the neural network model has learned that only the mouth area changes significantly while other areas remain basically unchanged, it is a generated attack video, while facial muscles and mouth muscles move together in real user videos. Therefore, the neural network model can be used to effectively classify the difference image into videos and obtain the detection result of whether the video is an attack, thereby effectively identifying attack videos synthesized using face images and mouth shape videos.

[0078] This application also provides a computer program product, including computer program instructions. These instructions are read and executed by a processor to perform the methods described above. It should be understood that this program product corresponds to the method embodiments described above and is capable of executing the various steps involved in the method embodiments. The specific functions of this program product can be found in the description above; detailed descriptions are omitted here to avoid repetition. The program product includes at least one software functional module that can be stored in memory or embedded in the operating system (OS) of the program product in the form of software or firmware.

[0079] This application also provides an attack detection device, including:

[0080] The face video acquisition module is used to acquire the face video of the person to be detected.

[0081] The detection result acquisition module is used to perform liveness detection on the face video to be detected based on the lip region in the face video to be detected, and to obtain the liveness detection result. It also uses a neural network model to perform mouth-swapping detection on the face video to be detected, and to obtain the mouth-swapping detection result. The neural network model is trained based on synthesized mouth-swapping video samples.

[0082] The video attack detection module is used to perform attack detection on the face video to be tested based on the liveness detection results and the mouth-swapping detection results, and obtain the attack detection results.

[0083] In one alternative implementation, the detection result acquisition module includes:

[0084] The target video frame determination module is used to determine multiple target video frames for mouth-swapping detection based on the video of the face to be detected.

[0085] The lip-swapping attack detection module is used to detect lip-swapping attacks based on multiple target video frames and a neural network model, and obtain lip-swapping detection results.

[0086] In one alternative implementation, the target video frame determination module includes:

[0087] The key point detection module is used to perform key point detection on each frame of a face video.

[0088] The open-mouth video frame filtering method is used to filter the number of target frames based on the distance between the upper lip keypoint and the lower lip keypoint. These frames are considered as multiple target video frames.

[0089] In one alternative implementation, the target video frame determination module includes:

[0090] The video frame extraction module is used to extract multiple video frames from the video of the face to be detected.

[0091] The difference image determination module is used to determine the difference image between two adjacent video frames in the extracted multi-frame video frames, and use it as the target video frame.

[0092] In one alternative implementation, the video attack detection module includes:

[0093] The target condition judgment module is used to determine whether the liveness detection result meets the first target condition, and whether the mouth-swapping detection result meets the second target condition; wherein, the first target condition includes: the face in the face video to be detected is a live face, and the second target condition includes: the face in the face video to be detected has not been mouth-swapped;

[0094] The video attack determination module is used to determine that the face video to be detected does not contain an attack if the liveness detection result meets the first target condition and the mouth-swapping detection result meets the second target condition; otherwise, it determines that the face video to be detected contains an attack.

[0095] In one alternative implementation, the attack detection device further includes:

[0096] The mouth region determination module is used to determine the first mouth region in the face sample image and the second mouth region in each frame of the target video.

[0097] The mask image generation module is used to generate a mask graphic corresponding to the mouth in the face sample image based on the first mouth region, and to fill each second mouth region into the mask graphic to generate the corresponding mask image.

[0098] The lip-swapping video synthesis module is used to fuse each mask image with a face sample image to obtain multiple frames of lip-swapping images, and then synthesize the multiple frames of lip-swapping images into a lip-swapping video sample.

[0099] This application also provides a nozzle-changing detection device, including:

[0100] The face video acquisition module is used to acquire the face video of the person to be detected.

[0101] The target video frame determination module is used to determine multiple target video frames for mouth-swapping detection in the face video to be detected.

[0102] The video lip-swapping detection module is used to detect lip-swapping in the video of the face to be detected based on multiple target video frames and a neural network model, and to obtain the lip-swapping detection results; wherein, the neural network model is trained based on synthesized lip-swapping video samples.

[0103] In one alternative implementation, the target video frame determination module includes:

[0104] The key point detection module is used to perform key point detection on each frame of a face video.

[0105] The video frame filtering module is used to filter video frames in an open-mouth state based on the distance between the upper lip keypoint and the lower lip keypoint, and select them as multi-frame target video frames.

[0106] In one alternative implementation, the target video frame determination module includes:

[0107] The video frame extraction module is used to extract multiple video frames from the video of the face to be detected.

[0108] The difference image determination module is used to determine the difference image between two adjacent video frames in the extracted multi-frame video frames, and use it as the target video frame.

[0109] Please see Figure 6 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 500 provided in this application includes a processor 510 and a memory 520. The memory 520 stores machine-readable instructions executable by the processor 510. When the machine-readable instructions are executed by the processor 510, the method described above is performed.

[0110] This application embodiment also provides a computer-readable storage medium 530, on which a computer program is stored, and the computer program is executed by a processor 510 to perform the above method.

[0111] The computer-readable storage medium 530 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0112] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending primarily on the functions involved.

[0113] In addition, the functional modules of each embodiment in the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0114] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0115] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. An attack detection method, characterized in that, include: Obtain the video of the face to be detected; Based on the lip region in the face video to be detected, liveness detection is performed on the face video to be detected to obtain liveness detection results. In addition, a neural network model is used to perform lip-swapping detection on the face video to be detected to obtain lip-swapping detection results. The neural network model is trained based on synthesized lip-swapping video samples. Based on the liveness detection results and the mouth-swapping detection results, attack detection is performed on the face video to be detected to obtain the attack detection results. The step of performing attack detection on the face video to be detected based on the liveness detection result and the mouth-swapping detection result, and obtaining the attack detection result, includes: The system determines whether the liveness detection result meets a first objective condition and whether the mouth-swapping detection result meets a second objective condition; wherein the first objective condition includes: the face in the face video to be detected is a live face, and the second objective condition includes: the face in the face video to be detected has not been mouth-swapped; If the liveness detection result satisfies the first target condition and the mouth-swapping detection result satisfies the second target condition, then it is determined that the face video to be detected is not attacked; otherwise, it is determined that the face video to be detected is attacked.

2. The method according to claim 1, characterized in that, The step of using a neural network model to perform mouth-swapping detection on the face video to be detected, and obtaining the mouth-swapping detection result, includes: Based on the face video to be detected, determine the multiple target video frames for mouth-swapping detection; Based on the multiple target video frames and the neural network model, a mouth-swapping attack detection is performed to obtain the mouth-swapping detection result.

3. The method as described in claim 2, characterized in that, The multi-frame target video frames determined based on the face video to be detected for mouth-swapping detection include: For each video frame in the face video, key point detection is performed on the video frame; Based on the distance between the upper lip key point and the lower lip key point, select a number of video frames in the open-mouth state as the target video frames.

4. The method as described in claim 2, characterized in that, The multi-frame target video frames determined based on the face video to be detected for mouth-swapping detection include: Extract multiple video frames from the video of the face to be detected; The difference image between two adjacent video frames in the extracted multi-frame video is determined as the target video frame.

5. The method according to any one of claims 1-4, characterized in that, The mouth-switching video sample is synthesized through the following process: Identify the first mouth region in the face sample image, and identify the second mouth region in each frame of the target video; Based on the first mouth region, a mask pattern corresponding to the mouth in the face sample image is generated, and each second mouth region is filled into the mask pattern to generate a corresponding mask image; Each of the mask images is fused with the face sample image to obtain multiple frames of lip-swapping images, and the multiple frames of lip-swapping images are then combined to form the lip-swapping video sample.

6. A method for detecting nozzle replacement, characterized in that, include: Obtain the video of the face to be detected; Identify multiple target video frames for mouth-swapping detection in the face video to be detected; Based on the multi-frame target video and the neural network model, the video of the face to be detected is subjected to lip-swapping detection to obtain lip-swapping detection results; wherein, the neural network model is trained based on synthesized lip-swapping video samples; The step of performing lip-swapping detection on the face video to be detected based on the multi-frame target video and the neural network model to obtain lip-swapping detection results includes: Based on the facial key points corresponding to the multi-frame target video, the facial region in the multi-frame target video is determined; The face regions in the multi-frame target video are input into a neural network model to obtain the mouth-swapping detection results output by the neural network model; The face region includes a mouth region and a face region. The step of inputting the face region from the multiple target video frames into a neural network model to obtain the mouth-swapping detection result output by the neural network model includes: If the neural network model identifies that there is movement in the mouth area between the multi-frame target video and no movement in the face area between the multi-frame target video, then the mouth-swapping detection result is determined to be a mouth-swapping attack video. If the neural network model identifies that there is movement in the mouth area and the face area between the multiple target video frames, then the mouth-swapping detection result is determined to be a normal video.

7. The method according to claim 6, characterized in that, The step of determining the multiple target video frames for mouth-swapping detection in the face video to be detected includes: For each video frame in the face video, key point detection is performed on the video frame; Based on the distance between the upper lip key point and the lower lip key point, select a number of video frames in the open-mouth state as the target video frames.

8. The method according to claim 6, characterized in that, The step of determining the multiple target video frames for mouth-swapping detection in the face video to be detected includes: Extract multiple video frames from the video of the face to be detected; The difference image between two adjacent video frames in the extracted multi-frame video is determined as the target video frame.

9. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method as described in any one of claims 1 to 5, or, when executed by the processor, perform the method as described in any one of claims 6 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in any one of claims 1 to 5, or, when executed by a processor, performs the method as described in any one of claims 6 to 8.

11. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1 to 5, or perform the method as described in any one of claims 6 to 8.

Images