Liveness detection methods and systems

Abstract

The liveness detection method and system provided in this specification acquire a target video obtained by compressing the original video captured from the target user, then partially decompresses the target video to obtain a compressed domain video, generates N uncompressed domain videos based on the compressed domain video, and performs liveness detection based on the compressed domain video and the N uncompressed domain videos to obtain and output the liveness detection result. The method and system partially decompress the target video, saving time compared to full video decompression. Subsequently, generating N uncompressed domain videos based on the compressed domain video compensates for the information loss caused by the compressed domain video, thereby improving liveness detection performance.

Description

Technical Field

[0001] This specification relates to the field of image recognition, and in particular to a liveness detection method and system. Background Technology

[0002] Compared to traditional identity verification methods such as passwords and verification codes, facial recognition is more efficient and convenient. Therefore, it is widely used in scenarios such as finance, transportation, and identity verification (verifying a user's true identity). However, facial recognition also faces security issues such as liveness detection attacks. To detect various types of liveness attacks and ensure the security of facial recognition systems, liveness detection has become a crucial component.

[0003] Existing liveness detection methods can perform liveness detection by collecting a video clip of the user. However, video data requires video decoding, which is time-consuming, resulting in low efficiency of liveness detection and consequently affecting the efficiency of face recognition. Summary of the Invention

[0004] This manual provides a more accurate method and system for liveness detection.

[0005] In a first aspect, this specification provides a liveness detection method, comprising: acquiring a target video of a target user, wherein the target video is obtained by compressing an original video collected from the target user; partially decompressing the target video to obtain a compressed domain video; generating N uncompressed domain videos based on the compressed domain video, wherein N is an integer greater than 0; and performing liveness detection based on the compressed domain video and the N uncompressed domain videos to obtain and output a liveness detection result.

[0006] In some embodiments, the compressed domain video includes multiple image frames, each image frame corresponding to a compression type of I-frame or non-I-frame, wherein the image frame with compression type of I-frame is a fully decompressed image, and the image frame with compression type of non-I-frame is a residual image and a motion vector image.

[0007] In some embodiments, generating N uncompressed domain videos based on the compressed domain video includes: generating M modal videos and their corresponding M confidence scores from the compressed domain video using a modal generation model, wherein the M modal videos are uncompressed domain videos, the M modal videos correspond to M modalities, M is an integer greater than 0, and M is greater than or equal to N; and selecting N modal videos from the M modal videos as the N uncompressed domain videos, wherein the confidence scores of the N uncompressed domain videos satisfy a preset rule.

[0008] In some embodiments, the preset rule includes the top N values ​​after the M confidence levels are sorted in descending order.

[0009] In some embodiments, the modality generation model is trained using the following steps: obtaining a first training sample and its corresponding first label, wherein the first training sample includes multiple first compressed domain training videos, the first compressed domain training videos are obtained by partially decompressing the original compressed training videos, and the first label includes M original video sets under the M modalities; and training a preset modality generation model based on the first training sample and the first label, with a first comprehensive loss less than a first preset value as a constraint objective, wherein the first comprehensive loss constrains the prediction result output by the preset modality generation model to approach the first label during the training.

[0010] In some embodiments, the first comprehensive loss includes: modality generation loss, including M sub-modality generation losses corresponding to the M modalities, wherein each of the M sub-modality generation losses constrains the difference between the training modality video output by the preset modality generation model and its corresponding original video during the training process.

[0011] In some embodiments, the first comprehensive loss further includes: a confidence regression loss, comprising M sub-confidence regression losses corresponding to the M modalities. Each of the M sub-confidence regression losses constrains the difference between the training confidence output by the preset modality generation model and its corresponding true value during training. The true value corresponding to the confidence is obtained based on its corresponding sub-modality generation loss.

[0012] In some embodiments, the step of performing liveness detection based on the compressed domain video and the N uncompressed domain videos, obtaining the liveness detection result and outputting it includes: using a liveness detection model to extract features from the compressed domain video and the N uncompressed domain videos to obtain compressed domain features corresponding to the compressed domain video and N uncompressed domain features corresponding to the N uncompressed domain videos; and performing liveness detection based on the compressed domain features and the N uncompressed domain features to obtain the liveness detection result and outputting it.

[0013] In some embodiments, the liveness detection model is trained using the following steps: obtaining a second training sample and its corresponding second label, wherein the second training sample includes multiple second compressed domain training videos and multiple corresponding uncompressed domain training video sets, each of the multiple uncompressed domain training video sets includes N uncompressed domain training videos, the second compressed domain training videos are obtained by partially decompressing the original compressed training videos, and the second label includes a liveness classification label; based on the second training sample and the second label, training a preset liveness detection model for liveness classification with a second comprehensive loss less than a second preset value as a constraint objective, thereby obtaining the liveness detection model, wherein the second comprehensive loss includes at least a liveness classification loss, and the liveness classification loss constrains the difference between the liveness classification training result output by the preset liveness detection model and its corresponding liveness classification label during the training process.

[0014] In some embodiments, the second comprehensive loss further includes at least one of compressed domain inter-frame consistency loss, uncompressed domain inter-frame consistency loss, and intermodal consistency loss; the compressed domain inter-frame consistency loss constrains the differences between different frames in the second compressed domain training features output by the preset liveness detection model during training; the uncompressed domain inter-frame consistency loss constrains the differences between different frames in each of the N uncompressed domain training features output by the liveness detection model during training; and the intermodal consistency loss constrains the differences between the second compressed domain training features and each of the N uncompressed domain training features during training.

[0015] Secondly, this specification also provides a liveness detection system, comprising: at least one storage medium storing at least one instruction set for performing liveness detection; and at least one processor communicatively connected to the at least one storage medium, wherein, when the liveness detection system is running, the at least one processor reads the at least one instruction set and executes the liveness detection method described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0016] As can be seen from the above technical solutions, the liveness detection method and system provided in this specification, after acquiring and compressing the original video collected from the target user, partially decompresses the target video to obtain a compressed domain video, generates N uncompressed domain videos based on the compressed domain video, and performs liveness detection based on the compressed domain video and the N uncompressed domain videos to obtain and output the liveness detection result. This solution saves time on full video decompression by partially decompressing the target video, and the subsequent generation of N uncompressed domain videos based on the compressed domain video compensates for the information loss caused by the compressed domain video, thereby improving liveness detection performance.

[0017] Other functions of the liveness detection methods and systems provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the liveness detection methods and systems provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating an application scenario of a liveness detection system provided according to an embodiment of this specification is shown.

[0020] Figure 2 A hardware structure diagram of a computing device provided according to an embodiment of this specification is shown;

[0021] Figure 3 A flowchart of a liveness detection method according to an embodiment of this specification is shown;

[0022] Figure 4 A flowchart of a method for generating N uncompressed domain videos according to embodiments of this specification is shown;

[0023] Figure 5 A flowchart illustrating a method for training a modality generation model according to embodiments of this specification is shown;

[0024] Figure 6 A schematic diagram of the structure of a preset modality generation model provided according to an embodiment of this specification is shown;

[0025] Figure 7 A schematic diagram of a pre-defined liveness detection model provided according to an embodiment of this specification is shown;

[0026] Figure 8 A schematic diagram of another pre-defined liveness detection model provided according to an embodiment of this specification is shown; and

[0027] Figure 9 An overall flowchart of a liveness detection method provided according to an embodiment of this specification is shown. Detailed Implementation

[0028] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0029] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0030] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0031] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0032] For ease of description, the terms that will appear in the following descriptions will be explained as follows:

[0033] Compression domain: refers to the data types obtained by partially decoding video, including I-frames, residual maps, and motion vector maps.

[0034] Cross-modal image generation: refers to the method of generating various modal images using compressed domain images. These modal images can be RGB images, depth images, etc.

[0035] Liveness detection: Algorithm technology used in facial recognition or other face-based verification systems to detect and intercept liveness attacks (including attacks using mobile phone photos, paper photos, and masks).

[0036] Before describing the specific embodiments in this specification, the application scenarios of this specification will be introduced as follows:

[0037] The liveness detection method provided in this specification can be applied to any liveness detection scenario in biometric processes, such as facial payment, access control, attendance, and identity verification. It can be used to perform liveness detection on target videos of target users. It can also be applied to any other liveness detection scenario, which will not be elaborated upon here. The biometric features may include, but are not limited to, one or more of facial images, iris, sclera, fingerprints, palm prints, voiceprints, and skeletal projections. For ease of description, this application will use the application of the liveness detection method in a facial recognition scenario to describe liveness detection of a face as an example.

[0038] Those skilled in the art should understand that the liveness detection methods and systems described in this specification are also within the scope of protection of this specification when applied to other application scenarios.

[0039] Figure 1 This diagram illustrates an application scenario of a liveness detection system 001 provided according to an embodiment of this specification. The liveness detection system 001 (hereinafter referred to as System 001) can be applied to liveness detection in any scenario, such as liveness detection in face payment scenarios, liveness detection in identity verification scenarios, liveness detection in other face recognition scenarios, etc. Figure 1 As shown, system 001 may include target user 100, client 200, server 300 and network 400.

[0040] Target user 100 can be a user who needs to undergo biometric identification, or a user who is currently undergoing biometric identification. Target user 100 can be the object detected by system 001. Target user 100 can initiate a biometric identification process, thereby triggering liveness detection of target user 100.

[0041] Client 200 can be a device for performing liveness detection on a target facial image in response to a liveness detection operation by target user 100. In some embodiments, the liveness detection method can be executed on client 200. In this case, client 200 may store data or instructions for executing the liveness detection method described herein, and may execute or be used to execute said data or instructions. In some embodiments, client 200 may include a hardware device with data information processing capabilities and the necessary programs required to drive the hardware device to operate. Figure 1As shown, client 200 can communicate with server 300. In some embodiments, server 300 can communicate with multiple clients 200. In some embodiments, client 200 can interact with server 300 via network 400 to receive or send messages, etc. In some embodiments, client 200 may include mobile devices, tablets, laptops, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, the mobile device may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, the smart home device may include smart TVs, desktop computers, etc., or any combination thereof. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, gaming devices, navigation devices, etc., or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include virtual reality headsets, virtual reality glasses, virtual reality patches, augmented reality headsets, augmented reality glasses, augmented reality patches, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, head-mounted displays, VR, etc. In some embodiments, the built-in device in the motor vehicle may include an in-vehicle computer, an in-vehicle TV, etc. In some embodiments, the client 200 may include an image acquisition device and an audio acquisition device for collecting user data of the account. In some embodiments, the image acquisition device may be a two-dimensional image acquisition device (such as an RGB camera), or a two-dimensional image acquisition device (such as an RGB camera) and a depth image acquisition device (such as a 3D structured light camera, a laser detector, etc.). In some embodiments, the client 200 may be a device with positioning technology for locating the position of the client 200.

[0042] In some embodiments, the client 200 may have one or more applications (APPs) installed. The APPs provide the target user 100 with the ability and interface to interact with the outside world via the network 400. The APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, video APPs, financial management APPs, instant messaging tools, email clients, social media platform software, etc. In some embodiments, the client 200 may have a target APP installed. The target APP can collect facial images and corresponding audio information of users corresponding to multiple accounts for the client 200, thereby creating a user data set. In some embodiments, the target object 100 can also trigger a liveness detection request through the target APP. The target APP can respond to the liveness detection request by executing the liveness detection method described in this specification. The liveness detection method will be described in detail later.

[0043] Server 300 may be a server providing various services, such as a backend server that supports liveness detection for multiple accounts collected from client 200, including user data sets and account login information. In some embodiments, the liveness detection method can be executed on server 300. In this case, server 300 may store data or instructions for executing the liveness detection method described herein, and may execute or be used to execute the data or instructions. In some embodiments, server 300 may include a hardware device with data processing capabilities and the necessary programs to drive the hardware device. Server 300 can communicate with multiple clients 200 and receive data sent by clients 200.

[0044] Network 400 serves as a medium to provide a communication connection between client 200 and server 300. Network 400 facilitates the exchange of information or data. For example... Figure 1 As shown, client 200 and server 300 can connect to network 400 and transmit information or data to each other through network 400. In some embodiments, network 400 can be any type of wired or wireless network, or a combination thereof. For example, network 400 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), and Bluetooth networks. TM ZigBee TM A network, a near-field communication (NFC) network, or a similar network. In some embodiments, network 400 may include one or more network access points. For example, network 400 may include wired or wireless network access points, such as base stations or internet exchange points, through which one or more components of client 200 and server 300 can connect to network 400 to exchange data or information.

[0045] It should be understood that Figure 1 The number of clients 200, servers 300, and networks 400 shown is merely illustrative. Depending on implementation needs, there can be any number of clients 200, servers 300, and networks 400.

[0046] It should be noted that the liveness detection method can be executed entirely on the client 200, entirely on the server 300, or partially on both the client 200 and the server 300. This paper will use the execution of the liveness detection method on the server 300 as an example for illustration.

[0047] Figure 2A hardware structure diagram of a computing device 600 provided according to an embodiment of this specification is shown. The computing device 600 can execute the liveness detection method described in this specification. The liveness detection method is described in other parts of this specification. When the liveness detection method is executed on a client 200, the computing device 600 can be the client 200. When the liveness detection method is executed on a server 300, the computing device 600 can be the server 300. When the liveness detection method can be executed partly on the client 200 and partly on the server 300, the computing device 600 can be both the client 200 and the server 300.

[0048] like Figure 2 As shown, the computing device 600 may include at least one storage medium 630 and at least one processor 620. In some embodiments, the computing device 600 may also include a communication port 650 and an internal communication bus 610. Additionally, the computing device 600 may include I / O components 660.

[0049] The internal communication bus 610 can connect different system components, including storage medium 630, processor 620 and communication port 650.

[0050] I / O component 660 supports input / output between computing device 600 and other components.

[0051] Communication port 650 is used for data communication between computing device 600 and external sources. For example, communication port 650 can be used for data communication between computing device 600 and network 400. Communication port 650 can be a wired communication port or a wireless communication port.

[0052] Storage medium 630 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 632, a read-only storage medium (ROM) 634, or a random access storage medium (RAM) 636. Storage medium 630 also includes at least one instruction set stored in the data storage device. The instructions are computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the liveness detection method provided in this specification.

[0053] At least one processor 620 can be communicatively connected to at least one storage medium 630 and a communication port 650 via an internal communication bus 610. At least one processor 620 is used to execute the at least one instruction set described above. When the computing device 600 is running, at least one processor 620 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the liveness detection method provided in this specification. Processor 620 can execute all the steps included in the liveness detection method. Processor 620 can be in the form of one or more processors. In some embodiments, processor 620 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof. For illustrative purposes only, only one processor 620 is described in this specification for the computing device 600. However, it should be noted that the computing device 600 may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described herein, or they may be executed jointly by multiple processors. For example, if processor 620 of the computing device 600 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 620 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0054] Figure 3 A flowchart of a liveness detection method P100 according to an embodiment of this specification is shown. As previously described, computing device 600 can execute the liveness detection method P100 of this specification. Specifically, processor 620 can read an instruction set stored in its local storage medium and then execute the liveness detection method P100 of this specification according to the instructions in the instruction set. Figure 3 As shown, method P100 may include:

[0055] S120: Obtain the target video for the target user.

[0056] For an introduction to the target user, please refer to the preceding content; it will not be repeated here. The target video is obtained by compressing the original video captured from the target user. When the target user's biometrics are triggered, the image acquisition module captures the user's original biometric video and compresses it to obtain the target video. The original video can be in any modality, such as RGB, NIR, or Depth modality. The target video is obtained by compressing the original video using video compression technology. Video compression technology can include at least one of the following video compression standards: MPEG, H.261, H.263, and H.264, to convert the original video file into another video format, thereby reducing the bandwidth occupied during transmission and the storage space required for storage.

[0057] The original video comprises multiple frames of original images, each frame containing the biometric features of the target user, i.e., the object of liveness detection. Biometric features are inherent physiological characteristics of the human body, which may include at least one of the following: face, iris, sclera, fingerprint, palm print, voiceprint, and skeletal projection; they may also include other inherent physiological characteristics of the human body capable of facial recognition. For ease of description, this specification will use the face as an example of biometric features. Those skilled in the art should understand that other biometric features are also within the scope of protection of this specification.

[0058] The methods for obtaining the target video may include the following: for example, receiving the target video of the target user uploaded through a local device or terminal device, or obtaining the target video from the storage address based on the received liveness detection request carrying the storage address of the target user's target video.

[0059] S140: Partially decompress the target video to obtain the compressed video domain.

[0060] The compressed domain video includes multiple image frames, each corresponding to either an I-frame or a non-I-frame compression type. Image frames with an I-frame compression type are fully decompressed images, while image frames with a non-I-frame compression type are residual maps and motion vector maps. Non-I-frames include at least one of P-frames and B-frames.

[0061] The residual map represents the difference between the image frame to be compressed and the compressed image frame. It can be obtained by subtracting the image frame to be compressed and the compressed image frame, for example, by difference operation.

[0062] Motion vector graphics represent the motion changes of the same target from a compressed image frame to a frame to be compressed.

[0063] Residual maps and motion vector maps can be obtained as follows: Taking H.264 as an example, in the target video, there are many similarities between adjacent macroblocks within an image, and also many similarities between two frames. Based on this characteristic, during video compression, the original video can be divided into multiple Groups of Pictures (GOPs). The first frame of each GOP is an I-frame image, and the images after the I-frame are non-I-frame images, such as P-frames or B-frames. For an I-frame, neighboring macroblocks adjacent to the macroblock to be compressed can be found in the compressed macroblocks of the I-frame (also called a keyframe or reference frame). Predictions are then made using some preset prediction modes to obtain the predicted blocks corresponding to the neighboring macroblocks. Finally, the predicted blocks are compared with the neighboring macroblocks to obtain the residual blocks corresponding to the neighboring macroblocks.

[0064] For non-I-frame images, the process involves iterating through each macroblock in the compressed image frame (which can be an I-frame, P-frame, or B-frame) to find macroblocks with high similarity to the macroblocks in the image to be compressed. These macroblocks are then used as prediction blocks. The difference between the macroblock to be compressed and the prediction block is calculated to obtain the residual block. The difference (x0-x1, y0-y1) between the coordinates (x0, y0) of the prediction block in the I-frame and the coordinates (x1, y1) of the macroblock to be compressed in the image to be compressed is called the motion vector. By performing the above process for each macroblock, the residual map and motion vector map corresponding to the image frame to be compressed can be obtained.

[0065] During full video decoding, I-frame images can be obtained by decompressing the compressed data of the I-frame. Non-I-frame images need to be obtained by reconstructing the image frame based on the residual map, motion vector map, and prediction image corresponding to the image frame to be compressed.

[0066] In this specification, partial decompression refers to decompressing the I-frame image while leaving the non-I-frame images uncompressed. That is, after decompressing the target video, the results are the reconstructed I-frame image, the residual image of the non-I-frames, the motion vector map, and the predicted image. Since partial decompression does not decompress the non-I-frame images, it can significantly reduce video decoding time. However, compared to fully decompressed video, partially decompressed compressed domain video suffers from information loss, which can affect subsequent liveness detection performance. To improve the liveness detection performance of compressed domain video while saving video decoding time, the following step S160 can also be used.

[0067] S160: Generate N uncompressed domain videos based on compressed domain videos.

[0068] Where N is an integer greater than 0. When N is greater than 1, the N uncompressed domain videos are multimodal videos, and each multimodal video corresponds to multiple modalities. Each modality corresponds to a modal video, which is the same as or close to the original video of the target video before compression.

[0069] As previously introduced, the original video can be of any modality. Compressing and partially decompressing the original video results in a partially decompressed video, unlike a fully decompressed video. To distinguish it from fully decompressed modal videos or the modality of the original video, the target video is considered to be in the compressed domain modality. Multimodal videos, on the other hand, are in the uncompressed domain modality.

[0070] The multiple modalities corresponding to a multimodal video may include at least one of RGB mode, NIR mode, and Depth mode, and may also include other modalities; this specification does not limit this. Those skilled in the art should understand that other modalities are also within the scope of protection of this specification. The multiple modalities corresponding to a multimodal video may include the modalities of the original video, or may not include the modalities of the original video; this embodiment does not limit this.

[0071] Figure 4 A flowchart illustrating a method for generating N uncompressed domain videos according to an embodiment of this specification is shown.

[0072] Figure 4 Corresponding to step S160. For example... Figure 4 As shown, step S160 may include:

[0073] S162: Use a modal generation model to generate M modal videos and their corresponding M confidence scores from the compressed domain video.

[0074] Among them, M modal videos are uncompressed domain videos, M modal videos correspond to M modalities, M is an integer greater than 0, and M is greater than or equal to N.

[0075] Step S162 includes: generating M modal videos from the compressed domain video using a modality generation model, and generating M confidence scores corresponding to the M modal videos. Currently, in the field of deep learning, models need to be trained before application. Therefore, before applying the modality generation model to generate M modal videos and their corresponding M confidence scores, the modality generation model needs to be trained. The execution entity of the training method for the model generation model can be the same as or different from the liveness detection method; this embodiment does not impose specific restrictions on this.

[0076] Figure 5 A flowchart illustrating a method for training a modality generation model according to embodiments of this specification is shown. Figure 5 As shown, the modality generation model can be trained using the following steps:

[0077] S162-2: Obtain the first training sample and its corresponding first label.

[0078] The first training sample includes multiple first compression domain training videos, which are obtained by partially decompressing the original compressed training videos. The first label includes M original video sets in M ​​modalities.

[0079] The first training sample can be obtained as follows: after obtaining multiple original training videos, compress the multiple original training videos, decompress the compressed multiple original training videos, and obtain the first training sample.

[0080] When obtaining multiple original training videos, videos of various attack materials (printed paper, mobile phone screens, masks, etc.) in RGB mode can be collected. For each type of attack material, 5-10 videos can be collected under different physical environments (e.g., lighting conditions). Additionally, liveness videos of multiple users in RGB mode (videos categorized as liveness) can be collected, for example, 5-10 videos of each user under different physical environments (e.g., lighting conditions). These collected attack material videos and user liveness videos are then used as multiple original training videos. Each video from the multiple videos corresponding to each type of attack material, or each video from the multiple videos corresponding to each user, is considered a single original training video.

[0081] It should be understood that the 5-10 videos mentioned here are illustrative examples, and those skilled in the art may select other numbers or ranges of videos as needed, and this specification does not impose any limitations on this.

[0082] After obtaining multiple original training videos, the video compression techniques described above can be used to compress each of the multiple original training videos to obtain multiple compressed original training videos.

[0083] By partially decompressing the multiple original training videos after compression, multiple first-compression domain training videos can be obtained.

[0084] Along with obtaining the first training sample, the first label also needs to be obtained. The first label can be obtained in the following way: For example, taking the original training video modality as RGB, multiple original training videos can be used as the original video set in the RGB modality. While collecting multiple original training videos in the RGB modality, M-1 original video sets in other modalities, such as NIR modality and Depth modality, can also be collected. That is, while collecting videos of various attack materials and liveness videos of multiple users in the RGB modality, videos of various attack materials and liveness videos of multiple users in other modalities will also be collected. The videos of various attack materials and liveness videos of multiple users in other modalities are used as M-1 original video sets, each original video set including videos of various attack materials and liveness videos of multiple users. After compression and full decompression of these M-1 original video sets and the original video sets in the RGB modality, the first label can be obtained, which serves as the supervision signal for training the preset modality generation model.

[0085] S162-4: Based on the first training sample and the first label, the preset modality generation model is trained with the constraint that the first comprehensive loss is less than the first preset value. During training, the first comprehensive loss constrains the prediction result output by the preset modality generation model to approach the first label.

[0086] In some embodiments, the first comprehensive loss may include a modality generation loss. The modality generation loss includes M sub-modality generation losses, corresponding to M modalities. Each of the M sub-modality generation losses constrains the difference between the training modality video output by the preset modality generation model and the corresponding original video during training. The modality generation loss aims to make the training modality video output by the preset modality generation model approximate the original video. Thus, a decompressed video close to that obtained by fully decompressing the target video can be obtained through deep learning. This achieves the goal of saving video decoding time while supplementing lost information in the compressed domain video, thereby improving liveness detection performance.

[0087] In some embodiments, the first comprehensive loss may further include a confidence regression loss. The confidence regression loss comprises M sub-confidence regression losses, corresponding to M modalities. Each of the M sub-confidence regression losses constrains the difference between the training confidence score output by the preset modality generation model and its corresponding true value during training. The true value corresponding to the confidence score is obtained based on its corresponding sub-modality generation loss.

[0088] Here, the true value corresponding to the confidence level can be obtained based on the difference between the pre-set confidence level and its corresponding sub-modality generation loss. For example, if the pre-set confidence level is 100, the true value corresponding to the confidence level = 100 - sub-modality generation loss. The smaller the sub-modality generation loss, the higher the confidence level. The purpose of the confidence level regression loss is to constrain the training confidence level to approach the true value.

[0089] The following section will describe the process of determining the generation loss of M sub-modes and the confidence regression loss, based on the network structure of the pre-defined modality generation model:

[0090] Figure 6 A schematic diagram of the structure of a preset modality generation model provided according to an embodiment of this specification is shown. For example... Figure 6 As shown, the preset modality generation model includes a preset feature encoding network, a preset modality generation network, and a preset modality confidence prediction network. The preset feature encoding network can be a CNN (Convolutional Neural Network), such as ResNet (Residual Network), VGGNet (A Deep Convolutional Network), or InceptionV3 (Asymmetric Convolutional Network). The preset modality generation network can be a Transformer. The preset modality confidence prediction network can be an MLP (Multilayer Perceptron).

[0091] The loss for generating M sub-modals and the confidence regression loss can be determined using the following steps:

[0092] 1. A preset feature encoding network is used to extract features from multiple training videos in the first compression domain to obtain multiple training features in the first compression domain.

[0093] In a set of multiple first-compression-domain training videos, each first-compression-domain training video corresponds to one of the multiple first-compression-domain training features. By using a pre-defined feature encoding network to extract features from each of the multiple first-compression-domain training videos, the first-compression-domain training feature corresponding to each first-compression-domain training video can be obtained.

[0094] Each training video in the first compression domain corresponds to an original training video before compression, which includes K first training images, where K is an integer greater than 0. For each GOP group, residual maps and motion vector maps can be added to the I-frame images. The residual maps and motion vector maps corresponding to the I-frame images are zero, and the I-frame images can be padded to other non-I-frame images. In this way, each first training image in the GOP group will correspond to an I-frame image, a residual map, and a motion vector map. Then, the I-frame image, residual map, and motion vector map corresponding to each first training image are input into a preset first feature extraction network for feature extraction.

[0095] The preset feature encoding network extracts features from each first compression domain training video, achieving the effect of obtaining K first sub-compression domain training features corresponding to K first training images, or at least a portion of first sub-compression domain training features corresponding to at least a portion of the first training images. These K first sub-compression domain training features or at least a portion of first sub-compression domain training features are used as the first compression domain training features corresponding to the first compression domain training video.

[0096] 2. A preset modality generation network is used to generate M training modal videos based on multiple first compression domain features.

[0097] The preset modality generation network can include M preset modality generation sub-networks, which can be regarded as M modality generation branches. Each of the M modality generation branches corresponds to one of the M modalities. Each modality generation branch is configured to generate a training modality video corresponding to that modality based on multiple first compression domain training features. For example, each of the M modality generation branches corresponds to one of the RGB, NIR, and Depth modalities, and the modalities corresponding to the M modality generation branches are not the same. By inputting the multiple first compression domain training features into the M modality generation branches, M training modality videos can be obtained.

[0098] In some embodiments, each of the M training modal videos may include K frames of RGB modal images, K frames of NIR modal images, or K frames of Depth modal images. In some embodiments, each of the M training modal videos may include at least a portion of RGB modal images, at least a portion of NIR modal images, or at least a portion of Depth modal images.

[0099] 3. A pre-defined modality confidence prediction network is used to determine the confidence of M training modality videos, resulting in M ​​training confidences.

[0100] Here, the preset modality confidence prediction network can include a single confidence prediction branch or M confidence prediction branches. When it includes a single branch, M training modality videos can be sequentially input into the preset modality confidence prediction network to determine the modality confidence, thereby obtaining M training confidences. When it includes M confidence prediction branches, the M confidence prediction branches correspond to M modalities. Each confidence prediction branch receives the training modality video under its corresponding modality and evaluates the confidence of the training modality video under that modality, thereby obtaining the training confidence corresponding to the training modality video under that modality. Compared with a single confidence prediction branch, M confidence prediction branches determine the confidence faster, which can further improve the efficiency of liveness detection.

[0101] The higher the training confidence of the corresponding modality output by the preset modality confidence prediction network, the better the image quality of the M training modality videos generated by the preset modality generation network, and the closer they are to the original video under the corresponding modality. This makes them more suitable for subsequent multimodal liveness detection, thereby improving the performance of multimodal liveness detection.

[0102] 4. For each of the M modalities, based on the difference between the training modal video corresponding to each modality and the original video set, determine the submodal generation loss corresponding to each modality, and based on the difference between the training confidence corresponding to each modality and its corresponding true value, determine the subconfidence regression loss corresponding to each modality.

[0103] After determining the M sub-modal generation losses and M sub-confidence regression losses, backpropagation can be performed based on these losses to update the parameters of the preset modality generation network. The above four training steps are then repeated until training is complete, resulting in the modality generation model. The training can end when the first comprehensive loss is less than a first preset value, the number of training iterations reaches a preset number, or the model accuracy reaches a preset accuracy.

[0104] There are several ways to perform backpropagation on a predefined modality generation model, as follows:

[0105] In some embodiments, backpropagation can be performed based on M sub-modal generation losses and M sub-confidence regression losses respectively. For example, the parameters of a preset modality generation network and a preset feature encoding network can be updated based on the M sub-modal generation losses, and the parameters of a preset confidence evaluation network, a preset modality generation network, and a preset feature encoding network can be updated based on the M sub-confidence regression losses.

[0106] Specifically, when updating the parameters of the preset mode generation network and the preset feature encoding network based on the M sub-mode generation losses, backpropagation is performed based on each of the M sub-mode generation losses to update the parameters of the preset mode generation network and the preset feature encoding network. For example, the parameters of the mode generation branch corresponding to the RGB mode can be updated based on the sub-mode generation loss corresponding to the RGB mode, the parameters of the mode generation branch corresponding to the NIR mode can be updated based on the sub-mode generation loss corresponding to the NIR mode, and the parameters of the mode generation branch corresponding to the Depth mode can be updated based on the sub-mode generation loss corresponding to the Depth mode.

[0107] When backpropagating based on M sub-confidence regression losses, the confidence prediction network for the preset modality can include either a single confidence prediction branch or M confidence prediction branches, which can be implemented in different ways. Specifically:

[0108] 1. When the preset modality confidence prediction network includes a single confidence prediction branch, the M sub-confidence regression losses can be weighted and summed, and the result of the weighted summation can be used as the confidence regression loss; and backpropagation can be performed based on the confidence regression loss to update the parameters of the preset modality confidence prediction network, the M preset modality generation sub-networks and the preset feature encoding network.

[0109] 2. When the preset mode confidence prediction network includes M confidence prediction branches, backpropagation can be performed based on each of the M sub-confidence regression losses to update the parameters of the corresponding mode confidence prediction branch, the corresponding mode preset mode generation sub-network, and the preset feature encoding network.

[0110] A modality generation model trained using deep learning can achieve near-full decompression of the target video; therefore, it can be considered a pseudo-decoder. In other words, by extracting features from each first compression domain training video using the modality generation model, the desired effect is to obtain K first sub-compression domain training features corresponding to K frames of the first training images.

[0111] In addition, based on Figure 6 The structure of the preset modality generation model shown is simple and consists of lightweight networks. Therefore, generating N uncompressed domain videos using deep learning takes less time than fully decompressing the target video. Furthermore, it can enrich the information of the input video used for liveness detection on top of the compressed domain videos, thus improving liveness detection performance.

[0112] In some embodiments, joint training can also be performed based on M sub-modal generation losses and M sub-confidence regression losses. For example, the modal generation loss is obtained by weighted summation of the M sub-modal generation losses, and the confidence regression loss is obtained by weighted summation of the M sub-confidence regression losses. The first comprehensive loss is obtained by weighted summation of the modal generation loss and the confidence regression loss, and the parameters of the preset modal confidence prediction network, the M preset modal generation sub-networks, and the preset feature encoding network are updated based on the first comprehensive loss.

[0113] It should be understood that the above-described implementation of backpropagation is an illustrative example. In addition to the above-described implementation of backpropagation, this specification may also include other implementations that can update the parameters of the preset mode generation network.

[0114] After training the modality generation model, the computing device 600 can input the compressed domain video into the modality generation model to generate M uncompressed domain videos corresponding to the compressed domain video. After obtaining the M uncompressed domain videos, the computing device 600 can determine N uncompressed domain videos based on the M uncompressed domain videos.

[0115] like Figure 4 As shown, step S160 may further include:

[0116] S164: Select N modal videos from M modal videos as N uncompressed domain videos.

[0117] Among them, the confidence scores of N uncompressed domain videos satisfy preset rules. The preset rules include sorting the M confidence scores from highest to lowest, and selecting the top N videos with the highest confidence scores as the N uncompressed domain videos. In other words, after sorting the M confidence scores from highest to lowest, the N modal videos corresponding to the top N confidence scores are selected as the N uncompressed domain videos. For example, the modal video with the highest confidence score is selected as the uncompressed domain video. Alternatively, after sorting the M confidence scores from lowest to highest, the N modal videos corresponding to the bottom N confidence scores are selected as the N uncompressed domain videos.

[0118] The preset rules can also include a confidence level greater than a pre-set confidence level. That is, N modal videos from M modal videos whose confidence level is greater than the pre-set confidence level are selected as N uncompressed domain videos. Alternatively, N modal videos can be randomly selected from the M modal videos as N uncompressed domain videos. Or, the M modal videos can be used as N uncompressed domain videos.

[0119] like Figure 3 As shown, the method P100 may further include:

[0120] S180: Perform liveness detection based on compressed domain video and N uncompressed domain videos, obtain the liveness detection results and output them.

[0121] Step S180 may include the following steps: using a liveness detection model to extract features from compressed domain video and N uncompressed domain videos to obtain compressed domain features corresponding to the compressed domain video and N uncompressed domain features corresponding to the N uncompressed domain videos; and performing liveness detection based on the compressed domain features and N uncompressed domain features to obtain and output the liveness detection results.

[0122] By inputting compressed domain video and N uncompressed domain videos into the liveness detection model, the liveness detection model can perform fusion liveness detection based on the compressed domain video and N uncompressed domain videos, thereby outputting liveness detection results.

[0123] The process of inputting compressed domain video and N uncompressed domain video samples into a liveness detection model to obtain liveness detection results can include: inputting compressed domain video and N uncompressed domain video samples into the liveness detection model to obtain a liveness attack probability P', and determining the liveness detection result based on the liveness attack probability P'. For example, if the liveness attack probability P' is greater than a set threshold T', the target user is identified as an attack category; if the liveness attack probability P' is less than the set threshold T', the target user is identified as a live user.

[0124] It should be noted that when the probability of a live attack, P', is equal to the set threshold, the target user can be identified as either a live user or an attack user. This manual does not impose any restrictions on this.

[0125] There are several ways to output the liveness detection results. For example, the processor 620 can directly return the liveness detection results to the client 200 or terminal device of the target user 100, or it can return the liveness detection results to the verification device that needs to perform facial recognition or verification so that the verification device can perform facial recognition based on the liveness detection results, or it can also visualize the liveness detection results.

[0126] There are several ways to visualize the liveness detection result. For example, the processor 620 can display the liveness detection result through a display, or it can issue a prompt message about the liveness detection result through sound and light, etc.

[0127] As described above, the model needs to be trained before application. Therefore, before applying the liveness detection model for liveness detection, a liveness detection model needs to be trained. The execution entity of the liveness detection model can be the same as or different from the liveness detection method. Similarly, the execution entity of the liveness detection method can be the same as or different from the execution entity of the modality generation model. This specification does not impose any restrictions on this.

[0128] The liveness detection model can be trained using the following steps: obtaining a second training sample and its corresponding second label, and training a preset liveness detection model for liveness classification based on the second training sample and the second label, with the second comprehensive loss being less than a second preset value as the constraint objective, to obtain the liveness detection model.

[0129] The second training samples include multiple training videos in the second compressed domain and their corresponding multiple sets of training videos in the uncompressed domain. Each set of uncompressed domain training videos includes N uncompressed domain training videos. The second compressed domain training videos are obtained by partially decompressing the original compressed training videos. The second label includes a liveness classification label.

[0130] In other words, there is a one-to-one correspondence between multiple sets of training videos in the second compressed domain and multiple sets of training videos in the uncompressed domain. Each set of training videos in the uncompressed domain includes N uncompressed domain training videos. Thus, the relationship between the training videos in the second compressed domain, the training videos in the uncompressed domain, and the N uncompressed domain training videos can be derived as follows: each training video in the second compressed domain corresponds to one set of training videos in the uncompressed domain, and one set of training videos in the uncompressed domain corresponds to N uncompressed domain training videos.

[0131] Each training video in the second compressed domain and each training video in the uncompressed domain are labeled with a liveness classification label. The liveness classification label indicates whether the liveness classification for each training video in the second compressed domain is the liveness category or the attack category, and it also indicates whether the liveness classification for each training video in the uncompressed domain is the liveness category or the attack category.

[0132] There are several ways to obtain multiple uncompressed domain training video sets, such as: for example, using a modality generation model to extract features from each of the multiple second compressed domain training videos to obtain the features corresponding to each second compressed domain training video, and generating M uncompressed domain training videos based on each feature, and selecting N uncompressed domain training videos that meet preset rules from the M uncompressed domain training videos, thereby obtaining the uncompressed domain training video set corresponding to each compressed domain training video.

[0133] For example, the second training sample can be obtained in a similar way to the method described above for obtaining the first training sample and the first label. Please refer to the method for obtaining the first training sample and the first label for details, which will not be repeated here.

[0134] After obtaining the second training samples and their corresponding second labels, a liveness detection model can be trained based on the second training samples, their corresponding second labels, and the second comprehensive loss. The second comprehensive loss can at least include a liveness classification loss, which constrains the difference between the liveness detection training results output by the pre-defined liveness detection model and their corresponding liveness classification labels during training. The following will combine... Figure 7 and Figure 8 The network structures of the two preset liveness detection models shown illustrate the training process of the liveness detection models.

[0135] Figure 7 A schematic diagram of a pre-defined liveness detection model provided according to an embodiment of this specification is shown. Figure 7As shown, the preset liveness detection model includes a preset first feature extraction network, a preset second feature extraction network, and a preset feature fusion and liveness classification network. The preset first feature extraction network and the preset second feature extraction network can be networks such as ResNet (residual network), VGGNet (a deep convolutional network), or InceptionV3 (asymmetric convolutional network) used for feature extraction. The preset feature fusion and liveness classification network can be multiple fully connected layers + Softmax.

[0136] based on Figure 7 The liveness detection model shown can be obtained by the following method for liveness classification loss:

[0137] 1. A preset first feature extraction network is used to extract features from each of the multiple second compression domain training videos to obtain multiple second compression domain training features corresponding to the multiple second compression domain training videos, with each second compression domain training video corresponding to one second compression domain training feature.

[0138] 2. A pre-defined second feature extraction network is used to extract features from multiple uncompressed domain training video sets to obtain multiple uncompressed domain training feature sets. Each uncompressed domain training feature set includes N uncompressed domain training features.

[0139] 3. A pre-defined feature fusion and liveness classification network is used to fuse multiple training features from the second compressed domain and multiple training feature sets from the uncompressed domain. Liveness classification is then performed based on the fused features to obtain the liveness classification training results.

[0140] 4. Determine the liveness classification loss based on the difference between the liveness classification training results and their corresponding liveness classification labels.

[0141] Figure 8 A schematic diagram of another preset liveness detection model provided according to an embodiment of this specification is shown. Figure 8 As shown, the preset liveness detection model includes a preset first feature extraction network, a preset second feature extraction network, multiple preset liveness classification networks, and a liveness classification fusion network. The preset first feature extraction network and the preset second feature extraction network can be networks used for feature extraction, such as ResNet (residual network), VGGNet (a deep convolutional network), or InceptionV3 (asymmetric convolutional network). The multiple preset liveness classification networks correspond to multiple modalities and compression domain modalities. Each preset liveness classification network can be multiple fully connected layers plus Softmax, and the liveness classification fusion network can be an attention fusion network.

[0142] based on Figure 8The liveness detection model shown can also have its liveness classification loss obtained in the following ways:

[0143] 1. A preset first feature extraction network is used to extract features from each of the multiple second compression domain training videos to obtain multiple second compression domain training features corresponding to the multiple second compression domain training videos, with each second compression domain training video corresponding to one second compression domain training feature.

[0144] 2. A pre-defined second feature extraction network is used to extract features from multiple uncompressed domain training video sets to obtain multiple uncompressed domain feature sets. Each uncompressed domain feature set includes N uncompressed domain training features.

[0145] 3. Using the pre-set liveness classification network corresponding to the compressed domain mode among multiple pre-set liveness classification networks, liveness classification is performed based on the second compressed domain training features to obtain the compressed domain liveness classification training results. Also, using the pre-set liveness classification network corresponding to each of the N modes, liveness classification is performed based on the uncompressed domain training features corresponding to that mode to obtain the uncompressed domain liveness classification training results corresponding to that mode.

[0146] 4. Based on the fusion of the compressed domain liveness classification training results and the N uncompressed domain liveness classification training results corresponding to the N modalities, the liveness classification training results are determined.

[0147] Here, fusion can be a weighted sum of the training results of liveness classification in the compressed domain and the training results of liveness classification in N uncompressed domains.

[0148] 5. Determine the liveness classification loss based on the difference between the liveness classification training results and the liveness classification labels.

[0149] In addition, modality consistency loss information can be determined based on the differences between the N+1 liveness classification training results corresponding to the N modalities and the compressed domain modality. This constrains the differences between the liveness classification training results of the N modalities and the compressed domain modality to be within a preset range, so that the liveness classification results of different liveness classification training results are relatively close, thereby maintaining prediction consistency.

[0150] In some embodiments, the second integrated loss may further include at least one of compressed domain inter-frame consistency loss, uncompressed domain inter-frame consistency loss, and inter-modal consistency loss.

[0151] The compressed domain inter-frame consistency loss constrains the differences between different frames in the second compressed domain training features output by the pre-set liveness detection model during training. The compressed domain inter-frame consistency loss can be obtained as follows:

[0152] 1. Each second compression domain training video includes P frames of second training images. A preset first feature extraction network is used to extract features from each second compression domain training video to obtain P sub-second compression domain training features. Based on the P sub-second compression domain training features, the pairwise correlation between the P sub-second compression domain training features can be obtained, where P is an integer greater than 0.

[0153] The pairwise relationships between the P sub-second compressed domain training features can be obtained by, for example, by determining the pairwise similarity between the P sub-second compressed domain training features to obtain the pairwise relationships between the P sub-second compressed domain training features.

[0154] The pairwise similarity between the P sub-second compression domain training features can be cosine similarity, Euclidean distance, Pearson correlation coefficient, etc.

[0155] Among them, the pairwise similarity between the P sub-second compression domain training features can be represented as a P*P relation matrix.

[0156] 2. Based on the pairwise correlation between the training features of the P sub-second compression domains, determine the inter-frame consistency loss of the compression domain.

[0157] Specifically, the inter-frame consistency loss of the compressed domain can be determined based on the weighted sum of the differences between the pairwise similarities of the P sub-second compressed domain training features and a preset similarity value. For example, if P is 3, the similarities between the first and second sub-second compressed domain training features, the first and third sub-second compressed domain training features, and the second and third sub-second compressed domain training features are first determined. Then, the differences between these three similarities and the preset similarity value are calculated respectively to obtain three similarity differences. Finally, the inter-frame consistency loss of the compressed domain is determined based on the weighted sum of these three similarity differences. This inter-frame consistency loss of the compressed domain constrains the pairwise similarities of the P sub-second compressed domain training features to approach the preset similarity value.

[0158] In the P-frame second training images, the sub-second compressed domain training features corresponding to each frame of the second training image represent whether the extracted features are liveness features or attack features. For videos, if the liveness classification label corresponding to the video is live, then the liveness classification label corresponding to each frame of the video should also be live. Therefore, the inter-frame relationship consistency loss in the compressed domain can constrain the consistency of predictions for liveness detection results among compressed domain image frames.

[0159] The uncompressed domain inter-frame consistency loss constrains the differences between different frames in each of the N uncompressed domain training features output by the liveness detection model during training. The uncompressed domain inter-frame consistency loss can be obtained as follows:

[0160] N uncompressed domain training features correspond to N uncompressed domain training videos. Each of the N uncompressed domain training videos includes T frames of uncompressed domain training images. The computing device 600 can use a preset second feature extraction network to extract features from each of the N uncompressed domain training videos, obtaining T sub-uncompressed domain training features corresponding to each uncompressed domain training video. Each of the T sub-uncompressed domain training features corresponds to one of the N sub-uncompressed domain training features, where T is an integer greater than 0.

[0161] 1. Each uncompressed domain training video includes T frames of uncompressed domain training images. A preset first feature extraction network is used to extract features from each uncompressed domain training video to obtain T sub-uncompressed domain training features. Based on the T sub-uncompressed domain training features, the pairwise correlation between the T sub-uncompressed domain training features can also be obtained.

[0162] The pairwise relationships between the T sub-uncompressed domain training features can be obtained by determining the pairwise similarity between the T sub-uncompressed domain training features.

[0163] The pairwise similarity between T uncompressed subdomain training features can be cosine similarity, Euclidean distance, Pearson correlation coefficient, etc.

[0164] Among them, the pairwise similarity between any two of the T sub-uncompressed domain training features can be represented as a T*T relation matrix.

[0165] 2. Based on the pairwise correlation between the training features of the T sub-uncompressed domains, determine the inter-frame consistency loss of the uncompressed domain.

[0166] The uncompressed domain inter-frame consistency loss can be determined by weighted summing the differences between the pairwise similarities of the T sub-uncompressed domain training features and a preset similarity value. For example, if T is 3, the similarities between the first and second sub-uncompressed domain training features, the first and third sub-uncompressed domain training features, and the second and third sub-uncompressed domain training features are first determined. Then, the differences between these three similarities and the preset similarity value are calculated to obtain three similarity differences. Finally, the weighted sum of these three similarity differences is used to determine the uncompressed domain inter-frame consistency loss for a single modality. This uncompressed domain inter-frame consistency loss constrains the pairwise similarities of the T sub-uncompressed domain training features to approach the preset similarity value.

[0167] Similarly, the consistency loss of uncompressed domain inter-frame relationships can be used to constrain the consistency of predictions for liveness detection results among uncompressed domain image frames.

[0168] The inter-modal consistency loss constraint constrains the difference between the second compressed domain training feature and each of the N uncompressed domain training features during training. This inter-modal consistency loss constraint can be obtained as follows:

[0169] 1. Sequentially label each of the N modalities as the target modality, and determine the similarity between the sub-second compressed domain training features corresponding to each frame of the second training image in the P-frame second training image of the second compressed domain training video and the sub-uncompressed domain training features under the target modality corresponding to each frame of the second training image. Based on the difference between the determined similarity and the preset similarity, obtain the similarity difference between single frames under the target modality.

[0170] 2. The similarity differences between P frames in the target modality are weighted and summed to obtain the consistency loss between submodalities in the target modality.

[0171] By performing step 2 on each of the N modes, we can obtain the consistency loss among the N sub-modes.

[0172] 3. The intermodal consistency loss is obtained by weighted summation of the consistency losses among the N submodals.

[0173] The modal consistency loss constraint ensures that the difference between the second compressed domain training feature and each of the N uncompressed domain training features is within a preset difference range.

[0174] In the P-frame second training images, the sub-second compressed domain training features corresponding to each frame of the second training image represent whether the extracted features are liveness features or attack features. Similarly, the sub-uncompressed domain training features under the target modality also represent whether the extracted features are liveness features or attack features. By using modality consistency loss, the liveness detection results of the compressed and uncompressed domains can be constrained to be consistent, avoiding inconsistencies between the two liveness detection results, enhancing the complementarity between modalities, and achieving the purpose of collaborative liveness detection.

[0175] After determining the second comprehensive loss, backpropagation can be performed on the preset liveness detection model based on the second comprehensive loss to update the parameters of the preset liveness detection model. This process is repeated until training is complete, resulting in the liveness detection model. The conditions for training to end can be that the first comprehensive loss is less than the first preset value, the number of training iterations reaches a preset number, or the accuracy of the trained model reaches a preset accuracy, etc.

[0176] There are several ways to perform backpropagation on a pre-defined liveness detection model, as follows:

[0177] When performing backpropagation based on liveness classification loss, for Figure 7 In other words, it updates the parameters of the preset feature fusion and liveness classification network, the preset first feature extraction network, and the preset second feature extraction network based on the liveness classification loss. Figure 8 In other words, it updates the parameters of the liveness classification fusion network, multiple preset liveness classification networks, preset second feature extraction network, and preset first feature extraction network based on the liveness classification loss.

[0178] When backpropagation is performed based on at least one of compressed domain inter-frame consistency loss, uncompressed domain inter-frame consistency loss, and inter-modal consistency loss, the following seven cases can be included:

[0179] 1. When backpropagating based on the compressed domain inter-frame consistency loss, the parameters of the preset first feature extraction network are updated based on the compressed domain inter-frame consistency loss.

[0180] 2. When backpropagating based on the uncompressed domain inter-frame consistency loss, the parameters of the preset second feature extraction network are updated based on the uncompressed domain inter-frame consistency loss.

[0181] 3. When performing backpropagation based on intermodal consistency loss, the parameters of the preset first feature network and the preset second feature extraction network are updated based on the intermodal consistency loss.

[0182] 4. When backpropagating based on the compressed domain inter-frame consistency loss and the uncompressed domain inter-frame consistency loss, the parameters of the preset first feature network and the preset second feature extraction network can be updated based on the weighted sum of the compressed domain inter-frame consistency loss and the uncompressed domain inter-frame consistency loss.

[0183] 5. When performing backpropagation based on the compressed domain inter-frame consistency loss and the inter-modal consistency loss, the parameters of the preset first feature network and the preset second feature extraction network can be updated based on the weighted sum of the compressed domain inter-frame consistency loss and the uncompressed domain inter-frame consistency loss.

[0184] 6. When backpropagating based on the uncompressed domain inter-frame consistency loss and the inter-modal consistency loss, the parameters of the preset first feature network and the preset second feature extraction network can be updated based on the weighted sum of the compressed domain inter-frame consistency loss and the uncompressed domain inter-frame consistency loss.

[0185] 7. When backpropagating based on the compressed domain inter-frame consistency loss, the uncompressed domain inter-frame consistency loss, and the inter-modal consistency loss, the parameters of the preset first feature network and the preset second feature extraction network can be updated based on the weighted sum of the compressed domain inter-frame consistency loss, the uncompressed domain inter-frame consistency loss, and the inter-modal consistency loss.

[0186] After training the liveness detection model based on the above training process, the compressed domain video and N uncompressed domain videos can be input into the liveness detection model to perform liveness detection and obtain the liveness detection results.

[0187] To improve liveness detection performance while saving video decompression time, this paper proposes a multi-frame liveness detection method based on cross-modal image generation in the compressed domain. The overall flowchart of this method is shown below. Figure 9 As shown, it can include four parts: data acquisition and preprocessing, modality generation, multimodal multiframe liveness detection model training, and model deployment and liveness detection, which can be specifically as follows:

[0188] 1. Data Acquisition and Preprocessing

[0189] This section primarily involves collecting various attack materials and video data from multiple users, compressing this collected video data, and uploading it to the server. The server then performs partial and full decompression to obtain training data.

[0190] Partial decompression can be achieved by improving the MPEG-4 decoder to support partial decoding, i.e., decoding I-frames but not non-I-frames. Alternatively, the compressed video data can be fully decompressed using the MPEG-4 decoder to obtain modal video data with the same modality as the original acquired data, which can then be used as a supervisory signal for training the preset modality generation model.

[0191] 2. Modal generation

[0192] In video-based liveness detection methods, a video clip of the user is often collected for liveness detection. Compared to images, videos contain more information, thus offering better liveness detection performance and are widely used in scenarios with high security requirements. However, these methods involve full video decompression, which is time-consuming, resulting in a long processing time and a poor user experience in biometrics. This solution uses partial decompression of the target video to obtain compressed domain video, saving time compared to full decompression. However, this also results in the loss of some information, leading to less information in the compressed domain video. Therefore, a deep learning-based pseudo-decoder is trained to generate uncompressed domain videos in multiple modalities to enrich the input video, thereby improving liveness detection performance while maintaining efficiency. 3. Training of Multimodal Multi-Frame Liveness Detection Model

[0193] The liveness detection model is trained based on the generated video data of multiple modalities. The specific training process can be found in the preceding description and will not be repeated here.

[0194] 4. Model Deployment and Liveness Detection

[0195] The trained liveness detection model is deployed to the server and applied to liveness detection.

[0196] In summary, the liveness detection method P100 and system 001 provided in this specification, after acquiring and compressing the original video collected from the target user, partially decompress the target video to obtain a compressed domain video, generate N uncompressed domain videos based on the compressed domain video, and perform liveness detection based on the compressed domain video and the N uncompressed domain videos to obtain and output the liveness detection result. The method and system partially decompress the target video, saving time compared to full video decompression. Furthermore, generating N uncompressed domain videos based on the compressed domain video compensates for the information loss caused by the compressed domain video, thereby improving liveness detection performance.

[0197] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for performing liveness detection. When the executable instructions are executed by a processor, they instruct the processor to implement the steps of the liveness detection method P100 described herein. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on a computing device 600, the program code causes the computing device 600 to perform the steps of the liveness detection method P100 described herein. The program product for implementing the above method may employ a portable compact disk read-only memory (CD-ROM) containing program code and may run on the computing device 600. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on computing device 600, partially on computing device 600, as a standalone software package, partially on computing device 600 and partially on a remote computing device, or entirely on a remote computing device.

[0198] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0199] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0200] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0201] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0202] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. All contents used for all purposes, except for any history of prosecution documents relating to it, that may be inconsistent with or conflict with this document, or any such history of prosecution documents that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0203] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for live detection, comprising: obtaining a target video of a target user, the target video being obtained by compressing a raw video collected from the target user; partially decompressing the target video to obtain a compressed domain video, the compressed domain video comprising a plurality of image frames composed of a fully decompressed image, a residual image and a motion vector image; generating N non-compressed domain videos based on the compressed domain video, N being an integer greater than 0, the non-compressed domain video comprising a plurality of image frames in a non-compressed state; and performing live detection based on the compressed domain video and the N non-compressed domain videos to obtain a live detection result and output the live detection result.

2. The method of claim 1, wherein, The compressed domain video comprises a plurality of image frames, each image frame corresponding to a compression type of I frame or non-I frame, wherein the image frame corresponding to the compression type of I frame is the fully decompressed image, and the image frame corresponding to the compression type of non-I frame is the residual image and the motion vector image.

3. The method of claim 1, wherein, The generating of the N non-compressed domain videos based on the compressed domain video comprises: generating M modal videos and their corresponding M confidence levels using a modal generation model, wherein the M modal videos are non-compressed domain videos, the M modal videos correspond to M modalities, M being an integer greater than 0, and M is greater than or equal to N; and selecting N modal videos from the M modal videos as the N non-compressed domain videos, wherein the confidence levels of the N non-compressed domain videos satisfy a preset rule.

4. The method of claim 3, wherein, The preset rule comprises that the M confidence levels are sorted in descending order, and the top N of the sorted confidence levels.

5. The method of claim 3, wherein, The modal generation model is trained by the following steps: obtaining a first training sample and its corresponding first label, the first training sample comprising a plurality of first compressed domain training videos, the first compressed domain training video being obtained by partially decompressing a compressed raw training video, and the first label comprising M original video sets under the M modalities; and training a preset modal generation model based on the first training sample and the first label, with a first comprehensive loss being less than a first preset value as a constraint target, the first comprehensive loss constraining the prediction result output by the preset modal generation model to be close to the first label in the training.

6. The method of claim 5, wherein, The first comprehensive loss comprises: a modal generation loss comprising M sub-modal generation losses corresponding to the M modalities, each sub-modal generation loss in the M sub-modal generation losses constraining the difference between a training modal video output by the preset modal generation model and its corresponding original video in the training process.

7. The method of claim 6, wherein, The first comprehensive loss further comprises: a confidence level regression loss comprising M sub-confidence level regression losses corresponding to the M modalities, each sub-confidence level regression loss in the M sub-confidence level regression losses constraining the difference between a training confidence level output by the preset modal generation model and its corresponding true value in the training process, the true value corresponding to the training confidence level being obtained based on its corresponding sub-modal generation loss.

8. The method of claim 3, wherein, The living body detection is performed based on the compressed domain video and the N non-compressed domain videos, the living body detection result is obtained and outputted, and the living body detection method comprises the following steps: The living body detection model is trained by the following steps: Second training samples and corresponding second labels are obtained, the second training samples comprise a plurality of second compressed domain training videos and corresponding sets of a plurality of non-compressed domain training videos, each set of non-compressed domain training videos in the sets of a plurality of non-compressed domain training videos comprises N non-compressed domain training videos, the second compressed domain training video is obtained by partially decompressing a compressed original training video, and the second label comprises a living body classification label; 9. The method of claim 8, wherein, The preset living body detection model is trained based on the second training samples and the second labels, and a second comprehensive loss is less than a second preset value as a constraint target, to obtain the living body detection model, the second comprehensive loss at least comprises a living body classification loss, and the living body classification loss constrains a difference between a living body classification training result outputted by the preset living body detection model in a training process and a corresponding living body classification label. The second comprehensive loss further comprises at least one of a compressed domain inter-frame consistency loss, a non-compressed domain inter-frame consistency loss and an inter-modal consistency loss; The compressed domain inter-frame consistency loss constrains a difference between different frames in the second compressed domain training feature outputted by the preset living body detection model in the training process; 10. The method of claim 9, wherein, The non-compressed domain inter-frame consistency loss constrains a difference between different frames in each non-compressed domain training feature of the N non-compressed domain training features outputted by the living body detection model in the training process; and The inter-modal consistency loss constrains a difference between the second compressed domain training feature and each non-compressed domain training feature of the N non-compressed domain training features in the training process.

11. A living body detection system, comprising: at least one storage medium storing at least one instruction set for performing living body detection; and at least one processor in communication connection with the at least one storage medium, wherein, when the living body detection system is running, the at least one processor reads the at least one instruction set, and performs the living body detection method according to the indication of the at least one instruction set in any one of claims 1-10. ​ ​

Images