A method and system for face liveness recognition based on multimodal large model

Abstract

The present application relates to a method and system for face liveness recognition based on a multimodal large model, wherein the method comprises: labeling the acquired unlabeled data through a first multimodal large model to obtain automatically labeled data; obtaining liveness labeled data, and using the liveness labeled data and the automatically labeled data as training data for a second multimodal large model including a visual encoder and a text encoder; based on the training data and the text encoder, training and adjusting the network parameters of the visual encoder through a loss function; and finally adaptively deploying the trained second multimodal large model to perform face liveness recognition. Through the present application, unlabeled data labeling based on a multimodal large model is realized, which effectively reduces the data collection cost, and the multimodal model training based on text features and visual features enhances the representation ability of the model, thereby improving the reliability of liveness recognition, and solving the problem of how to perform safe and reliable face liveness recognition.

Description

Technical Field

[0001] The present application relates to the technical field of liveness recognition, and in particular to a method and system for face liveness recognition based on a multimodal large model. Background Art

[0002] Liveness recognition technology aims to distinguish between real organisms and forged biometric features. In recent years, it has been widely used in financial payment, identity authentication, security monitoring and other fields. However, liveness recognition still faces many challenges in practical applications, such as the diversity of forgery methods, the complexity of environmental factors, the sensitivity of facial data, and the high cost of data collection.

[0003] The current mainstream liveness recognition solutions include those based on image texture, those based on motion information, and those based on deep learning. However, these solutions still have certain limitations in dealing with the above challenges. For example, the recognition solution based on image texture is easily affected by factors such as changes in lighting and angles, and it is difficult to distinguish highly simulated forged images; the recognition solution based on motion information requires users to perform specific actions or operations, which has a poor user experience and is easily deceived by means such as video attacks; the recognition solution based on deep learning requires the collection of a large amount of private facial data for different scenarios and modalities, which has poor generalization and requires continuous investment for targeted iterative upgrades.

[0004] Currently, no effective solution has been proposed for the problem of how to perform safe and reliable face liveness recognition in related technologies. Summary of the invention

[0005] The embodiments of the present application provide a method and system for face liveness recognition based on a multimodal large model, so as to at least solve the problem of how to perform safe and reliable face liveness recognition in the related art.

[0006] In a first aspect, an embodiment of the present application provides a method for face liveness recognition based on a multimodal large model, the method comprising:

[0007] Based on the preset propmt prompt word, the acquired unlabeled data is labeled by the first multimodal large model to obtain automatically labeled data, wherein the categories of the automatically labeled data include a live attack category and a live real person category;

[0008] Acquire living body annotated data, and use the living body annotated data and the automatic annotated data as training data for a second multimodal large model, wherein the second multimodal large model includes a visual encoder and a text encoder;

[0009] Based on the training data and the text encoder, training and adjusting the network parameters of the visual encoder through a first loss function, a second loss function, and a third loss function;

[0010] Based on the hardware computing power of the device to be deployed, the trained second multimodal large model is adaptively deployed to perform live face recognition.

[0011] In some of the embodiments, the visual encoder includes a first visual encoder for extracting local visual features of a face image, and a second visual encoder for extracting global visual features of a face image.

[0012] In some embodiments, based on the training data and the text encoder, training and adjusting the network parameters of the visual encoder by using a first loss function, a second loss function, and a third loss function includes:

[0013] Based on the training data and the text encoder, the network parameters of the first visual encoder and the second visual encoder are cross-trained and adjusted through a first loss function, a second loss function and a third loss function.

[0014] In some embodiments, based on the training data and the text encoder, cross-training and adjusting the network parameters of the first visual encoder and the second visual encoder by using a first loss function, a second loss function, and a third loss function includes:

[0015] Based on the training data, training and adjusting network parameters of the first visual encoder by using a first loss function, wherein the first visual encoder is constructed based on a ResNet50 network;

[0016] Based on the training data and the local visual features extracted by the first visual encoder, training and adjusting the network parameters of the second visual encoder through a second loss function, wherein the second visual encoder is constructed based on a Vision Transformer network;

[0017] Based on the training data and the text features extracted by the text encoder, the network parameters of the second visual encoder are trained and adjusted through a third loss function, wherein the text encoder is constructed based on the Bert model.

[0018] In some embodiments, based on the training data, training and adjusting the network parameters of the first visual encoder by using a first loss function includes:

[0019] Based on the training data, through the first loss function:

[0020]

[0021] The network parameters of the first visual encoder are trained and adjusted, where α is a hyperparameter, z i_1 is the local visual feature logits of the i-th category extracted by the four-category branch in the first visual encoder, z j_1 is the local visual feature logits of the jth class extracted by the four-classification branch, z i_2 is the local visual feature logits of the i-th category extracted by the seventeen classification branches in the first visual encoder, z j_2 The local visual feature logits of the jth class extracted by the seventeen classification branches.

[0022] In some embodiments, based on the training data and the local visual features extracted by the first visual encoder, training and adjusting the network parameters of the second visual encoder by a second loss function includes:

[0023] Based on the training data and the local visual features extracted by the first visual encoder, through a second loss function:

[0024]

[0025] The network parameters of the second visual encoder are trained and adjusted, where z i_0 is the local visual feature logits of the i-th category extracted by the first visual encoder, z j_0 is the local visual feature logits of the jth class extracted by the first visual encoder, y i is the global visual feature logits of the i-th category extracted by the second visual encoder, y j It is the global visual feature logits of the jth class extracted by the second visual encoder.

[0026] In some embodiments, based on the training data and the text features extracted by the text encoder, training and adjusting the network parameters of the second visual encoder by a third loss function includes:

[0027] Based on the training data and the text features extracted by the text encoder, through the third loss function:

[0028]

[0029] The network parameters of the second visual encoder are trained and adjusted, where q img is the image feature, k txt_pos is the positive text description feature corresponding to the image, k txt_iis the text description feature with index i, τ is the model training hyperparameter, and batch_size is the batch file size set during training.

[0030] In some of the embodiments, based on the hardware computing power of the device to be deployed, adaptively deploying the trained second multimodal large model to perform face liveness recognition includes:

[0031] Based on the hardware computing power of the device to be deployed, if the hardware computing power is greater than the preset computing power threshold, the first visual encoder and the second visual encoder in the second multimodal large model are jointly deployed to the device to be deployed; if the hardware computing power is less than the preset computing power threshold, the first visual encoder in the second multimodal large model is deployed alone to the device to be deployed.

[0032] In some embodiments, based on the preset propmt prompt word, the acquired unlabeled data is labeled by the first multimodal large model to obtain the automatically labeled data, including:

[0033] Crawl unlabeled data from the Internet based on keywords;

[0034] Based on the preset propmt prompt word, the unlabeled data is labeled by a first multimodal large model to obtain automatically labeled data, wherein the first multimodal large model is an InternVL large model.

[0035] In a second aspect, an embodiment of the present application provides a face liveness recognition system based on a multimodal large model, the system is used to execute the method described in the first aspect above, the system includes a training data construction module, a model training module and a liveness recognition execution module;

[0036] The training data construction module is used to annotate the acquired unlabeled data through the first multimodal large model according to the preset propmt prompt words to obtain automatically annotated data, wherein the categories of the automatically annotated data include a live attack category and a live real person category;

[0037] The training data construction module is used to obtain the living body annotated data, and use the living body annotated data and the automatic annotated data as training data for the second multimodal large model, wherein the second multimodal large model includes a visual encoder and a text encoder;

[0038] The model training module is used to train and adjust the network parameters of the visual encoder through a first loss function, a second loss function and a third loss function according to the training data and the text encoder;

[0039] The liveness recognition execution module is used to adaptively deploy the trained second multimodal large model to perform face liveness recognition according to the hardware computing power of the device to be deployed.

[0040] Compared with the related art, the embodiment of the present application provides a method and system for face liveness recognition based on a multimodal large model, wherein the method annotates the acquired unlabeled data through a first multimodal large model based on a preset propmt prompt word to obtain automatically annotated data, wherein the categories of the automatically annotated data include live attack categories and live real person categories; obtains liveness annotated data, and uses the liveness annotated data and the automatically annotated data as training data for a second multimodal large model, wherein the second multimodal large model includes a visual encoder and a text encoder; based on the training data and the text encoder, the network parameters of the visual encoder are trained and adjusted through a first loss function, a second loss function, and a third loss function; based on the hardware computing power of the device to be deployed, the trained second multimodal large model is adaptively deployed to perform face liveness recognition, thereby realizing unlabeled data labeling based on a multimodal large model, effectively reducing the data collection cost, and training the multimodal model based on text features and visual features, enhancing the representation ability of the model, thereby improving the reliability of liveness recognition, and the flexible decoupled adaptive deployment can make full use of hardware resources, solving the problem of how to perform safe and reliable face liveness recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 is a flowchart of the steps of a method for face liveness recognition based on a multimodal large model according to an embodiment of the present application;

[0043] Figure 2 is a schematic diagram of a process of labeling unlabeled data according to an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of the process of training the second multimodal large model according to an embodiment of the present application;

[0045] Figure 4 is a schematic diagram of a flexible decoupling deployment of a model according to an embodiment of the present application;

[0046] Figure 5 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0048] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0049] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0050] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0051] The present application embodiment provides a method for face liveness recognition based on a multimodal large model. Figure 1 is a flowchart of the steps of the face liveness recognition method based on a multimodal large model according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0052] Step S102, based on the preset propmt prompt word, annotate the acquired unlabeled data through the first multimodal large model to obtain automatically annotated data, wherein the categories of the automatically annotated data include a live attack category and a live real person category;

[0053] Step S102 specifically includes the following steps:

[0054] Step S1021, crawling unlabeled data from the Internet based on keywords;

[0055] Specifically, step S1021 first divides the data into categories to obtain four subcategories: real person, 2D print attack (such as newspapers, photos, picture books, scanned photos), 2D screen attack (such as mobile phones, computers, televisions), and 3D attack (such as masks, dolls, wax figures). Then, data is crawled and collected based on the subcategories as keywords; the crawled unlabeled data is filtered out of the unfaced data, and then the face integrity and clarity are detected to obtain high-quality unlabeled data.

[0056] It should be noted that the training data of the second multimodal large model involved in this embodiment includes two parts, one part is the massive unlabeled data crawled from the Internet according to keywords in step S1021, and the other part is the labeled living body annotated data obtained in step S104.

[0057] Step S1022: Based on the preset propmt prompt word, the unlabeled data is labeled by using the first multimodal large model to obtain automatically labeled data, wherein the first multimodal large model is the InternVL large model.

[0058] Specifically, in step S1022, the preset propmt prompt word may be:

[0059] prompt = """

[0060] Please look at the picture carefully and answer the following questions:

[0061] Is the person in the picture a real person or a live attack?

[0062] If you are a real person, please answer in the following JSON format:

[0063] {

[0064] "answer": "real",

[0065] }

[0066] -If it is an attack, please answer in the following JSON format:

[0067] {

[0068] "answer": "fake",

[0069] "caption": Carefully describe the attack method, attack type, and attack material.

[0070] }

[0071] """

[0072] Based on the above propmt prompt word, the first multimodal large model can be used to determine whether the category of the unlabeled data is a living attack category or a living real person category, and automatically label it; the automatically labeled data determined to be a real person is directly discarded, and the automatically labeled data determined to be an attack is mapped to the four subcategories of step S1021 (i.e., real person, 2D printing attack, 2D screen attack, and 3D attack).

[0073] It should be noted that Figure 2 is a schematic diagram of the process of labeling unlabeled data according to an embodiment of the present application, such as Figure 2As shown in the figure, the massive unlabeled data is denoised and labeled - ① massive unlabeled data is crawled from the Internet according to keywords, ② quality filtering is performed to obtain qualified unlabeled data, ③ the first multimodal large model is used to determine whether the category of the unlabeled data is real person or attack (at the same time, data classified as attack is also collected and input into the model), ④ the data classified as attack is retained and mapped to four subcategories (namely real person, 2D printing attack, 2D screen attack, 3D attack). After completion, it is used as part of the training data to reduce the high cost of data collection and the risk of facial privacy leakage, while effectively increasing the diversity of attack samples.

[0074] Step S104, obtaining living body annotated data, and using the living body annotated data and the automatic annotated data as training data for a second multimodal large model, wherein the second multimodal large model includes a visual encoder and a text encoder;

[0075] Specifically, in step S104, the visual encoder includes a first visual encoder for extracting local visual features of a face image, and a second visual encoder for extracting global visual features of a face image.

[0076] Step S106, based on the training data and the text encoder, the network parameters of the visual encoder are trained and adjusted by using a first loss function, a second loss function, and a third loss function;

[0077] Specifically, step S106 performs cross-training adjustment on network parameters of the first visual encoder and the second visual encoder based on the training data and the text encoder through the first loss function, the second loss function and the third loss function.

[0078] It should be noted that in step S106, the image description embedded features (text features) and visual features are aligned, which greatly improves the representation ability of the face liveness model. At the same time, the local visual features and the global visual features are fully interacted to effectively improve the model's generalization ability.

[0079] Step S106 preferably includes the following steps:

[0080] Step S1061, based on the training data, training and adjusting the network parameters of the first visual encoder by using a first loss function, wherein the first visual encoder is constructed based on a ResNet50 network;

[0081] Specifically, step S1061 is based on the training data, through the first loss function:

[0082]

[0083] The network parameters of the first visual encoder are trained and adjusted, where α is a hyperparameter and z i_1is the local visual feature logits of the i-th category extracted by the four-category branch in the first visual encoder, z j_1 is the local visual feature logits of the jth class extracted by the four-classification branch, z i_2 is the local visual feature logits of the i-th category extracted by the seventeen classification branches in the first visual encoder, z j_2 The local visual feature logits of the jth class extracted by the seventeen classification branches.

[0084] It should be noted that the first visual encoder uses the ResNet50 network. Figure 3 is a flow chart of the second multimodal large model training according to an embodiment of the present application, such as Figure 3 As shown in the figure, the first visual encoder includes a ConvNet module (convolutional neural network module) and a Reshape module (used to change the shape of a tensor without changing its data, such as flattening the feature map into a one-dimensional vector in CNN for input to the fully connected layer). It is mainly responsible for extracting local features in the image through local perception and weight sharing mechanisms. It has a dual-branch architecture (four-classification branch and seventeen-classification branch), and introduces a hyperparameter α to achieve curriculum learning, with the granularity of true and false judgment from coarse to fine.

[0085] Step S1062, based on the training data and the local visual features extracted by the first visual encoder, training and adjusting the network parameters of the second visual encoder through a second loss function, wherein the second visual encoder is constructed based on the VisionTransformer network;

[0086] Specifically, step S1062 uses a second loss function based on the training data and the local visual features extracted by the first visual encoder:

[0087]

[0088] The network parameters of the second visual encoder are trained and adjusted, where z i_0 is the local visual feature logits of the i-th category extracted by the first visual encoder, z j_0 is the local visual feature logits of the jth class extracted by the first visual encoder, y i is the global visual feature logits of the i-th category extracted by the second visual encoder, y j It is the global visual feature logits of the jth class extracted by the second visual encoder.

[0089] It should be noted that the second visual encoder uses the ViT (Vision Transformer) network, which can extract true and false features from a global perspective through the self-attention mechanism. Figure 3As shown in the figure, the second visual encoder includes a SelfAttention module (a core component of the Transformer architecture, which allows the model to focus on different parts of the sequence when processing sequence data, thereby capturing long-distance dependencies) and a Feed Forward module (plays an important role in the Transformer architecture, usually consisting of two fully connected layers). The local visual features feature_local extracted by the first visual encoder are jointly learned with the global features feature_global extracted by the second visual encoder, which effectively promotes the flow of feature information of different scales and makes full use of the large capacity of the ViT network and the strong feature representation learning ability.

[0090] Step S1063, based on the training data and the text features extracted by the text encoder, the network parameters of the second visual encoder are trained and adjusted through a third loss function, wherein the text encoder is constructed based on the Bert model.

[0091] Specifically, step S1063 uses a third loss function based on the training data and the text features extracted by the text encoder:

[0092]

[0093] The network parameters of the second visual encoder are trained and adjusted, where q img is the image feature, k txt_pos is the positive text description feature corresponding to the image, k txt_i is the text description feature with index i, τ is the model training hyperparameter, and batch_size is the batch file size set during training.

[0094] It should be noted that the text encoder uses the Bert model to describe the image (such as Figure 3 The "picture is a prosthetic attack, the attack method is 2D, and the attack material is a printed poster") in this article is coded. Figure 3As shown in the figure, the text encoder includes a Bi-Self Attention module (a bidirectional self-attention module, an extended self-attention mechanism designed to consider both forward and backward contextual information), a Cross-Attention module (an important component of the Transformer architecture, especially in multimodal learning, machine translation, text generation and other tasks) and a Feed Forward module (plays an important role in the Transformer architecture, usually composed of two fully connected layers). The CrossAttention module is added to achieve cross-modal cross attention. The specific implementation is to introduce the global visual feature feature_global extracted by the second visual encoder as the query q into the text encoding stage, align the global visual feature and the text description in the feature space, and give full play to the visual feature extraction capability of the second visual encoder.

[0095] Step S108, based on the hardware computing power of the device to be deployed, adaptively deploy the trained second multimodal large model to perform live face recognition.

[0096] Specifically, in step S108, based on the hardware computing power of the device to be deployed, if the hardware computing power is greater than the preset computing power threshold, the first visual encoder and the second visual encoder in the second multimodal large model are jointly deployed to the device to be deployed; if the hardware computing power is less than the preset computing power threshold, the first visual encoder in the second multimodal large model is deployed alone to the device to be deployed.

[0097] It should be noted that Figure 4 is a schematic diagram of a flexible decoupling deployment of a model according to an embodiment of the present application, such as Figure 4 As shown in the figure, traditional liveness recognition requires training two sets of liveness algorithms for the server and device for deployment. The multimodal large model provided in this embodiment only needs to be trained once, and the routing is selected according to the computing power of the hardware during deployment, so as to balance the speed and accuracy requirements and achieve one-time training and multi-terminal deployment. The device side (low computing power) has a relatively high speed requirement, such as Figure 4 As shown in the middle right side (low computing power deployment), you can choose to deploy only the first visual encoder (ConvNet module and Reshape module); the server (high computing power) has higher requirements for accuracy. Figure 4 As shown on the left (high computing power deployment), you can choose to jointly deploy the first visual encoder (ConvNet module and Reshape module) and the second visual encoder (Self Attention module and Feed Forward module). It can be seen that based on the flexible decoupling deployment of the second multimodal large model, hardware resources can be fully utilized, and flexible decoupling based on different hardware resources can be deployed on multiple terminals to achieve the optimal model judgment effect.

[0098] Through the above steps in the embodiments of the present application, unlabeled data labeling based on a multimodal large model is achieved, which effectively reduces the data collection cost. Multimodal model training based on text features and visual features enhances the representation ability of the model, thereby improving the reliability of liveness recognition. Flexible decoupled adaptive deployment can make full use of hardware resources and solve the problem of how to perform safe and reliable face liveness recognition.

[0099] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0100] The embodiment of the present application provides a face liveness recognition system based on a multimodal large model, the system comprising a training data construction module, a model training module and a liveness recognition execution module;

[0101] A training data construction module is used to annotate the acquired unlabeled data through the first multimodal large model according to the preset propmt prompt words to obtain automatically annotated data, wherein the categories of the automatically annotated data include a live attack category and a live real person category;

[0102] A training data construction module, used for obtaining live body annotated data, and using the live body annotated data and the automatic annotated data as training data for a second multimodal large model, wherein the second multimodal large model includes a visual encoder and a text encoder;

[0103] A model training module, used for training and adjusting the network parameters of the visual encoder through a first loss function, a second loss function and a third loss function according to the training data and the text encoder;

[0104] The liveness recognition execution module is used to adaptively deploy the trained second multimodal large model for face liveness recognition according to the hardware computing power of the device to be deployed.

[0105] Through the training data construction module, model training module and liveness recognition execution module in the embodiments of the present application, label-free data labeling based on a multimodal large model is realized, which effectively reduces the data collection cost. Multimodal model training based on text features and visual features enhances the representation ability of the model, thereby improving the reliability of liveness recognition. The flexible and decoupled adaptive deployment can make full use of hardware resources and solve the problem of how to perform safe and reliable face liveness recognition.

[0106] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0107] This embodiment further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0108] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0109] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0110] In addition, in combination with the face liveness recognition method based on a multimodal large model in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any of the face liveness recognition methods based on a multimodal large model in the above embodiment is implemented.

[0111] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for face liveness recognition based on a multimodal large model is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball, or touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0112] In one embodiment, Figure 5 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 5 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 5 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement a human face liveness recognition method based on a multimodal large model, and the database is used to store data.

[0113] Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0115] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for face liveness recognition based on a multimodal large model, characterized in that: The method comprises: Based on the preset propmt prompt word, the acquired unlabeled data is labeled by the first multimodal large model to obtain automatically labeled data, wherein the categories of the automatically labeled data include a live attack category and a live real person category; Acquire living body annotated data, and use the living body annotated data and the automatic annotated data as training data for a second multimodal large model, wherein the second multimodal large model includes a visual encoder and a text encoder; The visual encoder comprises a first visual encoder for extracting local visual features of a face image, and a second visual encoder for extracting global visual features of a face image; Based on the training data, training and adjusting network parameters of the first visual encoder by using a first loss function, wherein the first visual encoder is constructed based on a ResNet50 network; Based on the training data and the local visual features extracted by the first visual encoder, training and adjusting the network parameters of the second visual encoder through a second loss function, wherein the second visual encoder is constructed based on a VisionTransformer network; Based on the training data and the text features extracted by the text encoder, training and adjusting the network parameters of the second visual encoder through a third loss function, wherein the text encoder is constructed based on the Bert model; Based on the hardware computing power of the device to be deployed, the trained second multimodal large model is adaptively deployed to perform live face recognition.

2. The method according to claim 1, characterized in that Based on the training data, training and adjusting the network parameters of the first visual encoder by using a first loss function includes: Based on the training data, through the first loss function: The network parameters of the first visual encoder are trained and adjusted, where α is a hyperparameter, z i_1 is the local visual feature logits of the i-th category extracted by the four-category branch in the first visual encoder, z j_1 is the local visual feature logits of the jth class extracted by the four-classification branch, z i_2 is the local visual feature logits of the i-th category extracted by the seventeen classification branches in the first visual encoder, z j_2 The local visual feature logits of the jth class extracted by the seventeen classification branches.

3. The method according to claim 1, characterized in that Based on the training data and the local visual features extracted by the first visual encoder, training and adjusting the network parameters of the second visual encoder by using a second loss function includes: Based on the training data and the local visual features extracted by the first visual encoder, through a second loss function: The network parameters of the second visual encoder are trained and adjusted, where z i_0 is the local visual feature logits of the i-th category extracted by the first visual encoder, z j_0 is the local visual feature logits of the jth class extracted by the first visual encoder, y i is the global visual feature logits of the i-th category extracted by the second visual encoder, y j It is the global visual feature logits of the jth class extracted by the second visual encoder.

4. The method according to claim 1, characterized in that: Based on the training data and the text features extracted by the text encoder, training and adjusting the network parameters of the second visual encoder by a third loss function includes: Based on the training data and the text features extracted by the text encoder, through the third loss function: The network parameters of the second visual encoder are trained and adjusted, where q img is the image feature, k txt_pos is the positive text description feature corresponding to the image, k txt_i is the text description feature with index i, τ is the model training hyperparameter, and batch_size is the batch file size set during training.

5. The method according to claim 1, characterized in that: Based on the hardware computing power of the device to be deployed, adaptive deployment of the trained second multimodal large model for face liveness recognition includes: Based on the hardware computing power of the device to be deployed, if the hardware computing power is greater than the preset computing power threshold, the first visual encoder and the second visual encoder in the second multimodal large model are jointly deployed to the device to be deployed; if the hardware computing power is less than the preset computing power threshold, the first visual encoder in the second multimodal large model is deployed alone to the device to be deployed.

6. The method according to claim 1, characterized in that Based on the preset propmt prompt word, the acquired unlabeled data is annotated by the first multimodal large model, and the automatically annotated data obtained includes: Crawl unlabeled data from the Internet based on keywords; Based on the preset propmt prompt word, the unlabeled data is labeled by a first multimodal large model to obtain automatically labeled data, wherein the first multimodal large model is an InternVL large model.

7. A face liveness recognition system based on a multimodal large model, characterized in that: The system is used to execute the method according to any one of claims 1 to 6, and the system comprises a training data construction module, a model training module and a living body recognition execution module; The training data construction module is used to annotate the acquired unlabeled data through the first multimodal large model according to the preset propmt prompt words to obtain automatically annotated data, wherein the categories of the automatically annotated data include a live attack category and a live real person category; The training data construction module is used to obtain the living body annotated data, and use the living body annotated data and the automatic annotated data as training data for the second multimodal large model, wherein the second multimodal large model includes a visual encoder and a text encoder; The model training module is used to train and adjust the network parameters of the visual encoder through a first loss function, a second loss function and a third loss function according to the training data and the text encoder; The liveness recognition execution module is used to adaptively deploy the trained second multimodal large model to perform face liveness recognition according to the hardware computing power of the device to be deployed.

Images