Living body detection model training method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a living body detection model training method and device, equipment and a medium, and is used for improving generalization of an initial living body detection model so as to improve the accuracy of living body detection. The method comprises the following steps: in each round of iterative training of an initial living body detection model, performing feature extraction on a selected sample image, a content text and a classification text to obtain a sample image feature, a content text feature and an initial classification text feature; performing first feature fusion processing on the sample image feature and the content text feature to obtain a target image feature, and performing second feature fusion processing on the target image feature and the initial classification text feature to obtain a target classification text feature; and obtaining a living body detection result based on an initial classification probability determined by the target image feature and the similarity between the target image feature and the target classification text feature, and performing model parameter adjustment based on the difference between the living body detection result and the sample label.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, device, and medium for a liveness detection model. Background Technology

[0002] With the implementation of object recognition systems in production and daily life, object recognition algorithms are widely used. Liveness detection is a crucial step in object recognition algorithms, directly related to security issues in object recognition.

[0003] In related technologies, a trained liveness detection model is typically used to detect liveness in captured object images. Current liveness detection models usually consist of a feature extractor and a classifier; the feature extractor aims to transform the object image into high-dimensional features, and then these high-dimensional features are input into the classifier to obtain the liveness detection result.

[0004] However, in real-world scenarios, images of different objects may exhibit significant differences. These differences can stem from the object itself, environmental factors (such as lighting and background), and the shooting equipment. Consequently, when using a liveness detection model to detect liveness in real-world object images, the model's insufficient generalization ability can lead to low accuracy in liveness detection.

[0005] Therefore, improving the generalization ability of the liveness detection model to increase the accuracy of liveness detection is a problem that needs to be solved. Summary of the Invention

[0006] This application provides a training method, apparatus, device, and medium for a liveness detection model to improve the generalization of the liveness detection model and thus improve the accuracy of liveness detection.

[0007] On one hand, embodiments of this application provide a method for training a liveness detection model, the method comprising:

[0008] Obtain a sample dataset; wherein each sample dataset includes: a sample image, and a sample label indicating whether the sample image contains a live organism;

[0009] The initial liveness detection model is trained iteratively through multiple rounds based on the aforementioned sample dataset to obtain the target liveness detection model. Each iteration includes:

[0010] Feature extraction is performed on the sample images, content text, and classification text selected from the sample dataset to obtain the corresponding sample image features, content text features, and initial classification text features. The classification text contains content describing the liveness status of the target object, which includes whether the object is alive or not. The content text contains content describing the attribute information of the target object.

[0011] The sample image features and the content text features are subjected to a first feature fusion process to obtain target image features, and the target image features and the initial classification text features are subjected to a second feature fusion process to obtain target classification text features;

[0012] Based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features, the liveness detection result of the selected sample image is obtained, and the model parameters are tuned based on the difference between the liveness detection result and the corresponding sample label.

[0013] On one hand, embodiments of this application provide a liveness detection method, executed using a target liveness detection model, the method comprising:

[0014] Feature extraction is performed on the object image, content text, and classification text to be detected, respectively, to obtain corresponding object image features, content text features, and initial classification text features; the classification text contains content describing the liveness status of the target object, which includes live or non-liveness, and the content text contains content describing the attribute information of the target object;

[0015] The object image features and the content text features are subjected to a first feature fusion process to obtain target image features, and the target image features and the initial classification text features are subjected to a second feature fusion process to obtain target classification text features;

[0016] Based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features, a liveness detection result for the object image is obtained.

[0017] On one hand, embodiments of this application provide a training apparatus for a liveness detection model, the apparatus comprising:

[0018] The acquisition module is used to acquire a sample dataset; wherein each sample dataset includes: a sample image, and a sample label indicating whether the sample image contains a live organism;

[0019] The training module is used to perform multiple rounds of iterative training on the initial liveness detection model based on the sample dataset to obtain the target liveness detection model, wherein each round of iteration includes:

[0020] Feature extraction is performed on the sample images, content text, and classification text selected from the sample dataset to obtain the corresponding sample image features, content text features, and initial classification text features. The classification text contains content describing the liveness status of the target object, which includes whether the object is alive or not. The content text contains content describing the attribute information of the target object.

[0021] The sample image features and the content text features are subjected to a first feature fusion process to obtain target image features, and the target image features and the initial classification text features are subjected to a second feature fusion process to obtain target classification text features;

[0022] Based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features, the liveness detection result of the selected sample image is obtained, and the model parameters are tuned based on the difference between the liveness detection result and the corresponding sample label.

[0023] On one hand, the present application provides a liveness detection device that utilizes a target liveness detection model. The device includes:

[0024] The feature extraction module is used to extract features from the object image, content text, and classification text to be detected, respectively, to obtain corresponding object image features, content text features, and initial classification text features; the classification text contains content describing the liveness status of the target object, the liveness status including live or non-live, and the content text contains content describing the attribute information of the target object;

[0025] The feature fusion module is used to perform a first feature fusion process on the object image features and the content text features to obtain target image features, and to perform a second feature fusion process on the target image features and the initial classification text features to obtain target classification text features;

[0026] The liveness detection module is used to obtain a liveness detection result for the object image based on an initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features.

[0027] On one hand, an electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the above-mentioned training method or liveness detection method for the liveness detection model.

[0028] On one hand, embodiments of this application provide a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of the above-described training method or liveness detection method for the liveness detection model.

[0029] On one hand, embodiments of this application provide a computer program product, the computer program product including a computer program stored in a computer-readable storage medium; when the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the above-described training method or liveness detection method for the liveness detection model.

[0030] The solution proposed in this application has at least the following beneficial effects:

[0031] This application provides a training method, apparatus, device, and medium for a liveness detection model. When training the initial liveness detection model, the model input includes not only sample images but also content text and classification text. First, sample image features, content text features, and initial classification text features are extracted. Since the content text contains information describing the attribute information of the target object, the content text features can represent the attribute features of the target object. By fusing the sample image features and content text features using a first feature fusion, the attribute features of the target object can be integrated into the sample image features, effectively strengthening the sample image features and making the strengthened target image features more suitable for liveness detection tasks. Simultaneously, since the classification text contains information describing the liveness status of the target object, which includes liveness or non-liveness, the initial classification text features can represent the classification category features. By using the target image features as dynamic cues and fusing them with the initial classification text features using a second feature fusion, the initial classification text features are adaptively adjusted to obtain customized classification category features for the sample image, i.e., target classification text features.

[0032] Finally, based on the target image features more suitable for liveness detection tasks, the initial classification probability can be determined more accurately. Furthermore, by comprehensively considering this initial classification probability and the similarity between the target image features and the customized classification category features for the sample image, the liveness detection result corresponding to the sample image can be obtained adaptively and accurately. Through the above training process, the initial liveness detection model is tuned to obtain a target liveness detection model that can adapt to different object images, improving the model's generalization ability. This allows for accurate detection not only of sample images but also of object images in practical applications, thereby improving the accuracy of liveness detection.

[0033] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0035] Figure 1 This is a schematic diagram illustrating one application scenario in an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of the structure of an initial liveness detection model in an embodiment of this application;

[0037] Figure 3 This is a flowchart illustrating a training method for a liveness detection model in an embodiment of this application.

[0038] Figure 4 This is a schematic diagram of input data for an initial liveness detection model in an embodiment of this application;

[0039] Figure 5 This is a schematic diagram illustrating the training process of a liveness detection model in an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of feature filtering in an embodiment of this application;

[0041] Figure 7 This is a schematic diagram of feature fusion in an embodiment of this application;

[0042] Figure 8 This is another feature fusion diagram in the embodiments of this application;

[0043] Figure 9 This is a flowchart illustrating a liveness detection method in an embodiment of this application;

[0044] Figure 10 This is a schematic diagram of the overall logic of a liveness detection method in an embodiment of this application;

[0045] Figure 11 This is a schematic diagram of the composition of a training device for a liveness detection model according to an embodiment of this application;

[0046] Figure 12 This is a schematic diagram of the composition of a liveness detection device according to an embodiment of this application;

[0047] Figure 13 This is a schematic diagram of the composition structure of an electronic device according to an embodiment of this application;

[0048] Figure 14 This is a schematic diagram of the composition structure of another electronic device using an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0050] The following describes some of the concepts involved in the embodiments of this application.

[0051] Self-attention is a key technique in deep learning, particularly widely used in natural language processing. It allows models to focus on different parts of the input features and dynamically assign weights based on the relevance of these parts. This enables the model to better understand the relationships between the various feature elements in the input. This mechanism allows models to more effectively understand and generate complex sentence structures. In short, self-attention assigns different weights based on the varying importance of different parts of the input features.

[0052] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0053] Liveness detection is a crucial step in object recognition algorithms, directly impacting security. Related technologies typically employ trained liveness detection models to perform liveness detection on captured object images. Current liveness detection models generally consist of a feature extractor and a classifier; the feature extractor aims to transform the object image into high-dimensional features, which are then input into the classifier to obtain the liveness detection result. However, in real-world scenarios, different captured object images may exhibit significant differences, stemming from the object itself, environmental factors (e.g., lighting, background), and the capturing device. Thus, when using liveness detection models to perform liveness detection on object images in real-world scenarios, the insufficient generalization ability of the liveness detection models leads to low accuracy.

[0054] In view of this, in order to improve the generalization of the liveness detection model and thus improve the accuracy of liveness detection, this application provides a training method, apparatus, device, and medium for a liveness detection model. During the initial training process of the liveness detection model, by fusing sample image features with content text features as a first feature, the attribute features of the target object can be integrated into the sample image features, effectively strengthening the sample image features and making the strengthened target image features more suitable for the liveness detection task. Simultaneously, by fusing the target image features with the initial classification text features as a second feature, the initial classification text features are adaptively adjusted to obtain customized classification category features for the sample image, i.e., target classification text features. Finally, based on the aforementioned target image features more suitable for the liveness detection task, the initial classification probability can be determined more accurately. Furthermore, by comprehensively considering the initial classification probability and the similarity between the target image features and the customized classification category features for the sample image, the liveness detection result corresponding to the sample image can be adaptively and accurately obtained. By tuning the initial liveness detection model through the above training process, a target liveness detection model that can adapt to different object images can be obtained, improving the generalization of the model and thus improving the accuracy of liveness detection.

[0055] like Figure 1 The diagram shown illustrates an application scenario of an embodiment of this application. The application scenario diagram includes a terminal device 110 and a server 120. The terminal device 110 and the server 120 can communicate via a communication network.

[0056] In one alternative implementation, the communication network can be a wired network or a wireless network. Therefore, the terminal device 110 and the server 120 can be directly or indirectly connected via wired or wireless communication. This application embodiment does not impose specific limitations here.

[0057] In this application embodiment, the terminal device 120 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, access control devices, vehicle terminals, and other devices. The terminal device may have a client installed, which can use liveness detection functionality in some business scenarios. This client can be software (e.g., shopping software, social software, financial service software, security software, etc.), or it can be a webpage, mini-program, etc. The server 120 can be a backend server corresponding to the software, webpage, mini-program, etc., or a server specifically used for training liveness detection models and performing liveness detection; this application does not impose specific limitations. The server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0058] It should be noted that the training method and liveness detection of the liveness detection model in each embodiment of this application can be executed by the terminal device 110 or the server 120, or by both the terminal device 110 and the server 120.

[0059] In some embodiments, server 120 can perform training operations on a liveness detection model to obtain a trained target liveness detection model, and can use the target liveness detection model to perform liveness detection operations. Terminal device 110 has a client installed. When an object uses some services of the client, object recognition is required to ensure security. At this time, with the object's consent, terminal device 110 captures an image of the object and sends it to server 120 for liveness detection. Server 120 performs liveness detection on the received object image using the aforementioned target liveness detection model. If it determines that the object image contains a live object, it can further perform object recognition on the object image and finally return the object recognition result to terminal device 110.

[0060] In other embodiments, server 120 may perform training operations on a liveness detection model to obtain a trained target liveness detection model, and then send the target liveness detection model to terminal device 110; during the process of an object using terminal device 110, terminal device 110 may use the target liveness detection model to perform liveness detection operations.

[0061] In other embodiments, the terminal device 110 may perform training operations on a liveness detection model to obtain a trained target liveness detection model, and the terminal device 110 may perform liveness detection operations using the target liveness detection model while the object is using the terminal device 110.

[0062] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0063] The following describes the training method of the liveness detection model provided by the exemplary embodiments of this application, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0064] It is understood that in the specific embodiments of this application, data such as sample images and object images are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0065] The target liveness detection model used in this application embodiment is obtained by training an initial liveness detection model. First, the network structure of the initial liveness detection model to be trained in this application embodiment will be introduced by way of example.

[0066] In some embodiments, such as Figure 2 As shown, the initial liveness detection model in this application embodiment may include an image feature extraction layer, a text feature extraction layer, a first feature interaction layer, a second feature interaction layer, and a classification layer. These network layers can all adopt a neural network structure. The neural network structure of these network layers will be described by example below.

[0067] Optionally, the image feature extraction layer can employ a pre-trained image encoder, which can specifically be a convolutional neural network, an autoencoder, a residual neural network, etc., without limitation. The text feature extraction layer can employ a pre-trained text encoder, which can specifically be a convolutional neural network, a recurrent neural network, etc., without limitation.

[0068] The first feature interaction layer can use a neural network based on an attention mechanism (including a self-attention mechanism), such as Transformer; the second feature interaction layer can also use a neural network based on an attention mechanism (including a self-attention mechanism), such as Transformer.

[0069] The classification layer can use existing classification networks, such as those including fully connected layers and softmax layers.

[0070] The training process of the initial liveness detection model is described below.

[0071] See Figure 3 The diagram shown is a flowchart of a training method for a liveness detection model provided in this application. Taking a server as the execution entity, the specific implementation process of this method includes the following S31-S32:

[0072] S31. Obtain the sample dataset; where each sample dataset includes: a sample image, and a sample label indicating whether the sample image contains a live organism;

[0073] The sample images may contain target objects, including but not limited to physiological features such as faces, palms, and fingerprints. Sample images include, but are not limited to, RGB images and infrared images. The sample dataset may include positive and negative sample subsets. The sample images in the positive sample subset are images taken from real people; specifically, these can be obtained by cropping images taken from real people. The sample images in the negative sample subset are attack images that forge real people, including but not limited to images reproduced from printed photographs of real people, and images of 3D models of real people (such as masks, head models, palms, and fingerprints).

[0074] It should be noted that in this embodiment, the sample images in each sample dataset are obtained with the consent of the relevant users, and the collection, dissemination, and use of the sample data comply with relevant national laws and regulations. Furthermore, the sample datasets obtained in this embodiment are not saved and will be deleted at any time.

[0075] S32. Based on the sample dataset, the initial liveness detection model is trained in multiple rounds of iterations to obtain the target liveness detection model. Each round of iteration includes the following S321-S323.

[0076] S321. Extract features from the sample images, content text, and classification text selected from the sample dataset to obtain the corresponding sample image features, content text features, and initial classification text features.

[0077] The classification text contains content describing the liveness status of the target object, which includes whether the object is alive or not; the content text contains content describing the attribute information of the target object, which includes, but is not limited to, the components and appearance features of the target object.

[0078] Specifically, the categorized text may include one or both of a first categorized text and a second categorized text. The first categorized text contains content describing the target object as a living being, while the second categorized text contains content describing the target object as a non-living being. For example, if the target object is a face, the first categorized text may be expressed as "This is a real face," and the second categorized text may be expressed as "This is a fake face." If the target object is a hand, the first categorized text may be expressed as "This is a real hand," and the second categorized text may be expressed as "This is a fake hand." This application embodiment does not limit the specific content of the first categorized text and the second categorized text.

[0079] For example, if the target object is a face, the content text can be expressed as "a face is composed of eyes, nose, mouth and ears"; if the target object is a hand, the content text can be expressed as "a hand is composed of 5 fingers and palm". The specific content of the content text is not limited in the embodiments of this application.

[0080] Specifically, such as Figure 4 As shown, feature extraction is performed on the sample image through an image feature extraction layer to obtain sample image features. Similarly, feature extraction is performed on the content text and categorized text through a text feature extraction layer to obtain corresponding content text features and initial categorized text features. When the categorized text includes both first-category text and second-category text, the initial categorized text features include: first-category text sub-features corresponding to the first-category text and second-category text sub-features corresponding to the second-category text.

[0081] The sample image features described above are used to represent the semantic information of the sample images; the content text features described above are used to represent the semantic information of the content text; and the initial classification text features described above are used to represent the semantic information of the classification text. These features can specifically be represented as a feature matrix, where each feature element has a corresponding value. Furthermore, these features can also be represented as feature vectors (e.g., concatenating multiple feature vectors from the feature matrix into a single feature vector).

[0082] S323. Perform a first feature fusion process on the sample image features and content text features to obtain the target image features, and perform a second feature fusion process on the target image features and the initial classification text features to obtain the target classification text features.

[0083] Specifically, the first feature fusion processing of sample image features and content text features can be performed through the first feature interaction layer to obtain target image features.

[0084] On the one hand, considering that the attribute information of the target object, such as the components of a face, is irrelevant to liveness detection, filtering out this attribute information from the sample image features can make the sample image features more focused on the liveness detection task. Therefore, the first feature interaction layer can filter the sample image features based on the content text features to obtain the target image features. Optionally, attention mechanisms (including self-attention mechanisms) can also be used to perform attention weighting on the filtered sample image features to obtain the target image features, making the target image features more relevant to the liveness detection task.

[0085] On the other hand, the first feature interaction layer can also concatenate the content text features and sample image features, and then use an attention mechanism to apply first attention weighting to the obtained image concatenation features to obtain the target image features.

[0086] Furthermore, a second feature fusion process is performed on the target image features and the initial classification text features through a second feature interaction layer to obtain the target classification text features. Specifically, the target image features and the initial classification text features can be concatenated, and then an attention mechanism (including a self-attention mechanism) can be used to apply a second attention weighting to the obtained concatenated text features to obtain the target classification text features.

[0087] S323. Based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features, obtain the liveness detection results for the selected sample images, and perform model parameter tuning based on the difference between the liveness detection results and the corresponding sample labels.

[0088] The classification layer can classify the features of the target image to obtain the initial classification probability, which includes the initial probability of a live object and the initial probability of a non-live object.

[0089] In some embodiments, obtaining the liveness detection result in S323 above, based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classified text features, may include the following steps A1-A3:

[0090] A1. The target image features are predicted and processed through a fully connected layer to obtain the initial classification probability.

[0091] Specifically, the classification layer can include a fully connected layer and a softmax layer. The fully connected layer can map the features of the target image to a fixed-length vector for classification tasks. The softmax layer is used to convert the output vector of the fully connected layer into a probability distribution, that is, to convert the score of each category (live or inanimate) into a probability value between 0 and 1. The sum of the probabilities of all categories is 1, thus obtaining the initial classification probability.

[0092] A2. Determine the final classification probability based on the similarity between the target image features and the target classification text features, as well as the initial classification probability.

[0093] As mentioned in the above embodiments of this application, the categorized text may include one or both of the first categorized text and the second categorized text.

[0094] Optionally, when the target classification text features include either first-class or second-class text, existing similarity algorithms can be used to determine the similarity between the target image features and the target classification text features. For example, similarity algorithms include, but are not limited to, cosine similarity and Euclidean distance. When both the target image features and the target classification text features are in vector form, the similarity between the two vectors can be directly calculated using a similarity algorithm. When both the target image features and the target classification text features are in feature matrix form, they can be converted into feature vectors, for example, by concatenating multiple rows or columns of the feature matrix to obtain feature vectors, and then using a similarity algorithm to calculate the similarity between the two feature vectors. The similarity between the target image features and the target classification text features can represent the semantic matching degree.

[0095] Furthermore, the similarity and initial classification probabilities can be weighted and summed to obtain the final classification probability, which specifically includes the final probability of a live object and the final probability of a non-live object. The weights of similarity and initial classification probabilities can be the same or different, and can be set according to actual needs.

[0096] Optionally, when the categorized text includes a first categorized text and a second categorized text, the corresponding target categorized text features include: the first categorized text features corresponding to the first categorized text and the second categorized text features corresponding to the second categorized text. In this case, when determining the similarity between the target image features and the target categorized text features, a first similarity between the target image features and the first categorized text features, and a second similarity between the target image features and the second categorized text features can be determined. Then, based on the first similarity and the second similarity, the similarity between the target image features and the target categorized text features is obtained.

[0097] Specifically, the larger of the first and second similarity scores can be used as the similarity between the target image features and the target classification text features. For example, when the first similarity score is greater than the second similarity score, the first similarity score and the initial probability of a live object in the initial classification probability are weighted and summed to obtain the final probability of including a live object; conversely, when the second similarity score is greater than the first similarity score, the second similarity score and the initial probability of a non-live object in the initial classification probability are weighted and summed to obtain the final probability of a non-live object.

[0098] Furthermore, the first similarity and the second similarity can be used simultaneously as the similarity between the target image features and the target classified text features. In this case, the first similarity and the initial probability of a live object in the initial classification probability are weighted and summed to obtain the final probability of a live object, and the second similarity and the initial probability of a non-live object in the initial classification probability are weighted and summed to obtain the final probability of a non-live object.

[0099] In practical applications, the first and second similarities can be normalized to obtain normalized first and second similarities. Then, based on the normalized first and second similarities, the similarity between the target image features and the target classification text features can be determined.

[0100] A3. Based on the final classification probability, obtain the liveness detection result.

[0101] The final classification probability includes the final probability of a living organism and the final probability of a non-living organism.

[0102] Optionally, when the final probability of a live object reaches a first probability threshold, the live object detection result is determined to be that the sample image contains a live object; when the final probability of a live object does not reach the first probability threshold, the live object detection result is determined to be that the sample image does not contain a live object. Furthermore, when the final probability of a non-live object reaches a second probability threshold, the live object detection result is determined to be that the sample image does not contain a live object; when the final probability of a non-live object does not reach the second probability threshold, the live object detection result is determined to be that the sample image contains a live object. The first and second probability thresholds can be the same or different, and can be set as needed; there is no limitation on this.

[0103] In addition, when the final probability of a live object is greater than the final probability of a non-live object, the live object detection result is determined to be that the sample image contains a live object; when the final probability of a live object is not greater than the final probability of a non-live object, the live object detection result is determined to be that the sample image does not contain a live object.

[0104] In this embodiment, the final classification probability is determined based on the initial classification probability determined by the target image features and combined with the similarity (i.e. semantic matching degree) between the target image features and the target classification text features, which can more accurately obtain the liveness detection results.

[0105] In step S323 above, after obtaining the liveness detection result, the detection loss value can be determined based on the difference between the liveness detection result and the sample label corresponding to the sample image; specifically, the detection loss value can be calculated using an existing loss function; for example, the loss function includes, but is not limited to, the binary cross-entropy loss function.

[0106] In this embodiment of the application, in each round of iterative training, the parameters of the initial liveness detection model can be adjusted based on the detection loss value corresponding to a sample image; or the total detection loss value can be obtained based on the detection loss values ​​corresponding to multiple sample images. For example, the total detection loss value can be obtained by summing or averaging the detection loss values ​​corresponding to multiple sample images, and then the parameters of the initial liveness detection model can be adjusted.

[0107] For example, the total detection loss value can be calculated using the binary cross-entropy loss function shown in equation (1):

[0108]

[0109] Among them, l cls This represents the total detection loss value. Indicates the number of sample data. p represents the sample label of the i-th sample data. i This represents the liveness detection result of the i-th sample data.

[0110] Furthermore, assuming the parameters of the initial liveness detection model are adjusted based on the total detection loss value, specifically, the total detection loss value can be backpropagated, that is, starting from the output layer of the initial liveness detection model, the gradient of the total detection loss value with respect to the parameters of each network layer is calculated backwards. Then, gradient descent can be used to update the parameters of each network layer of the initial liveness detection model. For example, the parameters of each network layer can include weights, biases, and activation function parameters. Optionally, the parameters of the initial liveness detection model can be updated based on an optimizer, including but not limited to Adam (Adaptive Moment Estimation), RMSprop (Root Mean Square propagation), and Adagrad (Adaptive gradient algorithm).

[0111] In some embodiments, in order to further improve the generalization of the initial liveness detection model, multiple image enhancement methods can be used in each round of training iterations to enhance the sample images and obtain multiple enhanced images. Among them, multiple image enhancement methods can be selected as needed. For example, the sample images can be enhanced by random contrast, that is, different contrasts of the sample images are randomly set to obtain multiple enhanced images.

[0112] Subsequently, feature extraction is performed on multiple enhanced images to obtain multiple enhanced image features. Specifically, feature extraction can be performed on multiple enhanced images through the image feature extraction layer described above.

[0113] Next, based on the differences between multiple enhanced image features, the feature loss value is determined.

[0114] Specifically, existing loss functions can be used to determine the feature loss value based on the difference between every two enhanced image features. For example, loss functions include, but are not limited to, the Mean Squared Error (MSE) loss function.

[0115] Optionally, in each iteration of training, the parameters of the initial liveness detection model can be adjusted based on the feature loss value and detection loss value corresponding to a sample image. Specifically, the feature loss value and detection loss value are weighted and summed to obtain the total loss value corresponding to a sample image, and the parameters of the initial liveness detection model are adjusted based on this total loss value.

[0116] Furthermore, in each iteration of training, a total feature loss value can be obtained based on the feature loss values ​​corresponding to each of the multiple sample images. For example, the total feature loss value can be obtained by averaging or summing the feature loss values ​​corresponding to each of the multiple sample images. Then, the parameters of the initial liveness detection model can be adjusted based on the total feature loss value and the total detection loss value of the multiple sample images. For example, the total feature loss value and the total detection loss value can be weighted and summed to obtain the total loss value of the multiple sample images, and the parameters of the initial liveness detection model can be adjusted based on this total loss value.

[0117] Optionally, a contrastive loss function can also be used to determine the feature loss value of each sample image based on the differences between the multiple enhanced image features corresponding to each sample image. For example, assuming that each sample image corresponds to two enhanced image features, the contrastive loss function shown in equation (2) can be used to calculate the feature loss value corresponding to each sample image.

[0118]

[0119] Among them, l i σ represents the feature loss value corresponding to the i-th sample image; N represents the number of sample images, and σ 1 and σ 2 This represents different image enhancement methods; This represents an enhanced image feature corresponding to the i-th sample image; This represents another enhanced image feature corresponding to the i-th sample image; sim() represents cosine similarity; τ is a temperature parameter used to adjust the scale of similarity; 1 [k≠i] The function is an indicator function, which is 1 when k≠i and 0 otherwise; N is the number of sample images. This represents an enhanced image feature corresponding to the k-th sample image.

[0120] When each sample image corresponds to more than two enhanced image features, the sub-loss value corresponding to each pair of enhanced image features can be calculated based on the above formula (2). Based on the sub-loss value corresponding to each pair of enhanced image features, the feature loss value corresponding to each sample image can be obtained. For example, the average or summation of the sub-loss values ​​corresponding to each pair of enhanced image features can be used to obtain the feature loss value corresponding to each sample image.

[0121] Based on the feature loss values ​​corresponding to each of the multiple sample images, the total feature loss value can be obtained. For example, the average or sum of the feature loss values ​​corresponding to each of the multiple sample images can be used to obtain the total feature loss value. Then, the total feature loss value and the total detection loss value of the multiple sample images are weighted and summed to obtain the total loss value of the multiple sample images. Based on this total loss value, the parameters of the initial liveness detection model are adjusted.

[0122] Specifically, the total loss value can be backpropagated, that is, starting from the output layer of the initial liveness detection model, the gradient of the total loss value with respect to the parameters of each network layer can be calculated in reverse; then, the parameters of each network layer of the initial liveness detection model can be updated using gradient descent.

[0123] In this embodiment, during each round of iterative training, multiple enhanced images of each sample image are obtained, and feature extraction is performed on the multiple enhanced images to obtain multiple enhanced image features. Then, based on the differences between the multiple enhanced image features, a feature loss value is determined. This feature loss value aims to ensure that different image enhancement methods do not affect the expression of sample image features, so that the enhanced image features can also well represent the sample image, thereby further improving the generalization of the initial liveness detection model.

[0124] In this embodiment of the application, the iteration stopping condition for the initial liveness detection model includes at least one of the following conditions:

[0125] (1) The number of iterations reaches the set threshold.

[0126] (2) The model loss value is less than the set loss threshold; wherein, the model loss value may be the detection loss value or the total loss value (including the detection loss value and the feature loss value) corresponding to one sample image in the above embodiment, or it may be the total detection loss value or the total loss value (including the total detection loss value and the total feature loss value) corresponding to multiple sample images.

[0127] In this embodiment, when the number of iterations does not exceed a preset threshold and the model loss value is not less than a preset loss threshold, the current initial liveness detection model is considered not to have met the iteration termination condition, and the model parameters need to be adjusted to continue training. After parameter adjustment, the next iteration training process begins. When the number of iterations has exceeded the preset threshold, or the model loss value is less than the preset loss threshold, the current initial liveness detection model is considered to have met the convergence condition, and model training ends.

[0128] In this embodiment, during training, by fusing sample image features with content text features as a first feature, the attribute features of the target object can be integrated into the sample image features, effectively strengthening the sample image features and making the strengthened target image features more suitable for liveness detection tasks. Simultaneously, by fusing the target image features with initial classification text features as a second feature, the initial classification text features can be adaptively adjusted to obtain customized classification category features for the sample image, i.e., target classification text features. Finally, based on the aforementioned target image features more suitable for liveness detection tasks, the initial classification probability can be determined more accurately. Furthermore, by comprehensively considering this initial classification probability and the similarity between the target image features and the customized classification category features for the sample image, the liveness detection result corresponding to the sample image can be adaptively and accurately obtained. By tuning the initial liveness detection model through the above training process, a target liveness detection model capable of adapting to different object images can be obtained, improving the model's generalization ability. This not only allows for accurate detection of sample images but also accurate detection of object images to be detected in practical applications, thereby improving the accuracy of liveness detection.

[0129] The following is combined Figure 5 The training process of the liveness detection model in the embodiments of this application is described in general.

[0130] like Figure 5 As shown, in each iteration of the initial liveness detection model training process, sample images selected from the sample dataset are input into the image feature extraction layer to obtain sample image features fv. Additionally, content text and categorized text (including real and fake text) are input into the text feature extraction layer to obtain content text features fv. t1 and initial classification text features f t2 f t2 The dimensions are [N, C], where N is the number of categories and C is the number of channels. The sample image features fv and the content text features f... t1 The first feature interaction layer is used for the first feature fusion process to obtain the target image feature fv'. Then, the target image feature fv' and the initial classification text feature f are combined. t2The second feature interaction layer is used for second feature fusion processing to obtain the target classification text features f. t2 Next, the target image features fv' are input into the classification layer to obtain the initial classification probability, and the target image features fv' and target classification text features f are obtained. t2 The similarity between the samples is used to obtain the liveness detection result based on the initial classification probability and this similarity. Finally, the parameters of the initial liveness detection model are adjusted based on the difference between the liveness detection result and the sample label corresponding to the sample image.

[0131] The process of performing the first feature fusion processing on the sample image features and content text features in S323 above will be described in detail below.

[0132] In some embodiments, the content text contains multiple subtexts, each subtext containing content describing at least one attribute information of the target object, and the content text features contain features of the multiple subtexts; for example, each attribute information of the target object may be a component of the target object or an appearance feature, etc.

[0133] The first feature fusion process in S323 above, which involves fusing the sample image features and the content text features to obtain the target image features, may include the following steps B1-B2:

[0134] B1. Based on the similarity between the features of multiple image patches in the sample image features and the features of multiple sub-texts in the content text features, the filtering weights of multiple image patches are obtained.

[0135] The sample image features and content text features can be represented as feature matrices. In this case, each row of feature elements in the sample image features is considered a feature of an image patch, and each row of feature elements in the content text features is considered a feature of a sub-text. Alternatively, each column of feature elements in the sample image features can be considered a feature of an image patch, and each column of feature elements in the content text features can be considered a feature of a sub-text. Furthermore, the sample image features and content text features can also be represented as feature vectors. This can be considered as concatenating multiple feature vectors from the aforementioned feature matrix into a single feature vector. In this case, each sub-feature vector in the sample image features is considered a feature of an image patch, and each sub-feature vector in the content text features is considered a feature of a sub-text.

[0136] Then, for each image patch, existing similarity algorithms (such as cosine similarity, Euclidean distance, etc.) are used to calculate the similarity between the features of the image patch and the features of multiple sub-texts. Based on the obtained similarities, the filtering weights of the image patch can be obtained. These filtering weights can be represented as vectors, where each element represents a weight value. Specifically, for each image patch, preset operations can be performed on each similarity to obtain the filtering weights; for example, each similarity can be inverted (i.e., multiplied by -1) to obtain the weight values, which are then used to form the filtering weights. Other operations can also be performed on each similarity, such as subtracting a set value (e.g., 1) from each similarity. This is not limited.

[0137] In practical applications, the sample image features and content text features can be directly multiplied by a dot product (i.e., matrix multiplication) to obtain a similarity matrix. Then, the similarity matrix can be inverted to obtain a filtering matrix. Each row vector or column vector in the filtering matrix can be used as a filtering weight for an image patch.

[0138] For example, the features of the sample image are represented as Content text features are represented as Transpose the content text features to obtain Perform matrix multiplication (such as dot product) on the sample image features and the transposed text features to obtain the similarity matrix shown in equation (3).

[0139]

[0140] In addition, both sample image features and content text features can be in the form of feature vectors, which can also be regarded as one-dimensional feature matrices. In this case, the similarity algorithm is directly used to calculate the similarity between sample image features and content text features, and then the filtering weight of sample image features is obtained based on the similarity.

[0141] B2. By using the filtering weights of multiple image patches, the features of multiple image patches in the sample image features are weighted to obtain the target image features.

[0142] Optionally, for each image patch, the features of the image patch include multiple feature elements, and the corresponding filtering weights include multiple weight values, with each feature element corresponding to a weight value; based on the multiple weight values, the multiple feature elements are weighted and summed to obtain the filtered features of the image patch; and then based on the filtered features of the multiple image patches, the target image features are obtained.

[0143] In practical applications, the filtering weights of multiple image patches can be combined to form a filtering matrix, where each row or column represents the filtering weight of an image patch. Similarly, the features of multiple image patches in the sample image features can be combined to form a feature matrix, where each row or column represents a feature of an image patch. For example, performing matrix multiplication (such as dot product) on the filtering matrix and the feature matrix yields a new feature matrix that serves as the feature matrix for the target image.

[0144] In some embodiments, in step B2 above, the features of multiple image blocks in the object image features are weighted by the filtering weights of multiple image blocks to obtain intermediate image features; further, based on a self-attention mechanism, the intermediate image features are subjected to a first attention weighting to obtain target image features.

[0145] Specifically, the target image features are obtained by iteratively performing attention weighting on the intermediate image features at least once through the self-attention mechanism of the first feature interaction layer.

[0146] Specifically, perform the following steps B21-B23 at least once on the intermediate image features iteratively:

[0147] B21. Based on the intermediate image features of this input, and combined with the preset first query parameter set, first key parameter set, and first value parameter set, respectively, obtain the corresponding first query feature, first key feature, and first value feature.

[0148] The first set of query parameters can be a trainable query weight matrix W. Q The first set of key parameters can be a trainable key weight matrix W. K The first set of value parameters can be a trainable value weight matrix W. V Multiply the intermediate image features by the query weight matrix W. Q Obtain the first query feature Q, and multiply the intermediate image features by the key weight matrix W. K Obtain the first key feature K, and multiply the intermediate image features by the value weight matrix W. V The first feature value V is obtained. The intermediate image features and each weight matrix can be calculated using matrix multiplication (such as dot product).

[0149] B22. Based on the similarity between each query sub-feature in the first query feature and each key sub-feature in the first key feature, determine the first attention weight set of the first value feature.

[0150] The first query feature and the first key feature can be in the form of feature matrices. Each row or column element of the first query feature is treated as a query sub-feature, specifically a feature vector. Similarly, each row or column element of the first key feature is treated as a key sub-feature, also specifically a feature vector. For each query sub-feature in the first query feature, the similarity between that sub-feature and each key sub-feature in the first key feature is calculated to obtain a similarity matrix. For example, algorithms such as cosine similarity can be used to calculate the similarity between the query sub-feature and the key feature, or the dot product operation can be directly performed on the query sub-feature and the key feature to obtain the corresponding similarity. Then, the similarity values ​​in the similarity matrix can be normalized using the Softmax function to obtain a first attention weight set, which can be an attention weight matrix, where each element represents an attention weight.

[0151] Specifically, when calculating the similarity between each query sub-feature in the first query feature and multiple key sub-features in the first key feature, taking the first query feature as the query matrix and the first key feature as the key matrix as an example, matrix multiplication (such as dot product) can be performed between the query matrix and the transposed key matrix to obtain the similarity matrix.

[0152] B23. Based on the obtained first attention weight set, the first value features are weighted to obtain new intermediate image features.

[0153] In this context, both the first attention weight set and the first value feature are in the form of feature matrices. The first attention weight set and the first value feature are multiplied by matrix multiplication, specifically by dot product, to achieve weighted processing of the first value feature, thereby obtaining new intermediate image features.

[0154] For example, new intermediate image features can be calculated using the following equation (4):

[0155]

[0156] Among them, f v′ W represents the features of the intermediate image. Q f represents the query weight matrix. v′ W Q W represents the first query feature. K f represents the key weight matrix. v′ W K Denotes the first bond features K and W. V f represents the value weight matrix. v′ W V Let V represent the first key feature, and d represent the dimension of the first key feature K.

[0157] After iteratively performing steps B21-B23 on the intermediate image features, the latest intermediate image features obtained in the last iteration are used as the target image features.

[0158] For example, such as Figure 6 The diagram illustrates the process of feature filtering of sample image features based on content text features. The similarity between the sample image feature fv and the content text feature ft1 is calculated to obtain a similarity matrix. This similarity matrix is ​​then inverted to obtain a filtering matrix, which includes filtering weights for multiple image patches. The filtering matrix is ​​then multiplied by the sample image feature fv to obtain an intermediate image feature fv'. Finally, a self-attention mechanism is used to iteratively perform N self-attention operations on the intermediate image feature fv' to obtain the target image feature fv'.

[0159] In this embodiment, considering that the attribute information of the target object is irrelevant to liveness detection, feature filtering is performed on the sample image features based on the content text features. This filters out the attribute information of the target object from the sample image features, making the obtained target image features more focused on the liveness detection task. Furthermore, after feature filtering of the sample image features, a self-attention mechanism can be used to apply attention weighting to the filtered intermediate image features to enhance the parts of the intermediate image features relevant to the liveness detection task, which is beneficial for improving the generalization of the initial liveness detection model.

[0160] In other embodiments, the first feature fusion processing of the sample image features and content text features in S323 above to obtain the target image features may include the following steps C1-C2:

[0161] C1. The sample image features and content text features are concatenated to obtain image concatenation features.

[0162] Various concatenation methods can be used to concatenate sample image features and content text features. For example, if both sample image features and content text features are represented as feature matrices, they can be concatenated along the same dimension. Specifically, they can be concatenated row-wise or column-wise. When both target image features and initial classification text features are represented as feature vectors, vector concatenation can be performed directly. This application does not limit the concatenation method.

[0163] C2. Based on the self-attention mechanism, the image stitching features are subjected to a second attention weighting to obtain the target image features.

[0164] Specifically, the image stitching features are obtained by performing at least one attention weighting on the image stitching features through the self-attention mechanism of the first feature interaction layer.

[0165] Specifically, perform the following steps C21-C23 at least once on the image stitching features iteratively:

[0166] C21. Based on image stitching features, combine the preset second query parameter set, second key parameter set, and second value parameter set to obtain the corresponding second query features, second key features, and second value features.

[0167] C22. Based on the similarity between each query sub-feature in the second query feature and each key sub-feature in the second key feature, determine the second attention weight set of the second value feature;

[0168] C23. Based on the obtained second attention weight set, the second value features are weighted to obtain new image stitching features.

[0169] After iteratively performing steps C21-C23 on the image stitching features, the target image features are obtained based on the latest image stitching features obtained in the last iteration. Specifically, the latest image stitching features can be used as the target image features; alternatively, the latest image stitching features can be fused with the sample image features. For example, each feature element in the image stitching features can be added to the corresponding feature element in the sample image features to obtain the target image features. The image stitching features and sample image features can be represented as feature matrices, and the two feature matrices can be added together to obtain the target image features.

[0170] The specific implementation process of steps C21-C23 above is similar to that of steps B21-B23 above, and will not be repeated here.

[0171] In this embodiment, by concatenating the content text features and sample image features, and then applying attention weighting to the concatenated image features, the part of the image concatenated features related to the liveness detection task can be enhanced, which is beneficial to improving the generalization ability of the initial liveness detection model.

[0172] Figure 7 The first feature fusion process of sample image features and content text features is shown.

[0173] like Figure 7 As shown, after concatenating the sample image feature fv with the content text feature ft1, a self-attention mechanism is used to iteratively perform self-attention operations N times on the obtained image concatenation feature to obtain the latest image concatenation feature. Then, the latest image concatenation feature and the content text feature ft1 are added element by element to obtain the target image feature fv'.

[0174] In an optional implementation, in addition to using a self-attention mechanism to process the image stitching features, the sample image features and content text features can be processed without stitching them together. Instead, a cross-attention mechanism can be used to process the sample image features and content text features. This process is similar to steps C21-C23, except that in step C21, a second query feature is obtained based on the sample image features and the second query parameter set, a second key feature is obtained based on the content text features and the second key parameter set, and a second value feature is obtained based on the content text features and the second value parameter set. In the next iteration, the new sample image features and content text features can be processed again.

[0175] In another optional implementation, before concatenating the sample image features and content text features in step C1 above, the sample image features can be concatenated with a trainable first cue feature. Then, the concatenated sample image features are concatenated with the content text features. The initialized first cue feature can be set as needed, specifically in the form of a feature vector. The first cue feature can be adjusted simultaneously each time the model parameters are adjusted. In this way, by introducing a trainable first cue feature, the model can quickly adapt to the liveness detection task while maintaining its generalization ability.

[0176] In some embodiments, the second feature fusion processing of the target image features and the initial classified text features in S323 above to obtain the target classified text features may include the following steps D1-D2:

[0177] D1. Concatenate the target image features and the initial classified text features to obtain the text concatenation features.

[0178] Various concatenation methods can be used to concatenate the target image features and the initial classification text features. For example, when both the target image features and the initial classification text features are in the form of feature matrices, they can be concatenated along the same dimension, specifically by concatenating them row-wise or column-wise. When both the target image features and the initial classification text features are in the form of vectors, vector concatenation can be performed directly. This application does not limit the concatenation method.

[0179] D2. Based on the self-attention mechanism, the text splicing features are weighted by a third attention to obtain the target classification text features.

[0180] Specifically, the text splicing features are iteratively weighted at least once through the self-attention mechanism of the second feature interaction layer to obtain the target classification text features.

[0181] Specifically, the following steps D21-D23 can be performed at least once on the text concatenation features:

[0182] D21. Based on text concatenation features, combine the preset third query parameter set, third key parameter set, and third value parameter set to obtain the corresponding third query features, third key features, and third value features;

[0183] D22. Based on the similarity between each query sub-feature in the third query feature and each key sub-feature in the third key feature, determine the third attention weight set for each of the third value features.

[0184] D23. Based on the obtained third attention weight set, the third value features are weighted to obtain new text splicing features.

[0185] After iteratively performing steps D21-D23 on the text concatenation features, the target classification text features are obtained based on the latest text concatenation features obtained in the last iteration. Specifically, the latest text concatenation features can be used as the target classification text features; alternatively, the latest text concatenation features can be fused with the initial classification text features. For example, each feature element in the text concatenation features can be added to the corresponding feature element in the initial classification text features to obtain the target classification text features. The image concatenation features and sample image features can be represented as feature matrices, and the two feature matrices can be added together to obtain the target image features.

[0186] The specific implementation process of steps D21-D23 above is similar to that of steps B21-B23 above, and will not be repeated here.

[0187] In this embodiment, the target image features are concatenated with the initial classification text features, and the obtained text concatenation features are attention-weighted to use the target image features as dynamic cues to adaptively adjust the initial classification text features, thereby obtaining a classification category feature customized for the sample image, namely the target classification text feature. Subsequently, based on the similarity between the target image features and the target classification text feature, as well as the initial classification probability determined by the target image features, the liveness detection result corresponding to the sample image can be adaptively and accurately obtained, which is beneficial to improving the generalization of the initial liveness detection model.

[0188] In an optional implementation, in addition to using a self-attention mechanism to process the above-mentioned text concatenation features, the target image features and the initial classification text features can be processed without concatenating them. Instead, a cross-attention mechanism can be used to process the target image features and the initial classification text features. This processing is similar to steps D21-D23 above, except that in step D21, the third query feature is obtained based on the target image features and the third query parameter set, the third key feature is obtained based on the initial classification text features and the third key parameter set, and the third value feature is obtained based on the initial classification text features and the third value parameter set. In the next iteration, the new initial classification text features and target image features can be processed again.

[0189] In another optional implementation, before concatenating the target image features and the initial classification text features in step D1 above, the target image features can be concatenated with a trainable second cue feature. This second cue feature can be the same as or different from the first cue feature. Then, the concatenated target image features are concatenated with the initial classification text features. The initialized second cue feature can be set as needed, specifically in the form of a feature vector. The second cue feature can be adjusted simultaneously each time the model parameters are adjusted. In this way, by introducing a trainable second cue feature, the model can quickly adapt to the liveness detection task while maintaining its generalization ability.

[0190] Figure 8 The second feature fusion process of target image features and initial classification text features is shown.

[0191] like Figure 8 As shown, after concatenating the target image feature fv' with the initial classification text feature ft2, a self-attention mechanism is used to iteratively perform self-attention operations N times on the obtained text concatenation feature to obtain the latest text concatenation feature. Then, the latest text concatenation feature and the initial classification text feature ft2 are added element by element to obtain the target classification text feature ft2'.

[0192] Based on the same inventive concept, this application also provides a liveness detection method. The principle of this liveness detection method is similar to the training method of the liveness detection model in the above embodiments. Therefore, the implementation of this liveness detection method can refer to the implementation of the training method of the liveness detection model described above, and the repeated parts will not be described again.

[0193] like Figure 9 As shown, this application provides a liveness detection method, which is executed using a target liveness detection model. The method includes the following steps S91-S93:

[0194] S91. Extract features from the object image, content text, and classification text to be detected, respectively, to obtain the corresponding object image features, content text features, and initial classification text features.

[0195] The classification text contains information describing the liveness status of the target object, which may be either live or not. The content text contains information describing the attribute information of the target object. The image of the object to be detected can be an image taken of the object for liveness detection. After obtaining the image of the object, faces in the image can be cropped to obtain the image of the object to be detected.

[0196] S92. Perform a first feature fusion process on the content text features and object image features to obtain target image features, and perform a second feature fusion process on the target image features and initial classification text features to obtain target classification text features.

[0197] S93. Based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features, obtain the liveness detection result of the object image.

[0198] The specific implementation process of S91-S93 in the embodiments of this application can be referred to the specific implementation process of S321-S323 in the above embodiments of this application, and will not be repeated here.

[0199] In this embodiment, when using the target liveness detection model to perform liveness detection on the object image to be detected, content text and classification text are introduced. First, object image features, content text features, and initial classification text features are extracted. Since the content text contains information describing the attribute information of the target object, by fusing the object image features and content text features as a first feature, the attribute features of the target object can be integrated into the object image features, effectively strengthening the object image features and making the strengthened object image features more suitable for liveness detection tasks. Simultaneously, since the classification text contains information describing the liveness status of the target object, which includes whether the object is alive or not, the initial classification text features can... The classification features are represented by the target image features. By fusing the target image features with the initial classification text features using a second feature, the initial classification text features can be adaptively adjusted to obtain customized classification features for the object image, i.e., the target classification text features. Finally, based on the target image features that are more suitable for the liveness detection task, the initial classification probability can be determined more accurately. By comprehensively considering the initial classification probability and the similarity between the target image features and the customized classification features for the object image, the liveness detection result corresponding to the object image can be obtained adaptively and accurately. This allows the target liveness detection model to adapt to the object image, ensuring the model's generalization ability and thus improving the accuracy of liveness detection.

[0200] In some embodiments, the content text includes multiple subtexts, each subtext containing content describing at least one attribute information of the target object, and the content text features include features of the multiple subtexts; the first feature fusion processing of the content text features and the object image features in S92 above to obtain the target image features may include the following steps E1-E2:

[0201] E1. Obtain the similarity between the features of multiple image patches in the object image features and the features of multiple sub-texts in the content text features, and obtain the filtering weights of multiple image patches.

[0202] E2. By using the filtering weights of multiple image patches, the features of multiple image patches in the object image features are weighted to obtain the target image features.

[0203] The specific implementation process of steps E1-E2 in this embodiment is similar to that of steps B1-B2 in the above embodiment of this application, and will not be repeated here.

[0204] Optionally, in step E2, the features of multiple image blocks in the object image features are weighted by the filtering weights of multiple image blocks to obtain intermediate image features; further, based on the self-attention mechanism, the intermediate image features are subjected to first attention weighting to obtain target image features.

[0205] Specifically, the target image features are obtained by iteratively performing attention weighting on the intermediate image features at least once through the self-attention mechanism of the first feature interaction layer.

[0206] Specifically, the following steps E21-E23 are performed at least once on the intermediate image features, and the latest obtained intermediate image features are used as the target image features:

[0207] E21. Based on the intermediate image features of this input, and combined with the preset first query parameter set, first key parameter set, and first value parameter set, respectively, obtain the corresponding first query feature, first key feature, and first value feature;

[0208] E22. Based on the similarity between each query sub-feature in the first query feature and each key sub-feature in the first key feature, determine the first attention weight set of the first value feature;

[0209] E23. Based on the obtained first attention weight set, the first value features are weighted to obtain new intermediate image features.

[0210] The specific implementation process of steps E21-E23 in this embodiment is similar to that of steps B21-B23 in the above embodiment of this application, and will not be repeated here.

[0211] In other embodiments, the first feature fusion processing of the content text features and object image features in S92 above to obtain target image features may include the following steps F1-F2:

[0212] F1. Combine the object image features and content text features to obtain image stitching features.

[0213] The specific implementation process of step F1 in this embodiment is similar to that of step C1 in the above embodiment of this application, and will not be repeated here.

[0214] F2. Based on the self-attention mechanism, the image stitching features are subjected to a second attention weighting to obtain the target image features.

[0215] Specifically, the image stitching features are obtained by performing at least one attention weighting on the image stitching features through the self-attention mechanism of the first feature interaction layer.

[0216] Specifically, perform the following steps F21-F23 at least once on the image stitching features:

[0217] F21. Based on image stitching features, combine the preset second query parameter set, second key parameter set, and second value parameter set to obtain the corresponding second query features, second key features, and second value features.

[0218] F22. Based on the similarity between each query sub-feature in the second query feature and each key sub-feature in the second key feature, determine the second attention weight set of the second value feature;

[0219] F23. Based on the obtained second attention weight set, the second value features are weighted to obtain new image stitching features.

[0220] The specific implementation process of steps F21-F23 in the embodiments of this application is similar to the specific implementation process of steps C21-C23 in the above embodiments of this application, and will not be repeated here.

[0221] In addition to using a self-attention mechanism to process the above image stitching features, it is also possible to process the sample image features and content text features without stitching them together, instead using a cross-attention mechanism. This process is similar to steps F21-F23 above, except that in step F21, the second query feature is obtained based on the sample image features and the second query parameter set, the second key feature is obtained based on the content text features and the second key parameter set, and the second value feature is obtained based on the content text features and the second value parameter set. In the next iteration, the new sample image features and content text features can be processed again.

[0222] In some embodiments, the second feature fusion processing of the target image features and the initial classified text features in S92 above to obtain the target classified text features may include the following steps G1-G2:

[0223] G1. The target image features and the initial classification text features are concatenated to obtain the text concatenation features.

[0224] The specific implementation process of step G1 in this embodiment is similar to that of step D1 in the above embodiment of this application, and will not be repeated here.

[0225] G2. Based on the self-attention mechanism, the text splicing features are weighted by a third attention to obtain the target classification text features.

[0226] Specifically, the text splicing features are iteratively weighted at least once through the self-attention mechanism of the second feature interaction layer to obtain the target classification text features.

[0227] Specifically, the following steps G21-G23 are performed at least once on the text splicing features to obtain the target classification text features based on the latest obtained text splicing features;

[0228] G21. Based on text concatenation features, and combined with the preset third query parameter set, third key parameter set, and third value parameter set, respectively, obtain the corresponding third query features, third key features, and third value features;

[0229] G22. Based on the similarity between each query sub-feature in the third query feature and each key sub-feature in the third key feature, determine the third attention weight set of the third value feature;

[0230] G23. Based on the obtained third attention weight set, the third value features are weighted to obtain the target classification text features.

[0231] The specific implementation process of steps G21-G23 in this embodiment is similar to that of steps D21-D23 in the above embodiment of this application, and will not be repeated here.

[0232] In addition to using a self-attention mechanism to process the above-mentioned text concatenation features, the target image features and the initial classification text features can also be processed without concatenating them. Instead, a cross-attention mechanism can be used to process the target image features and the initial classification text features. This process is similar to steps G21-G23 above, except that in step G21, the third query feature is obtained based on the target image features and the third query parameter set, the third key feature is obtained based on the initial classification text features and the third key parameter set, and the third value feature is obtained based on the initial classification text features and the third value parameter set. In the next iteration, the new initial classification text features and target image features can be processed again.

[0233] In some embodiments, a liveness detection result is obtained based on an initial classification probability determined by the target image features and the similarity between the target image features and the target classified text features, including the following steps H1-H3:

[0234] H1. The target image features are predicted and processed through a fully connected layer to obtain the initial classification probability;

[0235] H2. Based on the similarity between the target image features and the target classified text features, and the initial classification probability, determine the final prediction probability;

[0236] Optionally, the classification text includes: a first classification text and a second classification text, wherein the first classification text contains content describing the target object as a living being, and the second classification text contains content describing the target object as a non-living being;

[0237] The target classification text features include: the first-class text features corresponding to the first-class text and the second-class text features corresponding to the second-class text;

[0238] The similarity between the target image features and the target classification text features is obtained in the following way:

[0239] Determine the first similarity between the target image features and the first category of text features, and determine the second similarity between the target image features and the second category of text features;

[0240] Based on the first similarity and the second similarity, the similarity between the target image features and the target classification text features is obtained.

[0241] H3. Based on the final classification probability, obtain the liveness detection result.

[0242] The specific implementation process of steps H1-H3 in the embodiments of this application can be referred to the specific implementation process of steps A1-A3 in the above embodiments of this application, and will not be repeated here.

[0243] Figure 10A schematic diagram of the overall logic of the liveness detection method according to an embodiment of this application is shown.

[0244] like Figure 10 As shown, when face detection is required during the use of a terminal device, the device captures an image of the object. The terminal device then transmits the object image to a server. The server inputs the object image into a target liveness detection model. Specifically, the object image is input into an image feature extraction layer, while the categorized text and content text are input into text feature extraction layers, respectively. Then, the image sequentially passes through a first feature interaction layer, a second feature interaction layer, and a classification layer, outputting the liveness detection result. If the liveness detection result indicates that the object image contains a live person, face recognition can be further performed based on the object image. If face recognition is successful, the face detection is complete.

[0245] The liveness detection method of this application can be applied to any scenario requiring liveness detection. In practical applications, liveness detection is usually combined with other technologies (such as face recognition, fingerprint recognition, palm recognition, iris recognition, etc.) as the first line of defense for identity verification security. Currently, liveness detection has been applied in many fields, such as remote identity verification in banking, registration and login of various software, facial recognition payment in payment software, and access control systems. The following uses the combination of liveness detection and face recognition as an example to illustrate the liveness detection process in some application scenarios.

[0246] In banking transactions, facial recognition technology, incorporating liveness detection algorithms, can be used to remotely verify the identity of an individual. The specific process is as follows: The individual first obtains an image containing their face through the camera of their terminal device. Then, the terminal device transmits the image to the server. The server uses a liveness detection model to perform liveness detection. If the image is determined to contain a live person, facial recognition can be performed. If facial recognition is successful, identity verification is passed; otherwise, it fails.

[0247] In the registration and login process of various software (such as social software, ride-hailing software, shopping software, etc.), in order to ensure the security of the user's account, when the user registers or logs in on the terminal device, the terminal device can capture an image of the user containing a face and transmit the image to the server. The server calls the target liveness detection model to perform liveness detection. If the image is determined to contain a live person, it can further perform face recognition. If the face recognition is successful, the registration or login is successful; otherwise, it fails.

[0248] In the facial recognition payment process of payment software, liveness detection and facial recognition play a crucial role. When a user makes a payment using a terminal device with the payment software installed, the device can capture an image of the user's face. The terminal device then transmits the image to a server, which uses a liveness detection model to perform a liveness check. If the image detects a live person, facial recognition is performed. If facial recognition is successful, the payment is completed; otherwise, the payment fails. High-precision liveness detection effectively prevents illegal transactions and ensures transaction security.

[0249] In access control systems, facial recognition can verify the identity of an individual. To improve the efficiency of identity verification, after acquiring an image of an object containing a face, the access control system can directly input the image into a target liveness detection model for liveness detection. If the image is determined to contain a live person, facial recognition can be performed. If facial recognition is successful, the access control is unlocked, and the result is immediately fed back.

[0250] Furthermore, the process of combining liveness detection with other technologies besides facial recognition is similar to the process of combining liveness detection with facial recognition, and will not be elaborated here.

[0251] Based on the same inventive concept, this application also provides a training device for a liveness detection model. The principle of this device in solving the problem is similar to the method in the above embodiments. Therefore, the implementation of this device can refer to the implementation of the above method, and the repeated parts will not be described again.

[0252] like Figure 11 As shown, this is a schematic diagram of the structure of a training device 1100 for a liveness detection model. The device may include:

[0253] The acquisition module 1101 is used to acquire a sample dataset; wherein each sample dataset includes: a sample image, and a sample label indicating whether the sample image contains a live organism;

[0254] Training module 1102 is used to perform multiple rounds of iterative training on the initial liveness detection model based on the sample dataset to obtain the target liveness detection model. Each round of iteration includes:

[0255] Feature extraction is performed on the sample images, content text, and classification text selected from the sample dataset to obtain the corresponding sample image features, content text features, and initial classification text features. The classification text contains content describing the liveness status of the target object, which includes whether the object is live or not. The content text contains content describing the attribute information of the target object.

[0256] The first feature fusion process is performed on the sample image features and the content text features to obtain the target image features, and the second feature fusion process is performed on the target image features and the initial classification text features to obtain the target classification text features.

[0257] Based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features, the liveness detection results of the selected sample images are obtained, and the model parameters are tuned based on the difference between the liveness detection results and the corresponding sample labels.

[0258] In some embodiments, the content text includes multiple subtexts, each subtext containing content describing at least one attribute information of the target object, and the content text features include features of the multiple subtexts; when performing a first feature fusion process on the sample image features and the content text features to obtain the target image features, the training module 1102 is specifically used for:

[0259] Based on the similarity between the features of multiple image patches in the sample image features and the features of multiple sub-texts in the content text features, the filtering weights of multiple image patches are obtained.

[0260] By using the filtering weights of multiple image patches, the features of multiple image patches in the sample image are weighted separately to obtain the features of the target image.

[0261] In some embodiments, when the features of multiple image patches in the sample image features are weighted by the filtering weights of multiple image patches to obtain the target image features, the training module 1102 is specifically used for:

[0262] By using the filtering weights of multiple image patches, the features of multiple image patches in the sample image features are weighted separately to obtain intermediate image features;

[0263] Based on the self-attention mechanism, the intermediate image features are first attention-weighted to obtain the target image features.

[0264] In some embodiments, when performing a first feature fusion process on sample image features and content text features to obtain target image features, the training module 1102 is specifically used for:

[0265] The image features and text features of the sample images are concatenated to obtain image concatenation features.

[0266] Based on the self-attention mechanism, the image stitching features are subjected to a second attention weighting to obtain the target image features.

[0267] In some embodiments, when performing a second feature fusion process on the target image features and the initial classified text features to obtain the target classified text features, the training module 1102 is specifically used for:

[0268] The target image features and the initial classified text features are concatenated to obtain the text concatenation features;

[0269] Based on the self-attention mechanism, the text splicing features are weighted by a third attention to obtain the target classification text features.

[0270] In some embodiments, when obtaining a liveness detection result based on an initial classification probability determined by the target image features and the similarity between the target image features and the target classified text features, the training module 1102 is specifically used for:

[0271] The target image features are predicted using a fully connected layer to obtain the initial classification probability.

[0272] The final classification rate is determined based on the similarity between the target image features and the target classification text features, as well as the initial classification probability.

[0273] Based on the final classification probability, the liveness detection result is obtained.

[0274] In some embodiments, the classification text includes: a first classification text and a second classification text, wherein the first classification text contains content describing the target object as a living being, and the second classification text contains content describing the target object as a non-living being.

[0275] The target classification text features include: the first-class text features corresponding to the first-class text and the second-class text features corresponding to the second-class text;

[0276] Training module 1102 is specifically used for:

[0277] Determine the first similarity between the target image features and the first category of text features, and determine the second similarity between the target image features and the second category of text features;

[0278] Based on the first similarity and the second similarity, the similarity between the target image features and the target classification text features is obtained.

[0279] In some embodiments, the apparatus further includes an enhancement module for:

[0280] Multiple image enhancement methods are used to enhance the sample images to obtain multiple enhanced images. Feature extraction is then performed on each of the multiple enhanced images to obtain multiple enhanced image features. Based on the differences between the features of the multiple enhanced images, the feature loss value is determined.

[0281] Training module 1102 is specifically used for:

[0282] Based on the difference between the liveness detection results and the corresponding sample labels, the detection loss value is determined, and the model parameters are tuned based on the detection loss value and the feature loss value.

[0283] Based on the same inventive concept, this application also provides a liveness detection device. The principle of this device in solving the problem is similar to the liveness detection method in the above embodiments. Therefore, the implementation of this device can refer to the implementation of the above method, and the repeated parts will not be described again.

[0284] like Figure 12 As shown, a liveness detection device 1200 provided in this application embodiment is executed using a target liveness detection model. The device includes:

[0285] The feature extraction module 1201 is used to extract features from the object image, content text and classification text to be detected, respectively, to obtain the corresponding object image features, content text features and initial classification text features; the classification text contains content describing the liveness status of the target object, the liveness status includes live or non-live, and the content text contains content describing the attribute information of the target object.

[0286] The feature fusion module 1202 is used to perform a first feature fusion process on the content text features and the object image features to obtain the target image features, and to perform a second feature fusion process on the target image features and the initial classification text features to obtain the target classification text features.

[0287] The liveness detection module 1203 is used to obtain the liveness detection result of the object image based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features.

[0288] In some embodiments, the content text includes multiple subtexts, each subtext containing content describing at least one attribute information of the target object, and the content text feature includes features of multiple subtexts.

[0289] When performing a first feature fusion process on the content text features and object image features to obtain the target image features, the feature fusion module 1202 is specifically used for:

[0290] Based on the similarity between the features of multiple image patches in the object image features and the features of multiple sub-texts in the content text features, the filtering weights of multiple image patches are obtained.

[0291] By using the filtering weights of multiple image patches, the features of multiple image patches in the object image feature are weighted separately to obtain the target image features.

[0292] In some embodiments, when the features of multiple image patches in the object image features are weighted by the filtering weights of multiple image patches to obtain the target image features, the feature fusion module 1202 is specifically used for:

[0293] By using the filtering weights of multiple image patches, the features of multiple image patches in the object image feature are weighted separately to obtain intermediate image features;

[0294] Based on the self-attention mechanism, the intermediate image features are first attention-weighted to obtain the target image features.

[0295] In some embodiments, when performing a first feature fusion process on object image features and content text features to obtain target image features, the feature fusion module 1202 is specifically used for:

[0296] The image features of the object and the text features of the content are concatenated to obtain the image concatenation features;

[0297] Based on the self-attention mechanism, the image stitching features are subjected to a second attention weighting to obtain the target image features.

[0298] In some embodiments, when feature fusion is performed on target image features and initial classified text features to obtain target classified text features, the feature fusion module 1202 is specifically used for:

[0299] The target image features and the initial classified text features are concatenated to obtain the text concatenation features;

[0300] Based on the self-attention mechanism, the text splicing features are weighted by a third attention to obtain the target classification text features.

[0301] In some embodiments, the liveness detection module 1203 is specifically used for:

[0302] The target image features are predicted using a fully connected layer to obtain the initial classification probability.

[0303] The final classification rate is determined based on the similarity between the target image features and the target classification text features, as well as the initial classification probability.

[0304] Based on the final classification probability, the liveness detection result is obtained.

[0305] In some embodiments, the classification text includes: a first classification text and a second classification text, wherein the first classification text contains content describing the target object as a living being, and the second classification text contains content describing the target object as a non-living being.

[0306] The target classification text features include: the first-class text features corresponding to the first-class text and the second-class text features corresponding to the second-class text;

[0307] The liveness detection module is also used for:

[0308] Determine the first similarity between the target image features and the first category of text features, and determine the second similarity between the target image features and the second category of text features;

[0309] Based on the first similarity and the second similarity, the similarity between the target image features and the target classification text features is obtained.

[0310] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0311] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0312] Having introduced the training method and apparatus for the liveness detection model according to exemplary embodiments of this application, and the liveness detection method and apparatus, the electronic device according to another exemplary embodiment of this application will now be described.

[0313] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 1 The server 120 is shown. In this embodiment, the structure of the electronic device can be as follows: Figure 13 As shown, it includes a memory 1301, a communication module 1303, and one or more processors 1302.

[0314] The memory 1301 is used to store computer programs executed by the processor 1302. The memory 1301 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0315] Memory 1301 may be volatile memory, such as random-access memory (RAM); memory 1301 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1301 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1301 may be a combination of the above-described memories.

[0316] The processor 1302 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1302 is used to implement the training method and liveness detection method of the aforementioned liveness detection model when it calls the computer program stored in the memory 1301.

[0317] The communication module 1303 is used to communicate with terminal devices and other servers.

[0318] This application embodiment does not limit the specific connection medium between the memory 1301, communication module 1303, and processor 1302. This application embodiment... Figure 13 The memory 1301 and the processor 1302 are connected via a bus 1304, and the bus 1304 is in Figure 13 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1304 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 13 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0319] The memory 1301 stores a computer-readable storage medium containing a computer program. The computer program implements the training method and liveness detection method of the liveness detection model according to embodiments of this application. The processor 1302 executes the aforementioned training method and liveness detection method of the liveness detection model, such as... Figure 3 and Figure 9 As shown.

[0320] In another embodiment, the electronic device may also be other electronic devices, such as... Figure 1 The terminal device 110 is shown. In this embodiment, the electronic device can be structured as follows: Figure 14As shown, it includes components such as: communication component 1410, memory 1420, display unit 1430, camera 1440, sensor 1450, audio circuit 1460, Bluetooth module 1470, processor 1480, etc.

[0321] The communication component 1410 is used to communicate with the server. In some embodiments, it may include a Circuit-Based Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology. Electronic devices can use the WiFi module to help users send and receive information.

[0322] The memory 1420 can be used to store software programs and data. The processor 1480 executes various functions of the terminal device 110 and performs data processing by running the software programs or data stored in the memory 1420. The memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 1420 stores an operating system that enables the terminal device 110 to run. In this application, the memory 1420 may store the operating system and various application programs, and may also store computer programs that execute the training method and liveness detection method of the liveness detection model of the embodiments of this application.

[0323] The display unit 1430 can also be used to display information input by the user or information provided to the user, as well as various menus of the terminal device 110, in a graphical user interface (GUI). Specifically, the display unit 1430 may include a display screen 1432 disposed on the front of the terminal device 110. The display screen 1432 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 1430 can be used to display the object image acquisition interface, etc., as described in the embodiments of this application.

[0324] The display unit 1430 can also be used to receive input digital or character information and generate signal inputs related to user settings and function control of the terminal device 110. Specifically, the display unit 1430 may include a touch screen 1431 disposed on the front of the terminal device 110, which can collect touch operations of the user on or near it, such as clicking buttons, dragging scroll boxes, etc.

[0325] The touchscreen 1431 can be placed on top of the display screen 1432, or the touchscreen 1431 and the display screen 1432 can be integrated to realize the input and output functions of the terminal device 110. After integration, it can be referred to as a touch display screen. In this application, the display unit 1430 can display the application program and the corresponding liveness detection operation steps.

[0326] Camera 1440 can be used to capture still images, which users can then share via an application. There can be one or multiple cameras 1440. An object is projected onto a photosensitive element through a lens, generating an optical image. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the processor 1480 for conversion into a digital image signal.

[0327] The terminal device may also include at least one sensor 1450, such as an accelerometer 1451, a proximity sensor 1452, a fingerprint sensor 1453, and a temperature sensor 1454. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0328] Audio circuitry 1460, speaker 1461, and microphone 1462 provide an audio interface between the user and terminal device 110. Audio circuitry 1460 converts received audio data into electrical signals, which are then transmitted to speaker 1461, where they are converted into sound signals for output. Terminal device 110 may also be equipped with volume buttons for adjusting the volume of the sound signal. Conversely, microphone 1462 converts collected sound signals into electrical signals, which are then received by audio circuitry 1460, converted back into audio data, and output to communication component 1410 for transmission to, for example, another terminal device 110, or to memory 1420 for further processing.

[0329] The Bluetooth module 1470 is used to interact with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through the Bluetooth module 1470, thereby exchanging data.

[0330] The processor 1480 is the control center of the terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 1420 and calling data stored in the memory 1420. In some embodiments, the processor 1480 may include one or more processing units; the processor 1480 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 1480. In this application, the processor 1480 can run the operating system, applications, user interface display and touch response, as well as the training method and liveness detection method of the liveness detection model in the embodiments of this application. Furthermore, the processor 1480 is coupled to the display unit 1430.

[0331] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when run on a computer device, causes the computer device to execute the steps in the training method and the liveness detection method of the liveness detection model of embodiments of this application.

[0332] In some possible implementations, the training method for the liveness detection model and various aspects of the liveness detection method provided in this application can also be implemented in the form of a computer program product, which includes a computer program. When the computer program product is run on an electronic device, the computer program is used to cause the electronic device to execute the steps in the training method for the liveness detection model and the liveness detection method of the embodiments of this application, such as... Figure 3 and Figure 9 The steps are shown in the figure.

[0333] Computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0334] The computer program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0335] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0336] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0337] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0338] It should be noted that although several modules of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0339] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0340] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0341] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A training method for a liveness detection model, characterized in that, The method includes: Obtain a sample dataset; wherein each sample dataset includes: a sample image, and a sample label indicating whether the sample image contains a live organism; The initial liveness detection model is trained iteratively through multiple rounds based on the aforementioned sample dataset to obtain the target liveness detection model. Each iteration includes: Feature extraction is performed on the sample images, content text, and classification text selected from the sample dataset to obtain the corresponding sample image features, content text features, and initial classification text features. The classification text contains content describing the liveness status of the target object, which includes whether the object is alive or not. The content text contains content describing the attribute information of the target object. The sample image features and the content text features are subjected to a first feature fusion process to obtain target image features, and the target image features and the initial classification text features are subjected to a second feature fusion process to obtain target classification text features; Based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features, the liveness detection result of the selected sample image is obtained, and the model parameters are tuned based on the difference between the liveness detection result and the corresponding sample label.

2. The method according to claim 1, characterized in that, The content text contains multiple subtexts, each subtext containing content describing at least one attribute information of the target object, and the content text features include features of the multiple subtexts; The first feature fusion process, which involves combining the sample image features and the content text features to obtain the target image features, includes: Based on the similarity between the features of multiple image patches in the sample image features and the features of multiple sub-texts in the content text features, the filtering weights of the multiple image patches are obtained; By using the filtering weights of the multiple image patches, the features of the multiple image patches in the sample image features are weighted respectively to obtain the target image features.

3. The method according to claim 2, characterized in that, The step of weighting the features of the multiple image patches in the sample image features by using the filtering weights of the multiple image patches to obtain the target image features includes: By using the filtering weights of the multiple image patches, the features of the multiple image patches in the sample image features are weighted respectively to obtain intermediate image features; Based on the self-attention mechanism, the intermediate image features are subjected to a first attention weighting to obtain the target image features.

4. The method according to claim 1, characterized in that, The first feature fusion process, which involves combining the sample image features and the content text features to obtain the target image features, includes: The sample image features and the content text features are concatenated to obtain image concatenation features; Based on the self-attention mechanism, the image stitching features are subjected to a second attention weighting to obtain the target image features.

5. The method according to claim 1, characterized in that, The second feature fusion process, which involves fusing the target image features and the initial classified text features to obtain the target classified text features, includes: The target image features and the initial classified text features are concatenated to obtain text concatenation features; Based on the self-attention mechanism, the text splicing features are subjected to a third attention weighting to obtain the target classification text features.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Multiple image enhancement methods are used to enhance the sample image to obtain multiple enhanced images. Feature extraction is then performed on each of the multiple enhanced images to obtain multiple enhanced image features. Based on the differences between the multiple enhanced image features, a feature loss value is determined. The step of tuning the model parameters based on the difference between the liveness detection results and the corresponding sample labels includes: Based on the difference between the liveness detection results and the corresponding sample labels, a detection loss value is determined, and model parameters are tuned based on the detection loss value and the feature loss value.

7. A method for detecting liveness, characterized in that, The method, performed using a target liveness detection model, includes: Feature extraction is performed on the object image, content text, and classification text to be detected, respectively, to obtain corresponding object image features, content text features, and initial classification text features; the classification text contains content describing the liveness status of the target object, which includes live or non-liveness, and the content text contains content describing the attribute information of the target object; The object image features and the content text features are subjected to a first feature fusion process to obtain target image features, and the target image features and the initial classification text features are subjected to a second feature fusion process to obtain target classification text features; Based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features, a liveness detection result for the object image is obtained.

8. The method according to claim 7, characterized in that, The content text contains multiple subtexts, each subtext containing content describing at least one attribute information of the target object, and the content text features include features of the multiple subtexts; The first feature fusion process, which involves fusing the object image features and the content text features to obtain target image features, includes: Based on the similarity between the features of multiple image patches in the object image features and the features of multiple sub-texts in the content text features, the filtering weights of the multiple image patches are obtained. By using the filtering weights of the multiple image patches, the features of the multiple image patches in the object image features are weighted respectively to obtain the target image features.

9. The method according to claim 8, characterized in that, The step of weighting the features of the multiple image patches in the object image features by means of filtering weights of the multiple image patches to obtain the target image features includes: By using the filtering weights of the multiple image patches, the features of the multiple image patches in the object image features are weighted respectively to obtain intermediate image features; Based on the self-attention mechanism, the intermediate image features are subjected to a first attention weighting to obtain the target image features.

10. The method according to claim 7, characterized in that, The first feature fusion process, which involves fusing the object image features and the content text features to obtain target image features, includes: The object image features and the content text features are concatenated to obtain image concatenation features; Based on the self-attention mechanism, the image stitching features are subjected to a second attention weighting to obtain the target image features.

11. The method according to any one of claims 7 to 10, characterized in that, The step of fusing the target image features and the initial classified text features to obtain the target classified text features includes: The target image features and the initial classified text features are concatenated to obtain text concatenation features; Based on the self-attention mechanism, the text splicing features are subjected to a third attention weighting to obtain the target classification text features.

12. A training device for a liveness detection model, characterized in that, The device includes: The acquisition module is used to acquire a sample dataset; wherein each sample dataset includes: a sample image, and a sample label indicating whether the sample image contains a live organism; The training module is used to perform multiple rounds of iterative training on the initial liveness detection model based on the sample dataset to obtain the target liveness detection model, wherein each round of iteration includes: Feature extraction is performed on the sample images, content text, and classification text selected from the sample dataset to obtain the corresponding sample image features, content text features, and initial classification text features. The classification text contains content describing the liveness status of the target object, which includes whether the object is alive or not. The content text contains content describing the attribute information of the target object. The sample image features and the content text features are subjected to a first feature fusion process to obtain target image features, and the target image features and the initial classification text features are subjected to a second feature fusion process to obtain target classification text features; Based on the initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features, the liveness detection result of the selected sample image is obtained, and the model parameters are tuned based on the difference between the liveness detection result and the corresponding sample label.

13. A liveness detection device, characterized in that, The apparatus, which utilizes a target liveness detection model, comprises: The feature extraction module is used to extract features from the object image, content text, and classification text to be detected, respectively, to obtain corresponding object image features, content text features, and initial classification text features; the classification text contains content describing the liveness status of the target object, the liveness status including live or non-live, and the content text contains content describing the attribute information of the target object; The feature fusion module is used to perform a first feature fusion process on the object image features and the content text features to obtain target image features, and to perform a second feature fusion process on the target image features and the initial classification text features to obtain target classification text features; The liveness detection module is used to obtain a liveness detection result for the object image based on an initial classification probability determined by the target image features and the similarity between the target image features and the target classification text features.

14. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods of claims 1 to 6 or any of the methods of claims 7 to 11.

15. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 6 or any of the methods described in claims 7 to 11.

Images