A method and system for joint verification of digital identity using a combined model

Abstract

The present application provides a method and system for joint verification of digital identity using a combined model, which relates to the field of intelligent verification. It collects the face image and fingerprint image of the object to be verified, and introduces an image processing and analysis algorithm based on artificial intelligence and deep learning at the back end to analyze the face image and the fingerprint image, so as to capture and characterize the multimodal identity joint representation between the face features and fingerprint features of the object to be verified, and use this identity joint representation to verify the digital identity to determine whether the object to be verified belongs to the authorized object. In this way, the digital identity can be more intelligently joint verified through the combination of the face recognition base model and the fingerprint recognition base model, so as to improve the intelligence level of identity authentication, enhance the security of the system, and prevent the risk of identity fraud.

Description

Technical Field

[0001] The present application relates to the field of intelligent verification, and more particularly, in an embodiment of the present application, to a method and system for jointly verifying a digital identity using a combined model. Background Art

[0002] In the current digital age, with the rapid development of information technology, network security and personal privacy protection have become the focus of social attention. Traditional single-factor authentication methods (such as passwords, PIN codes, etc.) can no longer meet security needs because they are easy to be cracked or forgotten. Therefore, multi-factor authentication has gradually become a mainstream trend, among which biometric authentication is favored because of its uniqueness and non-replicability. Biometric authentication uses a person's unique physiological or behavioral characteristics to authenticate identity, such as fingerprints, facial features, iris patterns, etc.

[0003] However, a single biometric recognition method may be affected by environmental factors or individual differences in some cases, resulting in poor verification results. Specifically, for biometrics that rely on visual information (such as facial information), changes in ambient light can significantly affect the recognition effect. Strong direct light or too dark an environment can cause recognition failure. For technologies that rely on skin features, such as fingerprints and palm geometry, if the user's skin is damaged (such as cuts, burns), too dry or wet, it may cause recognition difficulties. In addition, as people age, their biometrics will change, which will also affect the recognition effect.

[0004] Therefore, an optimized digital identity verification solution is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a method and system for joint verification of digital identity using a combined model, which collects the face image and fingerprint image of the object to be verified, and introduces an image processing and analysis algorithm based on artificial intelligence and deep learning at the back end to analyze the face image and the fingerprint image, so as to capture and characterize the multimodal identity joint representation between the face features and fingerprint features of the object to be verified, and use this identity joint representation to verify the digital identity to determine whether the object to be verified belongs to the authorized object. In this way, the digital identity can be more intelligently jointly verified through the combination of the face recognition base model and the fingerprint recognition base model, so as to improve the intelligence level of identity authentication, enhance the security of the system, and prevent the risk of identity fraud.

[0006] According to one aspect of the present application, a method for jointly verifying a digital identity using a combined model is provided, which includes: obtaining a face image and a fingerprint image of an object to be verified; performing face feature extraction on the face image of the object to be verified to obtain face semantic coding features; performing fingerprint feature extraction on the fingerprint image of the object to be verified to obtain fingerprint semantic coding features; performing joint encoding on the face semantic coding features and the fingerprint semantic coding features based on multimodal implicit association to obtain a joint representation of a multimodal identity of the object to be verified; and determining whether the object to be verified belongs to an authorized object based on the joint representation of the multimodal identity of the object to be verified.

[0007] Among them, the face semantic coding features and the fingerprint semantic coding features are jointly encoded based on multimodal implicit association to obtain a multimodal identity joint representation of the object to be verified, including: feature association encoding of the face semantic coding features and the fingerprint semantic coding features to obtain fingerprint-face semantic implicit common features; preliminary feature screening of the fingerprint-face semantic implicit common features to obtain preliminary screening fingerprint-face cross-modal joint coding features; secondary feature screening of the preliminary screening fingerprint-face cross-modal joint coding features to obtain a multimodal identity joint representation of the object to be verified.

[0008] In the above-mentioned method of jointly verifying digital identity using a combined model, facial features are extracted from the facial image of the object to be verified to obtain facial semantic coding features, including: inputting the facial image of the object to be verified into a facial feature extraction model based on a deep separable convolutional neural network model to obtain a facial semantic coding feature map as the facial semantic coding features.

[0009] In the above-mentioned method for jointly verifying digital identity using a combined model, fingerprint features are extracted from the fingerprint image of the object to be verified to obtain fingerprint semantic coding features, including: inputting the fingerprint image of the object to be verified into a fingerprint feature extraction model based on a multi-layer hole convolution layer to obtain a fingerprint semantic coding feature vector as the fingerprint semantic coding feature.

[0010] In the above-mentioned method of jointly verifying digital identity using a combined model, the face semantic coding features and the fingerprint semantic coding features are feature-associated encoded to obtain fingerprint-face semantic implicit common features, including: feature distillation of the face semantic coding feature map to obtain a face semantic distillation feature vector; inputting the fingerprint semantic coding feature vector and the face semantic distillation feature vector into a multimodal implicit association feature capture network to obtain a fingerprint-face semantic implicit common feature vector as the fingerprint-face semantic implicit common feature.

[0011] In the above-mentioned method for jointly verifying digital identity using a combined model, feature distillation is performed on the face semantic coding feature map to obtain a face semantic distillation feature vector, including: performing point convolution encoding on the face semantic coding feature map to obtain a face semantic coding channel modulation feature map; performing global mean pooling processing on each feature matrix along the channel dimension in the face semantic coding channel modulation feature map to obtain a face semantic channel feature vector; and inputting the face semantic channel feature vector into a Sigmoid function for processing to obtain the face semantic distillation feature vector.

[0012] In the above-mentioned method for jointly verifying digital identity using a combined model, the fingerprint semantic encoding feature vector and the face semantic distillation feature vector are input into a multimodal implicit association feature capture network to obtain a fingerprint-face semantic implicit common feature vector as the fingerprint-face semantic implicit common feature, including: cascading the fingerprint semantic encoding feature vector and the face semantic distillation feature vector to obtain a fingerprint-face semantic cascade feature vector; multiplying the fingerprint-face semantic cascade feature vector with a linear transformation weight matrix and then adding it to a linear transformation bias vector by position to obtain a linear transformation fingerprint-face semantic cascade feature vector; and concatenating the linear transformation fingerprint-face semantic cascade feature vector through The fingerprint-face semantic implicit common feature vector is obtained by processing in the function.

[0013] In the above-mentioned method for jointly verifying digital identity using a combined model, the fingerprint-face semantic implicit common features are preliminarily screened to obtain preliminary screening fingerprint-face cross-modal joint coding features, including: inputting the fingerprint-face semantic implicit common feature vector into a feature preliminary screening module based on a gating unit to obtain a first-level fingerprint-face semantic feature preliminary screening weight vector; based on the first-level fingerprint-face semantic feature preliminary screening weight vector, the face semantic coding feature map is preliminarily screened to obtain a preliminary screening fingerprint-face cross-modal joint coding feature map as the preliminary screening fingerprint-face cross-modal joint coding feature.

[0014] In the above-mentioned method for jointly verifying digital identity using a combined model, the primary screening fingerprint-face cross-modal joint coding features are subjected to secondary feature screening to obtain the multimodal identity joint representation of the object to be verified, including: inputting the fingerprint-face semantic implicit common feature vector into a feature secondary screening module based on a multi-level gating unit to obtain a secondary fingerprint-face semantic feature screening weight vector; based on the secondary fingerprint-face semantic feature screening weight vector, the primary screening fingerprint-face cross-modal joint coding feature map is subjected to secondary feature screening to obtain a fingerprint-face cross-modal significant fusion coding feature map as the multimodal identity joint representation of the object to be verified.

[0015] In the above-mentioned method of jointly verifying digital identities using a combined model, determining whether the object to be verified belongs to an authorized object based on the multimodal identity joint representation of the object to be verified includes: inputting the multimodal identity joint representation of the object to be verified into an identity identifier based on a classifier to obtain an identity recognition result, and the identity recognition result is used to indicate whether the object to be verified belongs to an authorized object.

[0016] According to another aspect of the present application, a system for jointly verifying digital identities using a combined model is provided, which includes: an identity verification data acquisition module for acquiring a face image and a fingerprint image of a subject to be verified.

[0017] The face feature extraction module is used to extract face features from the face image of the object to be verified to obtain face semantic coding features.

[0018] The fingerprint feature extraction module is used to extract fingerprint features from the fingerprint image of the object to be verified to obtain fingerprint semantic coding features.

[0019] The multimodal joint encoding module of the object to be verified is used to perform joint encoding on the face semantic encoding features and the fingerprint semantic encoding features based on multimodal implicit association to obtain a multimodal identity joint representation of the object to be verified.

[0020] The authorization object determination module is used to determine whether the object to be verified belongs to the authorization object based on the multimodal identity joint representation of the object to be verified.

[0021] Compared with the prior art, the present application provides a method and system for joint verification of digital identity using a combined model, which collects the face image and fingerprint image of the object to be verified, and introduces an image processing and analysis algorithm based on artificial intelligence and deep learning at the back end to analyze the face image and the fingerprint image, so as to capture and characterize the multimodal identity joint representation between the face features and fingerprint features of the object to be verified, and use this identity joint representation to verify the digital identity to determine whether the object to be verified belongs to the authorized object. In this way, the digital identity can be more intelligently joint verified through the combination of the face recognition base model and the fingerprint recognition base model, so as to improve the intelligence level of identity authentication, enhance the security of the system, and prevent the risk of identity fraud. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 The present invention is a flowchart of a method for jointly verifying a digital identity using a combined model according to an embodiment of the present application.

[0024] Figure 2 A data flow diagram of a method for jointly verifying digital identities using a combined model according to an embodiment of the present application.

[0025] Figure 3 The present invention is a flowchart of a method for jointly verifying a digital identity using a combined model according to an embodiment of the present application, for jointly encoding the face semantic coding features and the fingerprint semantic coding features based on multimodal implicit association to obtain a joint representation of the multimodal identity of the object to be verified.

[0026] Figure 4 The present invention is a flowchart of performing feature association coding on the face semantic coding features and the fingerprint semantic coding features in the method for jointly verifying digital identity using a combined model according to an embodiment of the present application to obtain fingerprint-face semantic implicit common features.

[0027] Figure 5 A flowchart of performing feature distillation on the face semantic encoding feature map to obtain a face semantic distillation feature vector in a method for jointly verifying a digital identity using a combined model according to an embodiment of the present application.

[0028] Figure 6 A system block diagram of a system for jointly verifying digital identities using a combined model according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0030] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0031] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0032] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0033] Digital identity verification refers to the process of verifying the user's identity through technical means to ensure the authenticity and security of the user when conducting online transactions, accessing sensitive information or using services. A single biometric recognition method is affected by environmental factors and individual differences, which may lead to poor verification results.

[0034] It should be understood that different digital identity verification technologies (such as facial recognition, fingerprint recognition and palmprint recognition, etc.) each serve as a base model. The combination of these base models for joint identity verification can effectively reduce the risk of forgery and misidentification by integrating different biometric information, thereby enhancing the security and reliability of identity authentication.

[0035] Based on this, the present application proposes a method for joint verification of digital identity using a combined model. It can perform joint verification of digital identity by adopting a combination of two identity authentication base models, face recognition and fingerprint recognition. Specifically, the technical concept of the present application is to collect the face image and fingerprint image of the object to be verified, and introduce an image processing and analysis algorithm based on artificial intelligence and deep learning at the back end to analyze the face image and the fingerprint image, so as to capture and characterize the multimodal identity joint representation between the face features and fingerprint features of the object to be verified, and use this identity joint representation to verify the digital identity to determine whether the object to be verified belongs to the authorized object. In this way, the digital identity can be more intelligently joint verified through the combination of the face recognition base model and the fingerprint recognition base model. This multimodal identity authentication method not only improves the intelligence level of identity authentication, but also enhances the security of the system and prevents the risk of identity fraud.

[0036] Figure 1 The present invention is a flowchart of a method for jointly verifying a digital identity using a combined model according to an embodiment of the present application. Figure 2Schematic diagram of data flow of a method for joint verification of digital identity using a combined model according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the method of using a combined model to jointly verify a digital identity according to an embodiment of the present application, the method includes: S110, obtaining a face image and a fingerprint image of an object to be verified; S120, performing face feature extraction on the face image of the object to be verified to obtain face semantic coding features; S130, performing fingerprint feature extraction on the fingerprint image of the object to be verified to obtain fingerprint semantic coding features; S140, performing joint encoding based on multimodal implicit association on the face semantic coding features and the fingerprint semantic coding features to obtain a multimodal identity joint representation of the object to be verified; S150, determining whether the object to be verified belongs to an authorized object based on the multimodal identity joint representation of the object to be verified.

[0037] In the above-mentioned method for jointly verifying digital identity using a combined model, the step S110 obtains the face image and fingerprint image of the object to be verified. It should be understood that both the face and the fingerprint are unique identifiers of an individual and can effectively distinguish different individuals. Among them, the face image can capture a person's facial features, including eyes, nose, mouth and facial contours. The fingerprint is composed of unique lines on an individual's fingers. Everyone's fingerprint is unique, and it is almost impossible for there to be exactly the same situation. By obtaining face images and fingerprint images, the identity authentication system can achieve more secure and efficient user identity confirmation. Specifically, in face recognition, the face image of the object to be verified is captured by a camera, and then the face feature data is converted into a digital signal and stored in a database. The fingerprint of the object to be verified is captured using a fingerprint scanner or sensor.

[0038] In the above-mentioned method for joint verification of digital identity using a combined model, the step S120 extracts facial features from the facial image of the object to be verified to obtain facial semantic coding features. It should be understood that the original data of the facial image often contains a lot of redundant information, and the extraction of facial features can focus on key information related to the identity. This feature extraction can greatly improve the recognition accuracy, so that the system can still accurately identify individuals in different environments (such as light changes, different angles).

[0039] In an embodiment of the present application, the step S120, extracting facial features from the facial image of the object to be verified to obtain facial semantic coding features, includes: inputting the facial image of the object to be verified into a facial feature extraction model based on a deep separable convolutional neural network model to obtain a facial semantic coding feature map as the facial semantic coding feature. It should be understood that considering that the deep separable convolutional neural network model can maintain high recognition accuracy and stability whether in indoor and outdoor environments with changing light or at different angles and postures, and can capture complex patterns and details in facial images, filter out noise while retaining key information to improve the accuracy of face recognition. Therefore, in the technical solution of the present application, facial features are extracted from the facial image of the object to be verified by a facial feature extraction model based on a deep separable convolutional neural network model. Specifically, first, the acquired facial image of the object to be verified is normalized so that it is within a unified range (such as 0 to 1) to ensure the consistency of the model input. Then, the input image is adjusted to the input size required by the deep separable convolutional network (such as 224x224) to ensure that the model can be processed correctly. Next, a facial feature extraction model based on a deep separable convolutional neural network model is constructed, including multiple convolutional layers, activation layers (such as ReLU), and pooling layers. Next, the preprocessed face image is input into the model for forward propagation calculation, and features are gradually extracted through each layer of convolution operations. Furthermore, before the last layer of the model, the output feature map is extracted. This feature map contains a high-level semantic representation of the input image and can reflect the characteristic information of the face. Finally, the feature map is further processed, such as global average pooling, to obtain a face semantic encoding feature map, thereby capturing key details of the face, such as facial contours, expressions, and age characteristics, to identify individuals under various conditions.

[0040] In the above-mentioned method for joint verification of digital identity using a combined model, the step S130 extracts fingerprint features from the fingerprint image of the object to be verified to obtain fingerprint semantic coding features. It should be understood that compared with traditional identity authentication methods such as passwords or smart cards, fingerprint recognition is more resistant to counterfeiting. In addition, the fingerprint image contains rich detailed information, and after the fingerprint features are extracted, they can be quickly compared and verified. Therefore, in the technical solution of the present application, fingerprint features are extracted and semantically encoded to effectively prevent identity fraud and improve security.

[0041] In an embodiment of the present application, the step S130, extracting fingerprint features from the fingerprint image of the object to be verified to obtain fingerprint semantic coding features, includes: inputting the fingerprint image of the object to be verified into a fingerprint feature extraction model based on a multi-layer hole convolution layer to obtain a fingerprint semantic coding feature vector as the fingerprint semantic coding feature. It should be understood that compared with traditional convolution, hole convolution can extract more context information while maintaining resolution. Fingerprint images contain rich texture information, and hole convolution layers can effectively retain this information, especially in extracting complex structures and details. Therefore, in the technical solution of the present application, fingerprint features are extracted from the fingerprint image of the object to be verified by a fingerprint feature extraction model based on a multi-layer hole convolution layer. Among them, the hole convolution can expand the receptive field without increasing the number of parameters by inserting holes between the convolution kernels (that is, skipping certain positions during the convolution process), so as to capture features in the fingerprint image within a larger range, thereby improving the ability to extract subtle features. In addition, the structure of the multi-layer hole convolution layer enables the network to process fingerprint features at different scales. Different convolutional layers can learn the local and global features of fingerprints, thereby more comprehensively characterizing the uniqueness of fingerprints. In this way, it can better adapt to the complexity of fingerprint data, thereby improving the accuracy and robustness of fingerprint recognition. Specifically, the fingerprint image of the object to be verified is first preprocessed, including denoising and contrast enhancement, to improve the recognizability of the fingerprint. Then, the fingerprint image is normalized to a fixed size and grayscale range. Next, the processed fingerprint image is input into multiple dilated convolutional layers of a fingerprint feature extraction model based on a multi-layer dilated convolutional layer to perform a convolution operation on the fingerprint image. The convolution kernel of each layer performs dilated convolution at different positions to extract local and global features. Next, the activation function (such as ReLU) of the fingerprint feature extraction model based on a multi-layer dilated convolutional layer is used to introduce nonlinearity. The feature map after the activation layer may pass through a pooling layer to further reduce the feature dimension while retaining important information. Finally, the feature map after the multi-layer dilated convolution and activation will be flattened to form a fingerprint semantic encoding feature to concisely represent the unique features of the fingerprint.

[0042] In the above-mentioned method for joint verification of digital identity using a combined model, the step S140 performs joint encoding of the face semantic coding features and the fingerprint semantic coding features based on multimodal implicit association to obtain a joint representation of the multimodal identity of the object to be verified. It should be understood that face recognition is usually more sensitive to environmental changes (such as lighting), while fingerprint recognition has strong robustness at different angles and skin conditions. Jointly encoding the features of the two can make full use of their respective advantages to form a more stable and comprehensive identity representation. In addition, a single modality (such as using only face or fingerprint) may be affected by environmental changes, counterfeiting technology or poor image quality. By fusing multimodal features (face and fingerprint), the system can combine different types of data, reduce the limitations of a single modality, and thus improve the overall identity authentication accuracy and security. In a specific embodiment of the present application, first, principal component analysis (PCA) is used to align the face semantic coding features and the fingerprint semantic coding features to map the extracted features to a unified feature space. Then, after feature alignment, the face features and fingerprint features are fused using a multimodal implicit association learning model. Specifically, the potential association between face and fingerprint features is automatically learned through the fusion network to generate a joint feature vector. Then, the obtained joint feature vector is matched with the features of the authorized object stored in the database using cosine similarity to obtain the similarity score between the object to be verified and the authorized object. Finally, according to the result of the similarity calculation, the corresponding decision is made. If the identity authentication is successful, the user will be granted access rights; if the verification fails, the system will send corresponding feedback to the user, prompting re-verification or using other verification methods, thereby effectively improving the accuracy and security of verification.

[0043] Figure 3 The present invention is a flowchart of a method for jointly verifying digital identities using a combined model according to an embodiment of the present application, wherein the face semantic coding features and the fingerprint semantic coding features are jointly encoded based on multimodal implicit association to obtain a multimodal identity joint representation of the object to be verified. Figure 3 As shown, the step S140, performs joint encoding based on multimodal implicit association on the face semantic coding features and the fingerprint semantic coding features to obtain a multimodal identity joint representation of the object to be verified, including: S141, performing feature association encoding on the face semantic coding features and the fingerprint semantic coding features to obtain fingerprint-face semantic implicit common features; S142, performing preliminary feature screening on the fingerprint-face semantic implicit common features to obtain preliminary screening fingerprint-face cross-modal joint coding features; S143, performing secondary feature screening on the preliminary screening fingerprint-face cross-modal joint coding features to obtain a multimodal identity joint representation of the object to be verified.

[0044] Specifically, in the step S141, the face semantic coding features and the fingerprint semantic coding features are feature-associated coded to obtain fingerprint-face semantic implicit common features. It should be understood that since the face semantic coding feature map and the fingerprint semantic coding feature vector respectively contain face semantic features and fingerprint semantic features related to the object to be verified, and identity feature information of different modalities is of great significance for the recognition and verification of digital identity, the accuracy of verification can obviously be improved by fusing the semantics of the two biometric features. However, in multimodal identity authentication, how to effectively fuse the features of two different modalities, face features and fingerprint features, is an important technical issue. Therefore, in the technical solution of the present application, the face semantic coding features and the fingerprint semantic coding features are further jointly encoded based on multimodal implicit association to obtain a multimodal identity joint representation of the object to be verified. By performing joint encoding based on multimodal implicit associations, it is possible to analyze and fuse biometric features from different perceptual channels (i.e., different modalities) to generate a unified and optimized multimodal identity joint representation. This encoding method aims to identify and utilize the potential connections between biometric features of different modalities (face semantic features and fingerprint semantic features), even if these connections are not directly apparent or obvious. This encoding method ensures that the final multimodal identity joint representation can not only capture the structural characteristics within a single bio-semantic feature modality, but also reflect the complementarity and consistency between biometric features across modalities.

[0045] Figure 4 The present invention is a flowchart of performing feature association coding on the face semantic coding features and the fingerprint semantic coding features to obtain fingerprint-face semantic implicit common features in the method for jointly verifying digital identities using a combined model according to an embodiment of the present application. Figure 4 As shown, in an embodiment of the present application, the step S141, performing feature association encoding on the face semantic coding features and the fingerprint semantic coding features to obtain fingerprint-face semantic implicit common features, includes: S1411, performing feature distillation on the face semantic coding feature map to obtain a face semantic distillation feature vector; S1412, inputting the fingerprint semantic coding feature vector and the face semantic distillation feature vector into a multimodal implicit association feature capture network to obtain a fingerprint-face semantic implicit common feature vector as the fingerprint-face semantic implicit common feature.

[0046] Specifically, in step S1411, feature distillation is performed on the face semantic coding feature map to obtain a face semantic distillation feature vector. It should be understood that feature distillation is performed on the face semantic coding feature map to extract a more compact and representative feature vector, namely, a face semantic distillation feature vector, through feature distillation technology. This process removes redundant information in the face semantic features and retains the most critical face information, so that the obtained face semantic distillation feature vector and the fingerprint semantic coding feature vector can be compared and fused in the same semantic space.

[0047] Figure 5 This is a flow chart of performing feature distillation on the face semantic encoding feature map to obtain a face semantic distillation feature vector in a method for jointly verifying digital identity using a combined model according to an embodiment of the present application. Figure 5 As shown, in an embodiment of the present application, the step S1411, performing feature distillation on the face semantic coding feature map to obtain a face semantic distillation feature vector, includes: S1411-1, performing point convolution encoding on the face semantic coding feature map to obtain a face semantic coding channel modulation feature map; S1411-2, performing global mean pooling processing on each feature matrix along the channel dimension in the face semantic coding channel modulation feature map to obtain a face semantic channel feature vector; S1411-3, inputting the face semantic channel feature vector into a Sigmoid function for processing to obtain the face semantic distillation feature vector.

[0048] Specifically, in step S1412, the fingerprint semantic encoding feature vector and the face semantic distillation feature vector are input into a multimodal implicit association feature capture network to obtain a fingerprint-face semantic implicit common feature vector as the fingerprint-face semantic implicit common feature. It should be understood that the face semantic distillation feature vector and the fingerprint semantic encoding feature vector are input into a multimodal implicit association feature capture network together. The task of this network is to discover and capture the potential common patterns or structures between the distillation features of face semantics and the fingerprint semantic features. This approach strengthens the consistent expression of cross-modal semantic fusion features through a collaborative attention mechanism, promotes the automatic discovery of the intrinsic connections between cross-modal biometric features from different sources, and constructs a fingerprint-face semantic implicit common feature vector as a fingerprint-face semantic implicit common feature.

[0049] In an embodiment of the present application, the step S1412, inputting the fingerprint semantic encoding feature vector and the face semantic distillation feature vector into a multimodal implicit association feature capture network to obtain a fingerprint-face semantic implicit common feature vector as the fingerprint-face semantic implicit common feature, includes: S1412-1, cascading the fingerprint semantic encoding feature vector and the face semantic distillation feature vector to obtain a fingerprint-face semantic cascade feature vector; S1412-2, multiplying the fingerprint-face semantic cascade feature vector with a linear transformation weight matrix and then adding the linear transformation bias vector by position to obtain a linear transformation fingerprint-face semantic cascade feature vector; S1412-3, concatenating the linear transformation fingerprint-face semantic cascade feature vector through The fingerprint-face semantic implicit common feature vector is obtained by processing in the function.

[0050] In an embodiment of the present application, the step S142, performing preliminary feature screening on the fingerprint-face semantic implicit common features to obtain preliminary screening fingerprint-face cross-modal joint coding features, includes: S1421, inputting the fingerprint-face semantic implicit common feature vector into a feature preliminary screening module based on a gating unit to obtain a first-level fingerprint-face semantic feature preliminary screening weight vector; S1422, based on the first-level fingerprint-face semantic feature preliminary screening weight vector, performing preliminary feature screening on the face semantic coding feature map to obtain a preliminary screening fingerprint-face cross-modal joint coding feature map as the preliminary screening fingerprint-face cross-modal joint coding feature.

[0051] Specifically, in step S1421, the fingerprint-face semantic implicit common feature vector is input into the feature preliminary screening module based on the gate control unit to obtain the primary fingerprint-face semantic feature preliminary screening weight vector. It should be understood that here, the gate control unit dynamically controls the semantic information flow by learning weight parameters, allowing important fingerprint-face semantic joint features to pass through, while suppressing parts that are not related to subsequent identity joint representation and identity authentication tasks, reducing noise interference, and thus calculating the primary fingerprint-face semantic feature preliminary screening weight vector.

[0052] Specifically, the step S1421 inputs the fingerprint-face semantic implicit common feature vector into a feature preliminary screening module based on a gate control unit to obtain a first-level fingerprint-face semantic feature preliminary screening weight vector, including: multiplying the fingerprint-face semantic implicit common feature vector with the preliminary screening weight matrix and then adding it to the preliminary screening bias vector by position to obtain a preliminary screening linear transformation fingerprint-face semantic implicit common feature vector; the preliminary screening linear transformation fingerprint-face semantic implicit common feature vector is passed through The function is used to process the primary fingerprint-face semantic feature preliminary screening modulation vector to obtain the primary fingerprint-face semantic feature preliminary screening modulation vector; the primary fingerprint-face semantic feature preliminary screening modulation vector is input into the mask function for processing to obtain the primary fingerprint-face semantic feature preliminary screening weight vector; wherein, in response to the eigenvalue in the primary fingerprint-face semantic feature preliminary screening modulation vector being greater than twice a preset threshold, the eigenvalue greater than twice the preset threshold is taken as the original value; in response to the eigenvalue in the primary fingerprint-face semantic feature preliminary screening modulation vector being less than the preset threshold, the eigenvalue less than the preset threshold is set to 0; in response to the eigenvalue in the primary fingerprint-face semantic feature preliminary screening modulation vector being greater than the preset threshold and less than twice the preset threshold, half of the eigenvalue is taken.

[0053] Specifically, in the step S1422, based on the primary fingerprint-face semantic feature preliminary screening weight vector, the face semantic coding feature map is subjected to preliminary feature screening to obtain a preliminary screening fingerprint-face cross-modal joint coding feature map as the preliminary screening fingerprint-face cross-modal joint coding feature. It should be understood that in the technical solution of the present application, since the fingerprint semantic feature information from another modality has been taken into account at this time, the obtained preliminary screening fingerprint-face cross-modal joint coding feature not only reflects the structural characteristics within a single modality, but also reflects the potential complementary effect between the two different modalities of face semantic features and fingerprint semantic features.

[0054] Specifically, the step S1422, based on the primary fingerprint-face semantic feature preliminary screening weight vector, performs preliminary feature screening on the face semantic coding feature map to obtain a preliminary screening fingerprint-face cross-modal joint coding feature map as the preliminary screening fingerprint-face cross-modal joint coding feature, including: using each eigenvalue in the primary fingerprint-face semantic feature preliminary screening weight vector as the channel weighting weight, performing weighted processing on the face semantic coding feature map based on the channel dimension to obtain a channel preliminary screening face semantic coding feature map; inputting the channel preliminary screening face semantic coding feature map into The function is used to process the initial screening fingerprint-face cross-modal joint encoding feature map.

[0055] In an embodiment of the present application, the step S143, performing secondary feature screening on the initial screening fingerprint-face cross-modal joint coding features to obtain the multi-modal identity joint representation of the object to be verified, includes: S1431, inputting the fingerprint-face semantic implicit common feature vector into a secondary feature screening module based on a multi-level gating unit to obtain a secondary fingerprint-face semantic feature screening weight vector; S1432, based on the secondary fingerprint-face semantic feature screening weight vector, performing secondary feature screening on the initial screening fingerprint-face cross-modal joint coding feature map to obtain a fingerprint-face cross-modal significant fusion coding feature map as the multi-modal identity joint representation of the object to be verified.

[0056] Specifically, in step S1431, the fingerprint-face semantic implicit common feature vector is input into a feature secondary screening module based on a multi-level gating unit to obtain a secondary fingerprint-face semantic feature screening weight vector. It should be understood that in order to further improve the model performance, the fingerprint-face semantic implicit common feature vector is again input into a feature secondary screening module comprising multiple hierarchical gating units. The design of the multi-level gating unit can manage information transmission more finely, with each layer focusing on the extraction or pattern discovery of a specific type of fingerprint-face semantic joint feature, and finally synthesizing a comprehensive weight allocation scheme, namely, a secondary fingerprint-face semantic feature screening weight vector. This method effectively enhances the ability to capture complex interaction patterns between fingerprint and face semantics, while also improving the quality of the final output results, which is helpful for subsequent digital identity authentication tasks.

[0057] Specifically, the step S1431 inputs the fingerprint-face semantic implicit common feature vector into a feature secondary screening module based on a multi-level gating unit to obtain a secondary fingerprint-face semantic feature screening weight vector, including: multiplying the fingerprint-face semantic implicit common feature vector with the primary screening weight matrix and then adding it to the primary screening bias vector by position to obtain the primary screening linear transformation fingerprint-face semantic implicit common feature vector; the primary screening linear transformation fingerprint-face semantic implicit common feature vector is obtained by The first-level fingerprint-human face semantic feature preliminary screening modulation vector is processed in the function to obtain the first-level fingerprint-human face semantic feature preliminary screening modulation vector; the first-level fingerprint-human face semantic feature preliminary screening modulation vector is input into the mask function for processing to obtain the second-level fingerprint-human face semantic feature screening weight vector.

[0058] Specifically, in the step S1432, based on the secondary fingerprint-face semantic feature screening weight vector, the primary screening fingerprint-face cross-modal joint coding feature map is subjected to secondary feature screening to obtain a fingerprint-face cross-modal significant fusion coding feature map as the multi-modal identity joint representation of the object to be verified. It should be understood that the previously formed primary screening fingerprint-face cross-modal joint coding feature map is subjected to a second refinement process using the latest determined secondary fingerprint-face semantic feature screening weight vector. After two consecutive targeted selections and reconstructions, a highly concentrated cross-modal significant fusion coding feature map rich in significant association information between the two modalities is finally obtained. After two targeted selections and reconstructions, a highly concentrated fingerprint-face cross-modal significant fusion coding feature map rich in significant association information between the two modalities of face semantics and fingerprint semantics is finally obtained as the multi-modal identity joint representation of the object to be verified. It is helpful to be more beneficial to subsequent identity recognition operations and identity verification tasks, because it not only contains the unique information of each biometric modality, but also highlights the complementarity and consistency between the two modalities of face and fingerprint.

[0059] Specifically, the step S1432, based on the secondary fingerprint-face semantic feature screening weight vector, performs secondary feature screening on the primary screening fingerprint-face cross-modal joint coding feature map to obtain a fingerprint-face cross-modal significant fusion coding feature map as the multimodal identity joint representation of the object to be verified, including: using each feature value in the secondary fingerprint-face semantic feature screening weight vector as the channel weighting weight, performing weighted processing based on the channel dimension on the primary screening fingerprint-face cross-modal joint coding feature map to obtain a channel-screened fingerprint-face cross-modal joint coding feature map; inputting the channel-screened fingerprint-face cross-modal joint coding feature map into the The fingerprint-face cross-modal significant fusion encoding feature map is processed in the function to obtain the fingerprint-face cross-modal significant fusion encoding feature map.

[0060] Specifically, the face semantic coding features and the fingerprint semantic coding features are jointly encoded based on multimodal implicit association using the following multimodal joint coding formula to obtain a multimodal identity joint representation of the object to be verified; wherein the multimodal joint coding formula is: .

[0061] in, is the face semantic coding feature map, is point convolutional coding, is the mean pooling process, for function, is the face semantic distillation feature vector, is the fingerprint semantic encoding feature vector, For vector cascade processing, and are the linear transformation weight matrix and the linear transformation bias vector respectively, for function, is the fingerprint-face semantic implicit common feature vector, and are the initial screening weight matrix and the initial screening bias vector respectively, is matrix multiplication, is the primary fingerprint-face semantic feature initial screening modulation vector, is the mask function, is the primary fingerprint-face semantic feature initial screening weight vector, is the channel point product by position, To initially screen the fingerprint-face cross-modal joint encoding feature map, and are the screening weight matrix and the screening bias vector respectively, Filter the modulation vector for the secondary fingerprint-face semantic features, Filter weight vectors for secondary fingerprint-face semantic features, is the predetermined threshold, A feature graph is provided for jointly representing the multimodal identity of the object to be verified.

[0062] In the above-mentioned method for joint verification of digital identity using a combined model, the step S150 determines whether the object to be verified belongs to an authorized object based on the joint representation of the multimodal identity of the object to be verified. It should be understood that by the joint representation of the multimodal identity of the object to be verified, the identity authentication system can more comprehensively and accurately determine whether the individual belongs to an authorized object. This method not only improves the reliability of identity recognition, but also effectively prevents various security risks. In one embodiment of the application, a fully connected neural network is used as a classifier to train a training data set containing multimodal feature vectors from different authorized users and their corresponding labels (such as "authorized" or "unauthorized"). By learning the training data, the classifier can identify which feature combinations represent authorized objects. Then, the feature vector of the object to be verified is compared with the classifier of the authorization feature library. Among them, if the classifier classifies the joint feature vector of the object to be verified as "authorized", the verification is passed; if it is classified as "unauthorized", access is denied. This method not only improves the accuracy of identity authentication, but also effectively reduces the false positive rate and false negative rate.

[0063] In an embodiment of the present application, the step S150, based on the multimodal identity joint representation of the object to be verified, determines whether the object to be verified belongs to the authorized object, including: inputting the multimodal identity joint representation of the object to be verified into an identity identifier based on a classifier to obtain an identity recognition result, and the identity recognition result is used to indicate whether the object to be verified belongs to the authorized object. In other words, the multimodal identity joint representation features between the fingerprint semantics and the face semantics of the object to be verified are used to perform digital identity recognition and verification to determine whether the object to be verified belongs to the authorized object. In this way, the digital identity can be more intelligently jointly verified through the combination of the face recognition base model and the fingerprint recognition base model. This multimodal identity authentication method not only improves the intelligence level of identity authentication, but also enhances the security of the system and prevents the risk of identity misuse.

[0064] Here, in the case where the face semantic encoding feature map and the fingerprint semantic encoding feature vector respectively represent the image semantic features of the face image and the fingerprint image based on different image semantic representation dimensions, when performing cross-modal joint encoding based on multimodal implicit association, the multimodal identity joint representation feature map of the object to be verified will also cause cross-modal joint encoding diversity due to the differences in the image semantic cross-modal implicit association patterns, affecting the classification mapping regression offset and the accuracy of the identity recognition result obtained by the identity recognizer based on the classifier.

[0065] Based on this, when the multimodal identity joint representation feature map of the object to be verified is input into the classifier-based identity identifier for classification probability regression, the multimodal identity joint representation feature map of the object to be verified is first optimized, and the optimization process specifically includes: expanding the multimodal identity joint representation feature map of the object to be verified into a multimodal identity joint representation feature vector of the object to be verified; determining the mean and variance of the feature set composed of the eigenvalues ​​of the multimodal identity joint representation feature vector of the object to be verified and dividing the mean by the variance to obtain a probability weight ;in, and represent the mean and variance of the feature set composed of the eigenvalues ​​of the eigenvector jointly representing the multimodal identity of the object to be verified, Represents the probability weight.

[0066] The first intermediate feature vector is obtained by multiplying the feature vector of the multimodal identity joint representation of the object to be verified by the inverse of its maximum eigenvalue. ,in, represents a multi-modal identity joint representation feature vector of the object to be verified, represents the maximum eigenvalue in the multi-modal identity joint representation eigenvector of the object to be verified, It means point multiplication by position. represents the first intermediate eigenvector.

[0067] The first intermediate feature vector is added with the probability weight and then the logarithm with base 2 is calculated to obtain the second intermediate feature vector ;in, represents the first intermediate eigenvector, represents the probability weight, Indicates adding by position point. represents the second intermediate eigenvector.

[0068] After subtracting the first intermediate eigenvector from the probability weight point, the inverse of each eigenvalue is calculated to obtain a third intermediate eigenvector ;in, represents the first intermediate eigenvector, represents the probability weight, It means to subtract by position point. represents the third intermediate eigenvector.

[0069] Finally, the second intermediate feature vector is added to the third intermediate feature vector to obtain an optimized multimodal identity joint representation feature vector of the object to be verified: ,in, represents the second intermediate eigenvector, Indicates adding by position point. represents the third intermediate eigenvector, Represents the optimized multimodal identity joint representation feature vector of the object to be verified.

[0070] Then, the optimized multi-modal identity joint representation feature vector of the object to be verified is input into an identity recognizer based on a classifier to obtain an identity recognition result.

[0071] That is, considering that when the weight matrix during classification probability regression acts on the feature vector of the multimodal identity joint representation of the object to be verified, the classification regression offset of the feature vector of the multimodal identity joint representation of the object to be verified causes the uncertainty of the weight stimulus parameters of the weight matrix, which makes the posterior probability density inference of the feature vector of the multimodal identity joint representation of the object to be verified missing through the weight matrix, which will affect the accuracy of the probability regression result. The above optimization process simulates the mesoscale interaction structure under probability constraints between the eigenvalue scale and the eigenvector scale of the feature vector of the multimodal identity joint representation of the object to be verified through the probability statistical characteristics of the feature vector of the multimodal identity joint representation of the object to be verified, so as to construct a bidirectional hidden variable motif based on the mesoscale short sequence relative to the probability statistical value to perform mesoscale bidirectional migration, and perform posterior recovery based on the sequence constraint on the interactive information, so as to improve the convergence effect of the feature vector of the multimodal identity joint representation of the object to be verified in the probability density domain, thereby improving the accuracy of the identity recognition result obtained by the identity recognizer based on the classifier. In this way, the digital identity can be automatically verified jointly through the combination of the face recognition base model and the fingerprint recognition base model. This multimodal identity authentication method not only improves the intelligence level of identity authentication, but also enhances the security of the system and prevents the risk of identity impersonation.

[0072] In summary, a method for joint verification of digital identity using a combined model based on an embodiment of the present application is explained, which collects the face image and fingerprint image of the object to be verified, and introduces an image processing and analysis algorithm based on artificial intelligence and deep learning at the back end to analyze the face image and the fingerprint image, so as to capture and characterize the multimodal identity joint representation between the face features and fingerprint features of the object to be verified, and use this identity joint representation to verify the digital identity to determine whether the object to be verified belongs to the authorized object. In this way, the digital identity can be more intelligently joint verified through the combination of the face recognition base model and the fingerprint recognition base model, so as to improve the intelligence level of identity authentication, enhance the security of the system, and prevent the risk of identity fraud.

[0073] Figure 6 FIG. 1 is a system block diagram of a system for jointly verifying digital identities using a combined model according to an embodiment of the present application. Figure 6As shown, according to an embodiment of the present application, a system 100 for jointly verifying a digital identity using a combined model includes: an identity authentication data acquisition module 110, used to acquire a face image and a fingerprint image of an object to be verified; a face feature extraction module 120, used to extract face features from the face image of the object to be verified to obtain face semantic coding features; a fingerprint feature extraction module 130, used to extract fingerprint features from the fingerprint image of the object to be verified to obtain fingerprint semantic coding features; a multimodal joint encoding module 140 for an object to be verified, used to perform joint encoding based on multimodal implicit association on the face semantic coding features and the fingerprint semantic coding features to obtain a multimodal identity joint representation of the object to be verified; an authorized object determination module 150, used to determine whether the object to be verified belongs to an authorized object based on the multimodal identity joint representation of the object to be verified.

[0074] Here, those skilled in the art will appreciate that the specific operations of each step in the system for jointly verifying digital identities using a combined model have been described above with reference to Figures 1 to 5 The method for jointly verifying digital identities using a combined model has been described in detail, and therefore, its repeated description will be omitted.

[0075] As described above, the system 100 for jointly verifying digital identities using a combination model according to the embodiment of the present application can be implemented in various terminal devices. In one example, the system 100 for jointly verifying digital identities using a combination model can be integrated into the terminal device as a software module and / or a hardware module. For example, the system 100 for jointly verifying digital identities using a combination model can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the system 100 for jointly verifying digital identities using a combination model can also be one of the many hardware modules of the terminal device.

[0076] Alternatively, in another example, the system 100 for jointly verifying digital identities using a combined model and the terminal device may be separate devices, and the system 100 for jointly verifying digital identities using a combined model may be connected to the terminal device via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0077] In summary, a system for jointly verifying digital identities using a combined model based on an embodiment of the present application is described, which collects facial images and fingerprint images of the object to be verified, and introduces an image processing and analysis algorithm based on artificial intelligence and deep learning at the back end to analyze the facial images and fingerprint images, so as to capture and characterize a multimodal identity joint representation between the facial features and fingerprint features of the object to be verified, and use this identity joint representation to verify the digital identity to determine whether the object to be verified belongs to an authorized object. In this way, the digital identity can be more intelligently jointly verified through the combination of the face recognition base model and the fingerprint recognition base model, so as to improve the intelligence level of identity authentication, enhance the security of the system, and prevent the risk of identity fraud.

[0078] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0079] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0080] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0081] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0082] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0083] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit of the technical solution of the present invention.

Claims

1. A method for joint verification of digital identity using a combined model, characterized in that: include: Acquire a face image and a fingerprint image of the object to be verified; extract face features from the face image of the object to be verified to obtain face semantic coding features; Extract fingerprint features from the fingerprint image of the object to be verified to obtain fingerprint semantic coding features; perform joint coding based on multimodal implicit association on the face semantic coding features and the fingerprint semantic coding features to obtain a joint representation of the multimodal identity of the object to be verified; determine whether the object to be verified belongs to an authorized object based on the joint representation of the multimodal identity of the object to be verified; wherein, perform joint coding based on multimodal implicit association on the face semantic coding features and the fingerprint semantic coding features to obtain a joint representation of the multimodal identity of the object to be verified, including: perform feature association coding on the face semantic coding features and the fingerprint semantic coding features to obtain fingerprint-face semantic implicit common features; perform preliminary feature screening on the fingerprint-face semantic implicit common features to obtain preliminary screening fingerprint-face cross-modal joint coding features; perform secondary feature screening on the preliminary screening fingerprint-face cross-modal joint coding features to obtain a joint representation of the multimodal identity of the object to be verified; Among them, the fingerprint-face semantic implicit common features are preliminarily screened to obtain preliminary screening fingerprint-face cross-modal joint coding features, including: inputting the fingerprint-face semantic implicit common feature vector into a feature preliminarily screened module based on a gate control unit to obtain a primary fingerprint-face semantic feature preliminarily screened weight vector; based on the primary fingerprint-face semantic feature preliminarily screened weight vector, the facial semantic coding feature map is preliminarily screened to obtain a preliminary screening fingerprint-face cross-modal joint coding feature map as the preliminary screening fingerprint-face cross-modal joint coding feature; Among them, the primary screening fingerprint-face cross-modal joint coding features are subjected to secondary feature screening to obtain the multi-modal identity joint representation of the object to be verified, including: inputting the fingerprint-face semantic implicit common feature vector into a feature secondary screening module based on a multi-level gating unit to obtain a secondary fingerprint-face semantic feature screening weight vector; based on the secondary fingerprint-face semantic feature screening weight vector, the primary screening fingerprint-face cross-modal joint coding feature map is subjected to secondary feature screening to obtain a fingerprint-face cross-modal significant fusion coding feature map as the multi-modal identity joint representation of the object to be verified.

2. The method for joint verification of digital identity using a combined model according to claim 1, characterized in that: Performing facial feature extraction on the facial image of the object to be verified to obtain facial semantic coding features, including: inputting the facial image of the object to be verified into a facial feature extraction model based on a deep separable convolutional neural network model to obtain a facial semantic coding feature map as the facial semantic coding features.

3. The method for joint verification of digital identity using a combined model according to claim 2, characterized in that: The fingerprint image of the object to be verified is subjected to fingerprint feature extraction to obtain fingerprint semantic coding features, including: inputting the fingerprint image of the object to be verified into a fingerprint feature extraction model based on a multi-layer dilated convolutional layer to obtain a fingerprint semantic coding feature vector as the fingerprint semantic coding feature.

4. The method for joint verification of digital identity using a combined model according to claim 3, characterized in that: The face semantic coding features and the fingerprint semantic coding features are subjected to feature association coding to obtain fingerprint-face semantic implicit common features, including: performing feature distillation on the face semantic coding feature map to obtain a face semantic distillation feature vector; inputting the fingerprint semantic coding feature vector and the face semantic distillation feature vector into a multimodal implicit association feature capture network to obtain a fingerprint-face semantic implicit common feature vector as the fingerprint-face semantic implicit common feature.

5. The method for joint verification of digital identity using a combined model according to claim 4, characterized in that: The face semantic coding feature map is subjected to feature distillation to obtain a face semantic distillation feature vector, including: performing point convolution coding on the face semantic coding feature map to obtain a face semantic coding channel modulation feature map; performing global mean pooling processing on each feature matrix along the channel dimension in the face semantic coding channel modulation feature map to obtain a face semantic channel feature vector; and inputting the face semantic channel feature vector into a Sigmoid function for processing to obtain the face semantic distillation feature vector.

6. The method for joint verification of digital identity using a combined model according to claim 5, characterized in that: The fingerprint semantic encoding feature vector and the face semantic distillation feature vector are input into a multimodal implicit association feature capture network to obtain a fingerprint-face semantic implicit common feature vector as the fingerprint-face semantic implicit common feature, including: cascading the fingerprint semantic encoding feature vector and the face semantic distillation feature vector to obtain a fingerprint-face semantic cascade feature vector; multiplying the fingerprint-face semantic cascade feature vector with a linear transformation weight matrix and then adding the linear transformation bias vector by position to obtain a linear transformation fingerprint-face semantic cascade feature vector; and concatenating the linear transformation fingerprint-face semantic cascade feature vector through The fingerprint-face semantic implicit common feature vector is obtained by processing in the function.

7. The method for joint verification of digital identity using a combined model according to claim 6, characterized in that: Based on the multimodal identity joint representation of the object to be verified, determining whether the object to be verified belongs to the authorized object includes: inputting the multimodal identity joint representation of the object to be verified into an identity identifier based on a classifier to obtain an identity recognition result, and the identity recognition result is used to indicate whether the object to be verified belongs to the authorized object.

8. A system for joint verification of digital identity using a combined model, characterized in that: include: The identity verification data acquisition module is used to obtain the face image and fingerprint image of the object to be verified; A face feature extraction module, used to extract face features from the face image of the object to be verified to obtain face semantic coding features; A fingerprint feature extraction module, used to extract fingerprint features from the fingerprint image of the object to be verified to obtain fingerprint semantic coding features; a multimodal joint coding module for the object to be verified, used to perform joint coding based on multimodal implicit association on the face semantic coding features and the fingerprint semantic coding features to obtain a multimodal identity joint representation of the object to be verified; an authorized object determination module, used to determine whether the object to be verified belongs to an authorized object based on the multimodal identity joint representation of the object to be verified; Among them, the face semantic coding features and the fingerprint semantic coding features are jointly encoded based on multimodal implicit association to obtain a multimodal identity joint representation of the object to be verified, including: performing feature association encoding on the face semantic coding features and the fingerprint semantic coding features to obtain fingerprint-face semantic implicit common features; performing preliminary feature screening on the fingerprint-face semantic implicit common features to obtain preliminary screening fingerprint-face cross-modal joint coding features; performing secondary feature screening on the preliminary screening fingerprint-face cross-modal joint coding features to obtain a multimodal identity joint representation of the object to be verified; Among them, the fingerprint-face semantic implicit common features are preliminarily screened to obtain preliminary screening fingerprint-face cross-modal joint coding features, including: inputting the fingerprint-face semantic implicit common feature vector into a feature preliminarily screened module based on a gate control unit to obtain a primary fingerprint-face semantic feature preliminarily screened weight vector; based on the primary fingerprint-face semantic feature preliminarily screened weight vector, the facial semantic coding feature map is preliminarily screened to obtain a preliminary screening fingerprint-face cross-modal joint coding feature map as the preliminary screening fingerprint-face cross-modal joint coding feature; Among them, the primary screening fingerprint-face cross-modal joint coding features are subjected to secondary feature screening to obtain the multi-modal identity joint representation of the object to be verified, including: inputting the fingerprint-face semantic implicit common feature vector into a feature secondary screening module based on a multi-level gating unit to obtain a secondary fingerprint-face semantic feature screening weight vector; based on the secondary fingerprint-face semantic feature screening weight vector, the primary screening fingerprint-face cross-modal joint coding feature map is subjected to secondary feature screening to obtain a fingerprint-face cross-modal significant fusion coding feature map as the multi-modal identity joint representation of the object to be verified.

Images