Multi-modal fusion identity authentication method for judicial scene, medium and equipment

Abstract

The application discloses a multi-modal fusion identity identification method for a judicial scene, a medium and equipment, and belongs to the field of identity identification. The application can extract features from various modal data such as face, iris and fingerprint representing the identity of a human body, and integrate facial micro-expression, wrinkle and key point information, so that misidentification problems caused by changes in facial texture are avoided, and finally, multi-modal features and facial micro-features are jointly learned with the aid of a self-attention mechanism to obtain a final identification result. The method can be applied to judicial identity identification scenes such as case filing, court trial and lawsuit service in intelligent justice, and excellent identity identification capability is achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of identity recognition, and particularly relates to a multi-modal fusion identity authentication method for a judicial scene, a medium and equipment. BACKGROUND

[0002] In the scenes of case filing, court trial, and lawsuit serving in the judicial field, the identities of parties, agents and other personnel need to be identified. Fingerprint, face, iris and other biological information are the most common biological information used for personnel identity authentication.

[0003] The accuracy of identity recognition in the scenes of the judicial field is extremely high. In theory, multi-modal data fusion can further improve the identity authentication capability. However, the multi-modal fusion scheme in the prior art has some problems in actual application. For example, the case closest to the present application is CN115565228A, which only uses two modal data (face and iris) for fusion, and is mainly used in special scenes to overcome the problem of inability to identify caused by local occlusion or damage of single biological characteristics. However, unlike ordinary identity recognition scenes, in the scenes of case filing, court trial, and lawsuit serving, the faces of personnel may be distorted or wrinkles may appear due to emotional excitement, long time in a high-pressure atmosphere and other reasons. For such situations, how to use multi-modal data to achieve accurate identity authentication is a technical problem to be solved at present. SUMMARY

[0004] The purpose of the present application is to solve the problems in the prior art and provide a multi-modal fusion identity authentication method for a judicial scene, a medium and equipment.

[0005] The specific technical solutions adopted by the present application are as follows:

[0006] In a first aspect, the present application provides a multi-modal fusion identity authentication method for a judicial scene, which comprises:

[0007] S1, acquiring multi-modal data of a target personnel, including a face image, an iris image or a fingerprint image;

[0008] S2, respectively extracting modal features of the fingerprint image, the iris image and the face image of the personnel to obtain first fingerprint features, first iris features and first face features, and further extracting micro-expression features and face key point features based on the face image;

[0009] S3, inputting the first fingerprint features, the first iris features and the first face features into a multi-modal feature fusion module learned by a sparse adaptive multi-task learning method, and respectively forming second fingerprint features, second iris features and second face features after further feature extraction and fusion;

[0010] S4, input the face key point feature and the second face feature into a graph convolutional neural network for face decoupling and structure learning under wrinkle decorrelation constraint to obtain wrinkle-removed face decoupling features;

[0011] S5, input the second fingerprint feature, the second iris feature, the wrinkle-removed face decoupling feature and the micro-expression feature into a self-attention module after splicing, form a multi-modal fusion feature through a self-attention mechanism, and perform feature matching in a personnel library based on the multi-modal fusion feature to obtain an identity authentication result.

[0012] As a preferred embodiment of the first aspect, the multi-modal feature fusion module comprises a basic network for extracting features and an adaptive expansion mechanism module for adapting to new tasks;

[0013] The basic network comprises a plurality of cascaded feature extraction modules, the input of the basic network is the extracted multi-modal features, and the output is multi-modal features obtained through further feature extraction and fusion;

[0014] The adaptive expansion mechanism module has multiple groups, each group of adaptive expansion mechanism modules corresponds to processing one feature modality in the multi-modal features; the number of modules in each group of adaptive expansion mechanism modules is the same as the number of feature extraction modules in the basic network, and the output end of each feature extraction module is connected to one adaptive expansion mechanism module; the original output of the feature extraction module is first subjected to the adaptive expansion mechanism module, and then the output of the adaptive expansion mechanism module is added to the original output in a residual connection manner and used as the input of the downstream cascaded module of the feature extraction module;

[0015] In the adaptive expansion mechanism module, the module input is first subjected to a feature adapter composed of three convolutional layers to obtain intermediate features, and then the intermediate features are input into a gated cropping strategy module to obtain a probability representing whether the current adaptive expansion mechanism module needs to be activated through a Gumbel-Softmax operation, and the probability is multiplied by the intermediate features to obtain the output of the current adaptive expansion mechanism module.

[0016] As a preferred embodiment of the first aspect, the three convolutional layers in the feature adapter have sizes of 1x1, 3x3 and 1x1, respectively.

[0017] As a preferred embodiment of the first aspect, the sparse adaptive multi-task learning method used by the multi-modal feature fusion module in joint learning is as follows:

[0018] Three different classification heads are connected at the end of the multi-modal feature fusion module, thereby forming a multi-task joint learning framework for multi-task joint learning.

[0019] For image samples of three different modalities of fingerprint images, iris images and face images, the extracted features are input into the multi-task learning framework for joint learning. During the joint learning process, after the features of each modality of image samples are input into the multi-task learning framework, they are first processed by the multi-modal feature fusion module, the base network and a set of adaptive expansion mechanism modules corresponding to the modality, and then output to the corresponding classification head to obtain the personnel identity identification result.

[0020] And during the multi-task joint learning, the loss function used for learning of each modality is represented as:

[0021]

[0022] Among them is the classification cross-entropy loss corresponding to personnel identity recognition; is the sparse constraint of the neural network, is the activation constraint of the neural network, and the calculation formula of the two constraints is:

[0023]

[0024]

[0025] In the formula, L represents the number of feature extraction modules in the base network, θ j，k represents the kth network parameter of the adaptive expansion mechanism module corresponding to the jth feature extraction module, |Ω j | represents the set of learnable weight parameters of the adaptive expansion mechanism module corresponding to the jth feature extraction module Ω j . and respectively represent the sparsity and activation, and the calculation formula is:

[0026]

[0027]

[0028] In the formula, represents the output probability of the gating clipping strategy module corresponding to the kth sample in the batch, |B| is the number of samples in a batch, represents the calculation probability, and σ represents the threshold value.

[0029] As a preferred embodiment of the first aspect, in the graph convolutional neural network, the method for generating wrinkle-removed face decoupling features from the face key point features and the second face features is as follows:

[0030] S401, use face key point features E4 to predict a set of face key region attention maps based on a multi-head attention mechanism:

[0031]

[0032] In the formula: W1 and W2 are learnable feature transformation matrices, tanh is a tanh activation function, and softmax represents a softmax operation; the obtained face key region attention map set A includes multiple attention maps, and the i-th attention map A i corresponds to a face key region where the i-th face key point is located;

[0033] S402, weighting each attention map in the face key region attention map set A to the second face feature to obtain a region face feature set V;

[0034] S403, judging whether there is a connection edge between any two region face features V i , j according to the coincidence degree between each attention map in the face key region attention map set A, if the coincidence degree of V i , j and the corresponding attention map A i ,A j exceeds a threshold, it is considered that there is a connection edge, otherwise it is considered that there is no connection edge;

[0035] S404, taking the region face feature set V as the node set in the graph, determining the connection edge between the nodes according to the judgment result in S403 and constructing an adjacency matrix, inputting the graph into a graph convolutional neural network (GCN), and performing graph mean pooling operation on the node features output by the graph convolutional neural network to obtain wrinkle-removed face decoupling features E3 * .

[0036] As a preferred embodiment of the first aspect, the coincidence degree between the two attention maps is calculated using the cross ratio.

[0037] As a preferred embodiment of the first aspect, the graph convolutional neural network needs to input a wrinkle feature extracted from a face image as an auxiliary supervision signal during training, and the graph convolutional neural network is trained jointly with a self-attention module to realize face decoupling and structure learning under wrinkle decorrelation constraint; the loss function used for training includes two loss terms, the first loss term is the Range Loss of the final identity discrimination result in S5, and the second loss term is the similarity between the face decoupling features output by the graph convolutional neural network and the wrinkle features.

[0038] In a second aspect, the present application provides a computer readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the multi-modal fusion identity authentication method for a judicial scene according to any one of the first aspect.

[0039] In a third aspect, the present application provides a computer electronic device, comprising a memory and a processor;

[0040] The memory is configured to store a computer program;

[0041] The processor is configured to, when executing the computer program, implement the multi-modal fusion identity authentication method for a judicial scene according to any one of the first aspect.

[0042] In a fourth aspect, the present application provides a hardware and software integrated identity authentication device for a judicial scene, comprising:

[0043] A fingerprint image acquisition device is configured to acquire a fingerprint image of a target person;

[0044] An iris image acquisition device is configured to acquire an iris image of the target person;

[0045] A face image acquisition device is configured to acquire a face image of the target person;

[0046] An identity recognition module is configured to obtain an identity authentication result of the target person according to the acquired fingerprint image, iris image and face image, and according to the multi-modal fusion identity authentication method for a judicial scene according to any one of the first aspect.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] 1) The present application provides an identity authentication method fusing multi-modal data such as face, iris and fingerprint, which can be applied in a smart judicial application scene, including judicial identity authentication scenes such as case filing, court trial and lawsuit service. The present application fuses based on multi-modal data, which can further improve the identity authentication capability.

[0049] 2) The present application designs an adaptive expansion mechanism module in combination with a feature adapter and a gating-based cropping strategy, and realizes joint learning of the multi-modal feature fusion module in different multi-modal data through the adaptive expansion mechanism module. The present application performs joint learning by means of a sparse adaptive multi-task learning method, so that one model can simultaneously perform joint learning on three modal data, and support one model to simultaneously output recognition results of three modalities.

[0050] 3)In order to accurately identify the identity information of the personnel, the face micro-expression, wrinkle and key point information are integrated, so that the misrecognition problem caused by the change of the face texture is avoided, and finally the multi-modal features and the face micro-features are jointly learned by means of the self-attention mechanism, so that the final recognition result is obtained, thereby greatly improving the identity recognition accuracy in the judicial scene. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The step flow chart of the multi-modal fusion identity identification method for the judicial scene is shown in the figure.

[0052] Figure 2 The multi-task learning framework schematic diagram formed after the multi-modal feature fusion module is connected with the classification head is shown in the figure.

[0053] Figure 3 The structure schematic diagram of the adaptive expansion mechanism module is shown in the figure.

[0054] Figure 4 The framework schematic diagram of the entire multi-modal fusion identity identification method during training is shown in the figure. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, therefore the present application is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present application can be combined accordingly without conflict.

[0056] In the description of the present application, it should be understood that the terms "first", "second" are only used for distinguishing description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can be explicitly or implicitly included at least one of the features.

[0057] In a preferred embodiment of the present application, a judicial scene-oriented identity identification method is provided, which comprises the following steps:

[0058] S1, acquiring multi-modal data of a target personnel, including a face image, an iris image or a fingerprint image.

[0059] S2, respectively extracting modal features of the fingerprint image, the iris image and the face image of the personnel to obtain first fingerprint features, first iris features and first face features, and further extracting micro-expression features and face key point features based on the face image.

[0060] S3, input the first fingerprint feature, the first iris feature and the first face feature into a multi-modal feature fusion module learned by a sparse adaptive multi-task learning method, and after further feature extraction and fusion, the second fingerprint feature, the second iris feature and the second face feature are formed.

[0061] S4, input the face key point feature and the second face feature into a graph convolutional neural network learned by face decoupling and structure learning under wrinkle decorrelation constraint, and obtain the wrinkle-removed face decoupling feature.

[0062] S5, input the second fingerprint feature, the second iris feature, the wrinkle-removed face decoupling feature and the micro-expression feature into a self-attention module after splicing, form a multi-modal fusion feature through a self-attention mechanism, and perform feature matching in a personnel library by using the multi-modal fusion feature to obtain an identity authentication result.

[0063] The application can fuse multi-modal data, realize the recognition of three modalities by means of a set of model architectures, further improve the overall identity authentication capability, and can be applied to the scenes of case filing, court trial, lawsuit and service in the judicial field to identify the identity of personnel.

[0064] It should be noted that the multi-modal data of the target personnel obtained in the S1 step of the application includes a face image, an iris image or a fingerprint image, which can be obtained online by a hardware device or offline by reading the corresponding image recorded by the hardware device, and no limitation is made.

[0065] In addition, the modal feature extraction of the fingerprint image, the iris image and the face image of the personnel in the application belongs to the prior art, and can be realized by referring to the existing scheme of each type of image in identity recognition. In the embodiments of the application, the fingerprint recognition system of the fingerprint image can adopt the FingerNet model, and the corresponding feature extraction model is referred to as FingerExt; the face recognition system of the face image can adopt the FaceNet model, and the corresponding feature extraction model is referred to as FaceExt; the iris recognition system of the iris recognition can adopt the Transformer model, and the corresponding feature extraction model is referred to as IrisExt. These feature extraction models need to be trained in advance, and the training method belongs to the prior art, which will not be described here. Therefore, the extraction processes of the first fingerprint feature, the first iris feature and the first face feature can be described as follows:

[0066] First fingerprint feature extraction: input the fingerprint image F1 into the trained model for the fingerprint recognition system, extract the feature expression of the corresponding fingerprint data, and further normalize it through LayerNorm, which is formalized as:

[0067] E1 = LayerNorm(FingerExt(F1))

[0068] First iris feature extraction: input the iris image I into the trained model in the iris recognition system, then take the third layer, the seventh layer and the second to last layer features (denoted as IrisExt_3, IrisExt_7, IrisExt_{-2} respectively) for feature extraction, which is formalized as:

[0069] E2 = LayerNorm(IrisExt_3(I) + IrisExt_7(I) + IrisExt_{-2}(I))

[0070] First face feature extraction: input the face image F2 into the trained model in the face recognition system, then take the second to last layer feature as the feature expression of the corresponding face data, and then perform LayerNorm further normalization, which is formalized as

[0071] E3 = LayerNorm(FaceExt(F2))

[0072] In addition, for the face image data collected, the corresponding pre-trained model (industry public or self-pretrained) can also be used to extract the face key point feature, wrinkle feature and micro-expression feature of the face image, denoted as E4, E5 and E6 respectively. The feature extraction of face key points, wrinkles and micro-expressions all exist in the prior art. For example, the recognition system of micro-expression recognition can use LBP-TOP (Local Binary Pattern), and the corresponding feature extraction model is referred to as MERExt. Similarly, expression feature extraction: based on the trained model in the expression recognition system, the third layer, the seventh layer and the second to last layer features (denoted as MERExt_3, MERExt_7, MERExt_{-2} respectively) are taken for feature extraction, which is formalized as:

[0073] E6 = LayerNorm(MERExt_3(I) + MERExt_7(I) + MERExt_{-2}(I))

[0074] It should be noted that in the present application, in order to map all extracted features to the same dimensional space, these features extracted from different modalities need to be linearly mapped.

[0075] In addition, in the embodiments of the present application, since multiple modal data need to be input simultaneously in the present application, a multi-task learning framework is designed to simultaneously learn and fuse face features, iris features and fingerprint features. Specifically, the multi-task learning framework is realized through the above-mentioned multi-modal feature fusion module. See Figure 2As shown, the multi-modal feature fusion module includes a base network for extracting features and an adaptively expanding mechanism module for adapting to new tasks.

[0076] The base network contains a plurality of cascaded feature extraction modules, the input of the base network is the extracted multi-modal features, and the output is the multi-modal features obtained through further feature extraction and fusion. The specific form of the base network is not limited, and in an embodiment, ResNet can be used to implement the base network. The intermediate convolutional part of the ResNet includes a total of four stages, i.e., Stage1 to Stage4. Each stage can correspond to a feature extraction module in the base network, so there are a total of four feature extraction modules in the base network.

[0077] In the application process, the present application attempts to jointly learn multiple modalities according to a multi-task learning framework, and the working principle is mainly realized through an adaptively expanding mechanism (AES) module. Continuing to refer to Figure 2 As shown, the AES module has multiple groups, and each group of AES modules corresponds to processing a feature modality in the multi-modal features, that is, for each additional feature modality in the multi-modal feature fusion module, a group of AES modules is correspondingly added to enable the neural network to adapt to the new modality identity recognition task. Moreover, the number of modules in each group of AES modules is the same as the number of feature extraction modules in the above-mentioned base network, and for the above-mentioned embodiment with four feature extraction modules, each group of AES modules includes four AES modules. The output end of each feature extraction module is connected to an AES module, and the original output H j First, the AES module is passed through to obtain the output of the AES module, denoted as F i,j Then, the output F i,j of the AES module is added to the above-mentioned original output H j in a residual connection manner, and is used as the input of the downstream cascaded module of the feature extraction module. Here, the downstream cascaded module of the feature extraction module needs to be determined according to the network structure. For the last feature extraction module, the downstream cascaded module thereof is the network module (classification head during training) cascaded at the rear end of the entire multi-modal feature fusion module, and the downstream cascaded module of the last feature extraction module is still the feature extraction module.

[0078] The AES module in the present application can play a switching role through a special mechanism inside. Referring to Figure 3 As shown, the network structure principle of the AES module is shown. In the jth AES module, the module input is the output H j of the corresponding jth feature extraction module, and H j is first passed through a feature adapter Di,j The intermediate feature D i,j (H j ) is obtained 1 In an embodiment of the present application, the three convolutional layer sizes in the feature adapter are 1x1, 3x3 and 1x1 respectively, and the corresponding weight matrices are denoted as W i,j 2 i,j 3 i,j .

[0079] The intermediate feature D i,j (H j ) is input into the gate-based pruning strategy module, and the probability (representing whether the current AES module needs to be activated) is obtained through Gumbel-Softmax operation (the probability takes the value of 0 or 1), and the probability is multiplied by the intermediate feature D i,j (H j ) to serve as the output of the current AES module.

[0080] As can be seen, the above AES module includes two important modules: the feature adapter is responsible for expanding and adapting to new learning tasks, and the gate-based pruning strategy is used to control whether the expanded neurons can be pruned, so as to achieve the best multi-task learning performance with minimal expansion cost. If the probability obtained through Gumbel-Softmax operation is 0, the intermediate feature D i,j (H j ) output by the feature adapter is not actually added to the feature extraction module output H j , but if the probability obtained through Gumbel-Softmax operation is 0, the intermediate feature D i,j (H j ) output by the feature adapter is added to H j , which can be formalized as

[0081]

[0082] In the gate-based pruning strategy of the present application, its main function is to determine whether the corresponding neurons need to be activated, but the activation operation is a binary operation and is not differentiable. Therefore, instead of using the traditional Softmax, the present application uses the Gumbel-Softmax estimator for approximate calculation. Gumbel-Softmax is a prior art, and its expression is as follows:

[0083] ​

[0084] where k represents 0 or 1, v represents the probability of k, and g represents a standard Gumbel (0, 1) distribution. Further, the one-hot form thereof is expressed as follows

[0085]

[0086] where τ is a corresponding temperature coefficient, and the default is 1.

[0087] In addition, since the multi-modal feature fusion module needs to perform multi-task learning on multi-modal data, the problem of increased computational cost caused by an excessively large network will be encountered in multi-task learning. In the embodiments of the present application, in order to overcome this problem, a sparsity-activation regularization mechanism is proposed to ensure. Specifically, the sparsity constraint is expressed as

[0088]

[0089] where L represents the number of feature extraction modules in the above basic network, θ j，k represents the kth network parameter of the AES module corresponding to the jth feature extraction module, and |Ω j | represents the set of learnable weight parameters of the AES module corresponding to the jth feature extraction module. j

[0090] In addition, the sparsity of the AES module corresponding to the jth feature extraction module is defined as

[0091]

[0092] where, represents the probability that the calculation formula is established, and σ represents a threshold value, which is a hyperparameter.

[0093] And the activation of the AES module corresponding to the jth feature extraction module can be defined as:

[0094]

[0095] where, represents the output probability of the gating clipping strategy module corresponding to the kth sample in the batch, and |B| is the number of samples in the batch.

[0096] Then the corresponding activation regularization can be expressed as

[0097]

[0098] ​Then, for the ith task, the training loss of the multi-task learning framework can be expressed as

[0099]

[0100] wherein, is the classification cross-entropy loss corresponding to personnel identity recognition; is the above-mentioned sparse constraint of the neural network, is the above-mentioned activation constraint of the neural network, and λ s and λ a are two weight hyperparameters, which can be optimized according to actual conditions.

[0101] In the above S3 step, when jointly learning the multi-modal feature fusion module, a sparse adaptive multi-task learning method can be used based on the above definition, as follows:

[0102] Three different classification heads are connected at the end of the multi-modal feature fusion module, thereby forming a multi-task learning framework for multi-task joint learning. For image samples of three different modalities of fingerprint images, iris images and face images, the extracted features are input into the multi-task learning framework after modal feature extraction, and the features are input into the corresponding classification head after being processed by the above multi-modal feature fusion module, the basic network and a group of AES modules corresponding to the modal, to obtain the personnel identity discrimination result.

[0103] After the multi-modal feature fusion module is subjected to the above-mentioned sparse adaptive multi-task learning, the three classification heads are removed when used for actual inference. The first fingerprint feature E1, the first iris feature E2 and the first face feature E3 are input into the multi-modal feature fusion module, and after further feature extraction and fusion, new three features E1', E2' and E3' are formed, i.e., the second fingerprint feature, the second iris feature and the second face feature.

[0104] In addition, in order to solve the situation that the face of the personnel is distorted or wrinkles that do not originally exist appear due to emotional excitement, long time in a high-pressure atmosphere and the like in the scene of case filing, court trial, lawsuit and the like, the graph convolutional neural network of the above S4 step needs to decouple the structure of the face and perform wrinkle removal processing. Therefore, in the embodiment of the present application, in the graph convolutional neural network of the above S4 step, the method for generating a wrinkle-removed face decoupling feature based on the face key point feature and the second face feature is as follows:

[0105] S401, use the face key point feature e4 to predict a multi-head additive attention mechanism (multi-head additive attention) based face key region attention map set:

[0106]

[0107] In the formula: W1 and W2 are learnable feature transformation matrices, tanh is the tanh activation function, and softmax represents the softmax operation.

[0108] It should be noted that since there are multiple facial landmarks, the aforementioned facial landmark feature E4 is actually a feature set. When using facial landmark feature E4 with a multi-head attention mechanism to predict the facial landmark region attention map, the resulting set of facial landmark region attention maps A also contains multiple attention maps, with the i-th attention map A... i This corresponds to the key facial region where the i-th facial key point is located. Therefore, each attention map represents a highlighted area of ​​a key region, achieving decoupling of facial features E1 from relevant regions.

[0109] S402. Weight each attention map in the set of attention maps for key facial regions A and apply it to the second facial feature to obtain the set of facial features for the regions V. This process can be expressed by the following formula:

[0110] V = f disen (E3′,A)

[0111] In the formula: f disen This means that the second face feature E3' is weighted and summed using probabilistic attention maps to obtain the region face feature set V. Similarly, since set A also contains multiple attention maps, the corresponding region face feature set V also contains a series of region face features V. i .

[0112] S403. Based on the overlap between the attention maps in the set A of facial key region attention maps, determine the facial features V of any two regions. i V j Does V have a connecting edge? i V j Corresponding attention map A i A j If the intersection ratio exceeds the threshold, it is considered that there is a connected edge; otherwise, it is considered that there is no connected edge.

[0113] In an embodiment of the present invention, the process of determining whether there are connecting edges between facial features in a region based on the overlap between attention maps can be described as follows:

[0114] c i,j =I(IoU(A) i A j > c ))

[0115] wherein: I (· represents an indicator function, returns 1 if the condition is met, and returns 0 if the condition is not met. IoU represents the intersection over union, which calculates the ratio of the overlapping area between the attention map and the merged area of the attention map. α c is a manually adjustable threshold, which is an optimizable hyperparameter.

[0116] S404, taking the region face feature set V as the node set in the graph, and determining the connection edges between the nodes according to the judgment result in S403 and constructing an adjacency matrix, inputting the graph into a graph convolutional neural network (GCN), and then performing graph mean pooling operation on the node features output by the graph convolutional neural network to obtain the wrinkle-removed face decoupling feature E3 * .

[0117] Specifically, the graph input into the GCN in this step takes the region face feature set V as the node set, and the connection between the nodes is specified by c i,j , and the structure learning of the face region and the whole is performed through the GCN:

[0118] V * = GCN (V, C)

[0119] E3 * = Pool (V * )

[0120] wherein C is an adjacency matrix arranged with c i,j as elements; the specific number of layers of the GCN can be optimized according to actual conditions, and in the embodiment of the present application, it is a three-layer graph convolutional neural network; and Pool is a graph mean pooling operation.

[0121] The above graph convolutional neural network needs to input the wrinkle feature E5 extracted from the face image as an auxiliary supervision signal during training, and the graph convolutional neural network is trained jointly with the self-attention module to realize face decoupling and structure learning under the constraint of wrinkle decorrelation; the loss function used for training contains two loss terms, the first loss term is the Range Loss of the identity discrimination result finally obtained in S5, and the second loss term is the similarity between the face decoupling feature output by the above graph convolutional neural network and the above wrinkle feature E5, and its formula can be expressed as:

[0122]

[0123] wherein φ is a similarity measure function, and in the present embodiment, the cosine similarity can be used. The introduction The reason is that the facial wrinkles of the biological characteristics in the scene of the judicial court trial are variable and will change with the mood change and other factors, so the constraint loss function needs to be used for scene-specific decorrelation, so that the similarity between the face decoupling features output by the graph convolutional neural network and the wrinkle features E5 is as low as possible, that is, the purpose of removing the wrinkle influence is achieved. In summary, the overall framework of the training of each model in the whole multi-modal fusion identity identification method in the application is as shown in Figure 4 Especially need to be explained that the multi-modal feature fusion module needs to be pre-learned by the multi-task learning framework for multi-task joint learning, and the parameters can be fixed in the joint training stage of the joint training of the graph convolutional neural network and the self-attention module, of course, the fine tuning can also be carried out on the basis of pre-training.

[0124] Similarly, in the judicial court scene, the face will have more micro expressions affected by emotional expression, which can be regarded as the characteristic information of the target identity. Therefore, the facial image micro expression features, that is, the E6 features, are extracted based on the collected face data and the pre-trained model in the application, and participate in the multi-modal feature fusion in the final biological feature identification link.

[0125] In the embodiment of the application, in the biological feature identification link of S6 step, the features of different modalities can be fused, that is, the obtained fusion features are emb=concat(E1',E2',E3 * ,E6), and input into the self-attention module to adopt the self-attention mechanism self-attention for fusion modeling, wherein the fusion features emb are used as K, Q and V values, and the specific expression is as follows:

[0126]

[0127] It should be noted that, That is, the output feature of the self-attention module, which can be directly calculated with the features of each person in the library, and the similarity is calculated according to the similarity.

[0128] In addition, it should be noted that in actual application, although the application needs to input multiple modal images, if there is missing data of a certain modality, it is directly defaulted to 1 (that is, not participating in fusion), and the identity identification result can also be output.

[0129] Therefore, based on the methods S1-S5, the application provides a multi-modal data fusion-based personnel identity recognition mode, which can simultaneously consider face, iris and fingerprint data recognition and be applied to different scenes (such as case filing, court trial and lawsuit service), and the fused model can realize better identity identification capability. The multi-modal data required by the application contains various human identity attribute information, and currently these identity recognition technologies are applied in the judicial field. For example, in the scenes of case filing, court trial and lawsuit service in the judicial field, personnel identity is identified, and multi-modal data (face, iris and fingerprint) based fusion can further improve the identity identification capability.

[0130] Similarly, based on the same inventive concept, another preferred embodiment of the application also provides an electronic device corresponding to the multi-modal fusion identity identification method for the judicial scene provided in the above embodiment, which comprises a memory and a processor;

[0131] The memory is used for storing a computer program;

[0132] The processor is used for realizing the multi-modal fusion identity identification method for the judicial scene described in any of the above embodiments when the computer program is executed.

[0133] In addition, the logical instructions in the above memory can be realized in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the application or parts of the application that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the application.

[0134] Therefore, based on the same inventive concept, another preferred embodiment of the application also provides a computer readable storage medium corresponding to the multi-modal fusion identity identification method for the judicial scene provided in the above embodiment, and the storage medium stores a computer program. When the computer program is executed by a processor, the multi-modal fusion identity identification method for the judicial scene described in any of the above embodiments can be realized.

[0135] Specifically, in the computer readable storage medium of the above two embodiments, the computer program stored therein can execute the steps S1-S5.

[0136] It can be understood that the storage medium can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Meanwhile, the storage medium can also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0137] It can be understood that the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0138] In addition, it should be noted that the skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here. In the embodiments provided in the present application, the division of steps or modules in the system and method is only a logical functional division, and another division mode can be used in actual implementation, for example, a plurality of modules or steps can be combined or integrated together, or a module or step can be split.

[0139] In addition, based on the same inventive concept, another preferred embodiment of the present application also provides a hardware-software integrated judicial scene identity authentication device corresponding to the multi-modal fusion identity authentication method for a judicial scene provided in the above embodiments, which comprises:

[0140] The fingerprint image acquisition device is configured to acquire a fingerprint image of the target person.

[0141] The iris image acquisition device is configured to acquire an iris image of the target person.

[0142] The face image acquisition device is configured to acquire a face image of the target person.

[0143] The identity recognition module is configured to obtain an identity authentication result of the target person according to the acquired fingerprint image, iris image and face image, according to the multi-modal fusion identity authentication method for a judicial scene described in any of the foregoing embodiments.

[0144] It should be noted that the fingerprint image acquisition device, the iris image acquisition device, and the face image acquisition device herein all exist as existing hardware devices, and any device capable of realizing the corresponding acquisition function can be adopted, and no limitation is made in this regard.

[0145] In addition, in the embodiments of the present application, in order to cope with the problem that computing resources are often limited in the scenes of case filing, lawsuit service, and court trial, the multi-modal data can be further adaptively divided, relatively separated stored, and typical feature cached by characteristic storage hardware, a physical constraint mechanism is introduced, and distributed parallel asynchronous feature matching calculation is adopted. At the same time, on the computing hardware, different scene requirements are adapted and combined with CPU, GPU, FPGA, and NPU hardware, and at the software coordination layer, resource scheduling, communication protocol, fault tolerance mechanism, and distributed framework are designed to support the whole life cycle of software and hardware integrated identity authentication of data acquisition, storage caching, scheduling load, training optimization, and inference prediction.

[0146] The above-described embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical scheme obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.

Claims

1. A multimodal fusion identity authentication method for judicial scenarios, characterized in that, include: S1. Obtain multimodal data of the target personnel, including facial images, iris images, or fingerprint images; S2. Modal features are extracted from the fingerprint image, iris image and face image of the person respectively to obtain the first fingerprint feature, the first iris feature and the first face feature. At the same time, micro-expression features and facial key point features are further extracted based on the face image. S3. Input the first fingerprint feature, the first iris feature, and the first face feature into the multimodal feature fusion module jointly learned by the sparse adaptive multi-task learning method. After further feature extraction and fusion, the second fingerprint feature, the second iris feature, and the second face feature are formed respectively. S4. Input the facial key point features and the second facial features into a graph convolutional neural network that has undergone facial decoupling and structure learning under the constraint of wrinkle decorrelation to obtain the face decoupling features with wrinkles removed. S5. The second fingerprint feature, the second iris feature, the wrinkle-removed face decoupling feature, and the micro-expression feature are spliced ​​together and input into the self-attention module. The self-attention mechanism is used to form a multimodal fusion feature. The multimodal fusion feature is used to perform feature matching in the personnel database to obtain the identity recognition result. The multimodal feature fusion module includes a base network for feature extraction and an adaptive expansion mechanism module for adapting to new tasks; The basic network contains multiple cascaded feature extraction modules. The input of the basic network is the extracted multimodal features, and the output is the multimodal features obtained after further feature extraction and fusion. The adaptive expansion mechanism module has multiple groups, and each group of adaptive expansion mechanism modules corresponds to one feature mode in the processing of the multimodal features. The number of modules in each group of adaptive expansion mechanism modules is the same as the number of feature extraction modules in the base network. The output of each feature extraction module is connected to an adaptive expansion mechanism module. The original output of the feature extraction module first passes through the adaptive expansion mechanism module, and then the output of the adaptive expansion mechanism module is added to the original output in a residual connection manner, and used as the input of the downstream cascaded module of the feature extraction module. In the adaptive expansion mechanism module, the module input first passes through a feature adapter consisting of three convolutional layers to obtain intermediate features. The intermediate features are then input into the gating pruning strategy module, where they undergo a Gumbel-Softmax operation to obtain the probability of a 0 or 1 value representing whether the current adaptive expansion mechanism module needs to be activated. This probability is multiplied by the intermediate features and used as the output of the current adaptive expansion mechanism module.

2. The multimodal fusion identity authentication method for judicial scenarios as described in claim 1, characterized in that, The three convolutional layers in the feature adapter have dimensions of 1×1, 3×3, and 1×1, respectively.

3. The multimodal fusion identity authentication method for judicial scenarios as described in claim 1, characterized in that, The sparse adaptive multi-task learning method used by the multimodal feature fusion module during joint learning is as follows: Three different classification heads are connected to the end of the multimodal feature fusion module to form a multi-task learning framework for joint multi-task learning. For image samples of three different modalities—fingerprint images, iris images, and face images—modal features are extracted separately, and the extracted features are input into the multi-task learning framework for joint learning. During the joint learning process, after the image sample features of each modality are input into the multi-task learning framework, they first pass through the multi-modal feature fusion module, and then through the basic network and a set of adaptive expansion mechanism modules of the corresponding modality. Finally, the output features are entered into the corresponding classification head to obtain the personnel identification results. Furthermore, in multi-task joint learning, the loss function used for learning each modality is expressed as: ； in The cross-entropy loss is used for personnel identification. For sparse constraints of neural networks, For the activation constraints of the neural network, the two constraints are calculated as follows: ； ； In the formula: L represents the number of feature extraction modules in the basic network. This represents the k-th network parameter of the adaptive expansion mechanism module corresponding to the j-th feature extraction module. This represents the set of learnable weight parameters for the adaptive expansion mechanism module corresponding to the j-th feature extraction module. The model; and Let these represent sparsity and activation, respectively, and the calculation formula is: ； ； In the formula: This represents the output probability of the gating pruning strategy module corresponding to the k-th sample in the batch, where |B| is the number of samples in a batch. This indicates the calculation of probability. This represents the threshold.

4. The multimodal fusion identity authentication method for judicial scenarios as described in claim 1, characterized in that, In the graph convolutional neural network, the method for generating wrinkle-removed facial decoupling features based on facial key point features and second facial features is as follows: S401, Using facial landmark features Predicting facial key region attention map sets based on multi-head attention mechanism: ； In the formula: and It is a learnable feature transformation matrix. for Activation function represent Operation; the resulting set of attention maps of key facial regions It contains multiple attention maps, the i-th attention map The facial key region corresponding to the location of the i-th facial key point; S402, Set up attention maps of key facial regions The attention maps in the first part are weighted and applied to the second face feature to obtain the set of face features in the region. ; S403, Based on the set of attention maps of key facial regions The degree of overlap between attention maps is used to determine facial features between any two regions. Are there any connecting edges between them? Corresponding attention map If the overlap exceeds the threshold, it is considered that there is a connected edge; otherwise, it is considered that there is no connected edge. S404, Based on regional facial feature sets As a set of nodes in the graph, and based on the judgment result in S403, the connection edges between nodes are determined and an adjacency matrix is ​​constructed. The graph is then input into a graph convolutional neural network (GCN), and graph mean pooling is performed on the node features output by the graph convolutional neural network to obtain the decoupled facial features with wrinkles removed. .

5. The multimodal fusion identity authentication method for judicial scenarios as described in claim 4, characterized in that, The overlap between the two attention maps is calculated using the intersection ratio.

6. The multimodal fusion identity authentication method for judicial scenarios as described in claim 1, characterized in that, During training, the graph convolutional neural network requires wrinkle features extracted from face images as auxiliary supervision signals. The graph convolutional neural network is trained in conjunction with a self-attention module to achieve facial decoupling and structure learning under wrinkle decorrelation constraints. The loss function used for training includes two loss terms: the first loss term is the Range Loss of the final identity recognition result obtained in S5, and the second loss term is the similarity between the face decoupling features output by the graph convolutional neural network and the wrinkle features.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the multimodal fusion identity authentication method for judicial scenarios as described in any one of claims 1 to 6.

8. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the multimodal fusion identity authentication method for judicial scenarios as described in any one of claims 1 to 7 when executing the computer program.

9. A hardware and software integrated identity verification device for judicial scenarios, characterized in that, include: A fingerprint image acquisition device used to acquire fingerprint images of target individuals; Iris image acquisition device, used to acquire iris images of target individuals; A facial image acquisition device used to acquire facial images of target individuals; The identity recognition module is used to obtain the identity recognition result of the target person based on the collected fingerprint image, iris image and face image, according to the multimodal fusion identity recognition method for judicial scenarios as described in any one of claims 1 to 7.

Images