Multi-mode identity authentication method and device, electronic equipment and storage medium

Abstract

The invention discloses a multi-mode identity authentication method and device, electronic equipment and a storage medium, and the identity authentication method comprises the steps: combining face recognition and palm vein recognition at the same time, extracting a to-be-authenticated image at a time, and carrying out the ROI (region of interest) extraction through a target monitoring network, thereby obtaining a face image region and a palm vein image region; multiple image acquisition devices or time-sharing acquisition are not needed, so that the image acquisition efficiency and experience are improved; the face image area and the palm vein image area are respectively input into a pre-trained face recognition network and a pre-trained palm vein recognition network to obtain a face recognition result and a palm vein recognition result, and meanwhile, the face recognition result and the palm vein recognition result are considered to determine an identity authentication result, so that the limitation of single-mode authentication is overcome, and the authentication efficiency is improved. And the security and the accuracy of the identity authentication system are obviously improved.

Description

Technical Field

[0001] The present invention relates to identity authentication technology, and in particular to a multimodal identity authentication method, device, electronic device and storage medium. Background Art

[0002] Identity authentication, a core technology in information security, is widely used in finance, government, healthcare, and other fields. Traditional authentication methods (such as passwords, cards, and fingerprints) have limitations in terms of security and accuracy, making them vulnerable to forgery, theft, or loss. With the rapid development of biometrics, authentication based on unique human characteristics has become a more secure and reliable option.

[0003] Common biometric technologies include facial recognition, fingerprint recognition, palm print recognition, and palm vein recognition. Facial recognition technology is widely used due to its non-contact nature and ease of use. However, in practice, the accuracy and robustness of facial recognition are often affected by ambient lighting, posture changes, expression changes, and forgery attacks (such as photos and masks), and still pose significant security risks. While fingerprint and palm vein recognition offer high security, they also have certain limitations due to their high equipment requirements and are affected by factors such as finger or palm state and acquisition angle.

[0004] Most identity recognition technologies rely on a single biometric feature for authentication. Due to the limitations of a single modality in specific environments (e.g., facial recognition is affected by lighting, and fingerprint recognition is affected by finger position), when recognition of a particular modality fails, the system cannot provide sufficient security, and the risk of authentication failure is high. Furthermore, while some technologies have proposed multimodal identity authentication solutions, they often adopt an "authentication by any modality" approach. Even if the recognition result of a particular modality is not completely accurate, as long as one modality passes authentication, the authentication is considered successful. This approach lacks optimization for inter-modal collaboration and can lead to system misjudgments when processing low-quality or ambiguous data. Summary of the Invention

[0005] Based on this, the present invention aims to propose a multimodal identity authentication method, device, electronic device and storage medium, which uses a face recognition model and a palm vein recognition model to perform identity matching authentication in parallel, and simultaneously considers the face recognition results and the palm vein recognition results to perform dual identity authentication on the authentication target.

[0006] In a first aspect, the present invention provides a multimodal identity authentication method, comprising:

[0007] Collect the image to be authenticated;

[0008] Use the pre-trained object detection network to process the image to be authenticated and extract the face image area and palm vein image area;

[0009] Input the face image area and the palm vein image area into the pre-trained face recognition network and palm vein recognition network respectively to obtain the face recognition result and the palm vein recognition result;

[0010] The face recognition network is an improved convolutional neural network that combines the MobileNet network and the attention module;

[0011] The identity authentication result is determined based on the face recognition results and palm vein recognition results.

[0012] Furthermore, the pre-trained object detection network is used to process the image to be authenticated, and the facial image area and the palm vein image area are extracted, including:

[0013] Input the image to be authenticated into the pre-trained YOLOv5s model to extract the initial face image area and the initial palm vein image area;

[0014] The validity of the initial face image area and the initial palm vein image area is verified, and the face image area and the palm vein image area are output when the verification passes.

[0015] Furthermore, the validity verification of the initial face image region and the initial palm vein image region includes:

[0016] Obtain the signal-to-noise ratio and target area ratio of the initial face image region and the initial palm vein image region;

[0017] The validity is verified based on the signal-to-noise ratio and the target area ratio.

[0018] Furthermore, the architecture of the face recognition network includes:

[0019] Along the feature propagation direction, it includes the input layer, MobileNetv3 network, SE attention module, global average pooling layer, fully connected layer and face classifier.

[0020] Furthermore, the architecture of the palm vein recognition network includes:

[0021] Along the feature propagation direction, it includes the input layer, the first convolutional layer, the asymmetric attention module, the second convolutional layer, the maximum pooling layer, the fully connected layer and the palm vein classifier.

[0022] Furthermore, the face classifier adopts ArcFace classifier, and the loss function is Focal Loss.

[0023] Furthermore, the palm vein classifier adopts the SVM classifier, and the loss function is Contrastive Loss.

[0024] In a second aspect, the present invention provides a model training method, wherein the model trained by the method is used for multimodal identity authentication, comprising:

[0025] Get the certified image training set;

[0026] Build a multimodal identity authentication network that includes an object detection network, a face recognition network, and a palm vein recognition network. The face recognition network is an improved convolutional neural network that combines the MobileNet network and the attention module.

[0027] Training the multimodal identity authentication network includes extracting the facial image region and the palm vein image region from the authentication image training set through the target detection network, passing the facial image region and the palm vein image region through the face recognition network and the palm vein recognition network respectively to obtain the facial recognition results and the palm vein recognition results, and determining the identity authentication result based on the facial recognition results and the palm vein recognition results;

[0028] The network parameters of the multimodal identity authentication network are updated according to the identity authentication result, and the trained multimodal identity authentication network is output as a multimodal identity authentication model.

[0029] In a third aspect, the present invention provides a multimodal identity authentication device, comprising:

[0030] Image acquisition module, used to acquire the image to be authenticated;

[0031] An image extraction module is used to process the image to be authenticated using a pre-trained object detection network and extract the facial image area and the palm vein image area;

[0032] The image recognition module is used to input the facial image area and palm vein image area into the pre-trained face recognition network and palm vein recognition network respectively to obtain face recognition results and palm vein recognition results; the face recognition network is an improved convolutional neural network that combines the MobileNet network and the attention module;

[0033] The identity authentication module is used to determine the identity authentication result based on the face recognition result and the palm vein recognition result.

[0034] In a fourth aspect, the present invention provides a model training device for training a multimodal identity authentication model, comprising:

[0035] A training sample acquisition module is used to obtain a certification image training set;

[0036] A network building module is used to construct a multimodal identity authentication network that includes an object detection network, a face recognition network, and a palm vein recognition network. The face recognition network is an improved convolutional neural network that combines the MobileNet network and the attention module.

[0037] The network training module trains the multimodal identity authentication network. Specifically, the authentication image training set is extracted through the target detection network to obtain the face image area and the palm vein image area. The face image area and the palm vein image area are respectively passed through the face recognition network and the palm vein recognition network to obtain the face recognition results and the palm vein recognition results. The identity authentication result is determined based on the face recognition results and the palm vein recognition results.

[0038] The network update module updates the network parameters of the multimodal identity authentication network according to the identity authentication results, and outputs the trained multimodal identity authentication network as a multimodal identity authentication model.

[0039] In a fifth aspect, the present invention provides an electronic device comprising a memory storing computer-executable instructions and a processor. When the computer-executable instructions are executed by the processor, the device executes the multimodal identity authentication method provided in the first aspect, and / or the various steps of the model training method provided in the second aspect.

[0040] In a sixth aspect, the present invention provides a readable storage medium storing a computer executable program, which, when executed, can implement the multimodal identity authentication method provided in the first aspect, and / or the various steps of the model training method provided in the second aspect.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention proposes a multimodal identity authentication method, device, electronic device and storage medium. The identity authentication method combines face recognition and palm vein recognition at the same time, extracts the image to be authenticated at one time, and uses the target monitoring network to extract ROI (region of interest) to obtain the face image area and the palm vein image area. There is no need to use multiple image acquisition devices or time-sharing acquisition, thereby improving image acquisition efficiency and experience; the face image area and the palm vein image area are respectively input into the pre-trained face recognition network and palm vein recognition network to obtain face recognition results and palm vein recognition results, and the face recognition results and palm vein recognition results are simultaneously considered to determine the identity authentication result, thereby overcoming the limitations of single-modality authentication and significantly improving the security and accuracy of the identity authentication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 This is a flowchart of a multi-modal identity authentication method according to an embodiment of the present invention;

[0045] Figure 2 This is a flow chart for implementing the model training method provided by an embodiment of the present invention;

[0046] Figure 3 is a structural diagram of a multimodal identity authentication device provided by an embodiment of the present invention;

[0047] Figure 4 is a structural diagram of a model training device provided by an embodiment of the present invention;

[0048] Figure 5 This is a diagram of the electronic device architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] See Figure 1 One embodiment of the present invention provides a multimodal identity authentication method, comprising:

[0051] Step S110: Capture the image to be authenticated.

[0052] In this step, an image acquisition device (such as a camera or scanner) is used to capture the image to be authenticated. Specifically, the image acquisition process can simultaneously capture RGB and infrared (IR) images using an RGB-IR Bayer format sensor. At the signal processing end, the RGB data stream corresponding to the facial image and the IR data stream corresponding to the palm vein image are split into two streams, ensuring high-quality acquisition of facial and palm vein images.

[0053] Image acquisition equipment should have high resolution and frame rate to ensure clear capture of the user's face and palm vein area. During this process, factors such as image clarity, contrast, and lighting conditions will affect subsequent recognition accuracy. Therefore, the acquisition equipment must be able to automatically adjust exposure and focus to adapt to different environmental conditions.

[0054] Step S120: Use the pre-trained target detection network to process the image to be authenticated and extract the face image area and the palm vein image area.

[0055] In this step, a pre-trained object detection network is used to process the captured image to be authenticated. The object detection network takes the image to be authenticated as input and outputs the locations of various target regions within the image, including facial and palm vein regions. This network can use a convolutional neural network (CNN) to extract features from the input image and use regression methods to predict bounding boxes containing the facial and palm vein regions.

[0056] Specifically, the image to be authenticated (typically an RGB image or a combination of RGB-IR images) is input into an object detection network. This network, which may be a YOLO-series model or a similar efficient detection model, has been pre-trained on large-scale datasets and can accurately identify and locate faces and palm vein regions against complex backgrounds. The object detection network outputs two regions: one containing the face, and the other containing the palm veins. Each region is represented by a bounding box that contains the location information of the detected face and palm veins. The extracted regions are preliminary ROIs but may contain some noise or low-quality areas, requiring further verification of their effectiveness.

[0057] The pre-training of the object detection network is usually based on large-scale datasets (such as COCO, ImageNet, etc.) to ensure that it can effectively detect faces and palm vein areas in various scenarios.

[0058] Furthermore, the target detection network uses YOLOv5s (or other efficient target detection networks), which can provide high detection accuracy while ensuring speed. After completing the detection, the corresponding ROI (Region of Interest), namely the image of the face and palm vein area, is extracted to provide basic data for subsequent recognition tasks.

[0059] Further, the effectiveness verification of the initial ROI includes:

[0060] Obtain the signal-to-noise ratio and target region area ratio of the face ROI and palm vein ROI. The signal-to-noise ratio (SNR) is an important indicator for measuring image quality, describing the ratio of effective information to noise in the image. In an image, a higher SNR indicates more effective information and less noise. The target region area ratio is used to measure the proportion of the extracted area in the entire image. A target region that is too small may mean that the region does not fully contain the face or palm vein information, which may lead to recognition failure.

[0061] If the signal-to-noise ratio is lower than a predetermined threshold (e.g., lower than 15dB), the region is considered to be of poor quality, likely containing excessive noise or distortion, and unsuitable for further processing. If the area ratio is lower than a predetermined threshold (e.g., 10%), the region is considered too small to provide sufficient features for recognition.

[0062] For each target area extracted, we first check whether its signal-to-noise ratio meets the predetermined quality standard. If the SNR is less than the preset value, the area is considered invalid and cannot enter the subsequent processing flow. At the same time, we check whether the area ratio of the target area is large enough. If the area ratio is lower than the threshold, it is also considered an invalid area. For areas that are considered invalid, the system will automatically adjust the image (such as refocusing or adjusting the lens position) to ensure that the area meets the validity standard. That is, if any area fails the validity verification, it will trigger the image reprocessing or re-capture process to ensure that the image area finally input into the recognition network is of sufficient quality.

[0063] If the signal-to-noise ratio and area ratio meet the requirements, the verified face image area and palm vein image area are output as input data for subsequent recognition. At this point, these areas are considered valid and suitable for face and palm vein recognition.

[0064] If verification fails, the system automatically makes image adjustments, such as changing the lens focal length, adjusting the exposure, or adjusting the shooting angle, until the image quality meets the requirements. If automatic adjustments cannot meet the requirements, the system may prompt the user to recapture or manually adjust the device settings.

[0065] Step S130. Input the face image area and the palm vein image area into the pre-trained face recognition network and palm vein recognition network respectively to obtain face recognition results and palm vein recognition results. The face recognition network is an improved convolutional neural network that combines the MobileNet network and the attention module.

[0066] In this step, the facial image region and palm vein image region extracted in step S120 are respectively input into the pre-trained facial recognition network and palm vein recognition network for further processing. The facial recognition network and palm vein recognition network each process their corresponding image regions and output corresponding recognition results, which are used for subsequent identity authentication.

[0067] Specifically, the face recognition network extracts facial features from an image and generates a facial feature vector. This feature vector is then compared with known facial templates stored in a face database (for example, using methods like Euclidean distance or cosine similarity) to identify the most similar ID. Similarly, the palm vein recognition network extracts the texture features of the palm veins and generates a palm vein feature vector. This vector is then matched with templates stored in the palm vein database and outputs the most similar ID.

[0068] The facial image region is then fed into an improved convolutional neural network based on the MobileNet network and an attention module. The MobileNet network extracts multi-scale features from the image, and the attention module dynamically adjusts the weights of feature channels to further enhance important features and suppress unimportant ones. Finally, classification is performed using global average pooling and fully connected layers. Finally, identity authentication is performed using the ArcFace classifier and the Focal Loss loss function, outputting the face recognition result (such as identity category or match score).

[0069] Furthermore, the architecture of the face recognition network includes the input layer, MobileNetv3 network, SE attention module, global average pooling layer, fully connected layer and face classifier in sequence along the feature propagation direction.

[0070] Specifically, the input layer receives facial ROI image data in Bayer format (a mixed image format of RGB and IR information). This image data is then processed. For example, the input image size is 640×480×3, where 640×480 represents the spatial resolution of the image and 3 represents the RGB channels (i.e., a color image).

[0071] The MobileNetv3 network parameter configuration is α = 0.75, with 16 layers. MobileNetv3 is a lightweight backbone network primarily used to extract multi-scale features from input facial images. By using a depthwise separable convolutional structure, MobileNetv3 can efficiently extract low-level and high-level geometric features, texture features, facial contours, and other information with low computational resource usage. Specifically, MobileNetv3 uses multiple convolutional layers to convert the spatial features of the input image into a deep feature map (sized 20×15×96), where 96 represents the number of extracted feature channels. α represents the network width coefficient, which is used to control the network width and computational complexity. The 0.75 configuration is lighter than the standard MobileNetv3 and is suitable for mobile devices or devices with limited resources.

[0072] The attention module uses the Squeeze-and-Excitation (SE) attention module, which dynamically adjusts the response of each channel in the feature map by learning the importance of each channel. This module first compresses the features of each channel into a scalar (Squeeze) through global average pooling. A small fully connected layer then generates activation coefficients for each channel (Excitation). Finally, a per-channel weighted operation adjusts the channel response of the feature map. A compression ratio of 16 can be selected for parameter configuration, which compresses the features of each channel by a factor of 16, reducing computational effort and enhancing model expressiveness. By adjusting the weights of each channel, this module emphasizes the feature responses of key areas (such as the eyes and mouth) and suppresses less important areas, improving the network's sensitivity to facial details. For example, the output feature size is 20×15×96.

[0073] The global average pooling operation performs an average operation on the spatial dimensions (height and width) of each channel, converting the feature map from 20×15×96 to a 1×1×576 feature representation. Global average pooling allows the network to aggregate information from different spatial locations into global features, avoiding excessive parameter calculation while preserving the robustness of the image's spatial features. Finally, a fully connected layer maps the 576-dimensional feature vector output by the global average pooling layer to a 128-dimensional feature space. This layer compresses the high-dimensional feature vector to a lower dimension using a weight matrix and bias term, providing input features for the subsequent classifier for identity recognition.

[0074] Furthermore, the palm vein image area is input into the pre-trained palm vein recognition network, which contains convolutional layers and asymmetric attention modules, which can effectively extract the vascular pattern features in the palm vein image. Feature extraction is performed through the convolutional layer, and the attention module weights important features to optimize the expression of features. Finally, identity authentication is performed through the SVM classifier and Contrastive Loss, and the palm vein recognition result is output.

[0075] Furthermore, the architecture of the palm vein recognition network includes an input layer, a first convolutional layer, an asymmetric attention module, a second convolutional layer, a maximum pooling layer, a fully connected layer, and a palm vein classifier in sequence along the feature propagation direction.

[0076] Specifically, the input layer receives palm vein ROI image data. The input image is a grayscale image. The vascular texture in palm vein images is typically very small and detailed, so the image quality is crucial for subsequent recognition. For example, the input image size is 320×240 and the number of channels is 1, representing a single-channel near-infrared image.

[0077] The first convolutional layer uses a 5×5 convolution kernel to perform preliminary feature extraction on the input palm vein image. The convolution operation scans the image with the kernel and extracts low-level features (such as vein texture, bifurcation points, and texture direction), which are crucial for subsequent recognition. 32 convolution kernels are used to capture more feature information, and padding (Padding=2) is used to maintain the spatial dimension of the image, ensuring the input and output images have the same spatial resolution. For example, the output size of this layer is 320×240×32, indicating that 32 feature channels are generated for each pixel.

[0078] The asymmetric attention module combines spatial and channel attention to further enhance meaningful texture features in vein images and suppress background noise. First, the asymmetric attention module uses a spatial attention mechanism to weight local regions in the image, focusing on areas with vein texture. Second, it uses a channel attention mechanism to dynamically adjust the responses of different feature channels, enabling the network to better identify key vein feature regions. For example, the output size remains 320×240×32, but after processing with the attention mechanism, important feature regions in the image are amplified and background noise is suppressed.

[0079] The second convolutional layer processes the input feature map using a 3×3 convolution kernel to further extract high-level features from the vein image. The number of convolution kernels in this layer is increased to 64 channels, enabling the capture of more complex vein texture structures and vascular topology. The convolution operation effectively extracts local patterns within an image within a larger receptive field, laying the foundation for vein recognition. The output size of this layer is 318×238×64, indicating a slight reduction in image size after convolution, while the number of feature channels increases.

[0080] The max pooling layer downsamples the feature map of the previous layer using a 2×2 pooling kernel with a stride of 2. The max pooling operation reduces the spatial dimension while retaining the most important features by selecting the maximum value in each pooling area. In this example, the pooling operation reduces the output size to 159×119, while maintaining the number of channels at 64. Finally, the pooled feature map is flattened in the flattening and fully connected layers, converting the two-dimensional feature map (159×119×64) into a one-dimensional vector. This operation compresses all spatial information into a long vector for processing in the fully connected layer. The fully connected layer then maps the flattened feature vector to a 64-dimensional feature space. These 64-dimensional features represent the deep features of the palm vein image, including the vein's geometric structure, topological relationships, and other important information.

[0081] Step S140: Determine the identity authentication result based on the face recognition result and the palm vein recognition result.

[0082] In this step, the facial recognition result and palm vein recognition result obtained in step S130 are comprehensively processed to obtain the final identity authentication result. If both recognition results meet the preset authentication standards (such as the matching degree exceeds a certain threshold), the identity authentication is considered successful; if either recognition result fails, the identity authentication fails.

[0083] Specifically, based on the accuracy or matching of facial and palm vein recognition, the system determines whether the face or palm vein matches a template in the registration database. The system then performs a logical analysis (e.g., a logical AND operation) on these two results. Only when both are verified is the user's identity authentication considered successful. This step ensures the high security and robustness of multimodal authentication, avoiding the potential risks of single-modality recognition systems by combining two biometric features.

[0084] Furthermore, the process of determining the identity authentication result by integrating the face recognition result and the palm vein recognition result in step S140 includes:

[0085] Based on the facial feature vector and palm vein feature vector, respectively, the system searches for the closest matching ID in the facial or palm vein database and determines whether the match is successful based on a set threshold (for example, if the degree of match exceeds a certain threshold). If successful, the matching ID is returned; if not, a failure result is returned. The facial and palm vein recognition IDs are compared. If the facial and palm vein recognition IDs match, the system considers authentication successful and the user's identity verified. If the facial and palm vein recognition IDs differ, the system considers authentication a failure and rejects the authentication.

[0086] By matching in the face database and palm vein database respectively, the present invention can effectively combine the two biometric features for dual verification, significantly improving the accuracy and security of identity authentication. This method, by combining multimodal recognition, ensures that even if one modality is affected by noise or other factors, the other modality can still provide reliable verification, thereby reducing the risk of forgery or impersonation.

[0087] See Figure 2 One embodiment of the present invention provides a model training method, wherein the model trained by the method is used for multimodal identity authentication, including:

[0088] Step S210: Obtain a certification image training set.

[0089] Step S220. Construct a multimodal identity authentication network including a target detection network, a face recognition network, and a palm vein recognition network. The face recognition network is an improved convolutional neural network that combines a MobileNet network and an attention module.

[0090] Step S230. Training the multimodal identity authentication network, specifically including extracting the authentication image training set through the target detection network to obtain the face image area and the palm vein image area, passing the face image area and the palm vein image area through the face recognition network and the palm vein recognition network respectively to obtain the face recognition result and the palm vein recognition result, and determining the identity authentication result based on the face recognition result and the palm vein recognition result.

[0091] Step S240: Update the network parameters of the multimodal identity authentication network according to the identity authentication result, and output the trained multimodal identity authentication network as a multimodal identity authentication model.

[0092] The disclosed method can be implemented using various devices, so the present invention also discloses a device corresponding to the method, which will be described in detail with specific embodiments given below.

[0093] like Figure 3 As shown, one embodiment of the present invention provides a multimodal identity authentication device, comprising:

[0094] Image acquisition module 302, used to acquire the image to be authenticated;

[0095] An image extraction module 304 is configured to process the image to be authenticated using a pre-trained object detection network to extract a facial image region and a palm vein image region;

[0096] Image recognition module 306, configured to input the facial image region and the palm vein image region into a pre-trained facial recognition network and a pre-trained palm vein recognition network, respectively, to obtain facial recognition results and palm vein recognition results; the facial recognition network is an improved convolutional neural network that combines a MobileNet network and an attention module;

[0097] The identity authentication module 308 is used to determine the identity authentication result based on the face recognition result and the palm vein recognition result.

[0098] like Figure 4 As shown, one embodiment of the present invention provides a model training device for training a multimodal identity authentication model, comprising:

[0099] The training sample acquisition module 402 is used to obtain a certification image training set;

[0100] A network construction module 404 is used to construct a multimodal identity authentication network including an object detection network, a face recognition network, and a palm vein recognition network. The face recognition network is an improved convolutional neural network that combines a MobileNet network and an attention module.

[0101] A network training module 406 trains the multimodal identity authentication network, specifically by extracting facial image regions and palm vein image regions from the authentication image training set through an object detection network, passing the facial image regions and palm vein image regions through a face recognition network and a palm vein recognition network, respectively, to obtain facial recognition results and palm vein recognition results, and determining an identity authentication result based on the facial recognition results and the palm vein recognition results;

[0102] The network update module 408 updates the network parameters of the multimodal identity authentication network according to the identity authentication result, and outputs the trained multimodal identity authentication network as a multimodal identity authentication model.

[0103] The device provided in the embodiment of the present application has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0104] The methods and related devices mentioned in the above embodiments are described with reference to the method flow charts and / or structural diagrams provided in the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0105] The following embodiments illustrate this method using a computer device as an example. It is understood that the computer device may be any device with computing and processing capabilities, including, but not limited to, a server or a personal laptop. In one embodiment, the computer device may be an application server, which may be a server for running the application under test.

[0106] See Figure 5 , which shows a hardware block diagram of an electronic device, which is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0107] like Figure 5As shown, the electronic device includes: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0108] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0109] The processor 1 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0110] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory;

[0111] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to: implement the aforementioned multimodal identity authentication method, and / or each step of the model training method.

[0112] An embodiment of the present invention also provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the multimodal identity authentication method provided in the above embodiment and / or any possible implementation method in combination with the embodiment, and / or the various processing flows of the model training method.

[0113] The above embodiments have described the invention in particular detail with respect to possible scenarios, and those skilled in the art will recognize that the invention can be practiced through other embodiments. The specific naming of components, capitalization of terms, attributes, data structures, or any other programming or structural aspects are not mandatory or important, and the mechanisms or features of the invention may have different names, forms, or procedures. The system may be implemented through a combination of hardware and software (as described), entirely through hardware elements, or entirely through software elements. The specific division of functions between the various system components described herein is exemplary only and not mandatory; rather, the functions performed by a single system component may be performed by multiple components, or the functions performed by multiple components may be performed by a single component.

[0114] Those skilled in the art will appreciate that the various steps of the method disclosed above can be implemented by a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the embodiments disclosed herein are not limited to any specific combination of hardware and software.

[0115] The programs executable by these computing devices (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0116] Certain aspects of the present invention include the process steps and instructions described herein in the form of algorithms. It should be noted that the process steps and instructions of the present invention can be implemented in software, firmware and / or hardware, and when implemented in software, they can be downloaded, stored on different platforms used by various operating systems, and operated from the platforms.

[0117] Those skilled in the art will understand that the structures shown in the accompanying drawings are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "possible design" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and features of different embodiments or examples, unless they are mutually inconsistent.

[0119] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0120] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal identity authentication method, characterized in that: include: Collect the image to be authenticated; Processing the image to be authenticated using a pre-trained object detection network to extract a face image region and a palm vein image region; Inputting the facial image region and the palm vein image region into a pre-trained face recognition network and a palm vein recognition network, respectively, to obtain a facial recognition result and a palm vein recognition result, wherein the face recognition network is an improved convolutional neural network that combines a MobileNet network and an attention module; The identity authentication result is determined based on the face recognition result and the palm vein recognition result.

2. The method according to claim 1, characterized in that The method of using a pre-trained target detection network to process the image to be authenticated and extracting the face image area and the palm vein image area includes: Input the image to be authenticated into the pre-trained YOLOv5s model to extract the initial face image area and the initial palm vein image area; The validity of the initial face image area and the initial palm vein image area is verified, and the face image area and the palm vein image area are output when the verification passes.

3. The method according to claim 2, characterized in that The validity verification of the initial face image area and the initial palm vein image area includes: Obtain the signal-to-noise ratio and target area ratio of the initial face image region and the initial palm vein image region; The validity is verified based on the signal-to-noise ratio and the target area ratio.

4. The method according to claim 1, wherein The architecture of the face recognition network includes: Along the feature propagation direction, it includes the input layer, MobileNetv3 network, SE attention module, global average pooling layer, fully connected layer and face classifier.

5. The method according to claim 1, wherein The architecture of the palm vein recognition network includes: Along the feature propagation direction, it includes the input layer, the first convolutional layer, the asymmetric attention module, the second convolutional layer, the maximum pooling layer, the fully connected layer and the palm vein classifier.

6. A model training method, characterized in that: The model trained by the method is used for multimodal identity authentication, including: Get the certified image training set; Build a multimodal identity authentication network that includes an object detection network, a face recognition network, and a palm vein recognition network. The face recognition network is an improved convolutional neural network that combines the MobileNet network and the attention module. Training the multimodal identity authentication network includes extracting the facial image region and the palm vein image region from the authentication image training set through the target detection network, passing the facial image region and the palm vein image region through the face recognition network and the palm vein recognition network respectively to obtain the facial recognition results and the palm vein recognition results, and determining the identity authentication result based on the facial recognition results and the palm vein recognition results; The network parameters of the multimodal identity authentication network are updated according to the identity authentication result, and the trained multimodal identity authentication network is output as a multimodal identity authentication model.

7. A multimodal identity authentication device, characterized in that: include: Image acquisition module, used to acquire the image to be authenticated; An image extraction module is used to process the image to be authenticated using a pre-trained object detection network and extract the facial image area and the palm vein image area; The image recognition module is used to input the facial image area and palm vein image area into the pre-trained face recognition network and palm vein recognition network respectively to obtain face recognition results and palm vein recognition results; the face recognition network is an improved convolutional neural network that combines the MobileNet network and the attention module; The identity authentication module is used to determine the identity authentication result based on the face recognition result and the palm vein recognition result.

8. A model training device, characterized in that: Used to train multimodal authentication models, including: A training sample acquisition module is used to obtain a certification image training set; A network building module is used to construct a multimodal identity authentication network that includes an object detection network, a face recognition network, and a palm vein recognition network. The face recognition network is an improved convolutional neural network that combines the MobileNet network and the attention module. The network training module trains the multimodal identity authentication network. Specifically, the authentication image training set is extracted through the target detection network to obtain the face image area and the palm vein image area. The face image area and the palm vein image area are respectively passed through the face recognition network and the palm vein recognition network to obtain the face recognition results and the palm vein recognition results. The identity authentication result is determined based on the face recognition results and the palm vein recognition results. The network update module updates the network parameters of the multimodal identity authentication network according to the identity authentication results, and outputs the trained multimodal identity authentication network as a multimodal identity authentication model.

9. An electronic device, characterized in that: The device comprises a memory storing computer-executable instructions and a processor, and when the computer-executable instructions are executed by the processor, the device executes the multimodal identity authentication method as described in any one of claims 1 to 5, and / or the model training method as described in claim 6.

10. A readable storage medium, characterized in that: A computer executable program is stored, which, when executed, can implement the multimodal identity authentication method as described in any one of claims 1 to 5, and / or the model training method as described in claim 6.

Images