Cross-modal living body fusion detection method, device, storage medium and computer equipment
By using feature difference and alignment loss functions in cross-modal live detection, and combining score functions and classifiers, the problem of low accuracy of cross-modal live detection in the existing technology is solved, and the problem of low accuracy of cross-modal live detection is improved Accuracy and robustness of the detection.
Patent Information
- Application Number
- CN202311450381.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-02
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-11-02
AI Technical Summary
The existing cross-modal live detection methods have low accuracy, and there are problems such as modal features interfering with each other and dependence on algorithm training effects.
The cross-modal live detection method is adopted to obtain visible light and near-infrared images, and the dual-modal fusion feature is optimized using preset feature difference loss function and alignment loss function, and the detection is combined with preset score function and classifier to avoid modal features. Mutual interference and complex model output combination strategy.
It improves the accuracy and robustness of live detection, reduces the impact of algorithm training effects and model fusion strategies on accuracy, and enhances the reliability of detection.
Smart Images

Figure CN117373138B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of biometric identification technology, and in particular to a cross-modal living body fusion detection method, device, storage medium and computer equipment. Background Art
[0002] With the continuous development and application of artificial intelligence technology, liveness detection, as an important biometric technology, has been widely used in face recognition, identity authentication and other fields. In liveness detection, in order to improve accuracy and security, it is often necessary to combine multiple modalities to obtain more comprehensive information. However, there are some problems with existing cross-modal liveness detection methods.
[0003] One method is to directly combine the target features in the visible light and near-infrared modes. However, this method does not interact with the target features of different modes, which may cause interference between the features of different modes, thus affecting the performance of liveness detection. Another method is to train the liveness detection algorithms in the visible light and near-infrared modes separately, and then fuse the prediction results of the two sets of modes in the inference stage. However, this method depends on the training effect of the two modal algorithms and also requires the design of a suitable model output combination strategy. Therefore, the accuracy and robustness of this method are affected by the algorithm training effect and the model fusion strategy.
[0004] In summary, existing cross-modal liveness detection methods have the problem of low accuracy. Summary of the invention
[0005] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defect that the cross-modal livingness detection method in the prior art has low accuracy.
[0006] In a first aspect, the present application provides a cross-modal living body fusion detection method, the method comprising:
[0007] Acquire visible light images and near infrared images of the living body to be detected;
[0008] Determine a cross-modal liveness detection model, the cross-modal liveness detection model is used to extract sample visible light image features of the sample visible light image and sample near-infrared image features of the sample near-infrared image, and use a preset feature difference loss function and a preset alignment loss function to supervise the optimization of bimodal fusion features corresponding to the sample visible light image features and the sample near-infrared image features, the sample visible light image features are extracted from the target visible light image block of the sample visible light image, the sample near-infrared image features are extracted from the target near-infrared image block of the sample near-infrared image, the bimodal fusion features are based on the original features corresponding to the target visible light image block and the original features corresponding to the target near-infrared image block, by randomly disrupting feature channels and merging them in series, the sample visible light image and the sample near-infrared image are different modalities of the same sample liveness image;
[0009] The cross-modal living body detection model is used to extract target visible light image features of the visible light image, target near-infrared image features of the near-infrared image, and target bimodal fusion features corresponding to the visible light image and the near-infrared image, and based on the target visible light image features, the target near-infrared image features and the target bimodal fusion features, a preset scoring function and a preset classifier are used to obtain the cross-modal detection result of the living body to be detected.
[0010] In one embodiment, the step of obtaining the cross-modal detection result of the living body to be detected according to the target visible light image feature, the target near-infrared image feature and the target bimodal fusion feature through a preset score function and a preset classifier includes:
[0011] According to the target visible light image feature, the target near infrared image feature and the target bimodal fusion feature, the visible light liveness score of the target visible light image feature, the near infrared liveness score of the target near infrared image feature and the bimodal liveness score of the target bimodal fusion feature are obtained by the preset score function;
[0012] According to the visible light live body score, the near infrared live body score and the dual-modal live body score, a cross-modal detection result of the live body to be detected is obtained through the preset classifier.
[0013] In one embodiment, the process of determining the visible light image feature and the near infrared image feature includes:
[0014] Determine a target visible light image block of the sample visible light image and a target near infrared image block of the sample near infrared image;
[0015] Inputting the target visible light image block into a pre-trained visible light image feature extraction network to obtain the visible light image feature;
[0016] The target near-infrared image block is input into a pre-trained near-infrared image feature extraction network to obtain the near-infrared image feature.
[0017] In one embodiment, the step of determining the target visible light image block of the sample visible light image and the target near infrared image block of the sample near infrared image comprises:
[0018] Dividing the sample visible light image and the sample near-infrared image respectively to obtain a plurality of initial visible light image blocks and a plurality of initial near-infrared image blocks, and determining an information amount probability corresponding to each of the initial visible light image blocks and an information amount probability corresponding to each of the initial near-infrared image blocks;
[0019] According to the order of the information amount probabilities corresponding to the initial visible light image blocks from large to small, the initial visible light image blocks corresponding to the first N information amount probabilities are selected as the self-selected visible light image blocks, and according to the order of the information amount probabilities corresponding to the initial near-infrared image blocks from large to small, the initial near-infrared image blocks corresponding to the first N information amount probabilities are selected as the self-selected near-infrared image blocks, where N is a positive integer greater than 1;
[0020] According to the pixel position of each of the self-selected near-infrared image blocks in the near-infrared image, select a visible light image block to be selected from each of the initial visible light image blocks, and use each of the self-selected visible light image blocks and each of the visible light image blocks to be selected as the target visible light image block;
[0021] According to the pixel position of each of the self-selected visible light image blocks in the visible light image, a near-infrared image block to be selected is selected from each of the initial near-infrared image blocks, and each of the self-selected near-infrared image blocks and each of the near-infrared image blocks to be selected are used as the target near-infrared image blocks.
[0022] In one embodiment, the expression of the preset feature difference loss function is as follows:
[0023]
[0024] Wherein, λ1 represents the preset feature difference loss function, Q represents the visible light embedding space feature, represents the original feature corresponding to the jth target visible light image block in the i-th sample visible light image, represents the original features corresponding to the jth target near-infrared image block in the i-th sample near-infrared image, represents the original feature corresponding to the jth target visible light image block in the tth sample visible light image, B represents the batch size of training, for The transpose of represents the original feature corresponding to the jth target near-infrared image block in the tth sample near-infrared image, where Q = P rgb (H rgb ), rgb A visible light feature map head representing the visible light image features, H rgb Represents the visible light image feature.
[0025] In one embodiment, the expression of the preset alignment loss function is as follows:
[0026]
[0027] Where λ2 represents the preset alignment loss function, S rgb represents the probability score of the visible light embedding space feature, S nir represents the feature probability score in the near-infrared embedding space.
[0028] In one embodiment, the bimodal fusion feature is determined by the following expression:
[0029]
[0030] In the formula, represents the bimodal fusion features corresponding to the jth target visible light image block in the i-th sample visible light image and the jth target near-infrared image block in the i-th sample near-infrared image, represents the original feature corresponding to the jth target visible light image block in the i-th sample visible light image, It represents the original features corresponding to the jth target near-infrared image block in the i-th sample near-infrared image, shuffle() represents randomly disrupting the feature channels, and cat() represents serial splicing and fusion.
[0031] In a second aspect, the present application provides a cross-modal living body fusion detection device, the device comprising:
[0032] An image acquisition module, used to acquire visible light images and near infrared images of the living body to be detected;
[0033] A cross-modal living body detection model determination module is used to determine a cross-modal living body detection model, wherein the cross-modal living body detection model is used to extract sample visible light image features of a sample visible light image and sample near-infrared image features of a sample near-infrared image, and adopt a preset feature difference loss function and a preset alignment loss function to supervise and optimize the bimodal fusion features corresponding to the sample visible light image features and the sample near-infrared image features, wherein the sample visible light image features are extracted from a target visible light image block of the sample visible light image, and the sample near-infrared image features are extracted from a target near-infrared image block of the sample near-infrared image, and the bimodal fusion features are obtained by randomly disrupting feature channels and merging them in series based on original features corresponding to the target visible light image block and original features corresponding to the target near-infrared image block, and the sample visible light image and the sample near-infrared image are different modalities of the same sample living body image;
[0034] A cross-modal detection result acquisition module is used to use the cross-modal living body detection model to extract the target visible light image features of the visible light image, the target near-infrared image features of the near-infrared image, and the target bimodal fusion features corresponding to the visible light image and the near-infrared image, and obtain the cross-modal detection result of the living body to be detected through a preset scoring function and a preset classifier based on the target visible light image features, the target near-infrared image features and the target bimodal fusion features.
[0035] In a third aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of any of the above-mentioned cross-modal living body fusion detection methods.
[0036] In a fourth aspect, the present application provides a computer device, comprising: one or more processors, and a memory;
[0037] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the cross-modal living body fusion detection method described in any one of the above embodiments are performed.
[0038] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0039] In the cross-modal living body fusion detection method, device, storage medium and computer equipment provided in the present application, a visible light image and a near-infrared image of a living body to be detected are first obtained; then a cross-modal living body detection model is determined, and the cross-modal living body detection model is used to extract sample visible light image features of the sample visible light image and sample near-infrared image features of the sample near-infrared image, and a preset feature difference loss function and a preset alignment loss function are used to supervise the optimization of the bimodal fusion features corresponding to the sample visible light image features and the sample near-infrared image features, wherein the sample visible light image features are extracted from the target visible light image block of the sample visible light image, and the sample near-infrared image features are extracted from the target near-infrared image block of the sample near-infrared image. The obtained bimodal fusion features are obtained by randomly disrupting the feature channels and merging them in series based on the original features corresponding to the target visible light image blocks and the original features corresponding to the target near-infrared image blocks, while the sample visible light image and the sample near-infrared image are different modalities of the same sample living image. Then, the cross-modal living body detection model is used to extract the target visible light image features of the visible light image, the target near-infrared image features of the near-infrared image, and the target bimodal fusion features corresponding to the visible light image and the near-infrared image. According to the target visible light image features, the target near-infrared image features and the target bimodal fusion features, the cross-modal detection results of the living body to be detected can be obtained through a preset scoring function and a preset classifier. In this method, the preset feature difference loss function and the preset alignment loss function are used to supervise and optimize the bimodal fusion features corresponding to the sample visible light image features and the sample near infrared image features, so as to solve the problem of mutual interference between different modalities, avoid training the liveness detection algorithm under the visible light and near infrared modalities separately, and do not need to design a complex model output combination strategy, reduce the influence of the algorithm training effect and the model fusion strategy on the accuracy and robustness, and improve the practicality of the method; the preset score function and the preset classifier are used to fuse the target visible light image features, the target near infrared image features and the target bimodal fusion features, which can solve the problem that the importance of visible light images and near infrared images is different in different application scenarios, and the bimodal fusion features may have feature loss and feature variation, thereby improving the robustness of liveness detection; combining the target features of multiple modalities, including visible light images, near infrared images and bimodal fusion features, increases the amount of information obtained during detection, and can more comprehensively determine the authenticity of liveness, thereby improving the reliability of liveness detection. In summary, this method can improve the accuracy of cross-modal liveness detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0041] Figure 1 A schematic diagram of the process of the cross-modal living body fusion detection method provided in the embodiment of the present application;
[0042] Figure 2 A schematic flow chart of a process for determining the visible light image features and the near infrared image features provided in an embodiment of the present application;
[0043] Figure 3 A schematic diagram of a flow chart of steps for determining a target visible light image block and a target near-infrared image block provided in an embodiment of the present application;
[0044] Figure 4 An algorithm flow chart of cross-modal living body fusion detection provided in an embodiment of the present application;
[0045] Figure 5 A schematic diagram of the structure of a cross-modal living body fusion detection device provided in an embodiment of the present application;
[0046] Figure 6 This is a diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0048] The present application provides a cross-modal fusion detection method for living bodies. The following embodiments are described by taking the method applied to a computer device as an example. It can be understood that the computer device can be any device with data processing functions, including but not limited to a single server, a server cluster, a personal laptop computer, a desktop computer, etc. Figure 1 As shown, the method may include the following steps:
[0049] S101: Acquire a visible light image and a near infrared image of a living body to be detected.
[0050] Among them, the living body to be detected refers to the face or other biological features that need to be detected for the purpose of determining whether it is a real biological feature, rather than a static or false image such as a photo or video. Visible light images refer to images obtained through the visible light band (usually 380-780nm). This image contains information such as color and texture, and is usually visible to the human eye. Near-infrared images refer to images obtained through the near-infrared band (usually 780-2500nm). This image can penetrate the surface of the skin, making it easier to obtain the characteristics of the tissue under the skin, and is more resistant to factors such as lighting conditions and changes in facial expressions.
[0051] In this step, the acquisition method of the visible light image and the near infrared image of the living body to be detected can be determined according to actual conditions, and this application does not impose specific restrictions on this. It can be understood that the visible light image and the near infrared image of the living body to be detected are two modes of the image of the same living body to be detected.
[0052] S102: Determine a cross-modal living body detection model.
[0053] Among them, the cross-modal living body detection model is used to extract sample visible light image features of the sample visible light image and sample near-infrared image features of the sample near-infrared image, and adopts a preset feature difference loss function and a preset alignment loss function to supervise the optimization of the bimodal fusion features corresponding to the sample visible light image features and the sample near-infrared image features. The sample visible light image features are extracted from the target visible light image block of the sample visible light image, and the sample near-infrared image features are extracted from the target near-infrared image block of the sample near-infrared image. The bimodal fusion features are based on the original features corresponding to the target visible light image block and the original features corresponding to the target near-infrared image block, by randomly disrupting the feature channels and merging them in series. The sample visible light image and the sample near-infrared image are different modalities of the same sample living body image.
[0054] Among them, the cross-modal liveness detection model is a liveness detection model for processing visible light images and near-infrared images at the same time. The sample visible light image feature refers to a numerical vector or feature map extracted from the sample visible light image to represent its feature. The sample near-infrared image feature refers to a numerical vector or feature map extracted from the sample near-infrared image to represent its feature. The preset feature difference loss function refers to a loss function used in the training process, which is used to measure the difference between the sample visible light image feature and the sample near-infrared image feature to enhance their distinguishing ability. The preset alignment loss function refers to a loss function used in the training process, which is used to measure the degree of alignment between the sample visible light image feature and the sample near-infrared image feature to improve their consistency and fusion effect. The bimodal fusion feature refers to a feature representation obtained by randomly disrupting the feature channels and concatenating and fusing them according to the sample visible light image feature and the sample near-infrared image feature. The target visible light image block refers to the region of interest extracted from the sample visible light image. The target near-infrared image block refers to the region of interest extracted from the near-infrared image of the sample, which is usually an image block corresponding to the target visible light image block. The original features corresponding to the target visible light image block refer to the original visible light image features used when extracting the target visible light image block, and generally refer to the features that have not been processed or transformed in any way. The original features corresponding to the target near-infrared image block refer to the features that have not been processed or transformed in any way when extracting the target near-infrared image block.
[0055] In this step, an initial cross-modal liveness detection model can be first constructed, and then the sample visible light image and the sample near-infrared image are input into the initial cross-modal liveness detection model, so that the model extracts the sample visible light image features of the sample visible light image and the sample near-infrared image features of the sample near-infrared image, and uses a preset feature difference loss function and a preset alignment loss function to supervise and optimize the bimodal fusion features corresponding to the sample visible light image features and the sample near-infrared image features, wherein the sample visible light image and the sample near-infrared image are different modalities of the same sample liveness image. Based on the output results of the model and the preset feature difference loss function and the preset alignment loss function, the model is iteratively trained until the model meets the preset training end conditions, and the final cross-modal fusion detection model is obtained.
[0056] It can be understood that the neural network adopted by the cross-modal fusion detection model of the present application can be selected according to actual conditions, and the present application does not impose any specific restrictions on this.
[0057] S103: Utilizing the cross-modal living body detection model, extracting target visible light image features of the visible light image, target near-infrared image features of the near-infrared image, and target bimodal fusion features corresponding to the visible light image and the near-infrared image, and obtaining a cross-modal detection result of the living body to be detected through a preset scoring function and a preset classifier based on the target visible light image features, the target near-infrared image features, and the target bimodal fusion features.
[0058] Among them, the preset score function refers to a function used to calculate the score of the living body to be detected. In one example, the preset score function may include a combination of several sub-functions, and the sub-function may be a linear or nonlinear function used to perform weighted summation or transformation on the input features. The preset classifier refers to a classifier used to perform classification tasks on the obtained features. In one example, common machine learning methods such as support vector machines, logistic regression, and decision trees can be used to build a classifier. The cross-modal detection result refers to the processing of the extracted target visible light image features, target near-infrared image features, and target bimodal fusion features according to the preset score function and the preset classifier to obtain the final liveness detection result. In one example, the cross-modal detection result can be a binary classification result, including the judgment of real biological people and prostheses.
[0059] In the above embodiment, a visible light image and a near-infrared image of a living body to be detected are first obtained; then a cross-modal living body detection model is determined, and the cross-modal living body detection model is used to extract sample visible light image features of the sample visible light image and sample near-infrared image features of the sample near-infrared image, and a preset feature difference loss function and a preset alignment loss function are used to supervise the optimization of bimodal fusion features corresponding to the sample visible light image features and the sample near-infrared image features, wherein the sample visible light image features are extracted from the target visible light image block of the sample visible light image, and the sample near-infrared image features are extracted from the target near-infrared image block of the sample near-infrared image, and the bimodal fusion features are based on According to the original features corresponding to the target visible light image blocks and the original features corresponding to the target near-infrared image blocks, the feature channels are randomly disrupted and fused through serial splicing, while the sample visible light image and the sample near-infrared image are different modalities of the same sample living image; then the cross-modal living body detection model is used to extract the target visible light image features of the visible light image, the target near-infrared image features of the near-infrared image, and the target bimodal fusion features corresponding to the visible light image and the near-infrared image, and according to the target visible light image features, the target near-infrared image features and the target bimodal fusion features, the cross-modal detection result of the living body to be detected can be obtained through a preset scoring function and a preset classifier. In this method, the preset feature difference loss function and the preset alignment loss function are used to supervise and optimize the bimodal fusion features corresponding to the sample visible light image features and the sample near infrared image features, so as to solve the problem of mutual interference between different modalities, avoid training the liveness detection algorithm under the visible light and near infrared modalities separately, and do not need to design a complex model output combination strategy, reduce the influence of the algorithm training effect and the model fusion strategy on the accuracy and robustness, and improve the practicality of the method; the preset score function and the preset classifier are used to fuse the target visible light image features, the target near infrared image features and the target bimodal fusion features, which can solve the problem that the importance of visible light images and near infrared images is different in different application scenarios, and the bimodal fusion features may have feature loss and feature variation, thereby improving the robustness of liveness detection; combining the target features of multiple modalities, including visible light images, near infrared images and bimodal fusion features, increases the amount of information obtained during detection, and can more comprehensively determine the authenticity of liveness, thereby improving the reliability of liveness detection. In summary, this method can improve the accuracy of cross-modal liveness detection.
[0060] In one embodiment, the step of obtaining the cross-modal detection result of the living body to be detected according to the target visible light image feature, the target near-infrared image feature and the target bimodal fusion feature through a preset score function and a preset classifier includes:
[0061] According to the target visible light image feature, the target near infrared image feature and the target bimodal fusion feature, the visible light liveness score of the target visible light image feature, the near infrared liveness score of the target near infrared image feature and the bimodal liveness score of the target bimodal fusion feature are obtained by the preset score function;
[0062] According to the visible light live body score, the near infrared live body score and the dual-modal live body score, a cross-modal detection result of the live body to be detected is obtained through the preset classifier.
[0063] Specifically, according to a preset score function, a visible light liveness score is calculated for the target visible light image feature, wherein the visible light liveness score can reflect the similarity between the biological features in the visible light image and the real human body. In one embodiment, the visible light liveness score can be obtained through a series of feature transformations, weighted summation or other function operations. Similarly, according to a preset score function, a near-infrared liveness score is calculated for the target near-infrared image feature, wherein the near-infrared liveness score can evaluate the correlation between the biological features in the near-infrared image and the real human body. For the target dual-modal fusion feature, the preset score function is also used for calculation, and a dual-modal liveness score is obtained, wherein the dual-modal liveness score comprehensively considers the complementary and synergistic information between the visible light image and the near-infrared image. The obtained visible light liveness score, near-infrared liveness score and dual-modal liveness score are used as inputs and classified by a preset classifier. In one example, the classifier can select a suitable algorithm according to different needs, such as a support vector machine or logistic regression. In another example, the classifier can determine whether the live body to be detected is a real biological person or a prosthesis according to the size of the score.
[0064] In one example, three feature storage units K can be established first rgb , K nir and K mix , used to save the original visible light image patch features, the original near infrared image patch features and the dual-modal patch fusion features respectively. Definition is the scoring function. Each feature storage unit predicts a liveness score, and then a learnable classifier φ is used to make the final classification decision. The specific expression is as follows:
[0065]
[0066] In the formula, H rgb represents the target visible light image feature, H nir represents the target near infrared image feature, H mix Represents the target bimodal fusion feature.
[0067] In this embodiment, the diversity of application scenarios is taken into account. For example, the imaging quality of visible light images is poor in dark or backlit environments. At this time, the classification decision is mainly based on near-infrared images; under normal clear lighting, since the features contained in visible light images are more obvious, the classification decision is mainly based on visible light images; dual-modal fusion features can be used for classification decisions in any environment, but the features fused from different modalities may have feature loss and feature variation. Therefore, the use of a preset scoring function and a preset classifier to obtain the final cross-modal detection result can solve the different importance of visible light images and near-infrared images, as well as the problem of feature loss and feature variation in dual-modal fusion features, thereby improving the robustness of liveness detection.
[0068] like Figure 2 As shown, in one embodiment, the process of determining the visible light image feature and the near infrared image feature includes:
[0069] S201: Determine a target visible light image block of the sample visible light image and a target near infrared image block of the sample near infrared image;
[0070] S202: Inputting the target visible light image block into a pre-trained visible light image feature extraction network to obtain the visible light image feature;
[0071] S203: Inputting the target near-infrared image block into a pre-trained near-infrared image feature extraction network to obtain the near-infrared image features.
[0072] Specifically, first, it is necessary to determine the target visible light image block and the target near-infrared image block in the sample. In one example, this can be achieved through a target detection algorithm or manual selection. Next, the target visible light image block is input into a pre-trained visible light image feature extraction network. In one example, the visible light image feature extraction network can be a deep learning model such as a convolutional neural network. Through forward propagation, the network will perform a series of convolution, pooling, and activation operations on the input image to extract feature information from the image. The final output is the feature representation of the target visible light image. Similarly, the target near-infrared image block is input into a pre-trained near-infrared image feature extraction network to obtain near-infrared image features.
[0073] In this embodiment, by determining the target image block in the visible light image and the near infrared image, the influence of invalid information can be reduced and the accuracy of living body detection can be improved. Using the pre-trained visible light image feature extraction network and the near infrared image feature extraction network, the biometric information can be quickly and accurately extracted from the target image block.
[0074] like Figure 3As shown, in one embodiment, the step of determining the target visible light image block of the sample visible light image and the target near infrared image block of the sample near infrared image includes:
[0075] S301: Dividing the sample visible light image and the sample near-infrared image respectively to obtain a plurality of initial visible light image blocks and a plurality of initial near-infrared image blocks, and determining an information amount probability corresponding to each of the initial visible light image blocks and an information amount probability corresponding to each of the initial near-infrared image blocks;
[0076] S302: selecting, in descending order of the information amount probabilities corresponding to the initial visible light image blocks, the initial visible light image blocks corresponding to the first N information amount probabilities as the self-selected visible light image blocks, and selecting, in descending order of the information amount probabilities corresponding to the initial near-infrared image blocks, the initial near-infrared image blocks corresponding to the first N information amount probabilities as the self-selected near-infrared image blocks, where N is a positive integer greater than 1;
[0077] S303: selecting a visible light image block to be selected from each of the initial visible light image blocks according to a pixel position of each of the self-selected near-infrared image blocks in the near-infrared image, and using each of the self-selected visible light image blocks and each of the visible light image blocks to be selected as the target visible light image block;
[0078] S304: selecting a near-infrared image block to be selected from each of the initial near-infrared image blocks according to the pixel position of each of the self-selected visible light image blocks in the visible light image, and using each of the self-selected near-infrared image blocks and each of the near-infrared image blocks to be selected as the target near-infrared image blocks.
[0079] Specifically, first, the sample visible light image and the sample near-infrared image are divided into a plurality of initial visible light image blocks and initial near-infrared image blocks. In one example, image processing techniques such as segmentation algorithms or sliding window methods can be used to achieve this. After the division, the amount of information contained in each initial visible light image block and initial near-infrared image block is calculated, and the corresponding information amount probability is determined accordingly. According to the information amount probability of the initial visible light image block and the initial near-infrared image block, the first N initial visible light image blocks with the highest information amount probability are selected in order from large to small as the self-selected visible light image blocks. Similarly, the first N initial near-infrared image blocks with the highest information amount probability are also selected as the self-selected near-infrared image blocks. N is a positive integer greater than 1 and can be determined according to specific needs. For the self-selected near-infrared image block, in the near-infrared image, the visible light image block to be selected can be selected in the initial visible light image block according to its pixel position. Specifically, the central pixel position of the self-selected near-infrared image block can be calculated, and the corresponding visible light image block can be found, so that the self-selected visible light image block and the visible light image block to be selected can be obtained as the target visible light image block. Similarly, for the self-selected visible light image block, in the visible light image, the near-infrared image block to be selected is selected in the initial near-infrared image block according to its pixel position, the central pixel position of the self-selected visible light image block can be calculated, and the corresponding near-infrared image block can be found, so that the self-selected near-infrared image block and the near-infrared image block to be selected can be obtained as the target near-infrared image block.
[0080] In an example, given a pair of raw visible light and near infrared images of palms, the palm ROI region is first extracted by the palm detection algorithm, and then each ROI region is segmented and patched. Here, the visible light palm image is taken as an example. is the visible light image patch set, P i represents the i-th patch, M represents the number of patches in a visible light roi image, and P i After vectorization, calculate the information amount δ of each patch i , including the texture clarity, brightness, and significant area of the patch block, as shown in the following expression:
[0081] δ i =‖|P i |·E‖1,i∈[1,M]
[0082] Where |P i | indicates P i In the vectorization process, E represents the unit vector, ‖·‖1 represents the L1 regularization process, and the amount of information δ for each patch iL1 regularization is performed to obtain the probability distribution of the information content of the image patch block, and then the probability distribution is sorted in descending order, and the top8 is selected as the self-selected patch block of the visible light palm. The information content of the near-infrared image patch is calculated in the same way, and the top8 is also selected as the self-selected patch block of the near-infrared palm. So far, 8 self-selected image patches of the visible light palm and the near-infrared palm have been obtained. Afterwards, in order to interact with the features between the corresponding patches of the subsequent dual-modal images, according to the pixel position of the self-selected patch block in the image, the image patch is selected as the selected patch at the corresponding position of the other modality. In this way, 8 image patches of the visible light palm and the near-infrared palm can be obtained respectively. The image patches finally selected include: 8 self-selected image patches of the visible light palm, 8 selected image patches, 8 self-selected image patches of the near-infrared palm, and 8 selected image patches. This method can select image patches with high discriminative features in at least one modality.
[0083] In this embodiment, the most representative and information-dense visible light image blocks and near-infrared image blocks for a specific task are selected by dividing the images of different modalities and calculating the probability of the amount of information. Compared with directly using the complete original image, this can effectively reduce the amount of calculation and time for processing, and can avoid unnecessary redundant information in the analysis process. In addition, according to the pixel position of the self-selected near-infrared image block in the near-infrared image, the visible light image block to be selected is selected in the visible light image, which can ensure that there is a clear correspondence between the selected visible light image block and the near-infrared image block, so that subsequent processing and feature extraction can be performed more accurately.
[0084] In one embodiment, the expression of the preset feature difference loss function is as follows:
[0085]
[0086] Wherein, λ1 represents the preset feature difference loss function, Q represents the visible light embedding space feature, represents the original feature corresponding to the jth target visible light image block in the i-th sample visible light image, represents the original features corresponding to the jth target near-infrared image block in the i-th sample near-infrared image, represents the original feature corresponding to the jth target visible light image block in the tth sample visible light image, B represents the batch size of training, for The transpose of represents the original feature corresponding to the jth target near-infrared image block in the tth sample near-infrared image, where Q = P rgb (H rgb), rgb A visible light feature map head representing the visible light image features, H rgb Represents the visible light image feature.
[0087] Specifically, considering that the difference between the original features of the image blocks of the same modality and the embedded space features after mapping by the mapping head should be as small as possible, the first goal of network optimization is to minimize the difference between the original features and the embedded space features of the image blocks of the same modality. At the same time, considering that there is complementary information between the features of different modalities, the feature difference between the original features of the image blocks at the same position of different modalities should also be as small as possible. Therefore, the second goal of network optimization is to minimize the feature difference between the original features of the image blocks at the same position of different modalities. Based on this, by designing a preset feature difference loss function, high discriminative features can be effectively retained, and the difference between the original features of the image blocks of the same modality and the embedded space features, as well as the difference between the original features of the image blocks at the same position of different modalities, can be minimized, thereby improving the performance and generalization ability of the model in multimodal tasks.
[0088] In one embodiment, the expression of the preset alignment loss function is as follows:
[0089]
[0090] Where λ2 represents the preset alignment loss function, S rgb represents the probability score of the visible light embedding space feature, S nir represents the feature probability score in the near-infrared embedding space.
[0091] Among them, S rgb The specific expression is as follows:
[0092]
[0093] Where i and j represent the subscripts of the i-th and j-th visible light image block embedding space features, respectively, T and U both represent the number of image blocks, and ‖·‖2 represents the L2 regularization process. Q is the visible light embedding space feature, Q = P rgb (H rgb ), P rgb Represents the visible light mapping head, H rgb Represents the visible light image features.
[0094] Similarly, S nir It can be determined by referring to the above expression, as shown below:
[0095]
[0096] Where R is the visible light embedding space feature, R = P nir(H nir ), P nir represents the near infrared mapping head, H nir Represents the near-infrared image features.
[0097] Specifically, there will be feature loss or feature variation during the optimization process. Therefore, considering the similarity between the near-infrared embedded features and the visible light embedded features, the optimization process of the bimodal feature fusion is supervised by adding the similarity between the corresponding bimodal image embedded features. Therefore, by designing a preset alignment loss function to ensure the similarity between the near-infrared embedded features and the visible light embedded features, feature loss or feature variation can be effectively reduced, and the model robustness and generalization ability can be improved.
[0098] In one embodiment, the bimodal fusion feature is determined by the following expression:
[0099]
[0100] In the formula, represents the bimodal fusion features corresponding to the jth target visible light image block in the i-th sample visible light image and the jth target near-infrared image block in the i-th sample near-infrared image, represents the original feature corresponding to the jth target visible light image block in the i-th sample visible light image, It represents the original features corresponding to the jth target near-infrared image block in the i-th sample near-infrared image, shuffle() represents randomly disrupting the feature channels, and cat() represents serial splicing and fusion.
[0101] Specifically, the shuffle() function randomly shuffles the input feature channels to increase the model's robustness and generalization ability to the features. The cat() function concatenates two feature vectors in series along the channel direction into a new feature vector to achieve the fusion of bimodal features. In this way, the information of visible light images and near-infrared images can be organically combined, thereby improving the processing capability and performance of multimodal data. In addition, by randomly shuffling the feature channels, the robustness and anti-interference ability of the model can also be enhanced, so that it can better adapt to different tasks and environments.
[0102] To facilitate understanding of the solution of this application, a specific example is provided below for illustration. Figure 4 As shown, the cross-modal living body fusion detection method provided in this example is as follows:
[0103] Set the living body to be detected as the palm, given a set of paired visible light and near-infrared original images, select 8 self-selected image patches of visible light palm, 8 image patches to be selected, 8 self-selected image patches of near-infrared palm, and 8 image patches to be selected. Definition For the jth selected patch of the i-th visible light and near infrared image, input the pre-trained feature extraction network respectively and Extracting image features and Input the visible light modality and near infrared modality patch features and The original features are obtained after the multi-layer perceptron This is shown in the following expression:
[0104]
[0105] Taking the visible light palm as an example, Represents the visible light mapping head, embedding spatial features Design feature difference loss function:
[0106]
[0107] Among them, B represents the training batch size, which is set to 64. By minimizing the above l2 loss, the features of the same patch block of different modalities can be effectively aggregated and interacted, and the bimodal features are fused by randomly disrupting the feature channels and then concatenating them in series:
[0108]
[0109] For visible light embedding space features The probability scores are calculated by fitting an exponential kernel combined with the cosine similarity between different patches:
[0110]
[0111] Among them, T and U both represent the number of image patches, which is set to 16.
[0112] make is the near infrared feature mapper, embedding spatial features The near-infrared embedding space feature probability score S is calculated in the same way. nir , then the alignment loss between the bimodal image embedding features is:
[0113]
[0114] By adding the similarity between the corresponding bimodal image embedding features, the optimization process of bimodal feature fusion is supervised, and the overall optimization loss is as follows:
[0115] l=l1+l2
[0116] Establish three feature storage units K rgb , K nir and K mix , used to save the original visible light image patch features, the original near infrared image patch features and the dual-modal patch fusion features respectively. Definition is the scoring function. Each feature storage unit predicts a liveness score, and then a learnable classifier φ is used to make the final classification decision. The specific expression is as follows:
[0117]
[0118]
[0119] The following is a description of the cross-modal living body fusion detection device provided in the embodiment of the present application. The cross-modal living body fusion detection device described below and the cross-modal living body fusion detection method described above can be referred to each other. Figure 5 As shown, the present application provides a cross-modal living body fusion detection device, the device comprising:
[0120] An image acquisition module 401 is used to acquire a visible light image and a near infrared image of a living body to be detected;
[0121] A cross-modal living body detection model determination module 402 is used to determine a cross-modal living body detection model, wherein the cross-modal living body detection model is used to extract sample visible light image features of a sample visible light image and sample near-infrared image features of a sample near-infrared image, and uses a preset feature difference loss function and a preset alignment loss function to supervise and optimize the bimodal fusion features corresponding to the sample visible light image features and the sample near-infrared image features, wherein the sample visible light image features are extracted from a target visible light image block of the sample visible light image, and the sample near-infrared image features are extracted from a target near-infrared image block of the sample near-infrared image, and the bimodal fusion features are obtained by randomly disrupting feature channels and merging them in series based on the original features corresponding to the target visible light image block and the original features corresponding to the target near-infrared image block, and the sample visible light image and the sample near-infrared image are different modalities of the same sample living body image;
[0122] The cross-modal detection result acquisition module 403 is used to use the cross-modal living body detection model to extract the target visible light image features of the visible light image, the target near-infrared image features of the near-infrared image, and the target bimodal fusion features corresponding to the visible light image and the near-infrared image, and obtain the cross-modal detection result of the living body to be detected through a preset scoring function and a preset classifier based on the target visible light image features, the target near-infrared image features and the target bimodal fusion features.
[0123] In one embodiment, the cross-modal detection result acquisition module 403 includes:
[0124] a liveness score acquisition unit, configured to obtain, according to the target visible light image feature, the target near infrared image feature and the target bimodal fusion feature, a visible light liveness score of the target visible light image feature, a near infrared liveness score of the target near infrared image feature, and a bimodal liveness score of the target bimodal fusion feature through the preset score function;
[0125] A cross-modal detection result acquisition unit is used to obtain the cross-modal detection result of the living body to be detected through the preset classifier according to the visible light living body score, the near-infrared living body score and the dual-modal living body score.
[0126] In one embodiment, the cross-modal living body detection model determination module 402 includes:
[0127] a target image block determining unit, configured to determine a target visible light image block of the sample visible light image and a target near infrared image block of the sample near infrared image;
[0128] A visible light image feature acquisition unit, used for inputting the target visible light image block into a pre-trained visible light image feature extraction network to obtain the visible light image feature;
[0129] The near infrared image feature acquisition unit is used to input the target near infrared image block into a pre-trained near infrared image feature extraction network to obtain the near infrared image feature.
[0130] In one embodiment, the target image block determination unit includes:
[0131] an information quantity probability determination subunit, used for dividing the sample visible light image and the sample near-infrared image respectively to obtain a plurality of initial visible light image blocks and a plurality of initial near-infrared image blocks, and determining an information quantity probability corresponding to each of the initial visible light image blocks and an information quantity probability corresponding to each of the initial near-infrared image blocks;
[0132] a self-selected image block selection unit, configured to select, in descending order of the information amount probabilities corresponding to the initial visible light image blocks, the initial visible light image blocks corresponding to the first N information amount probabilities as the self-selected visible light image blocks, and to select, in descending order of the information amount probabilities corresponding to the initial near-infrared image blocks, the initial near-infrared image blocks corresponding to the first N information amount probabilities as the self-selected near-infrared image blocks, where N is a positive integer greater than 1;
[0133] a target visible light image block determination subunit, configured to select a selectable visible light image block from each of the initial visible light image blocks according to a pixel position of each of the self-selected near-infrared image blocks in the near-infrared image, and use each of the self-selected visible light image blocks and each of the selectable visible light image blocks as the target visible light image block;
[0134] The target near-infrared image determination subunit is used to select a near-infrared image block to be selected from each of the initial near-infrared image blocks according to the pixel position of each of the self-selected visible light image blocks in the visible light image, and use each of the self-selected near-infrared image blocks and each of the near-infrared image blocks to be selected as the target near-infrared image block.
[0135] In one embodiment, the expression of the preset feature difference loss function is as follows:
[0136]
[0137] Wherein, λ1 represents the preset feature difference loss function, Q represents the visible light embedding space feature, represents the original feature corresponding to the jth target visible light image block in the i-th sample visible light image, represents the original features corresponding to the jth target near-infrared image block in the i-th sample near-infrared image, represents the original feature corresponding to the jth target visible light image block in the tth sample visible light image, B represents the batch size of training, for The transpose of represents the original feature corresponding to the jth target near-infrared image block in the tth sample near-infrared image, where Q = P rgb (H rgb ), rgb A visible light feature map head representing the visible light image features, H rgb Represents the visible light image feature.
[0138] In one embodiment, the expression of the preset alignment loss function is as follows:
[0139]
[0140] Where λ2 represents the preset alignment loss function, S rgb represents the probability score of the visible light embedding space feature, S nir represents the feature probability score in the near-infrared embedding space.
[0141] In one embodiment, the bimodal fusion feature is determined by the following expression:
[0142]
[0143] In the formula, represents the bimodal fusion features corresponding to the jth target visible light image block in the i-th sample visible light image and the jth target near-infrared image block in the i-th sample near-infrared image, represents the original feature corresponding to the jth target visible light image block in the i-th sample visible light image, It represents the original features corresponding to the jth target near-infrared image block in the i-th sample near-infrared image, shuffle() represents randomly disrupting the feature channels, and cat() represents serial splicing and fusion.
[0144] In one embodiment, the present application also provides a storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the cross-modal living body fusion detection method as described in any of the above embodiments.
[0145] In one embodiment, the present application also provides a computer device, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the cross-modal living body fusion detection method as described in any of the above embodiments.
[0146] Indicatively, Figure 6 As shown, Figure 6 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 500 may be provided as a server. Figure 6 The computer device 500 includes a processing component 502, which further includes one or more processors, and a memory resource represented by a memory 501, for storing instructions executable by the processing component 502, such as an application. The application stored in the memory 501 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 502 is configured to execute instructions to perform the cross-modal living body fusion detection method of any of the above embodiments.
[0147] The computer device 500 may further include a power supply component 503 configured to perform power management of the computer device 500, a wired or wireless network interface 504 configured to connect the computer device 500 to a network, and an input / output (I / O) interface 505. The computer device 500 may operate based on an operating system stored in the memory 501, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.
[0148] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0149] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, "one", "one", "said", "the" and "it" may also include plural forms, unless the context clearly indicates another way. A plurality refers to at least two cases, such as 2, 3, 5 or 8, etc. "And / or" includes any and all combinations of the relevant listed items.
[0150] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.
[0151] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A cross-modal living body fusion detection method, characterized in that: The method comprises: Acquire visible light images and near infrared images of the living body to be detected; Determine a cross-modal liveness detection model, the cross-modal liveness detection model is used to extract sample visible light image features of the sample visible light image and sample near-infrared image features of the sample near-infrared image, and use a preset feature difference loss function and a preset alignment loss function to supervise the optimization of bimodal fusion features corresponding to the sample visible light image features and the sample near-infrared image features, the sample visible light image features are extracted from the target visible light image block of the sample visible light image, the sample near-infrared image features are extracted from the target near-infrared image block of the sample near-infrared image, the bimodal fusion features are based on the original features corresponding to the target visible light image block and the original features corresponding to the target near-infrared image block, by randomly disrupting feature channels and merging them in series, the sample visible light image and the sample near-infrared image are different modalities of the same sample liveness image; The expression of the preset feature difference loss function is as follows: Wherein, λ1 represents the preset feature difference loss function, Q represents the visible light embedding space feature, represents the original feature corresponding to the jth target visible light image block in the i-th sample visible light image, represents the original features corresponding to the jth target near-infrared image block in the i-th sample near-infrared image, represents the original feature corresponding to the jth target visible light image block in the tth sample visible light image, B represents the batch size of training, for The transpose of represents the original feature corresponding to the jth target near-infrared image block in the tth sample near-infrared image, where Q = P rgb (H rgb ), rgb A visible light feature map head representing the visible light image features, H rgb represents the visible light image feature; The expression of the preset alignment loss function is as follows: Where λ2 represents the preset alignment loss function, S rgb represents the probability score of the visible light embedding space feature, S nir represents the feature probability score of near-infrared embedding space; The cross-modal living body detection model is used to extract target visible light image features of the visible light image, target near-infrared image features of the near-infrared image, and target bimodal fusion features corresponding to the visible light image and the near-infrared image, and based on the target visible light image features, the target near-infrared image features and the target bimodal fusion features, a preset scoring function and a preset classifier are used to obtain the cross-modal detection result of the living body to be detected.
2. The cross-modal living body fusion detection method according to claim 1, characterized in that: The step of obtaining the cross-modal detection result of the living body to be detected according to the target visible light image feature, the target near-infrared image feature and the target bimodal fusion feature through a preset score function and a preset classifier includes: According to the target visible light image feature, the target near infrared image feature and the target bimodal fusion feature, the visible light liveness score of the target visible light image feature, the near infrared liveness score of the target near infrared image feature and the bimodal liveness score of the target bimodal fusion feature are obtained by the preset score function; According to the visible light live body score, the near infrared live body score and the dual-modal live body score, a cross-modal detection result of the live body to be detected is obtained through the preset classifier.
3. The cross-modal living body fusion detection method according to claim 1, characterized in that: The process of determining the visible light image feature and the near infrared image feature includes: Determine a target visible light image block of the sample visible light image and a target near infrared image block of the sample near infrared image; Inputting the target visible light image block into a pre-trained visible light image feature extraction network to obtain the visible light image feature; The target near-infrared image block is input into a pre-trained near-infrared image feature extraction network to obtain the near-infrared image feature.
4. The cross-modal living body fusion detection method according to claim 3, characterized in that: The step of determining the target visible light image block of the sample visible light image and the target near infrared image block of the sample near infrared image comprises: Dividing the sample visible light image and the sample near-infrared image respectively to obtain a plurality of initial visible light image blocks and a plurality of initial near-infrared image blocks, and determining an information amount probability corresponding to each of the initial visible light image blocks and an information amount probability corresponding to each of the initial near-infrared image blocks; According to the order of the information amount probabilities corresponding to the initial visible light image blocks from large to small, the initial visible light image blocks corresponding to the first N information amount probabilities are selected as the self-selected visible light image blocks, and according to the order of the information amount probabilities corresponding to the initial near-infrared image blocks from large to small, the initial near-infrared image blocks corresponding to the first N information amount probabilities are selected as the self-selected near-infrared image blocks, where N is a positive integer greater than 1; According to the pixel position of each of the self-selected near-infrared image blocks in the near-infrared image, select a visible light image block to be selected from each of the initial visible light image blocks, and use each of the self-selected visible light image blocks and each of the visible light image blocks to be selected as the target visible light image block; According to the pixel position of each of the self-selected visible light image blocks in the visible light image, a near-infrared image block to be selected is selected from each of the initial near-infrared image blocks, and each of the self-selected near-infrared image blocks and each of the near-infrared image blocks to be selected are used as the target near-infrared image blocks.
5. The cross-modal living body fusion detection method according to claim 1, characterized in that: The dual-modal fusion feature is determined by the following expression: In the formula, represents the bimodal fusion features corresponding to the jth target visible light image block in the i-th sample visible light image and the jth target near-infrared image block in the i-th sample near-infrared image, represents the original feature corresponding to the jth target visible light image block in the i-th sample visible light image, It represents the original features corresponding to the jth target near-infrared image block in the i-th sample near-infrared image, shuffle() represents randomly disrupting the feature channels, and cat() represents serial splicing and fusion.
6. A cross-modal living body fusion detection device, characterized in that: The device comprises: An image acquisition module, used to acquire visible light images and near infrared images of the living body to be detected; A cross-modal living body detection model determination module is used to determine a cross-modal living body detection model, wherein the cross-modal living body detection model is used to extract sample visible light image features of a sample visible light image and sample near-infrared image features of a sample near-infrared image, and adopt a preset feature difference loss function and a preset alignment loss function to supervise and optimize the bimodal fusion features corresponding to the sample visible light image features and the sample near-infrared image features, wherein the sample visible light image features are extracted from a target visible light image block of the sample visible light image, and the sample near-infrared image features are extracted from a target near-infrared image block of the sample near-infrared image, and the bimodal fusion features are obtained by randomly disrupting feature channels and merging them in series based on original features corresponding to the target visible light image block and original features corresponding to the target near-infrared image block, and the sample visible light image and the sample near-infrared image are different modalities of the same sample living body image; The expression of the preset feature difference loss function is as follows: Wherein, λ1 represents the preset feature difference loss function, Q represents the visible light embedding space feature, represents the original feature corresponding to the jth target visible light image block in the i-th sample visible light image, represents the original features corresponding to the jth target near-infrared image block in the i-th sample near-infrared image, represents the original feature corresponding to the jth target visible light image block in the tth sample visible light image, B represents the batch size of training, for The transpose of represents the original feature corresponding to the jth target near-infrared image block in the tth sample near-infrared image, where Q = P rgb (H rgb ), rgb A visible light feature map head representing the visible light image features, H rgb represents the visible light image feature; The expression of the preset alignment loss function is as follows: Where λ2 represents the preset alignment loss function, S rgb represents the probability score of the visible light embedding space feature, S nir represents the feature probability score of near-infrared embedding space; A cross-modal detection result acquisition module is used to use the cross-modal living body detection model to extract the target visible light image features of the visible light image, the target near-infrared image features of the near-infrared image, and the target bimodal fusion features corresponding to the visible light image and the near-infrared image, and obtain the cross-modal detection result of the living body to be detected through a preset scoring function and a preset classifier based on the target visible light image features, the target near-infrared image features and the target bimodal fusion features.
7. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the cross-modal living body fusion detection method as described in any one of claims 1 to 5.
8. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the cross-modal living body fusion detection method as described in any one of claims 1 to 5 are performed.
