A biometric recognition method and system based on monocular depth-guided multimodal fusion

Abstract

The present application provides a biometric recognition method and system based on monocular depth-guided multimodal fusion. The method comprises acquiring an image of the organism to be recognized; performing depth estimation on the image of the organism to be recognized using a monocular depth model to obtain a pseudo-depth map; extracting features of multiple resolutions from the pseudo-depth map and the image of the organism to be recognized to obtain multi-level features; predicting the sampling position of each level of features through a convolutional layer, and sampling and fusing based on the predicted sampling positions to obtain multi-scale features; extracting and enhancing direction-specific features based on the image features and the depth features to obtain enhanced features; upsampling and enhancing boundary features based on the multi-scale features and the enhanced features to obtain decoded features; and performing pixel-level prediction based on the decoded features to obtain the category of the organism to be recognized. The present application can improve the accuracy of biological category prediction.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a biometric recognition method and system based on monocular depth-guided multimodal fusion. Background Art

[0002] In lightweight deployment scenarios such as biodiversity monitoring and smart agriculture, identifying organisms and their corresponding categories in the environment can help further protect or manage them. However, in nature, organisms often exhibit camouflage, such as camouflaged animals and hidden plants, making identification of organisms extremely difficult. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a biometric identification method and system based on monocular depth-guided multimodal fusion to solve the problem of how to improve the recognition efficiency of disguised creatures. The specific technical solution is as follows:

[0004] In a first aspect of the embodiments of the present application, a biometric recognition method based on monocular depth-guided multimodal fusion is provided, the method comprising:

[0005] Acquire an image of a to-be-identified organism, wherein the to-be-identified organism is an animal or a plant;

[0006] Performing depth estimation on the image of the to-be-identified organism using a monocular depth model to obtain a pseudo depth map, wherein the pseudo depth map includes image features and depth features;

[0007] Extracting features of multiple resolutions from the pseudo-depth map and the image of the organism to be identified to obtain multi-level features, wherein different levels of features in the multi-level features correspond to different resolutions;

[0008] For each level of features, the sampling position is predicted through the convolution layer, and sampling and fusion are performed according to the predicted sampling positions to obtain multi-scale features; direction-specific features are extracted and enhanced through the image features and the depth features to obtain enhanced features;

[0009] Perform upsampling and boundary feature enhancement according to the multi-scale features and the enhanced features to obtain decoding features;

[0010] Pixel-level prediction is performed based on the decoded features to obtain the category of the organism to be identified.

[0011] In a possible implementation, for each level of features, the sampling position is predicted through a convolutional layer, and sampling and fusing are performed according to the predicted sampling positions to obtain multi-scale features, including:

[0012] For each level of features, a dynamic sampling offset is predicted through a convolutional layer; a sampling position is adjusted according to the dynamic sampling offset, and sampling is performed according to the adjusted sampling position to obtain multiple sampling features;

[0013] For each sampled feature, global average pooling and multi-layer perception are performed to obtain the corresponding attention weight;

[0014] Modify the plurality of sampling features and the corresponding attention weights to obtain a plurality of modified sampling features;

[0015] The multiple modified sampling features are fused to obtain the multi-scale feature.

[0016] In a possible implementation, extracting and enhancing direction-specific features using the image features and the depth features to obtain enhanced features includes:

[0017] Extracting direction-specific features from the image features and the depth features;

[0018] Calculate the attention weight based on the extracted direction-specific features;

[0019] According to the calculated attention weight, the image feature and the depth feature,

[0020] The enhanced features are obtained by calculation.

[0021] In a possible implementation, performing upsampling and enhancing boundary features according to the multi-scale features and the enhanced features to obtain decoding features includes:

[0022] Upsampling the multi-scale features and the enhanced features by a first-stage decoder in the plurality of decoders to obtain an initial prediction result; calculating a first gating feature based on the initial prediction result; and calculating an output result of the first-stage decoder based on the initial prediction result and the first gating feature;

[0023] The remaining decoders at each level in the multiple decoders are up-sampled according to the output result of the previous level to obtain the prediction result of the level, wherein the remaining decoders at each level in the multiple decoders are the remaining decoders at each level except the first level decoder; the gating features of the level are calculated according to the prediction result of the level; the output result of the decoder at the level is calculated according to the prediction result of the level and the gating features of the level, and the output result of the last level is used as the decoding feature.

[0024] In a possible implementation, performing pixel-level prediction based on the decoded features to obtain the category of the organism to be identified includes:

[0025] Calculate the probability that each pixel value corresponds to a preset category based on the decoded features through a convolutional layer and a classifier;

[0026] The category of the organism to be identified is calculated based on the probability that each pixel value corresponds to a preset category.

[0027] A second aspect of the embodiments of the present application provides a biometric recognition system based on monocular depth-guided multimodal fusion, the system comprising:

[0028] An image acquisition module, configured to acquire an image of a to-be-identified organism, wherein the to-be-identified organism is an animal or a plant;

[0029] a depth estimation module, configured to perform depth estimation on the image of the to-be-identified organism using a monocular depth model to obtain a pseudo depth map, wherein the pseudo depth map includes image features and depth features;

[0030] a feature extraction module, configured to extract features of various resolutions from the pseudo-depth map and the image of the organism to be identified, to obtain multi-level features, wherein features of different levels in the multi-level features correspond to different resolutions;

[0031] A position prediction module is used to predict the sampling position of each level of features through the convolution layer, and to sample and fuse according to the predicted sampling positions to obtain multi-scale features; and to extract and enhance direction-specific features through the image features and the depth features to obtain enhanced features;

[0032] A feature decoding module, configured to perform upsampling and boundary feature enhancement based on the multi-scale features and the enhanced features to obtain decoded features;

[0033] The biological classification module is used to perform pixel-level prediction based on the decoded features to obtain the category of the biological to be identified.

[0034] In one possible implementation, the position prediction module is specifically used to predict the dynamic sampling offset through the convolution layer for each level of features; adjust the sampling position according to the dynamic sampling offset, and sample according to the adjusted sampling position to obtain multiple sampling features; perform global average pooling and multi-layer perception on each sampling feature to obtain corresponding attention weights; make corrections based on the multiple sampling features and the corresponding attention weights to obtain multiple modified sampling features; and fuse the multiple modified sampling features to obtain the multi-scale features.

[0035] In one possible embodiment, the position prediction module is specifically used to extract direction-specific features of the image features and the depth features; calculate attention weights based on the extracted direction-specific features; and calculate the enhanced features based on the calculated attention weights, the image features, and the depth features.

[0036] In one possible implementation, the feature decoding module is specifically used to upsample the multi-scale features and the enhanced features through a first-level decoder among multiple decoders to obtain an initial prediction result; calculate a first gating feature based on the initial prediction result; calculate the output result of the first-level decoder based on the initial prediction result and the first gating feature; upsample the output result of the previous level through the remaining decoders at each level among the multiple decoders to obtain the prediction result of the level, wherein the remaining decoders at each level among the multiple decoders are decoders at all levels except the first-level decoder; calculate the gating feature of the level based on the prediction result of the level; calculate the output result of the decoder at the level based on the prediction result of the level and the gating feature of the level, and use the output result of the last level as the decoding feature.

[0037] In one possible embodiment, the biological classification module is specifically used to calculate the probability that each pixel value corresponds to a preset category based on the decoding features through a convolutional layer and a classifier; and calculate the category of the organism to be identified based on the probability that each pixel value corresponds to the preset category.

[0038] Another aspect of the present application provides an electronic device, including:

[0039] Memory for storing computer programs;

[0040] The processor is configured to implement any of the above-mentioned biometric recognition methods based on monocular depth-guided multimodal fusion when executing a program stored in the memory.

[0041] In another aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it implements any of the above-mentioned biometric segmentation methods based on monocular depth-guided multimodal fusion.

[0042] In another aspect of the embodiments of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned biometric recognition methods based on monocular depth-guided multimodal fusion.

[0043] Beneficial effects of the embodiments of the present application:

[0044] An embodiment of the present application provides a biometric recognition method and system based on monocular depth-guided multimodal fusion, the method comprising: acquiring an image of a to-be-recognized organism, wherein the to-be-recognized organism is an animal or a plant; performing depth estimation on the image of the to-be-recognized organism using a monocular depth model to obtain a pseudo-depth map, wherein the pseudo-depth map includes image features and depth features; extracting features of multiple resolutions from the pseudo-depth map and the image of the to-be-recognized organism to obtain multi-level features, wherein different level features in the multi-level features correspond to different resolutions; predicting the sampling position of each level feature through a convolutional layer, and performing sampling and fusion based on the predicted sampling position to obtain multi-scale features; extracting and enhancing direction-specific features through the image features and the depth features to obtain enhanced features; performing upsampling and enhancing boundary features based on the multi-scale features and the enhanced features to obtain decoding features; and performing pixel-level prediction based on the decoding features to obtain the category of the to-be-recognized organism. Through the solution of the embodiment of the present application, not only can the depth of the image be estimated and multiple levels of features be extracted, but also the sampling accuracy can be improved by predicting and sampling the sampling position to adapt to the needs of organisms of different shapes, and direction-specific features can be extracted and enhanced to achieve the extraction of more detailed features, thereby improving the accuracy and efficiency of biological category prediction through pixel-level prediction.

[0045] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0047] Figure 1 A schematic diagram of a flow chart of a biometric recognition method based on monocular depth-guided multimodal fusion provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a process for obtaining multi-scale features provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of a process of deformable convolution provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a process for obtaining enhanced features provided in an embodiment of the present application;

[0051] Figure 5 An example diagram of a biometric recognition method based on monocular depth-guided multimodal fusion provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of a biometric recognition device based on monocular depth-guided multimodal fusion provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0055] In the first aspect of the embodiment of the present application, a biometric recognition method based on monocular depth-guided multimodal fusion is first provided. Figure 1 , Figure 1 A schematic flow chart of a biometric recognition method based on monocular depth-guided multimodal fusion provided in an embodiment of the present application, the method comprising:

[0056] Step S11, acquiring an image of a creature to be identified, wherein the creature to be identified is an animal or a plant;

[0057] Step S12, performing depth estimation on the image of the organism to be identified using a monocular depth model to obtain a pseudo depth map, wherein the pseudo depth map includes image features and depth features;

[0058] Step S13: extracting features of various resolutions from the pseudo-depth map and the image of the organism to be identified to obtain multi-level features, wherein different levels of features in the multi-level features correspond to different resolutions;

[0059] Step S14: For each level of features, the sampling position is predicted through the convolution layer, and sampling and fusion are performed according to the predicted sampling position to obtain multi-scale features; direction-specific features are extracted and enhanced through image features and depth features to obtain enhanced features;

[0060] Step S15: upsampling and enhancing boundary features are performed based on the multi-scale features and the enhanced features to obtain decoding features;

[0061] Step S16: Perform pixel-level prediction based on the decoded features to obtain the category of the organism to be identified.

[0062] Corresponding to step S11 above, the organism to be identified in the embodiment of the present application can be either an animal or a plant. Furthermore, to meet practical needs, the solution of the present application can be applied to the identification of disguised organisms. In this case, the organism to be identified can be disguised as a plant or other object. In actual use, when acquiring an image of the organism to be identified, an image of a specified format and rules can be acquired, such as pre-setting a corresponding resolution. In one example, to acquire an image of the organism to be identified, a single RGB (red, green, and blue) image can be acquired. The format of the image is: 3 channels, with a resolution of H (height) × W (width) × 3.

[0063] Corresponding to step S12 above, when a monocular depth model is used to estimate the depth of an image of a creature to be identified, the monocular depth model can predict the distance from each pixel in the scene to the camera based on a single RGB image, and obtain and output a pseudo depth map. The pseudo depth map includes image features and depth features. In one example, when a monocular depth model is used to estimate the depth of an image of a creature to be identified, a pre-trained MDE (monocular depth estimation model) can be called to generate a pseudo depth map (single channel, H×W×1). The pseudo depth map can include a multimodal input tensor, specifically, image features corresponding to red, green, and blue + Depth (depth features corresponding to spatial distance) bimodal data.

[0064] Corresponding to step S13 above, when extracting features at multiple resolutions from the pseudo-depth map and the image of the target object, PVTv2 (Pyramid Vision Transformer v2, a pyramid structure model) can be sampled to extract features at multiple resolutions. The extracted features at multiple resolutions can include both detail features and semantic features. Within the multi-level feature hierarchy, different levels of features correspond to different resolutions. In one example, four levels of features {F1, F2, F3, F4} can be extracted, where F1 is a high-resolution detail feature (in one example, the size of F1 is H / 2 × W / 2 × C1, where C1 = 64), preserving low-level information such as edges and textures; and F4 is a low-resolution semantic feature (in one example, the size of F4 is H / 16 × W / 16 × C4, where C4 = 512), capturing the overall contours of the target.

[0065] Corresponding to step S14 above, the sampling position is predicted through the convolution layer, and sampling and fusion are performed based on the predicted sampling position to obtain multi-scale features. The dynamic sampling offset can be predicted through the convolution layer, and the sampling position is adjusted based on the predicted dynamic sampling offset. Sampling is then performed at the adjusted sampling position, and finally the sampling results of each layer are fused to obtain multi-scale features. In actual use, because the body shapes of different organisms can vary greatly, such as being both slender and elliptical, the sampling position can be adjusted to accommodate the needs of different organisms, thereby improving the accuracy and efficiency of the sampling features. Direction-specific features are extracted and enhanced through image features and depth features. Direction-specific features can be extracted by convolving the image features and depth features. Finally, attention weights are predicted based on the extracted direction-specific features, and feature enhancement is performed based on the predicted weights to obtain enhanced features. In actual use, these direction-specific features represent the direction of animal fur texture, the direction of plant leaf depth gradient, etc. Therefore, by extracting these direction-specific features, the efficiency of organism recognition can be improved.

[0066] Corresponding to the above step S15, upsampling and boundary feature enhancement are performed based on the multi-scale features and enhanced features. Upsampling can be performed through multi-level cascade decoding, and during the upsampling process, the GGA (Gate Guided Attention) mechanism is introduced to enhance the attention to the target boundaries and details, thereby obtaining the decoding features.

[0067] Corresponding to the above step S16, pixel-level prediction is performed based on the decoded features, and each pixel in the image can be refinedly modeled and inferred to generate a probability map, in which each pixel value represents the probability of belonging to the target category, so that the category of the organism to be identified can be obtained through the probability map.

[0068] It can be seen that through the solution of the embodiment of the present application, not only can the depth of the image be estimated and multiple levels of features be extracted, but also the sampling accuracy can be improved by predicting and sampling the sampling position to adapt to the needs of organisms of different shapes, and direction-specific features can be extracted and enhanced to achieve the extraction of more detailed features, thereby improving the accuracy and efficiency of biological category prediction through pixel-level prediction.

[0069] In one possible implementation, see Figure 2 , for each level of features, the sampling position is predicted through the convolution layer, and sampling and fusion are performed according to the predicted sampling position to obtain multi-scale features, including:

[0070] Step S21: For each level of features, a dynamic sampling offset is predicted through a convolutional layer; the sampling position is adjusted according to the dynamic sampling offset, and sampling is performed according to the adjusted sampling position to obtain multiple sampling features;

[0071] Step S22: Perform global average pooling and multi-layer perception on each sampled feature to obtain the corresponding attention weight;

[0072] Step S23, performing corrections based on the multiple sampling features and the corresponding attention weights to obtain multiple modified sampling features;

[0073] Step S24: fuse the multiple modified sampling features to obtain multi-scale features.

[0074] In this application, for each level of features, the dynamic sampling offset is predicted through the convolution layer; the sampling position is adjusted according to the dynamic sampling offset, and sampling is performed according to the adjusted sampling position, which can be achieved through deformable convolution dynamic sampling. Specifically, each layer of feature F i , the dynamic sampling offset Δp(x) is predicted by the convolution layer, and then the convolution kernel sampling position is adjusted according to the offset to adapt to the target shape (such as bent limbs, overlapping leaves): , where p(k) is the standard sampling point; Δp(x) is the dynamic offset, which is predicted by Fi through 1×1 convolution; x is the input feature map F i The pixel value of is the original feature tensor to be sampled; k is the index of the sampling point in the convolution kernel, corresponding to the predefined position of the standard convolution kernel; is the weight parameter of the convolution kernel; y(x) is the sampling result. For each sampled feature, global average pooling and multi-layer perception are performed to obtain the corresponding attention weight. Modification based on multiple sampled features and corresponding attention weights can be achieved through cross-level channel attention. Specifically, the channel attention weight α can be generated through GAP (global average pooling) and MLP (multi-layer perception) c , suppress the background channel and enhance the target features: α c =MLP(GAP(F i )),Fi′ = Fi × α c The fusion of multiple modified sampling features to obtain multi-scale features can be achieved through multi-scale feature fusion. Specifically, {F1′, F2′, F3′, F4′} can be upsampled to F1 resolution (H / 2×W / 2), and multi-scale features F can be generated through bottom-up fusion. cup For example, the convolution kernel in this application can be found in Figure 3, including the input feature map; sampling region 1 through the deformable convolution kernel (where the deformable convolution kernel can adaptively adjust the size of the convolution kernel); the output of the deformable convolution kernel sampling region 1; and the deformable convolution kernel sampling region 2 (where the deformable convolution kernel can also adaptively adjust the size of the convolution kernel); the output of the deformable convolution kernel sampling region 2; and finally, the deformable convolution kernel output. In traditional solutions, the convolution kernel sampling is generally fixed.

[0075] In one possible implementation, see Figure 4 , through image features and depth features, direction-specific features are extracted and enhanced to obtain enhanced features, including:

[0076] Step S41, extracting direction-specific features from image features and depth features;

[0077] Step S42, calculating the attention weight based on the extracted direction-specific features;

[0078] Step S43: Calculate the enhanced features based on the calculated attention weights, image features, and depth features.

[0079] Extraction of direction-specific features from image features and depth features can be achieved through direction-sensitive grouped convolution. Specifically, 8 groups of 3×3 convolutions can be performed on RGB and depth features respectively to extract direction-specific features (such as the direction of animal fur texture and the direction of plant leaf depth gradient). The output direction-specific features include: F rgb-dir , F d-dir , where F rgb-dir is the directional feature of animal fur texture, F d-dir is the directional feature of the plant leaf depth gradient. The attention weight is calculated based on the extracted directional specific features, which can be achieved through the channel attention mask. Specifically, the attention weight β can be generated through the compression-expansion structure to amplify the target channel feature: β = Sigmoid(MLP( AvgPool(F rgb-dir ⊕F d-dir) ));F fused = β ⊙ (F rgb-dir + F d-dir ); AvgPool represents average pooling; Sigmoid represents activation function; MLP represents multi-layer perception. Based on the calculated attention weights, image features, and depth features, feature enhancement is performed to calculate the enhanced features, which can be achieved through detail hallucination recovery. Specifically, for the occluded area, a 3×3 convolution can be used to extract high-frequency edge features F hf , combined with the reverse mask Recovery details: F enhanced= F fused +(M inv ⊙ F hf ) , where F fused Indicates the fused features; F enhanced Indicates enhanced features; F hf Represents high-frequency edge features; M inv Indicates the reverse mask.

[0080] In one possible implementation, upsampling and boundary feature enhancement are performed based on multi-scale features and enhanced features to obtain decoding features, including: upsampling the multi-scale features and enhanced features through a first-level decoder among multiple decoders to obtain an initial prediction result; calculating a first gating feature based on the initial prediction result; calculating an output result of the first-level decoder based on the initial prediction result and the first gating feature; upsampling based on the output result of the previous level through the remaining decoders at each level among the multiple decoders to obtain a prediction result of the level, wherein the remaining decoders at each level among the multiple decoders are decoders at all levels except the first-level decoder; calculating the gating feature of the level based on the prediction result of the level; calculating the output result of the decoder at the level based on the prediction result of the level and the gating feature of the level, and using the output result of the last level as the decoding feature.

[0081] Specifically, multi-level feature fusion can be achieved, such as by upsampling the decoder at each level (bilinear interpolation, factor = 2) through a 4-level cascade decoder. Each decoder level fuses the features from the encoder with the features of the previous level decoder. In addition, a GGA (Gate Guided Attention) module is introduced in each level of decoding to achieve the gated feature F. gate Adaptively adjust feature responses to enhance attention to object boundaries and details. For example, in the first decoding stage, the GGA module generates a gating signal based on the initial prediction results to optimize the representation of low-level features. In subsequent cascades, the GGA module combines upsampled intermediate prediction results to iteratively refine the feature representation. Each decoder stage can be used to preserve original feature information through residual blocks and combine them with a gated attention mechanism to refine object boundaries and suppress background noise.

[0082] In one possible implementation, pixel-level prediction based on decoded features to determine the category of the organism to be identified includes: calculating the probability of each pixel value corresponding to a preset category based on the decoded features through a convolutional layer and a classifier; and calculating the category of the organism to be identified based on the probability of each pixel value corresponding to the preset category. Specifically, the output of the convolutional layer and the classifier ultimately generates a probability map P∈[0,1]H×W, where each pixel value represents the probability of belonging to the target category. The category of the organism to be identified is then determined using this probability map. Specifically, this can be determined using a maximum probability method, binary classification, or other methods.

[0083] For an example, see Figure 5 After obtaining the RGB image, a pseudo depth map is generated through monocular depth estimation, and then features are extracted through the backbone network. The backbone network includes a pyramid structure composed of F1, F2, F3, and F4. Multi-scale integration is performed through the CUP module to obtain image features and depth features. Then, the fusion and detail illusion mechanism is used to process the fused features X1, X2, X3, and X4. For X1, X2-feed is obtained by decoding, and then X2-gate is obtained by upsampling; X2 and X2-gate are fused to obtain X3-feed; X3-feed is upsampling to obtain X3-gate; X3 and X3-gate are fused to obtain X4-feed; X4-feed is upsampling to obtain X4-gate; X4 and X4-gate are fused and a preliminary coarse prediction is performed, followed by convolution, bilinear interpolation, and further fine prediction to generate a pixel-level camouflage prediction map. The input data in the embodiment of the present application is RGB+Depth (image features corresponding to red, green and blue + depth features corresponding to spatial distance). This input data can fuse two-dimensional vision and three-dimensional geometric information, replacing traditional hardware input and generating depth through monocular estimation, thus overcoming the dependence on dedicated equipment and reducing costs. The backbone network can be { The backbone network can realize multi-resolution feature extraction, and the backbone network can be a hierarchical Transformer architecture, which is better than the traditional CNN (Convolutional Neural Networks) for detecting small target features. The output of the CUP (Collaborative Unified Pyramid) module can be a multi-scale fusion feature F cup , can dynamically adapt to the target shape (deformable convolution + channel attention), and the number of deformable sampling points is better than that of traditional fixed convolution. The output of the ODEM (Occlusion-aware Detail Enhancement Fusion Module) module can be a cross-modal enhancement feature F enhanced , which can separate RGB texture and Depth gradient features and restore occluded details. The direction-sensitive group convolution (8 groups) is more sensitive to direction-specific information than the traditional single-group convolution, and is more suitable for use in scenarios such as animal camouflage and plant concealment. The output result can be a segmentation map P (single channel), which can achieve pixel-level target probability prediction. Sigmoid activation is used instead of threshold segmentation, which supports subsequent adaptive threshold adjustment and improves segmentation accuracy. The data flow and hierarchical relationship can be: input stage: RGB→monocular depth estimation→pseudo depth map; feature extraction: PVTv2 generates 4-level pyramid features ( , the resolution is halved step by step, and the semantic abstraction increases); Core processing: The CUP and ODEM modules sequentially process multi-level features, and transmit the multi-scale fusion features output by CUP to the ODEM module for further cross-modal feature enhancement; Decoding output: The cascade decoder gradually fuses the multi-scale features output by the CUP and ODEM modules, and dynamically filters key information through bilinear upsampling (factor = 2) and cross-level feature aggregation, combined with the GGA gated attention module. After optimizing the boundary details through the residual block, a pixel-level probabilistic segmentation map is generated.

[0084] The solution of this application not only achieves a breakthrough in 3D information fusion: The technical solution generates a pseudo-depth map using a monocular depth estimation model (such as DepthAnything V2). This innovative data structure enables the system to acquire 3D geometric cues in the scene. Compared to traditional methods relying solely on RGB, the structural metric of the present invention improved from 0.856 to 0.880, and the mean absolute error decreased from 0.065 to 0.039 on an animal dataset. This principle is based on the fact that depth information can help distinguish the spatial distance between the target and the background. In particular, for camouflaged animals with blurred outlines, segmentation accuracy is significantly improved by locating edges using depth gradients. Furthermore, cross-modal feature enhancement is achieved: The ODEM module achieves efficient cross-modal feature fusion through a three-stage architecture of directional decoupling, channel enhancement, and dynamic compensation. The constructed hallucination generator extracts high-frequency details and dynamically injects them into low-confidence regions using confidence masks, reducing missed detection rates in complex scenes such as camouflage. This module achieves breakthroughs in both cross-modal spatial alignment and occluded detail recovery with a minimal increase in computational complexity. Dynamic morphological adaptation is also possible: The CUP module utilizes deformable convolution to dynamically adjust sampling offsets, combined with cross-level channel attention, to adaptively process morphological differences between species. This improves contour integrity in scenes with overlapping plant leaves. This is achieved by dynamically expanding the receptive field based on the target morphology, while the channel attention mechanism enhances target feature channels and deemphasizes background noise channels.

[0085] In a specific example, the training hardware includes: GPU acceleration: Supports NVIDIA RTX 3090 / 4090 graphics cards, leveraging CUDA (Compute Unified Device Architecture) parallel computing to accelerate monocular depth estimation and feature extraction. Distributed training: Supports simultaneous training on 8 graphics cards, with a batch size of 32. The number of training rounds for plant cover detection, plant camouflage detection, and animal camouflage detection is 30, 120, and 100, respectively, adjusted based on dataset size, modality complexity, and task characteristics. Graphics memory usage is ≤ 20GB.

[0086] Research has found that traditional methods, often designed for a single species (such as animals or plants), suffer from missed detection rates exceeding 25% in mixed scenarios. This new method, leveraging the dynamic morphological adaptation of the CUP module and the cross-modal feature fusion of the ODEM module, achieves unified detection of both animal and plant camouflage for the first time. This significantly improves detection integrity in mixed test sets containing both animals and plants. The core principle behind this approach is the ability of deformable and direction-sensitive convolutions to adapt to the morphological and texture differences between species. Multimodal data fusion provides a richer information dimension for cross-species feature modeling, enabling the model to achieve stronger generalization capabilities.

[0087] The second aspect of the embodiment of the present application provides a biometric recognition system based on monocular depth-guided multimodal fusion, see Figure 6 , the system comprises:

[0088] An image acquisition module 601 is used to acquire an image of a to-be-identified organism, wherein the to-be-identified organism is an animal or a plant;

[0089] A depth estimation module 602 is configured to perform depth estimation on an image of a to-be-identified organism using a monocular depth model to obtain a pseudo depth map, wherein the pseudo depth map includes image features and depth features;

[0090] The feature extraction module 603 is used to extract features of multiple resolutions from the pseudo-depth map and the image of the organism to be identified to obtain multi-level features, wherein different levels of features in the multi-level features correspond to different resolutions;

[0091] The position prediction module 604 is used to predict the sampling position of each level of features through the convolution layer, and to sample and fuse according to the predicted sampling positions to obtain multi-scale features; and to extract and enhance direction-specific features through image features and depth features to obtain enhanced features;

[0092] A feature decoding module 605 is used to perform upsampling and boundary feature enhancement based on the multi-scale features and the enhanced features to obtain decoded features;

[0093] The biological classification module 606 is used to perform pixel-level prediction based on the decoded features to obtain the category of the biological being to be identified.

[0094] In one possible implementation, the position prediction module is specifically used to predict the dynamic sampling offset through the convolution layer for each level of features; adjust the sampling position according to the dynamic sampling offset, and sample according to the adjusted sampling position to obtain multiple sampling features; perform global average pooling and multi-layer perception on each sampling feature to obtain corresponding attention weights; make corrections based on the multiple sampling features and the corresponding attention weights to obtain multiple modified sampling features; and fuse the multiple modified sampling features to obtain multi-scale features.

[0095] In one possible implementation, the position prediction module is specifically used to extract direction-specific features from image features and depth features; calculate attention weights based on the extracted direction-specific features; and calculate enhanced features based on the calculated attention weights, image features, and depth features.

[0096] In one possible implementation, the feature decoding module is specifically used to upsample the multi-scale features and enhanced features through a first-level decoder among multiple decoders to obtain an initial prediction result; calculate a first gating feature based on the initial prediction result; calculate the output result of the first-level decoder based on the initial prediction result and the first gating feature; upsample the output result of the previous level through the remaining decoders at each level among the multiple decoders to obtain the prediction result of the level, wherein the remaining decoders at each level among the multiple decoders are the remaining decoders at each level except the first-level decoder; calculate the gating feature of the level based on the prediction result of the level; calculate the output result of the decoder at the level based on the prediction result of the level and the gating feature of the level, and use the output result of the last level as the decoding feature.

[0097] In one possible implementation, the biological classification module is specifically configured to calculate the probability that each pixel value corresponds to a preset category based on the decoded features through a convolutional layer and a classifier; and calculate the category of the organism to be identified based on the probability that each pixel value corresponds to the preset category.

[0098] It can be seen that through the solution of the embodiment of the present application, not only can the depth of the image be estimated and multiple levels of features be extracted, but also the sampling accuracy can be improved by predicting and sampling the sampling position to adapt to the needs of organisms of different shapes, and direction-specific features can be extracted and enhanced to achieve the extraction of more detailed features, thereby improving the accuracy and efficiency of biological category prediction through pixel-level prediction.

[0099] The present application also provides an electronic device, such as Figure 7 Shown, including:

[0100] Memory 701, used for storing computer programs;

[0101] The processor 702 is configured to execute the program stored in the memory 701 and implement the following steps:

[0102] Acquire an image of a to-be-identified organism, wherein the to-be-identified organism is an animal or a plant;

[0103] Depth estimation is performed on the image of the organism to be identified using a monocular depth model to obtain a pseudo depth map, wherein the pseudo depth map includes image features and depth features;

[0104] Extracting features of various resolutions from the pseudo-depth map and the image of the organism to be identified to obtain multi-level features, wherein different levels of features in the multi-level features correspond to different resolutions;

[0105] For each level of features, the sampling position is predicted through the convolution layer, and sampling and fusion are performed according to the predicted sampling position to obtain multi-scale features; direction-specific features are extracted and enhanced through image features and depth features to obtain enhanced features;

[0106] According to the multi-scale features and enhanced features, upsampling and boundary feature enhancement are performed to obtain decoding features;

[0107] Pixel-level prediction is performed based on the decoded features to obtain the category of the organism to be identified.

[0108] The communication bus mentioned in the electronic devices mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into address buses, data buses, control buses, etc. For ease of illustration, only a single thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0109] The communication interface is used for communication between the above electronic device and other devices.

[0110] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0111] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0112] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned biometric recognition methods based on monocular depth-guided multimodal fusion.

[0113] In another embodiment provided by the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the biometric recognition methods based on monocular depth-guided multimodal fusion in the above embodiments.

[0114] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or solid-state drive (SSD).

[0115] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0116] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system, electronic device, and storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, refer to the descriptions of the method embodiments.

[0117] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A biometric recognition method based on monocular depth-guided multimodal fusion, characterized in that: The method comprises: Acquire an image of a to-be-identified organism, wherein the to-be-identified organism is an animal or a plant; Performing depth estimation on the image of the to-be-identified organism using a monocular depth model to obtain a pseudo depth map, wherein the pseudo depth map includes image features and depth features; Extracting features of multiple resolutions from the pseudo-depth map and the image of the organism to be identified to obtain multi-level features, wherein different levels of features in the multi-level features correspond to different resolutions; For each level of features, the sampling position is predicted through the convolution layer, and sampling and fusion are performed according to the predicted sampling positions to obtain multi-scale features; direction-specific features are extracted and enhanced through the image features and the depth features to obtain enhanced features; Perform upsampling and boundary feature enhancement according to the multi-scale features and the enhanced features to obtain decoding features; Performing pixel-level prediction based on the decoded features to obtain the category of the organism to be identified; The method predicts the sampling position of each level feature through a convolution layer, and performs sampling and fusing according to the predicted sampling position to obtain multi-scale features, including: predicting a dynamic sampling offset for each level feature through a convolution layer; adjusting the sampling position according to the dynamic sampling offset, and sampling according to the adjusted sampling position to obtain multiple sampling features; performing global average pooling and multi-layer perception on each sampling feature to obtain a corresponding attention weight; performing correction according to the multiple sampling features and the corresponding attention weights to obtain multiple modified sampling features; and fusing the multiple modified sampling features to obtain the multi-scale features. The upsampling and boundary feature enhancement are performed according to the multi-scale features and the enhanced features to obtain decoding features, including: upsampling the multi-scale features and the enhanced features through a first-level decoder among multiple decoders to obtain an initial prediction result; calculating a first gating feature based on the initial prediction result; calculating an output result of the first-level decoder based on the initial prediction result and the first gating feature; upsampling according to the output result of the previous level through the remaining decoders at each level among the multiple decoders to obtain a prediction result of the level, wherein the remaining decoders at each level among the multiple decoders are decoders at each level except the first-level decoder; calculating the gating feature of the level based on the prediction result of the level; calculating the output result of the decoder at the level based on the prediction result of the level and the gating feature of the level, and using the output result of the last level as the decoding feature.

2. The method according to claim 1, characterized in that The step of extracting and enhancing direction-specific features through the image features and the depth features to obtain enhanced features includes: Extracting direction-specific features from the image features and the depth features; Calculate the attention weight based on the extracted direction-specific features; According to the calculated attention weight, the image feature and the depth feature, The enhanced features are obtained by calculation.

3. The method according to claim 1, characterized in that The performing pixel-level prediction based on the decoded features to obtain the category of the organism to be identified includes: Calculate the probability that each pixel value corresponds to a preset category based on the decoded features through a convolutional layer and a classifier; The category of the organism to be identified is calculated based on the probability that each pixel value corresponds to a preset category.

4. A biometric recognition system based on monocular depth-guided multimodal fusion, characterized in that: Applying the method according to any one of claims 1 to 3, the system comprises: An image acquisition module, configured to acquire an image of a to-be-identified organism, wherein the to-be-identified organism is an animal or a plant; a depth estimation module, configured to perform depth estimation on the image of the to-be-identified organism using a monocular depth model to obtain a pseudo depth map, wherein the pseudo depth map includes image features and depth features; a feature extraction module, configured to extract features of various resolutions from the pseudo-depth map and the image of the organism to be identified, to obtain multi-level features, wherein features of different levels in the multi-level features correspond to different resolutions; A position prediction module is used to predict the sampling position of each level of features through the convolution layer, and to sample and fuse according to the predicted sampling positions to obtain multi-scale features; and to extract and enhance direction-specific features through the image features and the depth features to obtain enhanced features; A feature decoding module, configured to perform upsampling and boundary feature enhancement based on the multi-scale features and the enhanced features to obtain decoded features; The biological classification module is used to perform pixel-level prediction based on the decoded features to obtain the category of the biological to be identified.

5. The system according to claim 4, characterized in that The position prediction module is specifically used to extract direction-specific features from the image features and the depth features; calculate attention weights based on the extracted direction-specific features; and calculate the enhanced features based on the calculated attention weights, the image features, and the depth features.

6. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 3 when executing a program stored in a memory.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images