An edge wild animal recognition method based on cross-modal alignment constraint
By employing a cross-modal alignment-constrained edge recognition method, combined with lightweight processing and feature fusion of image and voiceprint data, the accuracy and real-time performance issues of wildlife recognition on low-power edge devices are addressed, achieving efficient and low-power recognition results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAN QUELINGFEI INFORMATION TECH CO LTD
- Filing Date
- 2026-03-16
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies cannot achieve efficient, real-time multimodal wildlife recognition on low-power edge devices, and single-modal models have insufficient recognition accuracy in complex environments, lack adaptive capabilities, have high deployment costs, and cloud dependence leads to increased latency and costs.
An edge recognition method with cross-modal alignment constraints is adopted. Image and voiceprint data are collected by edge devices, and lightweight preprocessing and dual-modal joint low-rank decomposition are performed. Alignment weights are calculated using a cross-attention mechanism. Combined with lightweight feature extraction and a classifier, cross-modal feature fusion is achieved, and recognition is completed on the edge device.
It effectively improves the recognition accuracy in complex environments, reduces power consumption, extends device battery life, reduces reliance on the cloud and deployment costs, and enables real-time, low-power wildlife recognition.
Smart Images
Figure CN122173983A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to a method for identifying edge wildlife based on cross-modal alignment constraints. Background Technology
[0002] Real-time identification of wild animals in the wild is a core requirement for ecological monitoring, but existing technologies suffer from three contradictions: multimodal large-scale models (visual + acoustic) are robust, but their computing power / energy consumption is too high (usually ≥10W), making them unsuitable for deployment on low-power edge devices such as tree stumps and microcontrollers (e.g., ESP32); traditional edge recognition models are mostly unimodal, which are affected by environmental interference (visual failure at night, sound distortion in noisy environments), resulting in recognition accuracy of less than 60% in complex scenes; edge models lack adaptability after deployment, have poor generalization ability when facing new species / new environmental samples, and require re-burning of the model, which is costly and time-consuming.
[0003] One existing technology is a "cloud-driven multimodal wildlife recognition system". The core solution is that edge devices collect visual / acoustic data and upload it to the cloud, where a large cloud model completes the recognition and sends back the results.
[0004] This technology originates from publicly available literature on ecological monitoring. Its core limitations are its reliance on the cloud and inability to operate offline. It relies on network communication: it completely fails in remote, network-free environments; it has high latency: data transmission + cloud inference latency is ≥5 seconds, making real-time response impossible; and it has high deployment costs: cloud computing power and communication expenses increase monitoring costs.
[0005] The second existing technology is an "edge recognition device for single-modal model distillation". The core solution is to distill the output probability of a large single-modal visual model into a lightweight model and deploy it on an edge device to complete the recognition.
[0006] This technology originates from publicly available literature on edge machine learning, and its core limitations are its single-modality nature and lack of cross-modal adaptation. 1. Weak anti-interference capability in single-modality scenarios: recognition fails at night or in scenes with vegetation obstruction. 2. Lack of cross-modal information fusion: unable to utilize complementary information such as voiceprints to improve robustness. 3. Poor generalization ability: recognition accuracy decreases by ≥30% in new environmental samples. Summary of the Invention
[0007] To address the aforementioned shortcomings in existing technologies, this invention provides an edge wildlife identification method based on cross-modal alignment constraints, which solves the problem of weak anti-interference capabilities in single-modal edge models.
[0008] To achieve the aforementioned objectives, the present invention employs the following technical solution: a method for identifying edge wildlife based on cross-modal alignment constraints, comprising: S1: Use edge devices to collect image and voiceprint data of wild animals in the target area; S2: Lightweight preprocessing is performed on image data and voiceprint data to obtain standardized bimodal features; S3: Perform bimodal joint low-rank decomposition on the standardized bimodal features, and calculate the alignment weights based on the low-rank features obtained from the decomposition using a cross-attention mechanism to obtain cross-modal aligned fusion features; S4: Input the cross-modal aligned and fused features into the lightweight feature extraction network to extract features and obtain lightweight feature vectors; S5: Use a lightweight classifier to classify and identify lightweight feature vectors to obtain identification results for marginal wildlife, thus completing the identification of marginal wildlife.
[0009] The beneficial effects of this invention are as follows: This invention provides an edge wildlife recognition method based on cross-modal alignment constraints. By collecting image and voiceprint data, and utilizing a cross-attention mechanism to calculate alignment weights, it achieves deep fusion of cross-modal features. This effectively solves the limitations of single modality in complex environments (such as unclear nighttime images, noisy environments, and voiceprint interference), and significantly improves recognition accuracy by utilizing the complementarity between modalities. Before feature extraction, redundant information in the features is removed through dual-modal joint low-rank decomposition, and combined with lightweight preprocessing and a lightweight feature extraction network, the number of model parameters and computational complexity are significantly reduced. This allows the complex dual-modal recognition algorithm to be deployed on edge devices with limited computing power, effectively reducing power consumption and extending the battery life of field devices. By employing dual-modal joint low-rank decomposition, the main components (low-rank features) in image and voiceprint data can be extracted mathematically. While removing noise and non-critical information, the semantic association between modalities is strengthened through a cross-attention mechanism, thereby ensuring the integrity of recognition features while reducing data dimensionality. The entire process, from data collection and processing to classification and recognition, is integrated at the edge device, eliminating the need for cloud computing power or real-time uploading of large amounts of raw data. This not only reduces reliance on network bandwidth and cloud servers, lowering deployment costs, but also avoids recognition delays or interruptions caused by unstable networks in the field.
[0010] Further, S1 includes: By utilizing edge devices, we can obtain information such as ambient light intensity, noise intensity, and the remaining power of the edge devices. The sampling frequency is calculated based on the light intensity, noise intensity, remaining battery power, and the preset target species activity coefficient; the expression for the sampling frequency is: ; in, This indicates the dynamic acquisition frequency of wildlife image data and voiceprint data from the edge device in the target area. Indicates the basic sampling frequency. Indicates light intensity. Indicates noise intensity. Indicates the remaining battery power. Indicates the activity coefficient; Based on the calculated acquisition frequency, image data and voiceprint data of wild animals in the target area are collected synchronously.
[0011] Further, S2 includes: The image data is traversed, and the local mean and local variance of each pixel are calculated. Adaptive noise reduction is performed on the image data based on local mean, local variance, and noise variance. The denoised image data is shifted, scaled, and normalized to obtain standardized image features. Wavelet decomposition is performed on the voiceprint data to obtain approximate components and detail components; An adaptive threshold is obtained by calculating the length and median of the detail components of the voiceprint data. The detailed components are denoised using an adaptive threshold and then reconstructed by combining the approximate components to obtain the reconstructed speaker signal. The reconstructed voiceprint signal is normalized to obtain standardized voiceprint features; By combining standardized image features and standardized voiceprint features, standardized bimodal features are obtained.
[0012] Furthermore, the expression for the standardized bimodal feature is: ; ; ; ; ; ; ; ; ; ; in, Represents the local mean. Represents the original pixel values of the image. Represents local variance. This represents the pixel value after image noise reduction. Indicates the noise variance. Indicates light intensity. This represents the standardized pixel values of the image. Indicates an adaptive threshold. This represents the median calculation function. This represents the approximate components of the voiceprint. Indicates the voiceprint detail components. Represents a symbolic function. Indicates the length of the voiceprint data. This represents the median of the voiceprint detail components. This represents the reconstructed voiceprint signal value. This represents the function for calculating the minimum value. Indicates image quality score, Represents the characteristic target mean. This indicates the voiceprint signal-to-noise ratio. Indicates the intensity of ambient noise.
[0013] Further, S3 includes: The image features and voiceprint features in the standardized bimodal features are concatenated into a joint matrix; Perform singular value decomposition on the joint matrix and extract the singular vectors corresponding to the first few singular values as a shared low-rank basis; The image features and the voiceprint features are projected onto the shared low-rank basis to obtain the image low-rank features and the voiceprint low-rank features, respectively. A diagonal matrix is constructed based on the quality scores of image data and voiceprint data; The cross-attention weights are calculated using the low-rank features of the image, the low-rank features of the voiceprint, and the diagonal matrix. The cross-attention weights are used to interactively update the low-rank features of the image and the low-rank features of the voiceprint, and then weighted and fused based on the quality score to obtain cross-modal aligned fused features.
[0014] Furthermore, the expression for the cross-modal alignment fusion feature is: ; ; ; ; ; in, Denotes the truncated singular value decomposition basis of the joint matrix. Represents the feature projection error matrix. Let M represent the first r columns of singular vectors. The low-rank projected reconstruction value represents the feature. Represents the original feature vector of the bimodal mode. This represents the function for calculating cross-attention weights. Represents the image modality weight matrix. This represents the voiceprint modal weight matrix. Represents the normalized exponential function, This indicates the characteristic transpose operator. This represents the cross-modal attention weight matrix. This represents the low-rank features of the voiceprint after global alignment. This represents the low-rank features of the globally aligned image. The standardized normalized value represents the bimodal feature. Indicates image quality score, This indicates the voiceprint quality score.
[0015] Further, S4 includes: The cross-modal aligned and fused features are input into the adaptive lightweight channel segmentation and shuffling module. Based on the L1 norm of the channel weights, the feature channels are divided into important channel groups and secondary channel groups. Convolution enhancement is performed only on the important channel groups, and then channel shuffling and recombination are performed with the secondary channel groups to obtain the recombined features. The recombined features are input into the lightweight coordinate attention module, which generates attention weights along the feature length dimension and weights the features to obtain a lightweight feature vector. The adaptive lightweight channel segmentation and shuffling module and the lightweight coordinate attention module belong to the lightweight feature extraction network.
[0016] Further, S5 includes: Calculate the cosine similarity between the lightweight feature vector and the preset species prototype feature library, and convert it into an initial species probability distribution; Based on a lightweight classifier, a bimodal quality score is calculated according to image sharpness and voiceprint signal-to-noise ratio. The initial species probability distribution is then weighted and corrected using the bimodal quality score to obtain the identification results of marginal wildlife, thus completing the identification of marginal wildlife.
[0017] Furthermore, the expression for the power scheduling instruction is: ; ; ; ; ; ; in, This represents a placeholder for function input parameters. This represents the function for calculating standard deviation. This represents the weight calculation function. This represents the mean calculation function. This represents the function for calculating the quantization bit width. Indicates the total number of sampling points. Represents the diagonal mass matrix. This represents a numerical clipping function. Represents the feature channel weight vector. This represents the function for calculating the minimum weight. Represents the standardized scaling factor. This represents the feature zero-padding mapping function. This represents the weight parameters of the fully connected layer. Indicates the dimension index parameter. This indicates the percentage of remaining battery power for edge devices. This represents the function for calculating the confidence level. This indicates the real-time power consumption value of the edge device. This indicates the maximum rated power consumption of the edge device.
[0018] Furthermore, the expression for the loss function is: ; ; ; ; ; ; ; in, This represents the total loss function of the model. Represents the classification loss weight coefficient. This represents the alignment loss weighting coefficient. This represents the sparse loss weighting coefficient. This represents the power loss weighting coefficient. Represents classification loss. Indicates cross-modal alignment loss. This represents the feature sparsity loss. Indicates power consumption constraint loss, This represents the true label of the k-th species. This represents the probability that the model predicts for the k-th species, and its value ranges from [0,1]. This represents the total number of categories of the species to be identified. This represents the original feature vector of the image modality of the i-th input sample. This represents the original feature vector of the voiceprint modality of the i-th input sample. The squaring operation of the L2 norm of a vector is represented. This represents the dimension of the feature vector. This represents the high-frequency feature components in the d-th dimension, specifically referring to the detail components after wavelet decomposition of the speaker pattern or the high-frequency texture features of an image. The first norm operation of a vector is represented. This indicates the real-time power consumption of the edge recognition device during the current working cycle. This represents the minimum power consumption threshold required for edge recognition devices to maintain basic operation. This indicates the maximum power consumption threshold allowed by the hardware circuitry of the edge recognition device. This indicates the training rounds in the model training process. Attached Figure Description
[0019] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein: Figure 1 This is an exemplary flowchart illustrating a method for identifying edge wildlife based on cross-modal alignment constraints, according to some embodiments of this specification. Detailed Implementation
[0020] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0021] Example Figure 1 This is an exemplary flowchart illustrating an edge wildlife identification method based on cross-modal alignment constraints, according to some embodiments of this specification. Figure 1 As shown, the process includes the following steps. In some embodiments, the process may be executed by a processor.
[0022] Cross-modal alignment constraint bimodal distillation: This invention features a unique knowledge distillation method where the teacher's large model simultaneously outputs intermediate features of the visual and acoustic modalities and cross-modal relationships. Through alignment loss, it guides the student's TinyML model to learn bimodal unified perception technology.
[0023] Low-power edge micro-machine learning (TinyML) adaptation architecture: a compact model deployment framework that integrates pruning, quantization, and bimodal feature lightweighting for devices such as stumps / microcontrollers (e.g., ESP32) / single-board computers (e.g., Raspberry Pi).
[0024] Bimodal feature correlation pruning: A pruning strategy based on the correlation contribution of visual-acoustic features to evaluate redundant parameters, avoiding cross-modal information loss caused by single-modal pruning.
[0025] Edge bimodal divergence-triggered fine-tuning: a mechanism that updates parameters on edge devices by using the divergence between visual and acoustic branch recognition results as the trigger condition.
[0026] Energy consumption-dual-modal accuracy linkage verification: Iterative optimization of model lightweight parameters, while ensuring verification process of dual-modal recognition accuracy and edge device energy consumption threshold.
[0027] This invention addresses the core pain points of limited computing power, strict power consumption constraints, difficulties in aligning bimodal (image + voiceprint) features, and scarcity of wildlife identification samples in edge devices. It innovatively proposes an integrated technical solution combining "dynamic low-power acquisition + cross-modal low-rank alignment + a dedicated lightweight architecture for TinyML machine learning + end-to-end power consumption constraints." The invention is designed around a "precision-power consumption-real-time performance" triangle balance, achieving efficient bimodal wildlife identification on edge devices through the collaborative work of six functional modules. The system receives bimodal sensor data input and outputs identification results with low-power scheduling feedback, forming a closed-loop end-to-end process of "acquisition-preprocessing-alignment-feature extraction-identification-power optimization." The technical details are reproducible, do not duplicate existing patents or papers, and fully meet the core requirements of patent applications for "clarity, completeness, and support for claims."
[0028] The overall system architecture of this invention is based on the core logic of "low power consumption priority, lightweight design, and precise cross-modal alignment". It adopts a modular and layered architecture, which includes six functional modules in sequence according to the data flow direction. Each module achieves seamless connection through standardized low-power interfaces, adapting to the hardware constraints of edge microcontrollers (MCUs) such as microcontrollers (e.g., ≤512KBSRAM (Static Random Access Memory) and ≤4MB Flash memory) such as ESP32 and STM32. The specific architecture is as follows: 1. Core Architecture Logic: By "dynamically acquiring data to reduce redundant data, using lightweight algorithms to reduce computational overhead, using low-rank alignment to compress modal dimensions, using dedicated networks to adapt to edge computing power, and optimizing power consumption across the entire link," the core defects of traditional multimodal recognition systems, namely "high computing power requirements, high power consumption, and low cross-modal alignment accuracy," are addressed, enabling efficient deployment on edge devices.
[0029] 2. Module Composition and Interaction: First Layer: Dual-modal low-power acquisition layer, serving as the system data input, dynamically adjusting acquisition parameters, and outputting low-redundancy dual-modal raw data; Second Layer: Lightweight preprocessing layer, receiving data from the acquisition layer, performing data cleaning and standardization through low-computation algorithms, and outputting dual-modal features adapted for subsequent processing; Third Layer: Cross-modal low-rank alignment layer, based on low-rank decomposition and cross-attention, achieving semantic alignment of dual-modal features, and outputting a preliminary fused feature shape; Fourth Layer: TinyML dedicated lightweight feature extraction layer, employing an improved lightweight network to extract highly recognizable, low-dimensional wildlife-specific features, and outputting lightweight feature vectors; Fifth Layer: Efficient edge recognition layer, performing species recognition through a lightweight classifier, and outputting recognition results and confidence levels; Sixth Layer: End-to-End low-power optimization layer, spanning the first five modules, optimizing power consumption through dynamic quantization, computing power allocation, and other strategies, and outputting power scheduling instructions to be fed back to the acquisition layer and feature extraction layer.
[0030] 3. Data Flow and Control Flow: The data flow is unidirectional, following the path of "raw bimodal data → standardized features → aligned features → lightweight features → recognition result"; the control flow dynamically adjusts the acquisition parameters of the acquisition layer and the calculation accuracy of the feature extraction layer through scheduling instructions generated by the end-to-end low-power optimization layer, ensuring that the system maintains high recognition accuracy under low power constraints.
[0031] S1: Use edge devices to collect image and voiceprint data of wildlife in the target area.
[0032] In some embodiments, the edge device is used to obtain ambient light intensity, noise intensity, and the remaining power of the edge device; The sampling frequency is calculated based on the light intensity, noise intensity, remaining battery power, and the preset target species activity coefficient; the expression for the sampling frequency is: ; in, This indicates the dynamic acquisition frequency of wildlife image data and voiceprint data from the edge device in the target area. Indicates the basic sampling frequency. Indicates light intensity. Indicates noise intensity. Indicates the remaining battery power. Indicates the activity coefficient; Based on the calculated acquisition frequency, image data and voiceprint data of wild animals in the target area are collected synchronously.
[0033] S2: Lightweight preprocessing is performed on image data and voiceprint data to obtain standardized bimodal features.
[0034] In some embodiments, the image data is traversed to calculate the local mean and local variance of each pixel; Adaptive noise reduction is performed on the image data based on local mean, local variance, and noise variance. The denoised image data is shifted, scaled, and normalized to obtain standardized image features. Wavelet decomposition is performed on the voiceprint data to obtain approximate components and detail components; An adaptive threshold is obtained by calculating the length and median of the detail components of the voiceprint data. The detailed components are denoised using an adaptive threshold and then reconstructed by combining the approximate components to obtain the reconstructed speaker signal. The reconstructed voiceprint signal is normalized to obtain standardized voiceprint features; By combining standardized image features and standardized voiceprint features, standardized bimodal features are obtained.
[0035] In some embodiments, the expression for the standardized bimodal feature is: ; ; ; ; ; ; ; ; ; ; in, Represents the local mean. Represents the original pixel values of the image. Represents local variance. This represents the pixel value after image noise reduction. Indicates the noise variance. Indicates light intensity. This represents the standardized pixel values of the image. Indicates an adaptive threshold. This represents the median calculation function. This represents the approximate components of the voiceprint. Indicates the voiceprint detail components. Represents a symbolic function. Indicates the length of the voiceprint data. This represents the median of the voiceprint detail components. This represents the reconstructed voiceprint signal value. This represents the function for calculating the minimum value. Indicates image quality score, Represents the characteristic target mean. This indicates the voiceprint signal-to-noise ratio. Indicates the intensity of ambient noise.
[0036] S3: Perform bimodal joint low-rank decomposition on the standardized bimodal features, and calculate the alignment weights based on the low-rank features obtained by the decomposition using the cross-attention mechanism to obtain cross-modal aligned fusion features.
[0037] In some embodiments, the joint matrix is subjected to singular value decomposition, and the singular vectors corresponding to the first few singular values are extracted as a shared low-rank basis. The image features and the voiceprint features are projected onto the shared low-rank basis to obtain the image low-rank features and the voiceprint low-rank features, respectively. A diagonal matrix is constructed based on the quality scores of image data and voiceprint data; The cross-attention weights are calculated using the low-rank features of the image, the low-rank features of the voiceprint, and the diagonal matrix. The cross-attention weights are used to interactively update the low-rank features of the image and the low-rank features of the voiceprint, and then weighted and fused based on the quality score to obtain cross-modal aligned fused features.
[0038] In some embodiments, the expression for the cross-modal alignment fusion feature is: ; ; ; ; ; ; in, Denotes the truncated singular value decomposition basis of the joint matrix. Represents the feature projection error matrix. Let M represent the first r columns of singular vectors. The low-rank projected reconstruction value represents the feature. Represents the original feature vector of the bimodal mode. This represents the function for calculating cross-attention weights. Represents the image modality weight matrix. This represents the voiceprint modal weight matrix. Represents the normalized exponential function, This indicates the characteristic transpose operator. This represents the cross-modal attention weight matrix. This represents the low-rank features of the voiceprint after global alignment. This represents the low-rank features of the globally aligned image. The standardized normalized value represents the bimodal feature. Indicates image quality score, This indicates the voiceprint quality score.
[0039]
[0040] S4: Input the cross-modal aligned and fused features into the lightweight feature extraction network to extract features and obtain lightweight feature vectors.
[0041] In some embodiments, the cross-modal aligned fusion feature input adaptive lightweight channel segmentation and shuffling module divides the feature channels into important channel groups and secondary channel groups according to the L1 norm of the channel weights, performs convolution enhancement only on the important channel groups, and then performs channel shuffling and recombination with the secondary channel groups. The recombined features are input into the lightweight coordinate attention module, which generates attention weights along the feature length dimension and weights the features to obtain a lightweight feature vector.
[0042]
[0043] S5: Use a lightweight classifier to classify and identify lightweight feature vectors to obtain identification results for marginal wildlife, thus completing the identification of marginal wildlife.
[0044] In some embodiments, the cosine similarity between the lightweight feature vector and a preset species prototype feature library is calculated and converted into an initial species probability distribution; The bimodal quality score is calculated based on image sharpness and voiceprint signal-to-noise ratio. The initial species probability distribution is then weighted and corrected using the bimodal quality score to obtain the corrected species probability distribution and the corresponding species identification results.
[0045] In some embodiments, a computing power allocation coefficient is calculated using a weighted formula based on the remaining power, identification confidence, and real-time power consumption; based on the computing power allocation coefficient, the acquisition frequency in step S1, the quantization bit width of the feature extraction network in step S4, and the operating frequency of the edge device processor are dynamically adjusted to obtain a power scheduling instruction.
[0046] In some embodiments, the expression for the power scheduling instruction is: ; ; ; ; ; ; in, This represents a placeholder for function input parameters. This represents the function for calculating standard deviation. This represents the weight calculation function. This represents the mean calculation function. This represents the function for calculating the quantization bit width. Indicates the total number of sampling points. Represents the diagonal mass matrix. This represents a numerical clipping function. Represents the feature channel weight vector. This represents the function for calculating the minimum weight. Represents the standardized scaling factor. This represents the feature zero-padding mapping function. This represents the weight parameters of the fully connected layer. Indicates the dimension index parameter. This indicates the percentage of remaining battery power for edge devices. This represents the function for calculating the confidence level. This indicates the real-time power consumption value of the edge device. This indicates the maximum rated power consumption of the edge device.
[0047] In some embodiments, the expression for the loss function is: ; ; ; ; ; ; ; in, This represents the total loss function of the model. Represents the classification loss weight coefficient. This represents the alignment loss weighting coefficient. This represents the sparse loss weighting coefficient. This represents the power loss weighting coefficient. Represents classification loss. Indicates cross-modal alignment loss. This represents the feature sparsity loss. Indicates power consumption constraint loss, This represents the true label of the k-th species. This represents the probability that the model predicts for the k-th species, and its value ranges from [0,1]. This represents the total number of categories of the species to be identified. This represents the original feature vector of the image modality of the i-th input sample. This represents the original feature vector of the voiceprint modality of the i-th input sample. The squaring operation of the L2 norm of a vector is represented. This represents the dimension of the feature vector. This represents the high-frequency feature components in the d-th dimension, specifically referring to the detail components after wavelet decomposition of the speaker pattern or the high-frequency texture features of an image. The first norm operation of a vector is represented. This indicates the real-time power consumption of the edge recognition device during the current working cycle. This represents the minimum power consumption threshold required for edge recognition devices to maintain basic operation. This indicates the maximum power consumption threshold allowed by the hardware circuitry of the edge recognition device. This indicates the training rounds in the model training process.
[0048]
Claims
1. A method for identifying edge wildlife based on cross-modal alignment constraints, characterized in that, include: S1: Use edge devices to collect image and voiceprint data of wild animals in the target area; S2: Lightweight preprocessing is performed on image data and voiceprint data to obtain standardized bimodal features; S3: Perform bimodal joint low-rank decomposition on the standardized bimodal features, and calculate the alignment weights based on the low-rank features obtained from the decomposition using a cross-attention mechanism to obtain cross-modal aligned fusion features; S4: Input the cross-modal aligned and fused features into the lightweight feature extraction network to extract features and obtain lightweight feature vectors; S5: Use a lightweight classifier to classify and identify lightweight feature vectors to obtain identification results for marginal wildlife, thus completing the identification of marginal wildlife.
2. The edge wildlife identification method based on cross-modal alignment constraints according to claim 1, characterized in that, S1 includes: By utilizing edge devices, we can obtain information such as ambient light intensity, noise intensity, and the remaining power of the edge devices. The sampling frequency is calculated based on the light intensity, noise intensity, remaining battery power, and the preset target species activity coefficient; the expression for the sampling frequency is: ； in, This indicates the dynamic acquisition frequency of wildlife image data and voiceprint data from the edge device in the target area. Indicates the basic sampling frequency. Indicates light intensity. Indicates noise intensity. Indicates the remaining battery power. Indicates the activity coefficient; Based on the calculated acquisition frequency, image data and voiceprint data of wild animals in the target area are collected synchronously.
3. The edge wildlife identification method based on cross-modal alignment constraints according to claim 1, characterized in that, S2 includes: The image data is traversed, and the local mean and local variance of each pixel are calculated. Adaptive noise reduction is performed on the image data based on local mean, local variance, and noise variance. The denoised image data is shifted, scaled, and normalized to obtain standardized image features. Wavelet decomposition is performed on the voiceprint data to obtain approximate components and detail components; An adaptive threshold is obtained by calculating the length and median of the detail components of the voiceprint data. The detailed components are denoised using an adaptive threshold and then reconstructed by combining the approximate components to obtain the reconstructed speaker signal. The reconstructed voiceprint signal is normalized to obtain standardized voiceprint features; By combining standardized image features and standardized voiceprint features, standardized bimodal features are obtained.
4. The edge wildlife identification method based on cross-modal alignment constraints according to claim 1, characterized in that, The expression for the standardized bimodal feature is: ； ； ； ； ； ； ； ； ； ； in, Represents the local mean. Represents the original pixel values of the image. Represents local variance. This represents the pixel value after image noise reduction. Indicates the noise variance. Indicates light intensity. This represents the standardized pixel values of the image. Indicates an adaptive threshold. This represents the median calculation function. This represents the approximate components of the voiceprint. Indicates the voiceprint detail components. Represents a symbolic function. Indicates the length of the voiceprint data. This represents the median of the voiceprint detail components. This represents the reconstructed voiceprint signal value. This represents the function for calculating the minimum value. Indicates image quality score, Represents the characteristic target mean. This indicates the voiceprint signal-to-noise ratio. Indicates the intensity of ambient noise.
5. The edge wildlife identification method based on cross-modal alignment constraints according to claim 1, characterized in that, S3 includes: The image features and voiceprint features in the standardized bimodal features are concatenated into a joint matrix; Perform singular value decomposition on the joint matrix and extract the singular vectors corresponding to the first few singular values as a shared low-rank basis; The image features and the voiceprint features are projected onto the shared low-rank basis to obtain the image low-rank features and the voiceprint low-rank features, respectively. A diagonal matrix is constructed based on the quality scores of image data and voiceprint data; The cross-attention weights are calculated using the low-rank features of the image, the low-rank features of the voiceprint, and the diagonal matrix. The cross-attention weights are used to interactively update the low-rank features of the image and the low-rank features of the voiceprint, and then weighted and fused based on the quality score to obtain cross-modal aligned fused features.
6. The edge wildlife identification method based on cross-modal alignment constraints according to claim 1, characterized in that, The expression for the cross-modal alignment fusion feature is: ； ； ； ； ； ； in, Denotes the truncated singular value decomposition basis of the joint matrix. Represents the feature projection error matrix. Let M represent the first r columns of singular vectors. The low-rank projected reconstruction value represents the feature. Represents the original feature vector of the bimodal mode. This represents the function for calculating cross-attention weights. Represents the image modality weight matrix. This represents the voiceprint modal weight matrix. Represents the normalized exponential function, This indicates the characteristic transpose operator. This represents the cross-modal attention weight matrix. This represents the low-rank features of the voiceprint after global alignment. This represents the low-rank features of the globally aligned image. The standardized normalized value represents the bimodal feature. Indicates image quality score, This indicates the voiceprint quality score.
7. The edge wildlife identification method based on cross-modal alignment constraints according to claim 1, characterized in that, S4 includes: The cross-modal aligned and fused features are input into the adaptive lightweight channel segmentation and shuffling module. Based on the L1 norm of the channel weights, the feature channels are divided into important channel groups and secondary channel groups. Convolution enhancement is performed only on the important channel groups, and then channel shuffling and recombination are performed with the secondary channel groups to obtain the recombined features. The recombined features are input into the lightweight coordinate attention module, which generates attention weights along the feature length dimension and weights the features to obtain a lightweight feature vector. The adaptive lightweight channel segmentation and shuffling module and the lightweight coordinate attention module belong to the lightweight feature extraction network.
8. The edge wildlife identification method based on cross-modal alignment constraints according to claim 1, characterized in that, S5 includes: Calculate the cosine similarity between the lightweight feature vector and the preset species prototype feature library, and convert it into an initial species probability distribution; Based on a lightweight classifier, a bimodal quality score is calculated according to image sharpness and voiceprint signal-to-noise ratio. The initial species probability distribution is then weighted and corrected using the bimodal quality score to obtain the identification results of marginal wildlife, thus completing the identification of marginal wildlife.
9. The edge wildlife identification method based on cross-modal alignment constraints according to claim 1, characterized in that, The expression for the power scheduling instruction is: ； ； ； ； ； ； in, This represents a placeholder for function input parameters. This represents the function for calculating standard deviation. This represents the weight calculation function. This represents the mean calculation function. This represents the function for calculating the quantization bit width. Indicates the total number of sampling points. Represents the diagonal mass matrix. This represents a numerical clipping function. Represents the feature channel weight vector. This represents the function for calculating the minimum weight. Represents the standardized scaling factor. This represents the feature zero-padding mapping function. This represents the weight parameters of the fully connected layer. Indicates the dimension index parameter. This indicates the percentage of remaining battery power for edge devices. This represents the function for calculating the confidence level. This indicates the real-time power consumption value of the edge device. This indicates the maximum rated power consumption of the edge device.
10. The edge wildlife identification method based on cross-modal alignment constraints according to claim 1, characterized in that, The expression for the loss function of the lightweight classifier is: ； ； ； ； ； ； ； in, This represents the loss function of the lightweight classifier. Represents the classification loss weight coefficient. This represents the alignment loss weighting coefficient. This represents the sparse loss weighting coefficient. This represents the power loss weighting coefficient. Represents classification loss. Indicates cross-modal alignment loss. This represents the feature sparsity loss. Indicates power consumption constraint loss, This represents the true label of the k-th species. This represents the probability that the model predicts for the k-th species, and its value ranges from [0,1]. This represents the total number of categories of the species to be identified. This represents the original feature vector of the image modality of the i-th input sample. This represents the original feature vector of the voiceprint modality of the i-th input sample. The squaring operation of the L2 norm of a vector is represented. This represents the dimension of the feature vector. This represents the high-frequency feature components in the d-th dimension, specifically referring to the detail components after wavelet decomposition of the speaker pattern or the high-frequency texture features of an image. The first norm operation of a vector is represented. This indicates the real-time power consumption of the edge recognition device during the current working cycle. This represents the minimum power consumption threshold required for edge recognition devices to maintain basic operation. This indicates the maximum power consumption threshold allowed by the hardware circuitry of the edge recognition device. This indicates the training rounds in the model training process.