Animal scene-oriented adaptive multi-modal data fusion method

Abstract

The invention relates to the technical field of data fusion, and discloses an animal scene-oriented adaptive multi-modal data fusion method, which comprises the following steps of: extracting spatio-temporal characteristics from multi-source heterogeneous data such as visual sense, auditory sense and physiological sensing, constructing an animal-environment-group ternary spatio-temporal relation graph, and constructing an animal-environment-group ternary spatio-temporal relation graph; a pilot frequency sampling problem is solved through an adaptive interpolation algorithm, cross-modal projection alignment is completed in a public semantic space, unified space-time representation is output, and confidence coefficient weight is dynamically calculated based on uncertainty measurement of each modal feature. According to the method, accurate alignment of multi-modal data is realized through the cross-modal space-time attention network, the multi-modal feature alignment error is reduced compared with that of a traditional LSTM method, the training data volume of a federated element migration reinforcement learning framework is reduced compared with that of a traditional migration learning method, and the cross-species generalization performance of the model is improved. A multi-level causal inference engine quantitatively reveals causal association between environmental factors and animal diseases, and in combination with a dynamic decision tree visualization technology, the decision recognition degree is improved.

Description

Technical Field

[0001] This invention relates to the field of data fusion, and more specifically, to an adaptive multimodal data fusion method for animal scenarios. Background Technology

[0002] Traditional fusion methods for multimodal data in animal scenarios (such as weighted average and LSTM temporal fusion) have the following drawbacks: Existing methods suffer from high loss rates of key features when dealing with frequency differences of tens of thousands between vision and environmental sensing. Moreover, in cross-species scenarios (e.g., chicken → peacock), at least 200 target samples are required to maintain high accuracy, which far exceeds the amount of data that can be obtained in practice. Summary of the Invention

[0003] This invention provides an adaptive multimodal data fusion method for animal scenarios, which solves the problem of heterogeneous data fusion in related technologies and improves the robustness of health status judgment.

[0004] This invention provides an adaptive multimodal data fusion method for animal scenarios, comprising the following steps: S100, Multimodal Spatiotemporal Alignment: Extracts spatiotemporal features from multi-source heterogeneous data such as visual, auditory, and physiological sensing data, constructs a ternary spatiotemporal relationship graph of animal-environment-group, solves the problem of heterogeneous frequency sampling through adaptive interpolation algorithm, completes cross-modal projection alignment in the common semantic space, and outputs a unified spatiotemporal representation; S200, Dynamic Weight Fusion: Based on the uncertainty measure of each modality feature, the confidence weight is dynamically calculated. A hierarchical fusion strategy is adopted, which combines low-level feature weighted summation with high-level semantic attention mechanism. Temporal smoothing constraints are introduced to prevent weight abrupt changes, and multimodal fusion features are generated. S300, Enhanced Federated Transfer: Under the federated learning framework, cross-farm knowledge distillation is performed. Meta-learning methods are used to achieve rapid adaptation of support sets and generalization optimization of query sets. Differential privacy technology is combined to complete global model secure aggregation. At the same time, a species mapping matrix is ​​constructed to achieve cross-disease feature transfer. S400, Causal Decision Generation: Establishes a structural equation model and Bayesian network for environment-mediation-disease, quantifies the causal effects of key factors, generates heat maps based on gradient feature visualization technology, and finally outputs interpretable early warning reports by combining dynamic decision trees and natural language generation models.

[0005] Furthermore, S100 specifically includes the following steps: S110, Heterogeneous Data Representation Extraction: From the original visual, auditory and physiological sensor data, spatial features, temporal acoustic features and physiological signal encoding are extracted through deep networks to complete the preliminary feature representation of multimodal data; S120, Spatiotemporal Graph Construction: The extracted features are constructed into a spatiotemporal graph model. By quantifying the temporal difference and spatial distance relationship between individual animals, the dynamic attention weight between nodes is calculated to establish a three-dimensional spatiotemporal relationship network of animal-environment-group. S130, Adaptive Interpolation of Different Frequency Data: To address the non-uniform sampling characteristics of physiological sensor data, an exponential decay-based weight allocation strategy is adopted to perform spatiotemporal alignment of sensor data at different timestamps, generating a feature representation of a continuous time series. S140, Cross-modal projection alignment: Maps heterogeneous modal features to a unified semantic space, constrains the alignment consistency of visual, auditory and physiological data at the semantic level by calculating the feature similarity loss between modalities, and outputs a unified representation after spatiotemporal alignment; The unified representation after alignment is as follows: ; in The unified representation vector at time t, Represents the visual features after projection. This indicates the auditory characteristics after projection. Represents the physiological characteristics after projection. This indicates a feature splicing operation.

[0006] Furthermore, S200 specifically includes the following steps: S210, Modality-Specific Feature Extraction: Separate independent feature representations of visual, auditory and physiological modalities from uniform alignment features, and enhance the discriminative features of each modality through dedicated neural network layers; S220, Uncertainty Quantification: Based on the probability distribution of the output results of each modality, calculate the information entropy index to quantitatively evaluate the prediction confidence of visual, auditory and physiological modalities; S230, Adaptive weight allocation: The fusion weight coefficients are dynamically generated based on the modal confidence, and the weight enhancement of reliable modes is achieved by using a temperature-adjusted normalization method. S240, Hierarchical Feature Fusion: Through a dual-channel architecture of primary weighted fusion and advanced semantic fusion, combined with a gating mechanism, the optimal feature representation level is adaptively selected; S250, Dynamic Weight Decay: Introducing a sliding window temporal smoothing strategy, the fusion weights calculated in real time are weighted by historical states to generate multimodal fusion features; Primary feature fusion: ; in This represents the confidence level of the m-th mode. Represents the independent features of the m-th mode, and the input modal features. ,in Indicates visual characteristics, Indicates auditory characteristics, Represents physiological characteristics; weight ,in For visual modality confidence, For auditory modality confidence, For physiological modality confidence; Advanced semantic fusion: ; in The output features are for the Transformer, which has 4 layers: 8 heads and 512 hidden layers. Indicates a splicing operation; Gated fusion: ; ; in For the gated weight matrix, , It is the sigmoid activation function. For weighted summation features, g is the gating coefficient, which controls the fusion ratio of low-level and high-level features. This indicates the final fusion feature.

[0007] Furthermore, in S300, the specific steps include: S310, Cross-farm Feature Distillation: By comparing the difference in prediction distribution between the local model and the global model, the knowledge transfer loss is quantified to achieve implicit alignment of feature representations between farms; S320, Meta-transfer inner loop optimization: Performs fast parameter fine-tuning on the local support set, enabling the model to quickly adapt to the specific data distribution characteristics of the target farm; S330, Meta-transfer outer loop optimization: Based on the query set, the initially optimized model is adjusted a second time to enhance the model's generalization performance on the target task; S340, Differential Privacy Parameter Aggregation: Under the federated learning framework, noise perturbation and gradient pruning are applied to multi-node model updates to complete global model updates while ensuring data privacy and security. S350, Cross-Species Knowledge Projection: Utilizes species similarity matrix to guide feature space mapping, and achieves knowledge transfer and application in cross-species scenarios through distribution alignment technology.

[0008] Furthermore, in step S310, local fusion features are input. and global model ; Calculate the KL divergence loss: ; ; in This represents the fusion characteristic of the k-th farm. Let KL divergence loss be the value of the k-th farm. For local prediction probability distribution, For global prediction probability distribution; Feature similarity constraints: ; in To share the projection matrix, , To integrate feature dimensions, GAP represents global average pooling, maintaining dimensionality compatibility. For local predicted distribution, For global prediction distribution, Representation of local features, For global feature representation, Number the farms. This represents the feature similarity loss of the k-th farm.

[0009] Furthermore, in step S320, the input support set is... ; Inner loop parameter update: ; Among the losses incurred: ; in For the inner loop learning rate, Here, CE represents the distillation weighting coefficients, and CE denotes the cross-entropy loss function. Let these be the model parameters for the k-th farm. The parameters are updated for the inner loop. To support the set of input samples, To support tag collection, Let be the forward propagation function of the model. For the task loss function, Let represent the support set of the k-th farm.

[0010] Furthermore, in step S330, the query set is input. ; External loop parameter update: ; in The outer loop learning rate satisfies Meta-learning constraints, Input samples for the query set, To query set tags, The parameters are updated for the inner loop. These are the parameters updated after the outer loop.

[0011] Furthermore, in step S340, all farm parameters are entered. ; Generate noise disturbance parameters: ; Federal average aggregation: ; The cropping operation includes: ; in The standard deviation of Gaussian noise. K is the pruning threshold, and K is the number of farms participating in federated learning. The noise distribution is Gaussian. It is the L2 norm. These are the farm parameters after adding noise. For the updated global model, This represents a parameter pruning function used to limit the L2 norm of the parameters to no more than a threshold δ. This represents the parameter vector that needs to be clipped. Represents the identity matrix. This represents the updated model parameters for the k-th farm.

[0012] Furthermore, in step S350, the updated global model is input. Species mapping matrix ; Knowledge transfer loss: ; in This represents the feature extractor for species s, where MMD is the maximum mean difference and S is the number of species categories. This indicates the similarity between species s and t. This represents the feature representation of species s in the global model. Let be the feature representation of species t in the global model. This is the bandwidth parameter for the MMD kernel function.

[0013] Furthermore, in S400, the specific steps include: S410, Construction of Multidimensional Causal Graph: Establishing a causal relationship network between environmental parameters and disease characteristics, and quantifying the direct impact and mediating effect of environmental factors on animal health status through structural equation modeling; S420, Bayesian counterfactual reasoning: Based on observational data, it simulates the potential differences in outcomes under different environmental interventions and calculates the causal effect strength index between environmental factors and animal diseases; S430, Dynamic Weight Visualization: Utilizing an improved gradient heatmap generation technique, the key decision-making criteria of deep learning models are mapped back to the original input space, enabling an intuitive display of feature contributions. S440, Explainable Report Generation: Combining causal effect analysis and decision tree models, it automatically generates natural language diagnostic reports containing key pathogenic factors and causal path maps.

[0014] The beneficial effects of this invention are as follows: 1. Breakthrough in the accuracy of inter-frequency data fusion The Cross-Modal Spatiotemporal Attention Network (CMSTA-Net) achieves accurate alignment of visual, physiological, and environmental data, and reduces multimodal feature alignment error compared to traditional LSTM methods. 2. Innovation in small sample transfer efficiency The Federal Meta-Transfer Reinforcement Learning (MTRL) framework requires only 5 target farm samples to achieve the required detection accuracy, which reduces the amount of training data compared to traditional transfer learning methods and improves the model's cross-species generalization performance. 3. Decision interpretability crossing The Multilevel Causal Inference Engine (MCIE) is the first to quantitatively reveal the causal relationship between environmental factors and animal diseases. Combined with dynamic decision tree visualization technology, it enhances the acceptance of the system's decision-making by veterinary experts.

[0015] In summary, by using dynamic weight allocation, federated meta-transfer, and causal engine, we can overcome the challenges of cross-frequency fusion, small-sample migration, and edge deployment. Attached Figure Description

[0016] Figure 1 This is a flowchart of an adaptive multimodal data fusion method for animal scenarios proposed in this invention; Figure 2 This is the invention Figure 1 Flowchart of S100 in China; Figure 3 This is the invention Figure 1 Flowchart of S200; Figure 4 This is the invention Figure 1 Flowchart of S300; Figure 5 This is the invention Figure 1 Flowchart of S400. Detailed Implementation

[0017] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0018] like Figure 1 - Figure 5 As shown, an adaptive multimodal data fusion method for animal scenarios includes the following steps: S100, Multimodal Spatiotemporal Alignment: Extracts spatiotemporal features from multi-source heterogeneous data such as visual, auditory, and physiological sensing data, constructs a ternary spatiotemporal relationship graph of animal-environment-group, solves the problem of heterogeneous frequency sampling through adaptive interpolation algorithm, completes cross-modal projection alignment in the common semantic space, and outputs a unified spatiotemporal representation; In one embodiment of the present invention, the following steps are specifically included: S110, Heterogeneous Data Representation Extraction: From the original visual, auditory and physiological sensor data, spatial features, temporal acoustic features and physiological signal encoding are extracted through deep networks to complete the preliminary feature representation of multimodal data; Visual modality: ; in , For the image height and width, This represents the number of RGB channels. =3, Represents the RGB image at time t; Auditory modality: ; Where n is the number of audio frames, and F is the Mel-spectral dimension. This represents the Mel spectrum at time t'. This represents the feature vector of the i-th audio frame; Physiological sensing: in For the number of physiological sensors, For the set of sampling timestamps, Represents a heterogeneous set of sampling timestamps. This represents the sampled value of the i-th sensor at time τ; Extracting spatial features: ; Extracting temporal acoustic features: ; Encoding physiological signals: ; in Represents visual feature vectors. Represents auditory feature vectors, The physiological feature vector representing time τ; in For visual feature dimensions, , For auditory features, , Physiological characteristics dimension ; in Remove the pre-trained model from the last fully connected layer; The BiLSTM hidden layer has a dimension of 256; The MLP is a three-layer perceptron with hidden layer dimensions of 512, 384, and 256, respectively. S120, Spatiotemporal Graph Construction: The extracted features are constructed into a spatiotemporal graph model. By quantifying the temporal difference and spatial distance relationship between individual animals, the dynamic attention weight between nodes is calculated to establish a three-dimensional spatiotemporal relationship network of animal-environment-group. Input feature set ; Define spatiotemporal graph nodes: ; in This indicates a splicing operation. For time-dimensional mean pooling, This represents the i-th spatiotemporal graph node. Represents the i-th visual feature. This represents the i-th auditory feature. Represents the i-th physiological characteristic; Calculate the edge weight matrix: ; in This represents the edge weight from node i to node j; in , Encode the time difference; in , Encoding spatial distance; in , To standardize the time difference, Let i and j be the timestamps of nodes i and j, respectively. The maximum time difference; in , To standardize the Euclidean distance, Let i and j be the GPS coordinates of nodes i and j, respectively. in For querying the matrix, , The key matrix, , The total dimension of node features. , For the key / query dimension of the attention mechanism, , Let be the time normalization constant. Let be the space normalization constant. It is the softmax activation function; S130, Adaptive Interpolation of Different Frequency Data: To address the non-uniform sampling characteristics of physiological sensor data, an exponential decay-based weight allocation strategy is adopted to perform spatiotemporal alignment of sensor data at different timestamps, generating a feature representation of a continuous time series. Input physiological sensing sequence ; Execution content: Calculate the interpolation weights for the target time t: ; Generate aligned features: ; in The Gaussian kernel width parameter controls the time sensitivity. , These are the interpolation weighting coefficients. ,satisfy , Let τ be the target time and τ be the original sampling time. This represents the interpolated physiological characteristics at time t. Represents the original physiological characteristics at time τ; S140, Cross-modal projection alignment: Maps heterogeneous modal features to a unified semantic space, constrains the alignment consistency of visual, auditory and physiological data at the semantic level by calculating the feature similarity loss between modalities, and outputs a unified representation after spatiotemporal alignment; Input alignment features ; Constructing a public semantic space projection: ; Calculate the intermodal alignment loss: ; The similarity metric is: ; in The modal projection matrix includes For visual projection matrix, , For auditory projection matrix, , For physiological projection matrix, ; in For the public semantic space dimension, ; in For each modal bias term, , Original features Features after projection The cosine similarity metric function is used. For alignment loss; Output a unified representation after spatiotemporal alignment: ; in The unified representation vector at time t, Represents the visual features after projection. This indicates the auditory characteristics after projection. Represents the physiological characteristics after projection. Indicates feature concatenation operation; S200, Dynamic Weight Fusion: Based on the uncertainty measure of each modality feature, the confidence weight is dynamically calculated. A hierarchical fusion strategy is adopted, which combines low-level feature weighted summation with high-level semantic attention mechanism. Temporal smoothing constraints are introduced to prevent weight abrupt changes, and multimodal fusion features are generated. In one embodiment of the present invention, the following steps are specifically included: S210, Modality-Specific Feature Extraction: Separate independent feature representations of visual, auditory and physiological modalities from uniform alignment features, and enhance the discriminative features of each modality through dedicated neural network layers; Unified representation after input alignment ; Extract modal-independent features: ; in , This represents the independent features of the m-th mode. For visual feature projection matrix, , For auditory feature projection matrix, , The projection matrix of physiological characteristics. , For each modal bias term, , For the hidden layer dimension, ReLU is the modified linear unit activation function; S220, Uncertainty Quantification: Based on the probability distribution of the output results of each modality, calculate the information entropy index to quantitatively evaluate the prediction confidence of visual, auditory and physiological modalities; Input modal features ,in Indicates visual characteristics, Indicates auditory characteristics, Indicates physiological characteristics; Calculate the output distribution for each mode: in This represents the predicted probability distribution of the m-th mode. Denotes the category projection matrix, where For visual category projection matrix, , For auditory category projection matrix, , For physiological category projection matrix, ; Calculate the modal entropy value: ; Where C is the number of task categories, Let represent the predicted probability of the m-th modality for class c. For each modal feature, softmax is the normalized exponential function. This represents the information entropy of the m-th mode; S230, Adaptive weight allocation: The fusion weight coefficients are dynamically generated based on the modal confidence, and the weight enhancement of reliable modes is achieved by using a temperature-adjusted normalization method. Input entropy ; Calculate the normalized confidence score: ; in Let η be the temperature coefficient, and let η control the sensitivity to the influence of entropy. Optimization is achieved through grid search. For visual modality confidence, , For auditory modality confidence, , For physiological modality confidence, And satisfy , This represents the confidence level of the m-th mode. This represents the entropy value of the m'-th mode; S240, Hierarchical Feature Fusion: Through a dual-channel architecture of primary weighted fusion and advanced semantic fusion, combined with a gating mechanism, the optimal feature representation level is adaptively selected; Primary feature fusion: ; Advanced semantic fusion: ; The Transformer class contains 4 layers, with 8 head layers and 512 hidden layers. Indicates a splicing operation; Gated fusion: ; ; in For the gated weight matrix, , It is the sigmoid activation function. For the weighted summation characteristic, Here, g represents the output features of the Transformer, and g is the gating coefficient that controls the fusion ratio of low-level and high-level features. Indicates the final fusion characteristics; S250, Dynamic Weight Decay: Introducing a sliding window temporal smoothing strategy, the fusion weights calculated in real time are weighted by historical states to generate multimodal fusion features; Input historical weight sequence ; Calculate the time-series smoothing weights: ; in The current time weight is the smoothed weight. The original weight at the current time. The weights are the historical moments k steps ago. Maintaining a balance between current and historical weights. Indicates the attenuation factor. Indicates the size of the time window; S300, Enhanced Federated Transfer: Under the federated learning framework, cross-farm knowledge distillation is performed. Meta-learning methods are used to achieve rapid adaptation of support sets and generalization optimization of query sets. Differential privacy technology is combined to complete global model secure aggregation. At the same time, a species mapping matrix is ​​constructed to achieve cross-disease feature transfer. In one embodiment of the present invention, the specific steps are as follows: S310, Cross-farm Feature Distillation: By comparing the difference in prediction distribution between the local model and the global model, the knowledge transfer loss is quantified to achieve implicit alignment of feature representations between farms; Input local fusion features and global model ; Calculate the KL divergence loss: ; ; in This represents the fusion characteristic of the k-th farm. Let KL divergence loss be the value of the k-th farm. For local prediction probability distribution, For global prediction probability distribution; Feature similarity constraints: in To share the projection matrix, , To fuse feature dimensions, GAP indicates global average pooling, maintaining dimensionality compatibility. For local predicted distribution, For global prediction distribution, Representation of local features, For global feature representation, Number the farms. This represents the feature similarity loss of the k-th farm; S320, Meta-transfer inner loop optimization: Performs fast parameter fine-tuning on the local support set, enabling the model to quickly adapt to the specific data distribution characteristics of the target farm; Input support set ; Inner loop parameter update: ; Among the losses incurred: ; in For the inner loop learning rate, Here, CE represents the distillation weighting coefficients, and CE denotes the cross-entropy loss function. Let these be the model parameters for the k-th farm. The parameters are updated for the inner loop. To support the set of input samples, To support tag collection, Let be the forward propagation function of the model. For the task loss function, Represents the support set of the k-th farm; S330, Meta-transfer outer loop optimization: Based on the query set, the initially optimized model is adjusted a second time to enhance the model's generalization performance on the target task; Input query set ; External loop parameter update: ; in The outer loop learning rate satisfies Meta-learning constraints, Input samples for the query set, To query set tags, The parameters are updated for the inner loop. These are the parameters updated by the outer loop; S340, Differential Privacy Parameter Aggregation: Under the federated learning framework, noise perturbation and gradient pruning are applied to multi-node model updates to complete global model updates while ensuring data privacy and security. Enter all farm parameters ; Generate noise disturbance parameters: ; Federal average aggregation: ; The cropping operation includes: ; in The standard deviation of Gaussian noise. K is the pruning threshold, and K is the number of farms participating in federated learning. The noise distribution is Gaussian. It is the L2 norm. These are the farm parameters after adding noise. For the updated global model parameters, This represents a parameter pruning function used to limit the L2 norm of the parameters to no more than a threshold δ. This represents the parameter vector that needs to be clipped. Represents the identity matrix. This represents the updated model parameters for the k-th farm; S350, Cross-Species Knowledge Projection: Utilizes species similarity matrix to guide feature space mapping, and achieves knowledge transfer and application in cross-species scenarios through distribution alignment technology; Input the updated global model Species mapping matrix Knowledge transfer loss: ; in This represents the feature extractor for species s, where MMD is the maximum mean difference and S is the number of species categories. This indicates the similarity between species s and t. This represents the feature representation of species s in the global model. Let be the feature representation of species t in the global model. This refers to the bandwidth parameter of the MMD kernel function; S400, Causal Decision Generation: Establish a structural equation model and Bayesian network for environment-mediation-disease, quantify the causal effects of key factors, generate heat maps based on gradient feature visualization technology, and finally output interpretable early warning reports by combining dynamic decision trees and natural language generation models. In one embodiment of the present invention, the following steps are specifically included: S410, Construction of Multidimensional Causal Graph: Establishing a causal relationship network between environmental parameters and disease characteristics, and quantifying the direct impact and mediating effect of environmental factors on animal health status through structural equation modeling; Input environment parameters fusion features ; Constructing a structural equation model: ; Calculate the standardized path coefficients: ; ; in This is the output layer weight matrix. , This is the weight matrix of the intermediary layer. , This is a mapping matrix from the environment to the mediator. , The mapping matrix from features to mediators. , From the perspective of environmental factors, , As a mediating variable dimension, , For the output layer random error, For the random error of the intermediate layer, For covariance operators, For variance operators, For the path coefficients from the environment to the output, These are the path coefficients from the intermediary to the output; S420, Bayesian counterfactual reasoning: Based on observational data, it simulates the potential differences in outcomes under different environmental interventions and calculates the causal effect strength index between environmental factors and animal diseases; Input observation data ; Calculate the counterfactual probability: ; Estimating causal effects: ; in These are intervention values ​​for environmental parameters. These are the observed values ​​of environmental parameters. To intervene in changes and comply with environmental safety standards, Given the environment and intermediaries, the conditional probability is... For a given environment, the conditional probability of the mediating variable. For expectation operator, This is an estimate of the average causal effect; S430, Dynamic Weight Visualization: Utilizing an improved gradient heatmap generation technique, the key decision-making criteria of deep learning models are mapped back to the original input space, enabling an intuitive display of feature contributions. Input gradient features ; Calculate the weights for the improved Grad-CAM++: ; ; Generate a heatmap: ; in This is the activation map of the final convolutional layer. , The height and width of the feature map, The number of feature channels, The class activation mapping weight for the k-th channel at position (i, j) is... The feature weights for position (i, j) are... This indicates an element-wise multiplication operation. The original input image at time t. For the generated heatmap, For upsampling operations, the feature map size is restored to the original image size; S440, Explainable Report Generation: Combining causal effect analysis and decision tree model, it automatically generates natural language diagnostic reports containing key pathogenic factors and causal path maps; Input causal effect Weight matrix ; Constructing a dynamic decision tree: ; in , This represents the importance weight of the i-th node to the j-th feature; Generate natural language descriptions: in The decision threshold for the i-th node is determined through Gini coefficient optimization. Let i be the importance weight of the i-th node to the j-th feature. This is an indicator function, taking the value 0 or 1. For the j-th input feature, Generative models fine-tuned for the field of veterinary medicine To select the function of the top k features with the highest importance, The gradient of the environmental factors with respect to the output. A function to generate a causal Sankey diagram. For the generated natural language report, This represents the causal path from environmental factors to mediating variables and then to the output.

[0019] In this embodiment, an application example is given: early warning of respiratory diseases in pigs in a farm. Scenario: This system was deployed in an intensive pig farm. The environmental sensor detected that the ammonia concentration in pig house No. 3 was continuously exceeding the standard (28ppm > 25ppm threshold), triggering the early warning process.

[0020] Step 1, Multimodal Spatiotemporal Alignment: Data collection: Video: Infrared cameras captured an increase in the amount of time three pigs in pen number 5 spent lying down; Audio: Microphone recorded abnormal cough frequency (12 coughs / minute vs. baseline 5 coughs); Physiological: The ear tag sensor showed an average body temperature of 39.8℃ (normal range 38.5-39.5℃); Environment: Ammonia 28ppm / Temperature 29℃ / Humidity 80%; Alignment process: Align video frames (10fps), audio (1s window), and body temperature (5-minute interval) to the time baseline; Spatiotemporal modeling revealed that the spatial overlap between high ammonia areas and the distribution of diseased pigs reached 78%. Step 2, Dynamic Weight Fusion: Confidence calculation: Visual modality: 0.83 (high) confidence level for detecting decreased activity. Auditory modality: Cough sound recognition confidence level 0.67 (medium); Physiological modality: Confidence level of abnormal body temperature 0.91 (very high); Integrated Decision Making: Final weighting: Physiological (52%) > Visual (33%) > Auditory (15%); Overall anomaly score: 87.4 / 100 (threshold 70); Step 3, Enhanced Federal Migration: Knowledge transfer: Transferring common features from bovine respiratory disease models (correlation between cough frequency and lung function); After local fine-tuning, the model's F1-score for detecting porcine mycoplasma pneumonia was improved by 27%. Privacy Computing: After differential noise injection, the risk of re-identifying sensitive features (such as pig pen location) is reduced by 83%; Step 4, Causal Decision Generation: The explanatory report content is as follows: 1. Key pathogenic factors: Ammonia concentration exceeded the standard (contributing 62%). Sustained high temperature environment (contribution 28%); Pig density is too high (contribution 10%); 2. Causal pathway: Ammonia irritation (28ppm) → respiratory mucosal damage → increased cough frequency → thermoregulation imbalance; 3. Recommendations for handling the situation: Turn on the ventilation system immediately (ammonia levels are expected to drop to 18 ppm within 2 hours); Three individuals with a body temperature >39.5℃ were isolated and given nebulizer treatment; The stocking density in this area was adjusted from 3 heads / m² to 2.5 heads / m². Model confidence level: 92%; Accuracy rate of similar historical cases: 88% (correctly predicted 20 out of 23 cases in the past six months); A heatmap can be provided on the user display interface, and the feature heatmap shows the significant response areas of body temperature and cough characteristics; Results: After continuous monitoring for n hours, the system successfully reduced the ammonia concentration to a safe range, and the isolated sick pigs recovered after treatment, thus avoiding potential economic losses.

[0021] The embodiments of the present invention have been described above, but the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention, all of which are within the protection scope of the present invention.

Claims

1. An adaptive multimodal data fusion method for animal scenarios, characterized in that, Includes the following steps: S100, Multimodal Spatiotemporal Alignment: Extracts spatiotemporal features from multi-source heterogeneous data such as visual, auditory, and physiological sensing data, constructs a ternary spatiotemporal relationship graph of animal-environment-group, solves the problem of heterogeneous frequency sampling through adaptive interpolation algorithm, completes cross-modal projection alignment in the common semantic space, and outputs a unified spatiotemporal representation; S200, Dynamic Weight Fusion: Based on the uncertainty measure of each modality feature, the confidence weight is dynamically calculated. A hierarchical fusion strategy is adopted, which combines low-level feature weighted summation with high-level semantic attention mechanism. Temporal smoothing constraints are introduced to prevent weight abrupt changes, and multimodal fusion features are generated. S300, Enhanced Federated Transfer: Under the federated learning framework, cross-farm knowledge distillation is performed. Meta-learning methods are used to achieve rapid adaptation of support sets and generalization optimization of query sets. Differential privacy technology is combined to complete global model secure aggregation. At the same time, a species mapping matrix is ​​constructed to achieve cross-disease feature transfer. S400, Causal Decision Generation: Establishes a structural equation model and Bayesian network for environment-mediation-disease, quantifies the causal effects of key factors, generates heat maps based on gradient feature visualization technology, and finally outputs interpretable early warning reports by combining dynamic decision trees and natural language generation models.

2. The adaptive multimodal data fusion method for animal scenarios according to claim 1, characterized in that, S100 specifically includes the following steps: S110, Heterogeneous Data Representation Extraction: From the original visual, auditory and physiological sensor data, spatial features, temporal acoustic features and physiological signal encoding are extracted through deep networks to complete the preliminary feature representation of multimodal data; S120, Spatiotemporal Graph Construction: The extracted features are constructed into a spatiotemporal graph model. By quantifying the temporal difference and spatial distance relationship between individual animals, the dynamic attention weight between nodes is calculated to establish a three-dimensional spatiotemporal relationship network of animal-environment-group. S130, Adaptive Interpolation of Different Frequency Data: To address the non-uniform sampling characteristics of physiological sensor data, an exponential decay-based weight allocation strategy is adopted to perform spatiotemporal alignment of sensor data at different timestamps, generating a feature representation of a continuous time series. S140, Cross-modal projection alignment: Maps heterogeneous modal features to a unified semantic space, constrains the alignment consistency of visual, auditory and physiological data at the semantic level by calculating the feature similarity loss between modalities, and outputs a unified representation after spatiotemporal alignment; The unified representation after spatiotemporal alignment is as follows: ； in The unified representation vector at time t, Represents the visual features after projection. This indicates the auditory characteristics after projection. Represents the physiological characteristics after projection. This indicates a feature splicing operation.

3. The adaptive multimodal data fusion method for animal scenarios according to claim 2, characterized in that, S200 specifically includes the following steps: S210, Modality-Specific Feature Extraction: Separate independent feature representations of visual, auditory and physiological modalities from uniform alignment features, and enhance the discriminative features of each modality through dedicated neural network layers; S220, Uncertainty Quantification: Based on the probability distribution of the output results of each modality, calculate the information entropy index to quantitatively evaluate the prediction confidence of visual, auditory and physiological modalities; S230, Adaptive weight allocation: The fusion weight coefficients are dynamically generated based on the modal confidence, and the weight enhancement of reliable modes is achieved by using a temperature-adjusted normalization method. S240, Hierarchical Feature Fusion: Through a dual-channel architecture of primary weighted fusion and advanced semantic fusion, combined with a gating mechanism, the optimal feature representation level is adaptively selected; S250, Dynamic Weight Decay: Introducing a sliding window temporal smoothing strategy, the fusion weights calculated in real time are weighted by historical states to generate multimodal fusion features; Primary feature fusion: ； in This represents the confidence level of the m-th mode. Represents the independent features of the m-th mode, and the input modal features. ,in Indicates visual characteristics, Indicates auditory characteristics, Represents physiological characteristics; weight ,in For visual modality confidence, For auditory modality confidence, For physiological modality confidence; Advanced semantic fusion: ； in The output features are for the Transformer, which has 4 layers: 8 heads and 512 hidden layers. Indicates a splicing operation; Gated fusion: ； ； in For the gated weight matrix, , It is the sigmoid activation function. For weighted summation features, g is the gating coefficient, which controls the fusion ratio of low-level and high-level features. This indicates the final fusion feature.

4. The adaptive multimodal data fusion method for animal scenarios according to claim 3, characterized in that, In S300, the specific steps include: S310, Cross-farm Feature Distillation: By comparing the difference in prediction distribution between the local model and the global model, the knowledge transfer loss is quantified to achieve implicit alignment of feature representations between farms; S320, Meta-transfer inner loop optimization: Performs fast parameter fine-tuning on the local support set, enabling the model to quickly adapt to the specific data distribution characteristics of the target farm; S330, Meta-transfer outer loop optimization: Based on the query set, the initially optimized model is adjusted a second time to enhance the model's generalization performance on the target task; S340, Differential Privacy Parameter Aggregation: Under the federated learning framework, noise perturbation and gradient pruning are applied to multi-node model updates to complete global model updates while ensuring data privacy and security. S350, Cross-Species Knowledge Projection: Utilizes species similarity matrix to guide feature space mapping, and achieves knowledge transfer and application in cross-species scenarios through distribution alignment technology.

5. The adaptive multimodal data fusion method for animal scenarios according to claim 4, characterized in that, In step S310, input local fusion features. and global model ; Calculate the KL divergence loss: ； ； in This represents the fusion characteristic of the k-th farm. Let KL divergence loss be the value of the k-th farm. For local prediction probability distribution, For global prediction probability distribution; Feature similarity constraints: ； in To share the projection matrix, , To integrate feature dimensions, GAP represents global average pooling, maintaining dimensionality compatibility. For local predicted distribution, For global prediction distribution, Representation of local features, For global feature representation, Number the farms. This represents the feature similarity loss of the k-th farm.

6. The adaptive multimodal data fusion method for animal scenarios according to claim 5, characterized in that, In step S320, the input support set is... ; Inner loop parameter update: ； Among the losses incurred: ； in For the inner loop learning rate, Here, CE represents the distillation weighting coefficients, and CE denotes the cross-entropy loss function. Let these be the model parameters for the k-th farm. The parameters are updated for the inner loop. To support the set of input samples, To support tag collection, Let be the forward propagation function of the model. For the task loss function, Let represent the support set of the k-th farm.

7. The adaptive multimodal data fusion method for animal scenarios according to claim 6, characterized in that, In step S330, input the query set: ； External loop parameter update: ； in The outer loop learning rate satisfies Meta-learning constraints, Input samples for the query set, To query set tags, The parameters are updated for the inner loop. These are the parameters updated after the outer loop.

8. The adaptive multimodal data fusion method for animal scenarios according to claim 7, characterized in that, In step S340, enter all farm parameters. ; Generate noise disturbance parameters: ； Federal average aggregation: ； The cropping operation includes: ； in The standard deviation of Gaussian noise. K is the pruning threshold, and K is the number of farms participating in federated learning. The noise distribution is Gaussian. It is the L2 norm. These are the farm parameters after adding noise. For the updated global model, This represents a parameter pruning function used to limit the L2 norm of the parameters to no more than a threshold δ. This represents the parameter vector that needs to be clipped. Represents the identity matrix. This represents the updated model parameters for the k-th farm.

9. The adaptive multimodal data fusion method for animal scenarios according to claim 8, characterized in that, In step S350, the updated global model is input. Species mapping matrix ; Knowledge transfer loss: ； in This represents the feature extractor for species s, where MMD is the maximum mean difference and S is the number of species categories. This indicates the similarity between species s and t. This represents the feature representation of species s in the global model. Let be the feature representation of species t in the global model. This is the bandwidth parameter for the MMD kernel function.

10. The adaptive multimodal data fusion method for animal scenarios according to claim 9, characterized in that, In S400, the specific steps include: S410, Construction of Multidimensional Causal Graph: Establishing a causal relationship network between environmental parameters and disease characteristics, and quantifying the direct impact and mediating effect of environmental factors on animal health status through structural equation modeling; S420, Bayesian counterfactual reasoning: Based on observational data, it simulates the potential differences in outcomes under different environmental interventions and calculates the causal effect strength index between environmental factors and animal diseases; S430, Dynamic Weight Visualization: Utilizing an improved gradient heatmap generation technique, the key decision-making criteria of deep learning models are mapped back to the original input space, enabling an intuitive display of feature contributions. S440, Explainable Report Generation: Combining causal effect analysis and decision tree models, it automatically generates natural language diagnostic reports containing key pathogenic factors and causal path maps.

Images