Anomaly detection method and device for multimodal agricultural product data privacy protection

Abstract

This invention provides a privacy-preserving anomaly detection method and device for multimodal agricultural product data. The method comprises the following steps: a multimodal adaptive fusion method based on a meta-learning cross-modal attention mechanism to adaptively fuse multimodal agricultural product data features; a hierarchical adaptive differential privacy protection method based on a graph attention network to adaptively generate noise to protect the privacy of multimodal fused features; a privacy protection method based on knowledge distillation pre-training with dynamic weighting of modal under-optimization to protect model gradients from dynamic perturbations; and a multimodal agricultural product anomaly detection method based on a self-supervised denoising encoder to denoise pre-trained multimodal perturbation fusion data, effectively detecting agricultural product anomalies. This method enhances the dynamic correlation and adaptive fusion of multimodal features, dynamically adjusts the loss weights of weak modalities, provides dynamic noise protection, and achieves high-precision agricultural product anomaly detection.

Description

Technical Field

[0001] The present invention relates to the field of agricultural product data technology, and in particular to an anomaly detection method and device for multimodal agricultural product data privacy protection. Background Art

[0002] With the rapid development of agricultural big data and artificial intelligence technologies, agricultural product anomaly detection, as an important technical tool, has been widely applied in crop pest and disease monitoring, farmland environmental monitoring, and crop growth prediction. By collecting and analyzing data from different modalities, such as agricultural product images, text data, and soil data, agricultural product anomaly detection can effectively identify potential anomalies in farmland and promptly detect problems such as pests and diseases, mold, and moisture, thereby improving the efficiency and quality of agricultural production. However, with the increase in data volume, data diversity, and modality diversity, the accuracy of agricultural product anomaly detection faces significant challenges.

[0003] The process of detecting agricultural product anomalies involves a vast amount of multimodal data, including images, text descriptions, and soil information. This data not only contains rich information about the agricultural environment but also embodies the spatiotemporal dependencies within the agricultural system, potentially exposing farmers' planting habits, crop growth patterns, and sensitive farm information. For example, by fusing images of multiple batches of agricultural products from a farm with soil testing data, attackers can infer soil acidification and irrigation patterns, thereby inferring agricultural environmental information. By combining agricultural product growth cycle data with agricultural machinery operation logs, attackers can reverse engineer the patented growth technology and the locations of undisclosed sensitive plots on the farm, threatening the farm's commercial secrets. Cross-correlating multimodal agricultural product data amplifies the potential value of individual data, turning previously fragmented information into a vulnerability for privacy leaks. Therefore, ensuring the privacy of multimodal agricultural product data while performing agricultural product anomaly detection has become a critical and pressing challenge.

[0004] Existing privacy protection technologies include traditional differential privacy methods, homomorphic encryption methods, and data anonymization methods. However, traditional privacy protection methods face numerous challenges for large-scale multimodal agricultural product data. While traditional differential privacy methods can effectively protect data privacy, they fail to properly distribute noise based on data importance, making it difficult to balance the accuracy of agricultural product anomaly detection and privacy protection. For example, in pest and disease detection scenarios, the high-frequency texture features of agricultural product images are crucial for model recognition, but images may contain non-target sensitive information, such as the facial features of farm workers and landmarks. Traditional differential privacy, by applying global noise, results in the loss of edge details in anomaly detection, affecting detection accuracy. The addition of noise to highly sensitive features is insufficient, making it ineffective in protecting sensitive information. Although homomorphic encryption technology can ensure the computational security of data in an encrypted state, its computational efficiency and functional limitations make it difficult to meet the needs of multimodal agricultural product data analysis. For example, agricultural product anomaly detection requires cross-modal joint analysis of agricultural product images and text logs. However, homomorphic encryption cannot support complex multimodal data fusion operations within the encrypted domain. Furthermore, when processing large amounts of agricultural product data, encryption latency can lead to decision lags, making it difficult to meet the timeliness requirements of anomaly detection. While data anonymization methods can, to a certain extent, protect personal identity information from being exposed during data sharing and analysis, they cannot effectively prevent linking attacks when processing complex, multidimensional agricultural product data. For example, by analyzing agricultural product growth stage data and combining it with the publicly available gene expression cycle of a variety, an attacker could infer the undisclosed cultivation techniques used by the farm. Therefore, maintaining the high availability and accuracy of multimodal agricultural product data while protecting privacy has become a key challenge in the current application of this technology. Summary of the Invention

[0005] The present invention is made in view of the above problems, and its purpose is to provide an anomaly detection method and device for multimodal agricultural product data privacy protection, a multimodal agricultural product data adaptive fusion method based on meta-learning cross-modal attention mechanism, which enhances the dynamic correlation and adaptive fusion of multimodal features; a hierarchical adaptive differential privacy protection method based on graph attention network, which adds adaptive noise to multimodal fusion features through cross-level and intra-layer sensitivity; a knowledge distillation pre-training privacy protection method based on modal under-optimization dynamic weighting, which dynamically adjusts the loss weight of weak modalities and provides dynamic noise protection for the adaptively pruned local student model gradient; a multimodal agricultural product anomaly detection method based on self-supervised denoising encoder, which achieves high-precision agricultural product anomaly detection.

[0006] Specifically, a first aspect of the present invention provides an anomaly detection method for multimodal agricultural product data privacy protection, comprising the following steps:

[0007] Step 1: A multimodal adaptive fusion method based on a meta-learning cross-modal attention mechanism is used to mine the complex dynamic correlations between text and image features and adaptively fuse multimodal agricultural product data features.

[0008] Step 2: A hierarchical adaptive differential privacy protection method based on a graph attention network is used to construct a hypergeometric spherical graph structure for different detection tasks based on the obtained fusion features. Noise is adaptively generated based on the cross-layer and intra-layer sensitivity of the task nodes to effectively protect the privacy of the multimodal fusion features.

[0009] Step 3: Based on the knowledge distillation pre-training privacy protection method with dynamic weighting of modality under-optimization, the Shapley value is calculated to evaluate the degree of modality under-optimization, the loss weight of the weak modality is dynamically adjusted, and the adaptive gradient clipping differential privacy method is used to implement dynamic perturbation protection of the model gradient;

[0010] Step 4: A multimodal agricultural product anomaly detection method based on a self-supervised denoising encoder is used to denoise the pre-trained multimodal perturbation fusion data to achieve high-precision agricultural product anomaly detection.

[0011] Furthermore, the step one includes: extracting text and image features respectively through a text encoder and an image encoder, performing preliminary fusion using a cross-modal attention mechanism, and mining the dynamic correlation between multiple modalities through a meta-learning method to complete multimodal adaptive fusion for each different task type.

[0012] Furthermore, the step 1 specifically includes the following steps:

[0013] Step 1.1: Use the BERT pre-trained language model as the text encoder and the image encoder in CLIP to improve the model's focus on important image areas. Convert the image data into a vector form combined with the text representation. The formula is as follows:

[0014] ;

[0015] ;

[0016] ;

[0017] ;

[0018] in: Input text to the BERT model The final output representation generated contains information from multiple layers;

[0019] is the number of layers of the BERT model;

[0020] For the The weighting coefficient of the layer;

[0021] For the BERT model The output feature matrix of the layer;

[0022] For input text;

[0023] is the average pooling operation;

[0024] is the learnable weight vector;

[0025] is the normalization operation;

[0026] The final output representation generated by the CLIP model for the input image can adjust the contribution according to the importance of each image patch;

[0027] is the total number of image blocks;

[0028] For the The weighting coefficient of each image block;

[0029] They are query vector, key vector, and value vector respectively;

[0030] is the dimension size of the key vector;

[0031] Step 1.2: Utilize the key-value pair interaction of the cross-modal attention mechanism to enhance the complex association between text and image features and complete the preliminary multimodal agricultural product feature fusion. The formula is as follows:

[0032] ;

[0033] in: is the feature after preliminary fusion;

[0034] It is a cross-modal attention mechanism;

[0035] Input text to the BERT model The final output representation generated contains information from multiple layers;

[0036] The final output representation generated by the CLIP model for the input image can adjust the contribution according to the importance of each image patch;

[0037] Step 1.3: Use the multi-layer perceptron (MLP) to extract low-dimensional meta-information from the preliminary fusion features. Based on the task-dependent meta-information, adaptively calculate the dynamic weight matrix of different detection tasks. Combined with the static weight matrix, the multimodal agricultural product fusion matrix is ​​obtained. The formula is as follows:

[0038] Task extraction process:

[0039] ;

[0040] ;

[0041] ;

[0042] in: is the input layer of MLP;

[0043] is the input of MLP;

[0044] is the feature after preliminary fusion;

[0045] is the hidden layer of MLP;

[0046] is the activation function;

[0047] is the learnable weight matrix of the hidden layer;

[0048] is layer normalization;

[0049] is the bias vector of the hidden layer;

[0050] For the final extracted task The meta-information vector of

[0051] is the learnable weight matrix for the final extraction process;

[0052] is the bias vector of the final extraction process;

[0053] The process of obtaining task-specific dynamic weight matrix:

[0054] ;

[0055] ;

[0056] ;

[0057] in: They are query vector, key vector, and value vector respectively;

[0058] is the query matrix;

[0059] is the bond matrix;

[0060] is the value matrix;

[0061] For the final extracted task The meta-information vector of

[0062] is the attention matrix;

[0063] is a task-specific dynamic weight matrix;

[0064] To realize 1D vector by multiplying 3D tensors Mapping to 2D weight matrix;

[0065] The process of obtaining the multimodal agricultural product fusion matrix:

[0066] ;

[0067] ;

[0068] in: is the final multimodal fusion matrix;

[0069] is the static weight matrix;

[0070] is a task-specific dynamic weight matrix;

[0071] is the final multimodal fusion representation;

[0072] is the feature after preliminary fusion;

[0073] Furthermore, the step 2 includes the following steps:

[0074] Step 2.1: Consider each different detection task as a node in the graph, and the similarity between different tasks as an edge. Construct the graph structure. The formula is as follows:

[0075] ;

[0076] ;

[0077] ;

[0078] in: is a set of nodes;

[0079] For the Task nodes;

[0080] For computing nodes and The cosine similarity between

[0081] is the judgment threshold;

[0082] For the edge;

[0083] Step 2.2: Use a graph convolutional network to capture the relationship between nodes and neighboring nodes in the graph, embed the graph nodes into a sphere in the hypergeometric space, capture the hierarchical relationship in the graph structure, and optimize the graph structure embedding on the sphere by minimizing the loss function based on the hypergeometric distance. The formula is as follows:

[0084] ;

[0085] ;

[0086] in: is the embedding representation of all nodes in the graph;

[0087] is the set of embedding vectors corresponding to each node in the node set;

[0088] It is a graph convolutional network;

[0089] are the parameters of the graph convolutional network;

[0090] Represents each embedding vector Located on a unit hypersphere Inside;

[0091] is the dimension of the hypergeometric space;

[0092] are the parameters of the ball model;

[0093] are the optimized model parameters;

[0094] To minimize the loss function;

[0095] is the regularization parameter;

[0096] Indicates that when the following conditions are met, the parameters are met Each element in The paradigm is smaller than ;

[0097] For elements paradigm;

[0098] is a constant negative curvature;

[0099] Step 2.3: Calculate the cross-layer sensitivity and intra-layer sensitivity by obtaining the radius and angle of the node on the sphere. The formula is as follows:

[0100] The calculation process of cross-level sensitivity:

[0101] ;

[0102] ;

[0103] in: For the node The radius on the sphere;

[0104] The radius is n-dimensional sphere;

[0105] is the inverse hyperbolic tangent function;

[0106] is the Euclidean distance;

[0107] The Euclidean distance is retained additionally;

[0108] is the reference point;

[0109] is the weight coefficient;

[0110] for cross-level sensitivity;

[0111] Represents two adjacent node sets;

[0112] Representative Node The hypergeometric embedding vector of ;

[0113] The calculation process of intra-layer sensitivity:

[0114] ;

[0115] ;

[0116] ;

[0117] in: For nodes and nodes The angle between

[0118] For nodes and nodes The measurement between

[0119] For nodes self-measurement;

[0120] For nodes self-measurement;

[0121] For nodes and nodes The inner product of the embedding vector of

[0122] For nodes The norm of the embedding vector of ;

[0123] For nodes The norm of the embedding vector of ;

[0124] is the ternary loss function;

[0125] Anchor and positive sample Angle;

[0126] Anchor and negative samples Angle;

[0127] is the threshold parameter;

[0128] is the intra-layer sensitivity;

[0129] Step 2.4: Extend the Gaussian mechanism in Euclidean space to hypergeometric space, use the graph attention network to further consider the importance and similarity of task nodes at different levels, and adaptively generate cross-level noise and intra-level noise based on cross-level sensitivity and intra-level sensitivity to provide noise disturbance protection for multimodal agricultural product fusion data. The formula is as follows:

[0130] ;

[0131] ;

[0132] ;

[0133] in: Represents the point in the hypergeometric space Mapped to a reference point In the tangent space with the origin as

[0134] As a reference point is the tangent space at the origin;

[0135] It is a Gaussian noise mechanism;

[0136] represents the mean of the Gaussian distribution;

[0137] is the noise standard deviation;

[0138] is the identity matrix;

[0139] is the multi-scale normalized mapping function;

[0140] is the reference point calculated based on the Euclidean metric Arrive distance;

[0141] Indicates the use of hyperbolic sine function to transform distance into hypergeometric space metric;

[0142] is the number of scales, and the distance of each scale is adjusted using a different power;

[0143] ;

[0144] ;

[0145] ;

[0146] ;

[0147] in: For nodes and nodes The graph attention network weights between them;

[0148] is an exponential function;

[0149] is the activation function;

[0150] is a learnable linear transformation matrix;

[0151] is the learnable weight vector;

[0152] Representation node The eigenvector of

[0153] Representation node The eigenvector of

[0154] Representation node The eigenvector of

[0155] Representation node Neighbor nodes of

[0156] For nodes The set of neighbor nodes of

[0157] is cross-layer noise;

[0158] represents the normal distribution;

[0159] is the noise within the layer;

[0160] is the total privacy budget;

[0161] For the fusion characteristics after protection;

[0162] is the ratio parameter of cross-layer noise and intra-layer noise;

[0163] Furthermore, the nodes in the graph structure are represented by multimodal fusion vectors of corresponding tasks.

[0164] Furthermore, the edges in the graph structure are represented by calculating the similarity between task nodes using cosine similarity.

[0165] Furthermore, the step three includes the following steps:

[0166] The technical means used in step three include: through knowledge distillation, the student model learns the knowledge of the teacher model; based on the degree of modal under-optimization obtained by Shapley value evaluation, the loss weight of the weak mode is calculated to improve the pre-training accuracy of the local student model; the pre-training model gradient is adaptively clipped, and the clipped model gradient is provided with dynamic Gaussian noise protection.

[0167] Step 3.1: Deploy two unimodal teacher models on the cloud server and a multimodal student model on the edge layer;

[0168] Step 3.2: Set a triplet based on the knowledge distillation method, calculate the difference between the anomaly detection prediction probabilities of the teacher model and the student model, and optimize the pre-training learning process of the local student model through the KL divergence loss function. The formula is as follows:

[0169] ;

[0170] ;

[0171] ;

[0172] in: For modal The teacher model is used to train the data sample and The difference in predicted probabilities;

[0173] For modal The student model is good at predicting the data sample and The difference in predicted probabilities;

[0174] represents the predicted probability of the teacher model;

[0175] represents the predicted probability of the student model;

[0176] represents the KL divergence loss function;

[0177] represents the logarithmic function;

[0178] Represents agricultural products, Represents data samples , Represents data samples ;

[0179] is a set of triples, 、 、 constitute;

[0180] Step 3.3: Calculate the Shapley value based on the prediction accuracy of the local student model, evaluate the degree of under-optimization of each modality, calculate the loss weight of each modality, optimize the overall loss function of the local model, and improve the accuracy of the pre-trained model. The formula is as follows:

[0181] ;

[0182] ;

[0183] ;

[0184] ;

[0185] in: Prediction accuracy for the local student model;

[0186] is the data batch;

[0187] is the batch size;

[0188] For the The true labels of samples;

[0189] For the The mode of the samples Input features of

[0190] For modal Feature extraction function;

[0191] For modal Feature extraction function;

[0192] It is a cross-modal feature consistency module;

[0193] It is a simple multimodal fusion function;

[0194] [·] is the sigmoid activation function;

[0195] is the indicator function;

[0196] is the dynamic threshold;

[0197] is the minimum threshold;

[0198] is the maximum threshold;

[0199] is the current training round;

[0200] is the total number of training rounds;

[0201] is the Shapley value;

[0202] , indicating that in the modal If it already exists, introduce the modal The marginal contribution calculated later;

[0203] is the set of all modes;

[0204] The set of all modes does not contain A subset of

[0205] represents the total number of all modes;

[0206] is the normalization factor;

[0207] ;

[0208] ;

[0209] ;

[0210] in: For modal Degree of suboptimization;

[0211] is the Shapley value;

[0212] is the data batch;

[0213] is the batch size;

[0214] For the The true labels of samples;

[0215] is the indicator function;

[0216] For the The mode of the samples Input features of

[0217] For modal Feature extraction function;

[0218] For modal Feature extraction function;

[0219] It is a cross-modal feature consistency module;

[0220] It is a simple multimodal fusion function;

[0221] [·] is the sigmoid activation function;

[0222] For modal The loss function Relative to the sample The local gradient of

[0223] For modal The loss weight of

[0224] is the smoothing factor;

[0225] is the total optimization loss function of the local student model;

[0226] is the loss ratio parameter;

[0227] Based on the two-class loss function;

[0228] is the set of all modes;

[0229] For modal Degree of suboptimization;

[0230] represents the KL divergence loss function;

[0231] Step 3.4: Set the dynamic clipping threshold to adaptively clip the student model gradient. The formula is as follows:

[0232] ;

[0233] ;

[0234] in: is the clipping threshold after attenuation;

[0235] is the attenuation factor;

[0236] is the clipping threshold of the previous round;

[0237] Represents dynamic clipping gradient;

[0238] is the gradient of the tth round;

[0239] is the two-norm of the gradient of the t-th round;

[0240] is the minimum value function;

[0241] Step 3.5: Based on the dynamic noise scale factor, dynamically sample Gaussian noise and add dynamic noise to the clipped model gradient. The formula is as follows:

[0242] ;

[0243] ;

[0244] in: is the noise scaling factor;

[0245] is the initial noise scaling factor;

[0246] is the current training round;

[0247] is the total number of training rounds;

[0248] is the attenuation factor;

[0249] is Gaussian noise;

[0250] is the sample batch;

[0251] represents the normal distribution;

[0252] is the identity matrix;

[0253] is the clipping threshold after attenuation;

[0254] ;

[0255] ;

[0256] in: is the average clipped gradient;

[0257] is the sample batch size;

[0258] Represents dynamic clipping gradient;

[0259] is the gradient after adding noise;

[0260] is Gaussian noise;

[0261] Furthermore, the step 4 includes the following steps:

[0262] Step 4.1: The cloud server averages and aggregates the perturbed student model gradients, sends the teacher model parameters to the edge layer student model for update, and uses the momentum mechanism to improve the efficiency of local student model update. The formula is as follows:

[0263] ;

[0264] ;

[0265] ;

[0266] ;

[0267] in: is the momentum of the current round;

[0268] is the momentum coefficient;

[0269] is the momentum of the previous round;

[0270] is the number of edge nodes;

[0271] is the set of edge nodes participating in the calculation in round t;

[0272] is the gradient after adding noise;

[0273] is the updated global model parameter;

[0274] is the current global model parameter;

[0275] is the learning rate;

[0276] is the minimum value function;

[0277] is the standard deviation of the gradient;

[0278] is the Lipschitz constant of the gradient;

[0279] The latency of edge nodes participating in training is dynamically estimated by exponential averaging;

[0280] is the attenuation factor;

[0281] is the clipping threshold after attenuation;

[0282] is the logarithmic time decay term;

[0283] is the scale parameter;

[0284] The latency of the edge nodes participating in the training in the previous round is dynamically estimated by exponential averaging;

[0285] is the current observation delay;

[0286] Step 4.2: A denoising encoder based on residual learning is used to denoise the perturbed multimodal fusion features to improve anomaly detection accuracy.

[0287] ;

[0288] ;

[0289] in: is the noise residual;

[0290] It is a 1×1 convolution operation;

[0291] It is a 3×3 convolution operation;

[0292] is the activation function;

[0293] For the fusion characteristics after protection;

[0294] is the multimodal fusion feature after noise reduction;

[0295] Step 4.3: Use the self-supervised learning method to reconstruct the image features of the multimodal agricultural product fusion features after noise reduction, calculate the reconstruction error, and use the error threshold to determine whether the reconstructed agricultural product image area is abnormal. The formula is as follows:

[0296] ;

[0297] ;

[0298] in: is the reconstruction error;

[0299] To reconstruct image features;

[0300] is the multimodal fusion feature after noise reduction;

[0301] is the square of the norm;

[0302] is the regional anomaly map;

[0303] is the error threshold;

[0304] The second aspect of the present invention provides an anomaly detection device for multimodal agricultural product data privacy protection, comprising a multimodal adaptive fusion module of a meta-learning cross-modal attention mechanism, a hierarchical adaptive differential privacy protection module of a graph attention network, a modality under-optimization dynamic weighted knowledge distillation pre-training privacy protection module, a multimodal agricultural product anomaly detection module of a self-supervised denoising encoder, a personal mobile device, an edge node, and a cloud server;

[0305] The multimodal adaptive fusion module of the meta-learning cross-modal attention mechanism uses the cross-modal attention mechanism to perform preliminary fusion of text and image features extracted by the text and image encoders respectively, extracts meta-information through a multi-layer perceptron, obtains a multimodal agricultural product fusion matrix, and completes the adaptive fusion of multimodal features for each different task type;

[0306] The hierarchical adaptive differential privacy protection module of the graph attention network captures the associations between different tasks by constructing a graph structure, embedding graph nodes into a sphere in a hypergeometric space, capturing the hierarchical relationships in the graph structure, calculating cross-layer sensitivity and intra-layer sensitivity, and using the graph attention network to further consider the importance and similarity of task nodes at different levels. It generates noise through a designed hypergeometric Gaussian mechanism to adaptively protect tasks of different importance.

[0307] The modality under-optimization dynamic weighted knowledge distillation pre-training privacy protection module deploys two teacher models and one student model on the cloud server and edge layer respectively, performs knowledge distillation, calculates the Shapley value, evaluates the degree of modality under-optimization, obtains weak modality loss weights, optimizes the overall loss function of the local model, sets a dynamic clipping threshold, clips the student model gradient, designs a dynamic noise scaling factor, samples Gaussian noise, and adds noise to the model gradient;

[0308] In the multimodal agricultural product anomaly detection module of the self-supervised denoising encoder, the cloud server averages and aggregates the student model gradients, sends the teacher model parameters to the student model at the edge layer for updating, adopts the momentum mechanism to improve the model update efficiency, and uses the residual learning-based denoising encoder to denoise the multimodal fusion features after adding noise. The self-supervised learning method is used to detect anomalies on the denoised multimodal agricultural product fusion features.

[0309] The personal mobile device is used to fuse multimodal agricultural product features using a multimodal adaptive fusion method based on a meta-learning cross-modal attention mechanism, and to protect the privacy of the fused multimodal features using a hierarchical adaptive differential privacy protection method based on a graph attention network;

[0310] The edge node is used to pre-train the model of multimodal agricultural product fusion data uploaded by personal mobile devices, and to clip and add noise protection to the model gradient, and upload the disturbed gradient to the cloud.

[0311] The cloud server is used to aggregate all global perturbation gradients to perform global parameter optimization, and return the optimized parameters to each edge node to update the local student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0312] In order to more clearly illustrate the embodiments of the present drawings or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present drawings. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0313] Figure 1 A flowchart of the steps of an anomaly detection method for multimodal agricultural product data privacy protection of the present invention;

[0314] Figure 2 This is a comparison chart of the accuracy of the embodiment of the present invention and the traditional agricultural product anomaly detection method under different total privacy budgets;

[0315] Figure 3This is a comparison chart of the accuracy of the embodiment of the present invention and the traditional agricultural product anomaly detection method under different loss ratio parameters;

[0316] Figure 4 This is a comparison chart of the accuracy of the embodiment of the present invention and the traditional agricultural product anomaly detection method under different gradient smoothing coefficients;

[0317] Figure 5 This is a comparison chart of the accuracy of the embodiment of the present invention and the traditional agricultural product anomaly detection method under different initial clipping thresholds;

[0318] The purpose, features and advantages of this drawing will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0319] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0320] Obviously, the drawings described below are merely examples or embodiments of the present invention. Those skilled in the art can apply the present invention to other similar scenarios based on these drawings without inventive effort. Furthermore, it is understood that while the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the disclosure of the present invention, any design, manufacturing, or production changes based on the technical content disclosed in the present invention are merely conventional technical means and should not be construed as an inadequacy of the disclosure of the present invention.

[0321] Unless otherwise specified, all embodiments and optional embodiments of the present invention can be combined with each other to form new technical solutions.

[0322] Unless otherwise specified, all technical features and optional technical features of the present invention can be combined with each other to form a new technical solution.

[0323] Unless otherwise specified, all steps of the present invention may be performed sequentially or randomly, preferably sequentially. For example, the method includes steps (a) and (b), which means that the method may include steps (a) and (b) performed sequentially, or may include steps (b) and (a) performed sequentially. For example, the method may further include step (c), which means that step (c) may be added to the method in any order, for example, the method may include steps (a), (b) and (c), or may include steps (a), (c) and (b), or may include steps (c), (a) and (b), etc.

[0324] Unless otherwise specified, the terms "include" and "comprising" used in the present invention may be open-ended or closed-ended. For example, "include" and "comprising" may mean that other components not listed may also be included or that only the listed components are included.

[0325] Unless otherwise specified, the term "or" is inclusive in this disclosure. For example, the phrase "A or B" means "A, B, or both A and B." More specifically, the condition "A or B" is satisfied if any of the following conditions are met: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); or both A and B are true (or exist).

[0326] In order to better understand the solutions of the embodiments of the present invention, some relevant terms and concepts that may be involved in the embodiments of the present invention are first introduced below.

[0327] (1) Graph Attention Network (GAT) is a deep learning model based on graph-structured data that learns the relationships and feature representations between nodes through a self-attention mechanism. Unlike traditional graph convolutional networks (GCNs), GAT introduces learnable attention weights between the neighboring nodes of each node, allowing the model to automatically assign different weights to each neighboring node, thereby learning the relationships between nodes more accurately. GAT first calculates an attention coefficient for each node's neighboring nodes. The model then weights and sums the features of the neighboring nodes based on these coefficients and obtains the updated node representation through a nonlinear transformation. GAT not only makes full use of neighbor information, but also effectively solves the problem of importance differences between nodes, thereby improving the learning effect of graph data.

[0328] (2) Knowledge distillation is a model compression technique that aims to improve the performance of a student model by transferring the knowledge of a complex teacher model to a simpler student model. The teacher model is usually a large deep neural network that can handle complex tasks, while the student model is a relatively small and computationally efficient network. The core idea of ​​knowledge distillation is to guide the training of the student model through the prediction results of the teacher model. By minimizing the difference between the output of the student model and the output of the teacher model, the student model can achieve performance close to that of the teacher model with fewer parameters and computational overhead. Knowledge distillation is widely used in model compression, transfer learning, and improving the inference speed of deep learning models.

[0329] In this embodiment, Figure 1 As shown, a method for anomaly detection with privacy protection for multimodal agricultural product data includes the following steps:

[0330] Step 1: A multimodal adaptive fusion method based on a meta-learning cross-modal attention mechanism is used to mine the complex dynamic correlations between text and image features and adaptively fuse multimodal agricultural product data features.

[0331] Step 2: A hierarchical adaptive differential privacy protection method based on a graph attention network is used to construct a hypergeometric spherical graph structure for different detection tasks based on the obtained fusion features. Noise is adaptively generated based on the cross-layer and intra-layer sensitivity of the task nodes to effectively protect the privacy of the multimodal fusion features.

[0332] Step 3: Based on the knowledge distillation pre-training privacy protection method with dynamic weighting of modality under-optimization, the Shapley value is calculated to evaluate the degree of modality under-optimization, the loss weight of the weak modality is dynamically adjusted, and the adaptive gradient clipping differential privacy method is used to implement dynamic perturbation protection of the model gradient;

[0333] Step 4: A multimodal agricultural product anomaly detection method based on a self-supervised denoising encoder is used to denoise the pre-trained multimodal perturbation fusion data to achieve high-precision agricultural product anomaly detection.

[0334] Furthermore, step one includes: extracting text and image features respectively through a text encoder and an image encoder, performing preliminary fusion using a cross-modal attention mechanism, and mining the dynamic correlation between multiple modalities through a meta-learning method to complete multimodal adaptive fusion for each different task type.

[0335] Furthermore, step one specifically includes the following steps:

[0336] Step 1.1: Use the BERT pre-trained language model as the text encoder and the image encoder in CLIP to improve the model's focus on important image areas. Convert the image data into a vector form combined with the text representation. The formula is as follows:

[0337] ;

[0338] ;

[0339] ;

[0340] ;

[0341] Step 1.2: Utilize the key-value pair interaction of the cross-modal attention mechanism to enhance the complex association between text and image features and complete the preliminary multimodal agricultural product feature fusion. The formula is as follows:

[0342] ;

[0343] Step 1.3: Use the multi-layer perceptron (MLP) to extract low-dimensional meta-information from the preliminary fusion features. Based on the task-dependent meta-information, adaptively calculate the dynamic weight matrix of different detection tasks. Combined with the static weight matrix, the multimodal agricultural product fusion matrix is ​​obtained. The formula is as follows:

[0344] Task extraction process:

[0345] ;

[0346] ;

[0347] ;

[0348] The process of obtaining task-specific dynamic weight matrix:

[0349] ;

[0350] ;

[0351] ;

[0352] The process of obtaining the multimodal agricultural product fusion matrix:

[0353] ;

[0354] ;

[0355] Furthermore, step 2 includes the following steps:

[0356] Step 2.1: Consider each different detection task as a node in the graph, and the similarity between different tasks as an edge. Construct the graph structure. The formula is as follows:

[0357] ;

[0358] ;

[0359] ;

[0360] Step 2.2: Use a graph convolutional network to capture the relationship between nodes and neighboring nodes in the graph, embed the graph nodes into a sphere in the hypergeometric space, capture the hierarchical relationship in the graph structure, and optimize the graph structure embedding on the sphere by minimizing the loss function based on the hypergeometric distance. The formula is as follows:

[0361] ;

[0362] ;

[0363] Step 2.3: Calculate the cross-layer sensitivity and intra-layer sensitivity by obtaining the radius and angle of the node on the sphere. The formula is as follows:

[0364] The calculation process of cross-level sensitivity:

[0365] ;

[0366] ;

[0367] The calculation process of intra-layer sensitivity:

[0368] ;

[0369] ;

[0370] ;

[0371] Step 2.4: Extend the Gaussian mechanism in Euclidean space to hypergeometric space, use the graph attention network to further consider the importance and similarity of task nodes at different levels, and adaptively generate cross-level noise and intra-level noise based on cross-level sensitivity and intra-level sensitivity to provide noise disturbance protection for multimodal agricultural product fusion data. The formula is as follows:

[0372] ;

[0373] ;

[0374] ;

[0375] ;

[0376] ;

[0377] ;

[0378] ;

[0379] Furthermore, the nodes in the graph structure are represented by multimodal fusion vectors of the corresponding tasks.

[0380] Furthermore, the edges in the graph structure are represented by calculating the similarity between task nodes using cosine similarity.

[0381] Furthermore, step three includes the following steps:

[0382] The technical means used in step three include: through knowledge distillation, the student model learns the knowledge of the teacher model; based on the degree of modal under-optimization obtained by Shapley value evaluation, the loss weight of the weak mode is calculated to improve the pre-training accuracy of the local student model; the pre-training model gradient is adaptively clipped, and the clipped model gradient is provided with dynamic Gaussian noise protection.

[0383] Step 3.1: Deploy two unimodal teacher models on the cloud server and a multimodal student model on the edge layer;

[0384] Step 3.2: Set a triplet based on the knowledge distillation method, calculate the difference between the anomaly detection prediction probabilities of the teacher model and the student model, and optimize the pre-training learning process of the local student model through the KL divergence loss function. The formula is as follows:

[0385] ;

[0386] ;

[0387] ;

[0388] Step 3.3: Calculate the Shapley value based on the prediction accuracy of the local student model, evaluate the degree of under-optimization of each modality, calculate the loss weight of each modality, optimize the overall loss function of the local model, and improve the accuracy of the pre-trained model. The formula is as follows:

[0389] ;

[0390] ;

[0391] ;

[0392] ;

[0393] ;

[0394] ;

[0395] ;

[0396] Step 3.4: Set the dynamic clipping threshold to adaptively clip the student model gradient. The formula is as follows:

[0397] ;

[0398] ;

[0399] Step 3.5: Based on the dynamic noise scale factor, dynamically sample Gaussian noise and add dynamic noise to the clipped model gradient. The formula is as follows:

[0400] ;

[0401] ;

[0402] ;

[0403] ;

[0404] Furthermore, step four includes the following steps:

[0405] Step 4.1: The cloud server averages and aggregates the perturbed student model gradients, sends the teacher model parameters to the edge layer student model for update, and uses the momentum mechanism to improve the efficiency of local student model update. The formula is as follows:

[0406] ;

[0407] ;

[0408] ;

[0409] ;

[0410] Step 4.2: A denoising encoder based on residual learning is used to denoise the perturbed multimodal fusion features to improve anomaly detection accuracy.

[0411] ;

[0412] ;

[0413] Step 4.3: Use the self-supervised learning method to reconstruct the image features of the multimodal agricultural product fusion features after noise reduction, calculate the reconstruction error, and use the error threshold to determine whether the reconstructed agricultural product image area is abnormal. The formula is as follows:

[0414] ;

[0415] ;

[0416] The second aspect of the present invention provides an anomaly detection device for multimodal agricultural product data privacy protection, comprising a multimodal adaptive fusion module of a meta-learning cross-modal attention mechanism, a hierarchical adaptive differential privacy protection module of a graph attention network, a modality under-optimization dynamic weighted knowledge distillation pre-training privacy protection module, a multimodal agricultural product anomaly detection module of a self-supervised denoising encoder, a personal mobile device, an edge node, and a cloud server;

[0417] The multimodal adaptive fusion module of the meta-learning cross-modal attention mechanism uses the cross-modal attention mechanism to perform preliminary fusion of text and image features extracted by the text and image encoders respectively, extracts meta-information through a multi-layer perceptron, obtains a multimodal agricultural product fusion matrix, and completes the adaptive fusion of multimodal features for each different task type;

[0418] The hierarchical adaptive differential privacy protection module of the graph attention network captures the associations between different tasks by constructing a graph structure, embedding graph nodes into a sphere in a hypergeometric space, capturing the hierarchical relationships in the graph structure, calculating cross-layer sensitivity and intra-layer sensitivity, and using the graph attention network to further consider the importance and similarity of task nodes at different levels. It generates noise through a designed hypergeometric Gaussian mechanism to adaptively protect tasks of different importance.

[0419] The modality under-optimization dynamic weighted knowledge distillation pre-training privacy protection module deploys two teacher models and one student model on the cloud server and edge layer respectively, performs knowledge distillation, calculates the Shapley value, evaluates the degree of modality under-optimization, obtains weak modality loss weights, optimizes the overall loss function of the local model, sets a dynamic clipping threshold, clips the student model gradient, designs a dynamic noise scaling factor, samples Gaussian noise, and adds noise to the model gradient;

[0420] In the multimodal agricultural product anomaly detection module of the self-supervised denoising encoder, the cloud server averages and aggregates the student model gradients, sends the teacher model parameters to the student model at the edge layer for updating, adopts the momentum mechanism to improve the model update efficiency, and uses the residual learning-based denoising encoder to denoise the multimodal fusion features after adding noise. The self-supervised learning method is used to detect anomalies on the denoised multimodal agricultural product fusion features.

[0421] The personal mobile device is used to fuse multimodal agricultural product features using a multimodal adaptive fusion method based on a meta-learning cross-modal attention mechanism, and to protect the privacy of the fused multimodal features using a hierarchical adaptive differential privacy protection method based on a graph attention network;

[0422] The edge node is used to pre-train the model of multimodal agricultural product fusion data uploaded by personal mobile devices, and to clip and add noise protection to the model gradient, and upload the disturbed gradient to the cloud.

[0423] The cloud server is used to aggregate all global perturbation gradients to perform global parameter optimization, and return the optimized parameters to each edge node to update the local student model.

[0424] In order to ensure the privacy of multimodal agricultural product data and model gradients, it is theoretically proved that the anomaly detection method for privacy protection of multimodal agricultural product data proposed in this paper meets the requirements of differential privacy.

[0425] Proof: In the hierarchical adaptive differential privacy protection method of the graph attention network, the noise injected into the fusion feature satisfies the Gaussian distribution extended in the hypergeometric space, where the privacy budget is dynamically allocated by the ratio of cross-level and intra-level sensitivity, satisfying The Gaussian mechanism conditions of differential privacy realize hierarchical privacy protection of multimodal agricultural product fusion features.

[0426] In the modality-underoptimized dynamic weighted knowledge distillation pre-training privacy protection method, according to the dynamic noise scaling factor Control the intensity of the added noise. According to the differential privacy property of the Gaussian mechanism, when the noise standard deviation meets

[0427] ;

[0428] in: is the noise standard deviation;

[0429] is the dynamic noise scaling factor;

[0430] is the clipping threshold after attenuation;

[0431] is the sample batch;

[0432] When , the adaptive gradient clipping differential privacy method of this embodiment satisfies Differential privacy.

[0433] Based on the serial combination principle of differential privacy, the anomaly detection method for multimodal agricultural product data privacy protection proposed in this paper meets the requirements of Differential privacy, where , achieving privacy protection for local multimodal fusion embedding and model gradients.

[0434] In this example, based on real agricultural product datasets (Agri-vision, PlantVillage), different parameters are used: total privacy budget , loss ratio parameter , gradient smoothing coefficient and initial clipping bounds To evaluate the feasibility of the present invention for privacy protection of disturbed agricultural product data, the results of the comparative experiment are as follows: Figures 2 to 5 As shown, Figure 2 This is a comparison chart of the accuracy of the embodiment of the present invention and the traditional agricultural product anomaly detection method under different total privacy budgets; Figure 3 This is a comparison chart of the accuracy of the embodiment of the present invention and the traditional agricultural product anomaly detection method under different loss ratio parameters; Figure 4 This is a comparison chart of the accuracy of the embodiment of the present invention and the traditional agricultural product anomaly detection method under different gradient smoothing coefficients; Figure 5 The figure is a comparison chart of the accuracy of the embodiment of the present invention and the traditional agricultural product anomaly detection method under different initial clipping thresholds.

[0435] from Figures 2 to 5 It can be seen that the anomaly detection accuracy of this embodiment under different parameters is better than that of the traditional method. From the results of the various comparison graphs and the above-mentioned privacy protection proof, it can be seen that the anomaly detection method and device for multimodal agricultural product data privacy protection provided by the present invention can achieve high-precision agricultural product anomaly detection while ensuring the privacy protection of multimodal agricultural product data.

[0436] It should be noted that the present invention is not limited to the above-mentioned embodiments. The above-mentioned embodiments are merely examples, and any embodiments having substantially the same structure and effect as the technical concept within the scope of the technical solution of the present invention are all included in the technical scope of the present invention. In addition, without departing from the scope of the present invention, other embodiments that can be conceived by those skilled in the art and that combine some of the constituent elements in the embodiments are also included in the scope of the present invention.

Claims

1. A method for anomaly detection in multimodal agricultural product data privacy protection, characterized in that: The following steps are involved: Step 1: A multimodal adaptive fusion method based on meta-learning cross-modal attention mechanism is used to adaptively fuse multimodal agricultural product data features; Step 2: Adaptively generate noise based on the obtained fusion features, including: Step 2.1: Consider each different detection task as a node in the graph, and the similarity between different tasks as an edge to construct the graph structure; Step 2.2: Use a graph convolutional network to capture the relationship between nodes and their neighbors in the graph, embed the graph nodes into a sphere in a hypergeometric space, capture the hierarchical relationship in the graph structure, and optimize the graph structure embedding on the sphere by minimizing a loss function based on hypergeometric distance; Step 2.3: Calculate cross-layer sensitivity and intra-layer sensitivity by obtaining the radius and angle of the node on the sphere; Step 2.4: Extend the Gaussian mechanism in Euclidean space to hypergeometric space, use the graph attention network to further consider the importance and similarity of task nodes at different levels, and adaptively generate cross-level noise and intra-level noise based on cross-level sensitivity and intra-level sensitivity to provide noise disturbance protection for multimodal agricultural product fusion data; Step 3: Implement dynamic perturbation protection for model gradients, including: Step 3.1: Deploy two unimodal teacher models on the cloud server and a multimodal student model on the edge layer; Step 3.2: Set a triplet based on the knowledge distillation method, calculate the difference between the anomaly detection prediction probabilities of the teacher model and the student model, and optimize the pre-training learning process of the local student model using the KL divergence loss function; Step 3.3: Calculate the Shapley value based on the prediction accuracy of the local student model, evaluate the degree of under-optimization of each modality, calculate the loss weight of each modality, optimize the overall loss function of the local model, and improve the accuracy of the pre-trained model; Step 3.4: Set the dynamic clipping threshold to adaptively clip the student model gradient; Step 3.5: Based on the dynamic noise scale factor, dynamically sample Gaussian noise and add dynamic noise to the clipped model gradient; Step 4: A multimodal agricultural product anomaly detection method based on a self-supervised noise reduction encoder is used to reduce noise on the pre-trained multimodal perturbation fusion data, including: Step 4.1: The cloud server averages and aggregates the perturbed student model gradients and sends the teacher model parameters to the edge layer student model for update, using the momentum mechanism to improve the efficiency of local student model updates. Step 4.2: A denoising encoder based on residual learning is used to denoise the perturbed multimodal fusion features to improve anomaly detection accuracy. Step 4.3: Use the self-supervised learning method to reconstruct the image features of the denoised multimodal agricultural product fusion features, calculate the reconstruction error, and use the error threshold to determine whether the reconstructed agricultural product image area is abnormal.

2. The anomaly detection method for multimodal agricultural product data privacy protection according to claim 1 is characterized in that: The step one includes: extracting text and image features respectively through a text encoder and an image encoder, performing preliminary fusion using a cross-modal attention mechanism, and mining the dynamic correlation between multiple modalities through a meta-learning method to complete multimodal adaptive fusion for each different task type.

3. The anomaly detection method for multimodal agricultural product data privacy protection according to claim 2 is characterized in that: The step 1 specifically includes the following steps: Step 1.1: Use the BERT pre-trained language model as the text encoder and the image encoder in CLIP to improve the model's focus on important image regions and convert the image data into a vector form combined with the text representation; Step 1.2: Utilize the key-value pair interaction of the cross-modal attention mechanism to enhance the complex association between text and image features and complete the preliminary multimodal agricultural product feature fusion; Step 1.3: Use the multi-layer perceptron (MLP) to extract low-dimensional meta-information from the preliminary fusion features. Based on the task-dependent meta-information, adaptively calculate the dynamic weight matrix of different detection tasks and combine it with the static weight matrix to obtain the multimodal agricultural product fusion matrix.

4. The anomaly detection method for multimodal agricultural product data privacy protection according to claim 1 is characterized in that: The nodes in the graph structure are represented by multimodal fusion vectors of the corresponding tasks.

5. The anomaly detection method for multimodal agricultural product data privacy protection according to claim 1 is characterized in that: The edges in the graph structure are represented by the similarity between task nodes by calculating the cosine similarity.

6. An anomaly detection device for multimodal agricultural product data privacy protection, characterized in that: The device is used to implement the function of an anomaly detection method for multimodal agricultural product data privacy protection as described in any one of claims 1 to 5.

Images