Abnormality detection method and device for privacy protection of multi-modal agricultural product data

Abstract

The invention provides an anomaly detection method and device for privacy protection of multi-modal agricultural product data, and the method comprises the following steps: carrying out the adaptive fusion of the features of the multi-modal agricultural product data based on a multi-modal adaptive fusion method of a meta-learning cross-modal attention mechanism; the invention discloses a hierarchical adaptive differential privacy protection method based on a graph attention network. Noise is adaptively generated to protect multi-modal fusion feature privacy. Performing model gradient dynamic disturbance protection based on a knowledge distillation pre-training privacy protection method of modal under-optimization dynamic weighting; according to the multi-modal agricultural product anomaly detection method based on the self-supervised noise reduction encoder, noise reduction is carried out on pre-trained multi-modal disturbance fusion data, and the agricultural product anomaly condition is effectively detected. According to the method, the dynamic relevance and adaptive fusion of the multi-modal features are enhanced, the loss weight of the weak modal is dynamically adjusted, dynamic noise protection is provided, and high-precision agricultural product anomaly detection is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural product data, and particularly relates to an anomaly detection method and device for multi-modal agricultural product data privacy protection. Background Art

[0002] At present, with the rapid development of agricultural big data and artificial intelligence technologies, agricultural product anomaly detection, as an important technical means, has been widely applied in aspects such as crop pest and disease monitoring, farmland environment monitoring, and crop growth prediction. By collecting and analyzing data from different modalities, such as agricultural product images, text data, and soil data, agricultural product anomaly detection can effectively identify potential abnormal situations in farmland, timely detect problems such as pests and diseases, mildew, and dampness, and thus improve the efficiency and quality of agricultural production. With the increase in data volume, data diversity, and modality diversity, the accuracy of agricultural product anomaly detection faces huge challenges.

[0003] During the process of agricultural product anomaly detection, a large amount of multi-modal data is involved, including agricultural product images, text descriptions, soil information, etc. These data not only contain rich agricultural environment information but also contain spatio-temporal dependence relationships in the agricultural system, and thus may expose farmers' planting habits, crop growth patterns, and sensitive information of the farm. For example, an attacker can reverse infer the degree of soil acidification and irrigation patterns by fusing multi-batch agricultural product images and soil detection data of a certain farm, and then infer agricultural environment information; by combining agricultural product growth cycle data and agricultural machinery operation logs, it may be possible to reverse engineer the patent technology of the agricultural product growth pattern and the location of sensitive plots not publicly disclosed by the farm, threatening its trade secrets. Through the cross-correlation of multi-modal agricultural product data, the potential value of a single data will be amplified, making the originally fragmented information a breakthrough point for privacy leakage. Therefore, how to ensure the privacy of multi-modal agricultural product data while performing agricultural product anomaly detection has become an important challenge that urgently needs to be solved.

[0004] Existing privacy protection technologies, such as traditional differential privacy methods, homomorphic encryption methods, and data anonymization methods. However, for large-scale multi-modal agricultural product data, traditional privacy protection methods face many challenges. Although traditional differential privacy methods can effectively protect data privacy, they fail to reasonably allocate noise according to the importance of data, resulting in difficulties in balancing the accuracy of agricultural product anomaly detection and privacy protection. For example, in the scenario of pest and disease detection, the high-frequency texture features of agricultural product images are crucial for model recognition, but the images may contain non-target sensitive information, such as the facial features of farm workers and iconic landforms. Traditional differential privacy adds global noise, resulting in the loss of edge details in anomaly detection, affecting the detection accuracy, and insufficient noise addition to highly sensitive features, unable to effectively protect sensitive information. Although homomorphic encryption technology can ensure the computational security of data in the encrypted state, its computational efficiency and functional limitations are difficult to meet the needs of multi-modal agricultural product data analysis. For example, in the task of agricultural product anomaly detection, cross-modal joint analysis of agricultural product images and text logs is required, but homomorphic encryption cannot support complex multi-modal data fusion operations within the encrypted domain, and when dealing with large-scale agricultural product data, decision-making will be delayed due to encryption latency, making it difficult to meet the timeliness requirements of anomaly detection. Data anonymization methods can, to a certain extent, ensure that personal identity information is not exposed during data sharing and analysis, but when dealing with multi-dimensional and complex agricultural product data, they cannot effectively prevent link attacks. For example, attackers may reverse-engineer the undisclosed planting techniques used by the farm by analyzing agricultural product growth stage data and combining the publicly available gene expression cycles of varieties. Therefore, how to maintain the high availability and accuracy of multi-modal agricultural product data while protecting privacy has become a key challenge in current technology applications. Summary of the Invention

[0005] The present invention is made in view of the above problems, and its purpose is to provide an anomaly detection method and device for privacy protection of multi-modal agricultural product data, a multi-modal agricultural product data adaptive fusion method based on meta-learning cross-modal attention mechanism, which enhances the dynamic relevance and adaptive fusion of multi-modal features; a hierarchical adaptive differential privacy protection method based on graph attention network, which adds adaptive noise to multi-modal fusion features through cross-level and intra-level sensitivities; a knowledge distillation pre-training privacy protection method based on modality under-optimization dynamic weighting, which dynamically adjusts the loss weight of weak modalities and provides dynamic noise protection for the gradients of locally cropped student models; a multi-modal agricultural product anomaly detection method based on self-supervised denoising encoder, which realizes high-precision agricultural product anomaly detection.

[0006] Specifically, the first aspect of the present invention provides an anomaly detection method for privacy protection of multi-modal agricultural product data, including the following steps: Step 1: A multi-modal adaptive fusion method based on meta-learning cross-modal attention mechanism to mine the complex dynamic correlation between text and image features and adaptively fuse multi-modal agricultural product data features; Step 2: A hierarchical adaptive differential privacy protection method based on graph attention network to construct a hypergeometric spherical graph structure for different detection tasks for the obtained fusion features, and adaptively generate noise according to the cross-layer and intra-layer sensitivity of task nodes to effectively protect the privacy of multi-modal fusion features; Step 3: A knowledge distillation pre-training privacy protection method based on modality under-optimization dynamic weighting to calculate the Shapley value to evaluate the degree of modality under-optimization, dynamically adjust the loss weight of weak modalities, and use the adaptive gradient clipping differential privacy method to achieve dynamic perturbation protection of model gradients; Step 4: A multi-modal agricultural product anomaly detection method based on self-supervised denoising encoder to denoise the pre-trained multi-modal perturbed fusion data and achieve high-precision agricultural product anomaly detection.

[0007] Further, the first step includes: extracting text and image features through a text encoder and an image encoder respectively, performing preliminary fusion using a cross-modal attention mechanism, and mining the dynamic correlation between multi-modalities through a meta-learning method to complete the multi-modal adaptive fusion of each different task type.

[0008] Further, the first step specifically includes the following steps: Step 1.1: Adopt the BERT pre-trained language model as the text encoder, use the image encoder in CLIP to enhance the model's attention to important image regions, and convert the image data into a vector form combined with text representation. The formula is as follows: ; ; ; ; Where: is the final output representation generated by the BERT model for the input text containing information from multiple layers; is the number of layers of the BERT model; is the weighting coefficient of the th layer; is the output feature matrix of the th layer of the BERT model; is the input text; is the average pooling operation; is a learnable weight vector; is a normalization operation; is the final output representation generated by the CLIP model for the input image, and the contribution can be adjusted according to the importance of each image patch; is the total number of image patches; is the weighted coefficient of the are the query vector, key vector, and value vector respectively; is the dimensionality size of the key vector; Step 1.2: Utilize the key-value pair interaction of the cross-modal attention mechanism to enhance the complex association between text and image features, and complete the preliminary multi-modal agricultural product feature fusion. The formula is as follows: ; where: is the feature after preliminary fusion; is the cross-modal attention mechanism; is the final output representation generated by the BERT model for the input text and contains information from multiple layers; is the final output representation generated by the CLIP model for the input image, and the contribution can be adjusted according to the importance of each image patch; Step 1.3: Extract the low-dimensional meta-information in the preliminary fusion features through a multi-layer perceptron MLP. Based on the task-dependent meta-information, adaptively calculate the dynamic weight matrix for different detection tasks, and combine it with the static weight matrix to obtain the multi-modal agricultural product fusion matrix. The formula is as follows: Task extraction process: ; ; ; where: is the input layer of the MLP; is the input of the MLP; is the feature after preliminary fusion; is the hidden layer of the MLP; is the activation function; is the learnable weight matrix of the hidden layer; is layer normalization; is the bias vector of the hidden layer; is the finally extracted task 's meta-information vector; is the learnable weight matrix of the final extraction process; is the bias vector of the final extraction process; The process of obtaining the task-specific dynamic weight matrix: ; ; ; where: are the query vector, key vector, and value vector respectively; is the query matrix; is the key matrix; is the value matrix; is the finally extracted task 's meta-information vector; is the attention matrix; is the task-specific dynamic weight matrix; is to map the 1D vector to the 2D weight matrix through three-dimensional tensor multiplication; The process of obtaining the multi-modal agricultural product fusion matrix: ; ; where: is the final multi-modal fusion matrix; is the static weight matrix; is the task-specific dynamic weight matrix; is the final multi-modal fusion representation; is the feature after preliminary fusion; Furthermore, step two includes the following steps: Step 2.1: For each different detection task, consider it as a node in the graph, and consider the similarity between different tasks as an edge, constructing a graph structure. The formula is as follows: ; ; ; Where: is the set of nodes; is the th task node; is the cosine similarity calculated between node and ; is the judgment threshold; is the edge; Step 2.2: Use the graph convolutional network to capture the relationship between nodes and neighbor nodes in the graph, embed the graph nodes into the sphere in the hypergeometric space, capture the hierarchical relationship in the graph structure, and optimize the graph structure embedding of the sphere by minimizing the loss function based on the hypergeometric distance. The formula is as follows: ; ; Where: is the embedding representation of all nodes in the graph; is the set of embedding vectors corresponding to each node in the set of nodes; is the graph convolutional network; is the parameter of the graph convolutional network; indicates that each embedding vector is located within a unit hypersphere ; is the dimensionality of the hypergeometric space; is the spherical model parameter; is the optimized model parameter; is the minimized loss function; is the regularization parameter; indicates that under the condition of satisfying the following, that is, satisfying that each element in the parameter has a norm less than ; For elements paradigm; is a constant negative curvature; Step 2.3: Calculate the cross-layer sensitivity and intra-layer sensitivity by obtaining the radius and angle of the node on the sphere. The formula is as follows: The calculation process of cross-level sensitivity: ; ; in: For the node The radius on the sphere; The radius is n-dimensional sphere; is the inverse hyperbolic tangent function; is the Euclidean distance; The Euclidean distance is retained additionally; is the reference point; is the weight coefficient; for cross-level sensitivity; Represents two adjacent node sets; Representative Node The hypergeometric embedding vector of ; The calculation process of intra-layer sensitivity: ; ; ; in: For nodes and nodes The angle between For nodes and nodes The measurement between For nodes self-measurement; For nodes self-measurement; For nodes and nodes Inner product of the embedding vectors; is the node norm of the embedding vector; is the node norm of the embedding vector; is the triplet loss function; is the anchor point and the positive sample angle; is the anchor point and the negative sample angle; is the threshold parameter; is the intra-layer sensitivity; Step 2.4: Extend the Gaussian mechanism in Euclidean space to hypergeometric space, further consider the importance and similarity of task nodes at different levels using graph attention network, adaptively generate cross-layer noise and intra-layer noise based on cross-layer sensitivity and intra-layer sensitivity, and provide noise perturbation protection for multi-modal agricultural product fusion data. The formula is as follows: ; ; ; Where: represents mapping the point in hypergeometric space to the tangent space with the reference point as the origin; is the reference point is the tangent space with the origin; is the Gaussian noise mechanism; represents the mean of the Gaussian distribution; is the noise standard deviation; is the identity matrix; is the multi-scale normalization mapping function; is the reference point calculated based on Euclidean metric to the point distance; represents using the hyperbolic sine function to convert the distance to hypergeometric space metric; is the number of scales, and the distances of each scale are adjusted using different powers; ; ; ; ; where: is the graph attention network weight between node and node ; is the exponential function; is the activation function; is the learnable linear transformation matrix; is the learnable weight vector; represents the feature vector of node ; represents the feature vector of node ; represents the feature vector of node ; represents the neighbor nodes of node ; is the set of neighbor nodes of node ; is the cross-level noise; represents the normal distribution; is the intra-level noise; is the total privacy budget; is the protected fused feature; is the ratio parameter of cross-level noise and intra-level noise; Furthermore, the nodes in the graph structure are represented by multi-modal fusion vectors corresponding to tasks.

[0009] Furthermore, the edges in the graph structure are represented by calculating the similarity between task nodes through cosine similarity.

[0010] Furthermore, step three includes the following steps: The technical means used in Step 3 include: through knowledge distillation, the student model learns the knowledge of the teacher model; based on the degree of sub-optimization of the modality evaluated by the Shapley value, the loss weight of the weak modality is calculated to improve the pre-training accuracy of the local student model; the gradients of the pre-trained model are adaptively clipped, and dynamic Gaussian noise protection is provided for the clipped model gradients.

[0011] Step 3.1: Deploy two single-modal teacher models on the cloud server and deploy a multi-modal student model on the edge layer; Step 3.2: Set a triple based on the knowledge distillation method, calculate the difference in the anomaly detection prediction probabilities of the teacher model and the student model, and optimize the pre-training learning process of the local student model through the KL divergence loss function. The formula is as follows: ; ; ; Where: is the difference in the prediction probabilities of the teacher model of modality for the data samples and ; is the difference in the prediction probabilities of the student model of modality for the data samples and ; represents the prediction probability of the teacher model; represents the prediction probability of the student model; represents the KL divergence loss function; represents the logarithmic function; represents agricultural products, represents the data sample , represents the data sample ; is the set of triples, composed of , , ; Step 3.3: Calculate the Shapley value through the prediction accuracy of the local student model, evaluate the degree of sub-optimization of each modality, calculate the loss weight of each modality, and optimize the overall loss function of the local model to improve the accuracy of the pre-trained model. The formula is as follows: ; ; ; ; Wherein: is the prediction accuracy of the local student model; is the data batch; is the batch size; is the true label of the is the modal input feature of the is the feature extraction function of the modality; is the feature extraction function of the modality; is the cross-modal feature consistency module; is a simple multi-modal fusion function; [·] is the sigmoid activation function; is the indicator function; is the dynamic threshold; is the minimum threshold; is the maximum threshold; is the current training epoch; is the total number of training epochs; is the Shapley value; , indicating the marginal contribution calculated after introducing modality when modality already exists; is the set of all modalities; is the subset of the set of all modalities that does not contain ; represents the total number of all modalities; is the normalization factor; ; ; ; in: For modal Degree of suboptimization; is the Shapley value; is the data batch; is the batch size; For the The true labels of samples; is the indicator function; For the The mode of the samples Input features of For modal Feature extraction function; For modal Feature extraction function; It is a cross-modal feature consistency module; It is a simple multimodal fusion function; [·] is the sigmoid activation function; For modal The loss function Relative to the sample The local gradient of For modal The loss weight of is the smoothing factor; is the total optimization loss function of the local student model; is the loss ratio parameter; Based on the two-class loss function; is the set of all modes; For modal Degree of suboptimization; represents the KL divergence loss function; Step 3.4: Set the dynamic clipping threshold and adaptively clip the gradients of the student model. The formula is as follows: ; ; Where: is the clipped threshold after attenuation; is the attenuation factor; is the clipped threshold of the previous round; represents the dynamically clipped gradient; is the gradient of the t-th round; is the L2 norm of the gradient of the t-th round; is the minimum function; Step 3.5: Based on the dynamic noise ratio factor, dynamically sample Gaussian noise and add dynamic noise to the clipped model gradients. The formula is as follows: ; ; Where: is the noise ratio factor; is the initial noise ratio factor; is the current training round; is the total number of training rounds; is the attenuation factor; is the Gaussian noise; is the sample batch; represents the normal distribution; is the identity matrix; is the clipped threshold after attenuation; ; ; Where: is the average clipped gradient; is the sample batch size; represents the dynamically clipped gradient; is the gradient after adding noise; is Gaussian noise; Further, step four includes the following steps: Step 4.1: The cloud server distributes the teacher model parameters to the student models in the edge layer for update by averaging the perturbed student model gradients, and adopts the momentum mechanism to improve the local student model update efficiency. The formula is as follows: ; ; ; ; Where: is the momentum of the current round; is the momentum coefficient; is the momentum of the previous round; is the number of edge nodes; is the set of edge nodes participating in the calculation in the t-th round; is the gradient after adding noise; is the updated global model parameter; is the current global model parameter; is the learning rate; is the minimum function; is the standard deviation of the gradient; is the Lipschitz constant of the gradient; is the delay degree dynamically estimated by exponential averaging for edge nodes participating in training; is the decay factor; is the clipped threshold after decay; is the logarithmic time decay term; is the proportional parameter; is the delay degree dynamically estimated by exponential averaging for edge nodes participating in training in the previous round; is the current observed delay degree; Step 4.2: Based on the denoising encoder with residual learning, denoise the perturbed multi-modal fusion features to improve the accuracy of anomaly detection; ; ; Where: is the noise residual; is the 1×1 convolution operation; is the 3×3 convolution operation; is the activation function; is the protected fusion feature; is the denoised multi-modal fusion feature; Step 4.3: Adopt a self-supervised learning method to reconstruct the image features of the denoised multi-modal agricultural product fusion features, calculate the reconstruction error, and determine whether the reconstructed agricultural product image area is abnormal through an error threshold. The formula is as follows: ; ; Where: is the reconstruction error; is the reconstructed image feature; is the denoised multi-modal fusion feature; is the square of the norm; is the regional anomaly map; is the error threshold; The second aspect of the present invention provides an anomaly detection device for multi-modal agricultural product data privacy protection, including a multi-modal adaptive fusion module with a meta-learning cross-modal attention mechanism, a hierarchical adaptive differential privacy protection module of a graph attention network, a knowledge distillation pre-training privacy protection module with modal sub-optimization dynamic weighting, a multi-modal agricultural product anomaly detection module with a self-supervised denoising encoder, a personal mobile device, an edge node, and a cloud server; The multi-modal adaptive fusion module with the meta-learning cross-modal attention mechanism uses the cross-modal attention mechanism to preliminarily fuse the text and image features extracted by the text and image encoders respectively, extracts meta-information through a multi-layer perceptron, obtains a multi-modal agricultural product fusion matrix, and completes the adaptive fusion of multi-modal features for each different task type; The hierarchical adaptive differential privacy protection module of the graph attention network captures the associations between different tasks by constructing a graph structure, embeds the graph nodes into the sphere of the hypergeometric space to capture the hierarchical relationships in the graph structure, calculates the cross-level sensitivity and intra-level sensitivity, further considers the importance and similarity of task nodes at different levels using the graph attention network, generates noise through the designed hypergeometric Gaussian mechanism, and adaptively protects tasks of different importance; The knowledge distillation pre-training privacy protection module with modality sub-optimal dynamic weighting deploys two teacher models and one student model on the cloud server and the edge layer respectively for knowledge distillation, calculates the Shapley value, evaluates the degree of modality sub-optimality, obtains the weak modality loss weight, optimizes the overall loss function of the local model, sets the dynamic pruning threshold to prune the student model gradient, designs the dynamic noise ratio factor, samples Gaussian noise, and adds noise to the model gradient; The multi-modal agricultural product anomaly detection module of the self-supervised denoising encoder has the cloud server average aggregate the student model gradient, send the teacher model parameters to the student model on the edge layer for update, adopts the momentum mechanism to improve the model update efficiency, denoises the noisy multi-modal fusion features through the denoising encoder based on residual learning, and uses the self-supervised learning method to perform anomaly detection on the denoised multi-modal agricultural product fusion features; The personal mobile device is used for the multi-modal adaptive fusion method based on the meta-learning cross-modal attention mechanism to fuse the multi-modal agricultural product features, and protects the privacy of the fused multi-modal features based on the hierarchical adaptive differential privacy protection method of the graph attention network; The edge node is used to pre-train the model on the multi-modal agricultural product fusion data uploaded by the personal mobile device, clip and add noise protection to the model gradient, and upload the perturbed gradient to the cloud layer.

[0012] The cloud server is used to aggregate all the global perturbed gradients for global parameter optimization and return the optimized parameters to each edge node to update the local student model. Brief Description of the Drawings

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present drawings or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present drawings. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the structures shown in these drawings.

[0014] Figure 1 It is the step flow chart of an anomaly detection method for multi-modal agricultural product data privacy protection of the present invention; Figure 2 The figure shows the comparison of the accuracy between the embodiments of the present invention and traditional agricultural product anomaly detection methods under different total privacy budgets; Figure 3 The figure shows the comparison of the accuracy between the embodiments of the present invention and traditional agricultural product anomaly detection methods under different loss ratio parameters; Figure 4 The figure shows the comparison of the accuracy between the embodiments of the present invention and traditional agricultural product anomaly detection methods under different gradient smoothing coefficients; Figure 5 The figure shows the comparison of the accuracy between the embodiments of the present invention and traditional agricultural product anomaly detection methods under different initial pruning thresholds; The realization, functional features and advantages of this attached figure will be further described in combination with embodiments with reference to the attached drawings. Detailed implementation manners

[0015] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be described and explained below in combination with the attached drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0016] Obviously, the attached drawings in the following description are only some examples or embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, the present invention can also be applied to other similar scenarios based on these attached drawings. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for those of ordinary skill in the art related to the content disclosed by the present invention, some design, manufacturing or production changes based on the technical content disclosed by the present invention are only conventional technical means and should not be understood that the content disclosed by the present invention is insufficient.

[0017] If there is no special description, all implementation manners and optional implementation manners of the present invention can be combined with each other to form a new technical solution.

[0018] If there is no special description, all technical features and optional technical features of the present invention can be combined with each other to form a new technical solution.

[0019] Unless otherwise specified, all steps of the present invention may be carried out sequentially or randomly, preferably sequentially. For example, when the method includes steps (a) and (b), it means that the method may include steps (a) and (b) carried out sequentially, or may also include steps (b) and (a) carried out sequentially. For another example, when it is mentioned that the method may further include step (c), it means that step (c) can be added to the method in any order. For example, the method may include steps (a), (b) and (c), or may also include steps (a), (c) and (b), or may also include steps (c), (a) and (b), etc.

[0020] Unless otherwise specified, the terms "comprising" and "including" mentioned in the present invention are open-ended and may also be closed-ended. For example, the terms "comprising" and "including" may mean that other components not listed may also be included or comprised, or may only include or comprise the listed components.

[0021] Unless otherwise specified, in the present invention, the term "or" is inclusive. For example, the phrase "A or B" means "A, B, or both A and B". More specifically, any of the following conditions satisfies the condition "A or B": A is true (or exists) and B is false (or does not exist); A is false (or does not exist) while B is true (or exists); or both A and B are true (or exist).

[0022] To better understand the solutions of the embodiments of the present invention, some related terms and concepts that may be involved in the embodiments of the present invention will be introduced below.

[0023] (1) The Graph Attention Network (GAT) is a deep learning model based on graph-structured data, which learns the relationships and feature representations between nodes through a self-attention mechanism. Different from the traditional Graph Convolutional Network (GCN), GAT introduces learnable attention weights between the neighbor nodes of each node, enabling the model to automatically assign different weights to each neighbor node, and thus more precisely learn the relationships between nodes. GAT first calculates an attention coefficient for the neighbor nodes of each node, and then the model weights and sums the features of the neighbor nodes according to these coefficients, and obtains the updated node representation through a non-linear transformation. GAT can not only make full use of neighbor information, but also effectively solve the problem of differences in importance between nodes, thereby improving the learning effect of graph data.

[0024] (2) Knowledge Distillation is a model compression technique aimed at improving the performance of a student model by transferring the knowledge of a complex teacher model to a simple student model. The teacher model is usually a large deep neural network capable of handling complex tasks, while the student model is a relatively smaller and computationally more efficient network. The core idea of knowledge distillation is to guide the training of the student model through the prediction results of the teacher model. By minimizing the difference between the output of the student model and the output of the teacher model, the student model can achieve performance close to that of the teacher model with fewer parameters and computational overhead. Knowledge distillation is widely applied in model compression, transfer learning, and improving the inference speed of deep learning models.

[0025] In this embodiment, as Figure 1 shown, an anomaly detection method for multi-modal agricultural product data privacy protection includes the following steps: Step 1: A multi-modal adaptive fusion method based on a meta-learning cross-modal attention mechanism is used to mine the complex dynamic correlation between text and image features and adaptively fuse the multi-modal agricultural product data features. Step 2: A hierarchical adaptive differential privacy protection method based on a graph attention network is used to construct a hypergeometric spherical graph structure for different detection tasks for the obtained fusion features, and noise is adaptively generated according to the cross-level and intra-level sensitivity of task nodes to effectively protect the privacy of multi-modal fusion features. Step 3: A knowledge distillation pre-training privacy protection method based on modal under-optimization dynamic weighting is used to calculate the Shapley value to evaluate the degree of modal under-optimization, dynamically adjust the loss weight of weak modalities, and use an adaptive gradient clipping differential privacy method to achieve dynamic perturbation protection of model gradients. Step 4: A multi-modal agricultural product anomaly detection method based on a self-supervised denoising encoder is used to denoise the pre-trained multi-modal perturbed fusion data to achieve high-precision agricultural product anomaly detection.

[0026] Furthermore, Step 1 includes: extracting text and image features through a text encoder and an image encoder respectively, performing preliminary fusion using a cross-modal attention mechanism, and mining the dynamic correlation between multi-modalities through a meta-learning method to complete the multi-modal adaptive fusion of each different task type.

[0027] Furthermore, Step 1 specifically includes the following steps: Step 1.1: Use the BERT pre-trained language model as the text encoder, and use the image encoder in CLIP to enhance the model's attention to important image regions, and convert the image data into a vector form combined with text representation. The formula is as follows: ; ; ; ; Step 1.2: Use the key-value pair interaction of the cross-modal attention mechanism to enhance the complex association between text and image features, and complete the preliminary multi-modal agricultural product feature fusion. The formula is as follows: ; Step 1.3: Use a multi-layer perceptron (MLP) to extract the low-dimensional meta-information in the preliminary fusion features. Based on the task-dependent meta-information, adaptively calculate the dynamic weight matrix for different detection tasks, and combine it with the static weight matrix to obtain the multi-modal agricultural product fusion matrix. The formula is as follows: [[ID=~16]]Task extraction process: ; ; ; Process of obtaining the task-specific dynamic weight matrix: ; ; ; Process of obtaining the multi-modal agricultural product fusion matrix: ; ; Furthermore, Step 2 includes the following steps: Step 2.1: Consider each different detection task as a node in the graph, and consider the similarity between different tasks as an edge to construct a graph structure. The formula is as follows: ; ; ; Step 2.2: Use a graph convolutional network to capture the relationship between nodes and neighbor nodes in the graph, embed the graph nodes into the sphere of the hypergeometric space, capture the hierarchical relationship in the graph structure, and optimize the graph structure embedding of the sphere by minimizing the loss function based on the hypergeometric distance. The formula is as follows: ; ; Step 2.3: Calculate the cross-level sensitivity and intra-level sensitivity by obtaining the radius and angle of the node on the sphere. The formula is as follows: Calculation process of cross-level sensitivity: ; ; Calculation process of intra-layer sensitivity: ; ; ; Step 2.4: Extend the Gaussian mechanism in Euclidean space to hypergeometric space, further consider the importance and similarity of task nodes at different levels using a graph attention network, adaptively generate cross-layer noise and intra-layer noise based on cross-layer sensitivity and intra-layer sensitivity, and provide noise perturbation protection for multi-modal agricultural product fusion data. The formula is as follows: ; ; ; ; ; ; ; Furthermore, the nodes in the graph structure are represented by multi-modal fusion vectors corresponding to the tasks.

[0028] Furthermore, the edges in the graph structure are represented by calculating the similarity between task nodes through cosine similarity.

[0029] Furthermore, Step 3 includes the following steps: The technical means used in Step 3 include: through knowledge distillation, the student model learns the knowledge of the teacher model; based on the degree of modal sub-optimization evaluated by the Shapley value, calculate the loss weight of the weak modality to improve the pre-training accuracy of the local student model; adaptively clip the gradients of the pre-trained model and provide dynamic Gaussian noise protection for the clipped model gradients.

[0030] Step 3.1: Deploy two single-modal teacher models on the cloud server and a multi-modal student model on the edge layer; Step 3.2: Set a triple based on the knowledge distillation method, calculate the difference between the anomaly detection prediction probabilities of the teacher model and the student model, and optimize the pre-training learning process of the local student model through the KL divergence loss function. The formula is as follows: [[ID=5�]]; ; ; Step 3.3: Calculate the Shapley value through the prediction accuracy of the local student model, evaluate the degree of under-optimization of each modality, calculate the loss weight of each modality, optimize the overall loss function of the local model, and improve the accuracy of the pre-trained model. The formula is as follows: ; ; ; ; ; ; ; Step 3.4: Set the dynamic clipping threshold to adaptively clip the gradients of the student model. The formula is as follows: ; ; Step 3.5: Dynamically sample Gaussian noise based on the dynamic noise scale factor and add dynamic noise to the clipped model gradients. The formula is as follows: ; ; ; ; Furthermore, Step Four includes the following steps: Step 4.1: The cloud server updates the student model in the edge layer by averaging and aggregating the perturbed gradients of the student model and sending down the teacher model parameters. The momentum mechanism is used to improve the update efficiency of the local student model. The formula is as follows: ; ; ; ; Step 4.2: Based on the residual learning denoising encoder, denoise the perturbed multi-modal fusion features to improve the anomaly detection accuracy; ; ; Step 4.3: Adopt the self-supervised learning method to reconstruct the image features of the denoised multi-modal agricultural product fusion features, calculate the reconstruction error, and determine whether the reconstructed agricultural product image area is abnormal through the error threshold. The formula is as follows: ; ; The second aspect of the present invention provides an anomaly detection device for multi-modal agricultural product data privacy protection, including a multi-modal adaptive fusion module with a meta-learning cross-modal attention mechanism, a hierarchical adaptive differential privacy protection module with a graph attention network, a knowledge distillation pre-training privacy protection module with modal sub-optimal dynamic weighting, a multi-modal agricultural product anomaly detection module with a self-supervised denoising encoder, a personal mobile device, an edge node, and a cloud server; The multi-modal adaptive fusion module with a meta-learning cross-modal attention mechanism uses a cross-modal attention mechanism to preliminarily fuse the text and image features extracted by a text encoder and an image encoder respectively, extracts meta-information through a multi-layer perceptron, obtains a multi-modal agricultural product fusion matrix, and completes the multi-modal feature adaptive fusion for each different task type; The hierarchical adaptive differential privacy protection module with a graph attention network captures the associations between different tasks by constructing a graph structure, embeds the graph nodes into the sphere of a hypergeometric space to capture the hierarchical relationships in the graph structure, calculates the cross-layer sensitivity and intra-layer sensitivity, further considers the importance and similarity of task nodes at different levels using a graph attention network, and generates noise through a designed hypergeometric Gaussian mechanism to adaptively protect tasks with different importance; The knowledge distillation pre-training privacy protection module with modal sub-optimal dynamic weighting conducts knowledge distillation by deploying two teacher models and one student model on a cloud server and an edge layer respectively, calculates the Shapley value, evaluates the degree of modal sub-optimization, obtains the weak modal loss weight, optimizes the overall loss function of the local model, sets a dynamic pruning threshold to prune the student model gradient, designs a dynamic noise ratio factor, samples Gaussian noise, and adds noise to the model gradient; The multi-modal agricultural product anomaly detection module with a self-supervised denoising encoder averages and aggregates the student model gradients on the cloud server, sends the teacher model parameters to the student model on the edge layer for update, adopts a momentum mechanism to improve the model update efficiency, denoises the noisy multi-modal fusion features through a denoising encoder based on residual learning, and uses a self-supervised learning method to detect anomalies in the denoised multi-modal agricultural product fusion features; The personal mobile device is used to fuse multi-modal agricultural product features based on the multi-modal adaptive fusion method with a meta-learning cross-modal attention mechanism, and protect the privacy of the fused multi-modal features based on the hierarchical adaptive differential privacy protection method with a graph attention network; The edge node is used to pre-train the model on the multi-modal agricultural product fusion data uploaded by the personal mobile device, prune and add noise protection to the model gradient, and upload the perturbed gradient to the cloud layer.

[0031] The cloud server is used to aggregate all global perturbation gradients for global parameter optimization and return the optimized parameters to each edge node to update the local student model.

[0032] To ensure the privacy of multimodal agricultural product data and model gradients, it is theoretically proven that an anomaly detection method for multimodal agricultural product data privacy protection proposed by the present invention meets the requirements of differential privacy.

[0033] Proof: In the hierarchical adaptive differential privacy protection method of the graph attention network, the noise injected into the fused features satisfies the Gaussian distribution of hypergeometric space expansion, where the privacy budget is dynamically allocated by the ratio of cross-layer and intra-layer sensitivities, meeting the Gaussian mechanism condition of differential privacy and realizing hierarchical privacy protection of multimodal agricultural product fused features.

[0034] In the knowledge distillation pre-training privacy protection method of modal sub-optimization dynamic weighting, according to the dynamic noise ratio factor to control the magnitude of the added noise intensity. According to the differential privacy property of the Gaussian mechanism, when the noise standard deviation conforms to ; where: is the noise standard deviation; is the dynamic noise ratio factor; is the clipped threshold after attenuation; is the sample batch; when, the differential privacy method based on adaptive gradient clipping of this embodiment meets differential privacy.

[0035] Based on the serial combination principle of differential privacy, an anomaly detection method for multimodal agricultural product data privacy protection proposed by the present invention meets differential privacy, where , realizing privacy protection for local multimodal fusion embedding and model gradients.

[0036] In this embodiment, based on real agricultural product datasets (Agri-vision, PlantVillage), different parameters are adopted: the total privacy budget , the loss ratio parameter , the gradient smoothing coefficient and the initial clipping bound to evaluate the usability of the privacy protection of the perturbed agricultural product data of the present invention. The results of the comparative experiment are as Figures 2 to 5 shown, where Figure 2Comparison chart of the accuracy between the embodiments of the present invention and traditional agricultural product anomaly detection methods under different total privacy budgets; Figure 3 Comparison chart of the accuracy between the embodiments of the present invention and traditional agricultural product anomaly detection methods under different loss ratio parameters; Figure 4 Comparison chart of the accuracy between the embodiments of the present invention and traditional agricultural product anomaly detection methods under different gradient smoothing coefficients; Figure 5 Comparison chart of the accuracy between the embodiments of the present invention and traditional agricultural product anomaly detection methods under different initial clipping thresholds.

[0037] From Figures 2 to 5 It can be seen that the anomaly detection accuracy of this embodiment under different parameters is better than that of traditional methods. From the results of each comparison chart and the above privacy protection proof, it can be seen that an anomaly detection method and device for multi-modal agricultural product data privacy protection provided by the present invention achieve high-precision agricultural product anomaly detection while ensuring the privacy protection of multi-modal agricultural product data.

[0038] It should be noted that the present invention is not limited to the above embodiments. The above embodiments are only examples, and embodiments with the same structure and the same function and effect as the technical idea within the technical solution scope of the present invention are all included in the technical scope of the present invention. In addition, within the scope not departing from the gist of the present invention, various deformations that can be thought of by those skilled in the art applied to the embodiments, and other ways constructed by combining some constituent elements of the embodiments are also included in the scope of the present invention.

Claims

1. An anomaly detection method for multi-modal agricultural product data privacy protection, characterized in that, It includes the following steps: Step 1: A multi-modal adaptive fusion method based on a meta-learning cross-modal attention mechanism adaptively fuses multi-modal agricultural product data features; Step 2: A hierarchical adaptive differential privacy protection method based on a graph attention network constructs a hypergeometric spherical graph structure for different detection tasks for the obtained fusion features, and adaptively generates noise according to the cross-level and intra-level sensitivities of task nodes; Step 3: A knowledge distillation pre-training privacy protection method based on modality sub-optimal dynamic weighting calculates the Shapley value to evaluate the degree of modality sub-optimality, dynamically adjusts the loss weight of weak modalities, and uses an adaptive gradient clipping differential privacy method to achieve dynamic perturbation protection of model gradients; Step 4: A multi-modal agricultural product anomaly detection method based on a self-supervised denoising encoder denoises the pre-trained multi-modal perturbed fusion data.

2. The anomaly detection method for multi-modal agricultural product data privacy protection according to claim 1, characterized in that The first step includes: extracting text and image features through a text encoder and an image encoder respectively, performing preliminary fusion using a cross-modal attention mechanism, and mining the dynamic relevance between multi-modalities through a meta-learning method to complete the multi-modal adaptive fusion of each different task type.

3. An anomaly detection method for multi-modal agricultural product data privacy protection according to claim 2, characterized in that, The first step specifically includes the following steps: Step 1.1: Use the BERT pre-trained language model as the text encoder, and use the image encoder in CLIP to enhance the model's attention to important image regions, and convert the image data into a vector form combined with text representation; Step 1.2: Use the key-value pair interaction of the cross-modal attention mechanism to enhance the complex association between text and image features, and complete the preliminary multi-modal agricultural product feature fusion; Step 1.3: Extract the low-dimensional meta-information in the preliminary fusion features through a multi-layer perceptron MLP. Based on the task-dependent meta-information, adaptively calculate the dynamic weight matrix for different detection tasks, and combine it with the static weight matrix to obtain the multi-modal agricultural product fusion matrix.

4. The anomaly detection method for multimodal agricultural product data privacy protection according to claim 1, characterized in that, The second step includes the following steps: Step 2.1: Consider each different detection task as a node in the graph, and regard the similarity between different tasks as an edge to construct a graph structure; Step 2.2: Use a graph convolutional network to capture the relationship between nodes and neighbor nodes in the graph, embed the graph nodes into the sphere of the hypergeometric space, capture the hierarchical relationship in the graph structure, and optimize the graph structure embedding of the sphere by minimizing the loss function based on the hypergeometric distance; [[ID=!3]]Step 2.3: Calculate the cross-level sensitivity and intra-level sensitivity by obtaining the radius and angle of the node on the sphere; Step 2.4: Extend the Gaussian mechanism in the Euclidean space to the hypergeometric space, use the graph attention network to further consider the importance and similarity of different hierarchical task nodes, and adaptively generate cross-level noise and intra-level noise based on the cross-level sensitivity and intra-level sensitivity to provide noise perturbation protection for the multi-modal agricultural product fusion data.

5. The anomaly detection method for multimodal agricultural product data privacy protection according to claim 4, characterized in that, The nodes in the graph structure are represented by multi-modal fusion vectors corresponding to tasks.

6. The anomaly detection method for multi-modal agricultural product data privacy protection according to claim 4, characterized in that, The edges in the graph structure are represented by calculating the similarity between task nodes through cosine similarity.

7. An anomaly detection method for multi-modal agricultural product data privacy protection according to claim 1, characterized in that, The third step includes the following steps: Step 3.1: Deploy two single-modal teacher models on the cloud server and deploy a multi-modal student model on the edge layer; Step 3.2: Set up a triple based on the knowledge distillation method, calculate the difference in the anomaly detection prediction probabilities between the teacher model and the student model, and optimize the pre-training learning process of the local student model through the KL divergence loss function; Step 3.3: Calculate the Shapley value through the prediction accuracy of the local student model, evaluate the degree of under-optimization of each modality, calculate the loss weight of each modality, optimize the overall loss function of the local model, and improve the accuracy of the pre-trained model; Step 3.4: Set a dynamic clipping threshold to adaptively clip the gradients of the student model; Step 3.5: Based on the dynamic noise ratio factor, dynamically sample Gaussian noise and add dynamic noise to the clipped model gradients.

8. An anomaly detection method for multi-modal agricultural product data privacy protection according to claim 1, characterized in that The fourth step includes the following steps: Step 4.1: The cloud server updates the student model at the edge layer by averaging and aggregating the perturbed gradients of the student model and sending down the teacher model parameters, and uses the momentum mechanism to improve the update efficiency of the local student model; Step 4.2: Based on the residual learning denoising encoder, denoise the perturbed multi-modal fusion features to improve the anomaly detection accuracy; Step 4.3: Adopt the self-supervised learning method to reconstruct the image features of the denoised multi-modal agricultural product fusion features, calculate the reconstruction error, and determine whether the reconstructed agricultural product image area is abnormal through the error threshold.

9. An anomaly detection device for multi-modal agricultural product data privacy protection, characterized in that, The device is used to implement the functions of an anomaly detection method for multi-modal agricultural product data privacy protection according to any one of claims 1 to 8.

Images