A non-corresponding review system for internet of things data sharing

Abstract

The present application relates to the technical field of intelligent agricultural data auditing, in particular to a non-corresponding auditing system for Internet of Things data sharing, which comprises data collection and association, multi-modal data fusion, self-adaptive calibration and calibration evaluation feedback units. The present application collects and associates multi-modal data through the data collection and association unit, extracts features using a convolutional neural network, obtains precise fusion feature vectors through deep data fusion by means of a multi-modal feature fusion model and various optimization techniques, dynamically adjusts calibration model parameters in combination with historical data by means of a reinforcement learning algorithm and a deep Q network in the self-adaptive calibration unit, and evaluates the calibration effect in the calibration evaluation feedback unit. If the effect is not as expected, optimization is fed back. Meanwhile, decision support is provided based on the calibrated data, effectively solving the problem of data accuracy caused by the complexity of the agricultural environment and improving the reliability of data auditing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent agricultural data auditing, in particular to a non-corresponding auditing system for Internet of Things data sharing. BACKGROUND

[0002] Intelligent agricultural data auditing is an important technology. In the field of intelligent agriculture, in order to realize precision agricultural management, a large number of Internet of Things devices are deployed in farmland, greenhouses and other agricultural production environments to collect various crop growth-related data. These data are crucial for agricultural decision-making.

[0003] Intelligent agriculture relies on a large number of Internet of Things sensors to collect data to assist decision-making. However, sensors are subject to complex agricultural environments and their own conditions, and data accuracy is a concern. In natural environments, heavy rain can cause soil moisture sensors to be submerged, wind and sand can clog air quality sensors, and extreme temperatures can interfere with the accuracy of temperature and humidity sensors. Such interference causes large fluctuations in data and deviation from reality, affecting irrigation, fertilization and pest control decisions. Existing single-modal data processing techniques are difficult to comprehensively consider multiple factors, leading to a decline in data accuracy and affecting the reliability of data sharing and agricultural decision-making. To solve this technical problem, we provide a non-corresponding auditing system for Internet of Things data sharing. SUMMARY

[0004] The present application aims to provide a non-corresponding auditing system for Internet of Things data sharing to solve the problems raised in the background.

[0005] To achieve the above-mentioned purpose, a non-corresponding auditing system for Internet of Things data sharing is provided, which includes a data collection and association unit, a multi-modal data fusion unit, an adaptive calibration unit and a calibration evaluation and feedback unit.

[0006] The data collection and association unit deploys sensors and cameras in farmland and greenhouse sheds. Each sensor collects data at a predetermined time interval and attaches a timestamp and device ID, and records the operating state data of the sensor. A distributed data association network is used to associate multi-modal data within the same time window and in the same farmland area and greenhouse, forming an associated data set.

[0007] The multi-modal data fusion unit directly extracts numerical data as features, and for crop image data, a feature conversion module is used to extract texture, color distribution and leaf shape features using a convolutional neural network and a multi-branch parallel architecture, and encode them into a feature vector. The sensor state data is converted into a feature representation according to a pre-set encoding rule, and a multi-modal feature fusion model of a feature enhancement module is introduced. The multi-modal feature fusion model uses a feature enhancement method based on a generative adversarial network and an attention mechanism. According to the importance of different modal data for growth condition judgment in different growth stages of crops, the weight is dynamically allocated, and the allocated weight result is input into the weight optimization module. The weight optimization module uses a quantum particle swarm optimization-based attention mechanism weight optimization method to deeply fuse multi-modal data when fusing multi-modal data. Finally, a feature fusion module uses a deep belief network-based feature pre-training method to obtain a fusion feature vector that comprehensively reflects the accuracy of crop growth environment data.

[0008] The adaptive calibration unit constructs a data calibration model, combines the fusion feature vector with historical data, and dynamically adjusts the calibration model parameters according to the current data actual situation and the calibration effect with the help of a reinforcement learning algorithm.

[0009] The calibration evaluation feedback unit sets a calibration effect quantitative index, evaluates the calibrated data according to the calibration effect quantitative index, and if the expected result is not achieved, it feeds back to the multi-modal data fusion unit to adjust the feature extraction strategy and feature weight allocation strategy, and to the adaptive calibration unit to optimize the parameter adjustment strategy. At the same time, based on the calibrated data, decision support is provided for agricultural production.

[0010] As a further improvement of the technical solution, the feature conversion module specifically operates as follows:

[0011] Three parallel branches are constructed. The first branch uses a convolutional neural network to extract multi-scale texture features by setting the size of the convolution kernel. After extracting the features, they are input into the normalization layer and the activation function, and are reduced in dimension through the max-pooling layer.

[0012] The second branch uses a residual network to extract image semantic features, and sets a residual block. The residual block is used to solve the gradient vanishing problem in deep neural networks. Each residual block contains two convolution layers and a jump connection. The jump connection is used to directly connect the output of a certain layer with the input of a certain layer behind it.

[0013] The third branch uses a deep learning network to extract overall color distribution and rough shape features of the image. The features extracted by the three branches are spliced and fused at the end of the network to form the crop image features.

[0014] As a further improvement of the technical solution, the feature conversion module adopts a feature dimension reduction method based on kernel principal component analysis after converting the sensor state data into feature representation, specifically as follows:

[0015] First, the kernel matrix of the sensor state data feature matrix is calculated, a Gaussian kernel function is selected, and then the eigenvalues and eigenvectors of the kernel matrix are solved;

[0016] The eigenvalues corresponding to the first several eigenvectors whose cumulative contribution rate reaches a predetermined threshold are selected to construct a projection matrix, and the original sensor state data features are projected onto the projection matrix to obtain the reduced feature representation.

[0017] As a further improvement of the technical solution, the feature enhancement module operates as follows:

[0018] A generator network and a discriminator network are constructed, the generator network takes a random noise vector and a fused feature vector as input, the random noise vector is used to generate a new data sample, and a new feature vector is generated through a deconvolution layer and an upsampling operation;

[0019] The discriminator network receives a real fused feature vector and a feature vector generated by the generator, extracts and discriminates features through a convolution network, and outputs a discrimination result representing the authenticity probability of the input feature vector, the real fused feature vector refers to a new feature vector formed by integrating multiple feature vectors;

[0020] During training, the generator and the discriminator are trained in an adversarial manner, through this adversarial training mechanism, the generator generates a secondary feature vector, and the secondary feature vector is spliced with the original fused feature vector and then input into the fusion model again for deep fusion.

[0021] As a further improvement of the technical solution, the multi-modal data fusion unit adopts an adaptive learning rate adjustment strategy during the training of the multi-modal feature fusion model, specifically as follows:

[0022] Set the initial learning rate η0, during training, every t training iterations, calculate the loss function value L t of the current model on the validation set The ratio of the moving average value of the loss function value of the first k training batches

[0023] If r t < θ1, increase the learning rate by α%, if r t > θ2, decrease the learning rate by β%, if θ1≤ r t ≤ θ2, the learning rate remains unchanged, θ1 and θ2 are preset thresholds.

[0024] As a further improvement of the technical solution, the weight optimization module specifically operates as follows:

[0025] Encode the weight parameters in the attention mechanism into quantum bit form, construct a quantum particle swarm, and each quantum particle represents a set of possible weight parameter combinations. The position and velocity of the particle correspond to the value of the weight parameter and the update direction, respectively.

[0026] During the quantum particle swarm optimization process, the evaluation index of the multi-modal data fusion model is calculated according to the current position of the particle. The evaluation index is the similarity between the fusion feature vector and the true crop growth condition label.

[0027] Update the individual optimal position and the group optimal position of the particle according to the evaluation index, update the position and velocity of the particle through the quantum rotation gate operation, and after multiple iterations, the optimal weight parameter combination is obtained.

[0028] As a further improvement of the technical solution, the feature fusion module specifically operates as follows:

[0029] Construct a deep belief network, which includes a restricted Boltzmann machine layer. For numerical data, crop image data extracted features, and sensor state data converted features, input them into the corresponding restricted Boltzmann machine layer for pre-training.

[0030] During the pre-training process, the contrast divergence algorithm is used to train the restricted Boltzmann machine layer. By adjusting the network parameters to reconstruct the input data, the second parameters are obtained, which are used as the initial parameters of the multi-modal feature fusion model. Then, the labeled crop growth condition data is used for training to obtain the fusion feature vector that comprehensively reflects the accuracy of the crop growth environment data.

[0031] As a further improvement of the technical solution, the reinforcement learning algorithm in the adaptive calibration unit uses a deep Q network to dynamically adjust the calibration model parameters, specifically as follows:

[0032] Construct a deep Q network structure, including an input layer, a hidden layer, and an output layer. The input layer receives the combined feature representation of the fusion feature vector and the historical data. The output layer corresponds to the calibration model parameter adjustment strategy, and the number of output layer nodes is the same as the number of adjustable parameter types.

[0033] Quantify the actual situation and calibration effect of the current data as the reward signal of the deep Q network. The deep Q network is trained through the experience replay mechanism, and stores the historical state, action, and reward information. During the training process, a batch of data is randomly sampled from the experience replay buffer, and the target Q value is calculated.

[0034] According to the calculated target Q value, the gradient descent algorithm is used to update the parameters of the main network.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] In an Internet of Things data sharing non-correspondence auditing system, by integrating various modal data, the crop growth environment conditions can be more comprehensively and accurately reflected, features are extracted using convolutional neural network technology, deep-level data information is effectively mined, feature richness and integrity are improved, providing a solid basis for accurate judgment of crop growth, the attention mechanism dynamically allocates weights according to the growth stage of crops, focuses on key modal data, enhances data relevance and effectiveness, a multi-branch parallel architecture processes crop image features, analyzes images from multiple dimensions such as texture, semantics, and color distribution, provides more comprehensive visual features, and based on the pre-training of a deep belief network and the weight adjustment of a quantum particle swarm optimization, the learning ability and fusion effect of the model are improved, and it is ensured that the fused feature vector can accurately represent the crop growth environment data accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The present application is a whole block diagram.

[0038] The meanings of the various labels in the figure are as follows:

[0039] 1, data acquisition association unit; 2, multi-modal data fusion unit; 21, feature conversion module; 22, feature enhancement module; 23, weight optimization module; 24, feature fusion module; 3, adaptive calibration unit; 4, calibration evaluation feedback unit. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0041] The present application provides an Internet of Things data sharing non-correspondence auditing system, as shown in Figure 1 The present application provides an Internet of Things data sharing non-correspondence auditing system, as shown in

[0042] The data acquisition association unit 1 respectively deploys sensors and cameras in farmland and greenhouse sheds, each sensor collects data at a predetermined time interval and attaches a timestamp and device ID, and records the running state data of the sensor, and then uses a distributed data association network to associate multi-modal data within the same time window and located in the same farmland area and greenhouse, forming an associated data set.

[0043] The multi-modal data fusion unit 2 directly extracts numerical data as features, and for crop image data, uses a convolutional neural network through a feature conversion module 21 and adopts a multi-branch parallel architecture to extract texture, color distribution, and leaf shape features and encode them as a feature vector. Sensor state data is converted into feature representations according to a pre-set encoding rule, and a multi-modal feature fusion model of a feature enhancement module 22 is introduced. The multi-modal feature fusion model uses a feature enhancement method based on a generative adversarial network and an attention mechanism, dynamically allocates weights according to the importance of different modal data for growth condition judgment at different stages of crop growth, and inputs the allocation weight result into a weight optimization module 23. The weight optimization module 23 uses a quantum particle swarm optimization-based attention mechanism weight optimization method to deeply fuse multi-modal data when fusing multi-modal data. Finally, a feature fusion module 24 uses a deep belief network-based feature pre-training method to obtain a fusion feature vector that comprehensively reflects the accuracy of crop growth environment data.

[0044] The feature conversion module 21 specifically operates as follows:

[0045] First, a convolutional neural network branch is constructed, with the input of the network being crop image data, the image size being uniformly adjusted to m x n, and the uniform image size facilitating network processing, ensuring that each image data is extracted at the same scale, and improving the reliability of the entire image recognition system. Different sizes of convolution kernels are used to extract multi-scale texture features, with the convolution kernel sizes being set to 3 x 3, 5 x 5, and 7 x 7. For the 3 x 3 convolution kernel, the number of convolution kernels is set to c1, the step size s1 = 1, and the padding p1 = 1. The convolution operation formula is where O ij is the pixel value of the output feature map at position (i, j), is the weight of the kth 3 x 3 convolution kernel, I i+u，j+v is the pixel value of the input image at the corresponding position, and b k is the bias term. Using multi-scale convolution kernels can capture texture at different levels of detail, and different scale textures have unique representations for crop growth conditions, fully extracting image texture information, enriching feature dimensions, and enhancing the ability to recognize subtle texture changes such as diseases, pests, and nutrient deficiencies.

[0046] Next is the batch normalization layer, which normalizes the features output by the convolution layer. The formula is where x i is the input feature, μ B is the mean of the batch data, is the variance of the batch data, and ε is a small value to prevent the denominator from being zero. After normalization, the Scaling and translation are performed, where γ, β are learning parameters, and then an activation function layer is passed, with the formula f(x) = max(0, x); the non-linear expression capacity of the enhancement network is enhanced, the data distribution change in neural network training affects convergence, batch normalization stabilizes distribution, accelerates network training convergence speed, improves training stability, and reduces the need for hyperparameter adjustment.

[0047] For the convolutional layers with 5x5 and 7x7 convolutional kernels, the parameters are set in the same way as the 3x3 convolutional kernel, and the same convolution, batch normalization, and activation operations are performed. The multi-scale convolutional kernels work together to comprehensively capture textures. The advantage is that different scale texture features are fully exploited, the feature richness and completeness are improved, and comprehensive texture basis is provided for comprehensive evaluation of crop growth conditions. Finally, dimensionality reduction is performed through a max pooling layer, with a pooling kernel size of 2x2 and a step size of s p = 2, and the formula is where O pool (i, j) is the pixel value of the pooling layer output at position (i, j), I 2i+u，2j+v is the pixel value of the feature map before pooling at the corresponding position, and the feature output F1 of the first branch is obtained. The pooling layer reduces dimensionality to reduce computational complexity, reduces data dimensionality under the premise of retaining key information, improves computational efficiency, and avoids overfitting.

[0048] A residual network branch is constructed, with the same crop image data as input. The deep structure of the residual network can extract deep semantic features of the image, as crop growth conditions are closely related to deep semantic features of the image. The advantage is that more abstract and high-level semantic features can be extracted, providing key semantic basis for precision agriculture decision-making. Multiple residual blocks are set, and the first residual block is taken as an example. It contains two 3x3 convolutional layers. The number of convolutional kernels in the first convolutional layer is set to d1, and the step size is s r1 = 1, and the padding is p r1 = 1, and the convolution operation formula is where O r1，ij is the pixel value of the first convolutional layer output at position (i, j), is the weight of the kth convolutional kernel of the first convolutional layer, is the bias term. The convolutional layers in the residual block gradually extract features. Multi-layer convolution can gradually abstract image information, effectively extract image semantic features, improve feature expression accuracy and depth, and enhance the system's understanding and judgment ability for complex crop growth conditions.

[0049] A batch normalization layer and an activation function layer are connected after the first convolutional layer. In the residual network branch, the training process is also stabilized, the convergence is accelerated, and the non-linear expression is enhanced, so that the residual network can efficiently and accurately learn image semantic features, improve image semantic feature extraction quality and efficiency. The number of convolutional kernels in the second convolutional layer is set to d2, and the step size is s r2= 1, padding p r2 = 1, the convolution formula is where O r2，ij is the pixel value of the output of the second convolution layer at the (i, j) position, is the weight of the kth convolution kernel of the second convolution layer, is the bias term, the second convolution layer further refines the features, and the multi-layer convolution cooperatively constructs a complex feature mapping, enhances the feature abstraction capability, improves the representativeness and discriminativeness of the semantic features, and improves the recognition accuracy and sensitivity of the system to the abnormal growth of crops.

[0050] The skip connection directly adds the input to the output of the second convolution layer, and the formula is F res = O r2 + I; where F res is the output of the residual block, and I is the input of the residual block, which can effectively solve the gradient disappearance problem and enable the network to train deeper structures. Multiple residual blocks are connected in turn, and finally the feature output F2 of the second branch is obtained, which can effectively solve the gradient disappearance problem and enable the network to train deeper structures. The skip connection bypasses the gradient transmission, guarantees the expansion of the network depth, and fully learns the deep semantic features of the image.

[0051] A deep learning network is used as the third branch network, which inputs the crop image data. The deep learning network needs to quickly process images due to the limited resources of agricultural Internet of Things devices, and quickly extracts the overall features of the image under the premise of ensuring a certain accuracy, reducing the demand for computing resources. The deep learning network mainly reduces the computational complexity through deep separable convolution. For the deep convolution layer, the convolution kernel size is 3x3, and the step size is s m1 = 1, padding p m1 = 1, and the deep convolution formula is where O d，ij is the pixel value of the output of the deep convolution at the (i, j) position, W d is the weight of the deep convolution kernel, and I i+u，j+v is the pixel value of the input image at the corresponding position. The deep separable convolution decomposes the convolution operation to reduce the computational complexity, greatly reduces the occupation of computing resources, and improves the image processing speed.

[0052] The deep convolution is followed by a batch normalization layer and an activation function layer. The pointwise convolution layer has a convolution kernel size of 1x1 and a step size of s m2 = 1, and the convolution formula is where O p，ij is the pixel value of the output of the pointwise convolution at the (i, j) position, is the weight of the pointwise convolution kernel, is the bias term, e is the number of point-by-point convolution kernels. After several depth-wise separable convolution combinations, the overall color distribution and rough shape features of the image are extracted to obtain the feature output F3 of the third branch. This branch focuses on the overall features of the image. Since the overall features reflect the macroscopic state of the crop, the overall visual information of the image is quickly obtained, providing an intuitive basis for judging the growth status.

[0053] At the end of the network, the features F1, F2, and F3 obtained from the three branches are concatenated and fused. Assume that the dimension of F1 is h1×ω1×c f1 , the dimension of F2 is h2×ω2×c f2 , the dimension of F3 is h3×ω3×c f3 , first adjust the dimensions to make them consistent in the channel dimension, and then splice them along the channel dimension to obtain the final crop image feature F = [F1+F2+F3]; its dimension is h×ω×(c f1 +c f2 +c f3 ), this fused feature will be input into the subsequent processing flow as part of the multimodal data fusion, which can comprehensively reflect the multi-faceted information of crop images and provide rich visual feature basis for accurately judging the growth status of crops.

[0054] After converting the sensor state data into feature representation, the feature conversion module 21 adopts a feature dimension reduction method based on kernel principal component analysis, as follows:

[0055] Assume that the sensor state data sample set is {x1, x2, ..., x m}, where x i ∈R n Represents the eigenvector of the i-th sample, m is the number of samples, and n is the original feature dimension. First, calculate the kernel matrix K and select the Gaussian kernel function. The Gaussian kernel function can map the original feature space to a high-dimensional feature space. In the high-dimensional space, it may make the originally linearly inseparable data become linearly separable, which is more conducive to mining the intrinsic structure of the data. Where σ is the kernel bandwidth parameter, which controls the range of the kernel function and calculates the element k of the kernel matrix K. ij =K(x i , x j ), for all i, j = 1, 2, ..., m, it can effectively handle the nonlinear relationship that may exist in the sensor state data, improve the dimensionality reduction effect, re-represent the sensor state data in a high-dimensional space, and lay the foundation for the subsequent accurate extraction of main features.

[0056] Calculate the mean vector of the kernel matrix K where k iis the i-th row vector of the kernel matrix K, and then the kernel matrix is ​​centralized to obtain the centralized kernel matrix Its elements where μ i and μ j are the i-th and j-th elements of μ, respectively, In order to eliminate the influence of the mean in the data, the subsequent eigenvalue decomposition can more accurately reflect the variance information of the data, improve the accuracy of the eigenvalue decomposition, so that the main eigenvectors can be determined more accurately, and provide a guarantee for the data after dimensionality reduction to better reflect the key information of the sensor status data.

[0057] Solving the centralized kernel matrix The eigenvalues ​​λ1, λ2, ..., λ m and the corresponding eigenvectors v1, v2, ..., v m , that is, solving the equation The eigenvalues ​​and eigenvectors can reflect the variance of the data in various directions. The larger the eigenvalue, the more important the information contained in the corresponding eigenvector. By solving the eigenvalues ​​and eigenvectors, the main characteristic directions of the data can be determined, providing a basis for dimensionality reduction, ensuring that the data after dimensionality reduction can still accurately reflect the core characteristics of the sensor state, improving data processing efficiency while reducing information loss.

[0058] Sort the eigenvalues ​​and eigenvectors in descending order, set the predetermined threshold as α, and calculate the cumulative contribution rate Select the first k eigenvectors v1, v2, ..., v that make the cumulative contribution rate reach α k , then construct the projection matrix P = [v1, v2, ..., v k ]; By setting a threshold to select the main eigenvectors, the data dimension can be reduced while retaining sufficient information, avoiding excessive information loss caused by excessive dimensionality reduction, and also reducing unnecessary calculations. A balance is found between data information and computing resources, which ensures that the data after dimensionality reduction can effectively represent the sensor status and improves the speed and efficiency of data processing.

[0059] The original sensor state data feature vector x i Projected onto the projection matrix P, the reduced-dimensional feature representation y is obtained i =P T x i ; where y i ∈R kIt is the eigenvector of the i-th sample after dimensionality reduction. The original high-dimensional data is mapped to the low-dimensional space through projection operation to achieve the purpose of dimensionality reduction. A low-dimensional feature representation that retains key information is obtained, which is convenient for subsequent fusion processing with other modal data, reduces the complexity of data fusion, and provides a more concise and powerful data basis for accurately evaluating the accuracy of crop growth environment data.

[0060] The feature enhancement module 22 specifically operates as follows:

[0061] The input of the generator network G includes the random noise vector z and the fused feature vector x f , combining the randomness of the noise and the existing fusion feature information, guide the generator to generate a feature vector that is related to the original data but has new characteristics. First, the random noise vector z is transformed into a fully connected layer, and the weight matrix of the fully connected layer is set to W z , the bias vector is b z , then output h z =ReLU(W z z+b z ); ReLU(x) = max(0, x) is used to introduce nonlinearity. Random noise itself lacks structural information. A meaningful feature representation is initially constructed through the fully connected layer and activation function, so that the generator can better utilize the noise information to generate diversified features, providing potential basic features for the subsequent generation of feature vectors.

[0062] The fully connected layer outputs h z and the fused feature vector x f Splicing is performed, and the dimension of the spliced ​​vector is d h +d xf , let the concatenated vector be h con =[h z ;x f ]; The multi-source information is integrated to enrich the connotation of the generated features, so that the generated feature vectors can retain some features of the original data while also having innovative changes, thereby improving feature diversity and providing an effective way for data enhancement.

[0063] The concatenated vector h con After a series of deconvolution layers and upsampling operations, taking the first deconvolution layer as an example, let the deconvolution kernel size be k1, the step size be s1, the padding be p1, and the deconvolution layer weight matrix be W d1 , the bias vector is b d1 , then the output o1=ReLU(W d1 CT(h con ,k1,s1,p1)+d d1); wherein CT represents a deconvolution operation, and the deconvolution layer and the upsampling operation gradually increase the feature map size and enrich the feature details, gradually recovering from a low-dimensional feature space to a high-dimensional feature space similar to the original data, which can generate a feature vector with spatial structure and detailed information, making the generated feature vector more similar to the real data feature in form and content.

[0064] After multiple deconvolution layers and upsampling operations, the final secondary feature vector x g is generated. c1 , the stride is s c1 , the convolution layer weight matrix is W c1 , and the bias vector is b c1 . For the input feature vector x, the convolution layer can extract local features of the feature vector, and the output f1=LReLU(W c1 Con(x, k c1 , s c1 )+b c1 ); wherein Con represents a convolution operation, and the LReLU activation function helps the network to stabilize training in backpropagation, effectively extracts key feature information of the feature vector, and ensures training stability, so that the discriminator can accurately perceive the feature details of the feature vector.

[0065] After multiple convolution layers, the extracted features are mapped to a single output value through a fully connected layer, representing the authenticity probability of the input feature vector. Let the fully connected layer weight matrix be W fc , and the bias vector be b fc , then the output p=Sd(W fc f n +b fc ); wherein f n is the output of the last convolution layer, The output value is compressed to the interval (0, 1), representing the authenticity probability. The closer to 1 indicates the more likely it is a real feature vector, and the closer to 0 indicates the more likely it is a feature vector generated by the generator. The discriminator network can accurately determine the authenticity of the input feature vector through such a structure, providing feedback to the generator in the adversarial training, prompting the generator to generate more realistic feature vectors, thereby improving the effectiveness of the entire feature enhancement process, making the generated feature vector better integrate into multi-modal data fusion, improving the representation ability of crop growth environment data, and providing more reliable data basis for agricultural production decision-making.

[0066] Initialize the network parameters of the generator G and the discriminator D. In the training process, first fix the parameters of the discriminator D, and train the generator G. The goal of the generator G is to generate as realistic feature vectors as possible to deceive the discriminator D. Let the loss function of the generator be LG = -log(D(G(z, x f ))) where z is a random noise vector, x f is the fused feature vector, by minimizing this loss function, the generator adjusts its network parameters so that the generated feature vector x g = G(z, x f ) in the discriminator D is closer to the real feature vector, i.e. D(x g ) is as large as possible, through adversarial training, the generator can learn the latent distribution characteristics of the data, generate more consistent with the real data feature vector, improve the quality and diversity of the generated feature vector, provide more valuable complementary features for multi-modal data fusion.

[0067] Then fix the parameters of the generator G, train the discriminator D, the goal of the discriminator D is to accurately distinguish between the real fused feature vector x f and the feature vector x g generated by the generator, let the loss function of the discriminator be L D = log(D(x f )) + log(1-D(x g )) by minimizing this loss function, the discriminator adjusts its network parameters to improve the discrimination ability of real and generated feature vectors, i.e. D(x f ) is as close to 1 as possible, D(x g ) is as close to 0 as possible, such adversarial training process alternately, in each iteration, the generator tries to generate more realistic feature vectors, the discriminator tries to improve the discrimination ability, after several iterations, the generator can generate high-quality feature vectors, these feature vectors are spliced with the original fused feature vector and input to the fusion model for deep fusion.

[0068] After adversarial training, the generator G generates secondary feature vector x g , the secondary feature vector x g is spliced with the original fused feature vector x f , let the spliced feature vector be x new = [x f ; x g ]; combine the generated feature vector with innovation and diversity with the original information-rich feature vector, fuse the advantages of both, form a more comprehensive and powerful feature representation, provide more abundant input for multi-modal data fusion model.

[0069] The spliced feature vector x new is input again to the multi-modal data fusion model for deep fusion. In the fusion model, weights are assigned according to the importance of different modal data features, let the fusion model be M, the fused feature vector be xfinal = M(x new ); through such a deep fusion process, the generated feature vector is fully fused with the original data, further improving the effect of multi-modal data fusion, and can more accurately reflect the accuracy of crop growth environment data.

[0070] During the multi-modal feature fusion model training process, the multi-modal data fusion unit 2 adopts an adaptive learning rate adjustment strategy, as follows:

[0071] When the multi-modal data fusion unit 2 adopts the adaptive learning rate adjustment strategy during the multi-modal feature fusion model training process, if the loss function value L t of the current model on the validation set is calculated , the ratio r of the moving average value of the loss function value of the first k training batches t satisfies θ1≤r t ≤θ2, the learning rate remains unchanged, and the specific operation steps are as follows:

[0072] Before training begins, set the initial learning rate η0, and determine the parameters θ1, θ2, and the value k required for calculating the moving average value, set η0=0.001, k=10, and after every t training iterations, the moving average value of the loss function value can be calculated by a simple weighted average method, such as Calculate the ratio r Determine whether r t satisfies θ1≤r t ≤θ2, if it does, do not adjust the learning rate, continue to use the current learning rate η0=η t for the next batch of training; if it does not, adjust the learning rate according to the relationship between r t and the threshold value, when θ1≤r t ≤θ2, it indicates that the training state of the model is relatively stable, and the current learning rate can maintain a good balance between convergence speed and accuracy, and there is no need to adjust the learning rate, avoiding the instability of training caused by frequent adjustment of the learning rate, and ensuring the smoothness of the model training process.

[0073] The weight optimization module 23 specifically operates as follows:

[0074] First, for the weight parameters in the attention mechanism, encode them into quantum bit form, assume that the attention mechanism has n weight parameters ω = {ω1, ω2,... ω n}, and each weight parameter ω i is represented by a quantum bit as where α i and β irespectively represent the probability amplitude of the quantum bit in the |0> state and the |1> state, the quantum bit can represent the superposition of multiple states, which increases the diversity of the search space and can explore a wider range of possible weight combinations, providing more possibilities for finding the optimal weight parameter combination.

[0075] A quantum particle swarm is constructed, with the particle swarm size m, and each quantum particle q j (j = 1, 2, …, m) represents a set of possible weight parameter combinations, i.e. The position x j and the speed v j of the particle correspond to the value of the weight parameter and the update direction respectively, and the quantum bit state of the quantum particle is randomly generated at initialization, providing a starting point for subsequent optimization search. Random initialization helps to explore different areas and improve the probability of finding the global optimal solution.

[0076] During the quantum particle swarm optimization process, the evaluation index of the multi-modal data fusion model is calculated according to the current position of the particle, with the fusion feature vector F and the true crop growth condition label T, and the cosine similarity is used as the evaluation index, with the calculation formula being k is the dimension of the feature vector, and the cosine similarity is selected as the evaluation index, which can effectively measure the similarity of two vectors in direction, and is more appropriate for evaluating the matching degree of the fusion feature vector and the true label, and intuitively reflects the closeness of the model output and the true situation. Through this index, the pros and cons of the model under the current weight parameter combination can be accurately judged.

[0077] For each particle q j , the evaluation index sim j corresponding to its current position is calculated, and if sim j > sim(pbest j ); then the individual optimal position pbest j = x j is updated, where sim(pbest j ) is the evaluation index corresponding to the individual optimal position of particle j. Then, among all the individual optimal positions, the particle position with the optimal evaluation index is found. By constantly comparing and updating the individual optimal and group optimal positions, the particle swarm can gradually move to better spatial regions, guiding the particle swarm to focus on more potential weight parameter combinations, accelerating the optimization process and improving the efficiency of finding the optimal weight parameter combination.

[0078] The quantum rotation gate is used to update the quantum bit state of the particle, thereby updating the position and speed of the particle, with the particle q j The i-th quantum bit is The update formula is where θ jiis a rotation angle, which is calculated with the current position x of the particle j , the individual optimal position pbest j , the group optimal position gbest and a random number r, the quantum rotation gate can rotate the quantum bit state according to the current state and optimal state information of the particle, update the position and velocity of the particle, retain certain historical optimal information while exploring new weight parameter combinations, avoid blind search, and enable the particle swarm to search more intelligently in the solution space and gradually approach the optimal weight parameter combination.

[0079] Repeat the above steps for multiple iterations. As the number of iterations increases, the particle swarm converges to a more optimal weight parameter combination region. In each iteration, the particle adjusts its position and velocity according to the evaluation index and quantum rotation gate operation, and the individual optimal position and group optimal position are also updated. After N iterations, the weight parameter combination corresponding to the final group optimal position gbest is the optimal weight parameter combination. The complex multi-modal data fusion model and attention mechanism weight optimization problem cannot be solved by one calculation, and needs to be gradually approached through iterative improvement, which can improve the probability of finding the global optimal solution or approaching the global optimal solution.

[0080] The feature fusion module 24 specifically operates as follows:

[0081] A deep belief network is constructed, which is stacked by multiple restricted Boltzmann machine layers. First, the network structure is determined, and three RBM layers are set. The RBM layer is a restricted Boltzmann machine layer, which corresponds to the numerical data, the features extracted from the crop image data, and the features converted from the sensor state data, respectively. Let the dimension of the numerical data be n1, the dimension of the features extracted from the crop image data be n2, and the dimension of the features converted from the sensor state data be n3. For the numerical data input layer, n1 visible units are set. For the crop image data feature input layer, n2 visible units are set. For the sensor state data feature input layer, n3 visible units are set. Different types of data have different feature dimensions and information representation forms, and correspond to different input layers, which can better process and learn their own features, so that the deep belief network can perform specialized feature extraction and learning for different modal data, and improve the adaptability and feature learning ability of the network to multi-modal data.

[0082] For the first layer RBM corresponding to the numerical data input, let the visible layer unit state vector be v, the hidden layer unit state vector be h, m1 is the number of hidden layer units, and the energy function is defined as:

[0083]

[0084] where ω ij is the connection weight between visible unit i and hidden unit j, b i is the bias of visible unit i, c j is the bias of hidden unit j.

[0085] The activation probability of visible unit i at given hidden state h is calculated as where is the activation function. Similarly, the activation probability of hidden unit j at given visible state v is calculated as

[0086] The RBM is trained using the contrastive divergence algorithm. First, the connection weights ω ij , the visible biases b i and the hidden biases c j are initialized. Then, a numerical data sample v 0 is selected from the training data as the initial visible state.

[0087] The hidden state h is calculated as 0 For each hidden unit j, the following is sampled based on to get based on h 0 The reconstructed visible state v is calculated as 1 For each visible unit i, the following is sampled based on to get based on v 1 The hidden state h is calculated as 1 For each hidden unit j, the following is sampled based on to get

[0088] The connection weights ω ij , the visible biases b i and the hidden biases c j are updated according to the following formula:

[0089]

[0090] where ε is the learning rate, <·> denotes the expectation over the samples. The above steps are repeated to complete the pre-training of the first layer RBM on the numerical data using the contrastive divergence algorithm. The RBM is able to learn the intrinsic feature representation of the data by adjusting the network parameters to reconstruct the input data. The advantage is that the data features are effectively extracted and good initial parameters are provided for the subsequent network training. The effect is that the obtained parameters can reflect some key feature structures of the data, laying a foundation for the deep understanding of the numerical data by the deep belief network.

[0091] For the RBM layer corresponding to the extracted features of crop image data and the converted features of sensor state data, repeat the pre-training process of the above steps to obtain the respective pre-training parameters. The difference is that in the steps of calculating the energy function, the activation probability, etc., according to the respective data dimensions and features, the corresponding adjustment is made. Different modal data has its own unique features, and needs to be pre-trained specifically, so that each RBM layer can learn the effective features of the corresponding modal data, and the effect is to comprehensively extract the feature information of multi-modal data.

[0092] The parameters obtained by pre-training each layer of RBM are used as the initial parameters of the multi-modal feature fusion model. Then, the extracted features of numerical data, crop image data and converted features of sensor state data are simultaneously input into the multi-modal feature fusion model, and the labeled crop growth condition data is used for training. The output of the multi-modal feature fusion model is y, and the labeled crop growth condition data is t. The cross-entropy loss function is used where K is the number of categories. Through the back propagation algorithm, the gradient is calculated according to the loss function and the model parameters are updated, so that the prediction output of the model is as close to the true label as possible. The initial parameters obtained by pre-training provide a good starting point for the model, but still need to be fine-tuned according to the specific task. Combining the advantages of unsupervised pre-training and supervised fine-tuning, the complex relationship between multi-modal data and crop growth conditions can be better learned, and the fusion feature vector obtained can comprehensively reflect the accuracy of crop growth environment data.

[0093] The adaptive calibration unit 3 constructs a data calibration model, combines the fusion feature vector with historical data, and then dynamically adjusts the calibration model parameters according to the current data actual situation and calibration effect with the help of reinforcement learning algorithm.

[0094] The reinforcement learning algorithm in the adaptive calibration unit 3 dynamically adjusts the calibration model parameters using a deep Q network, as follows:

[0095] The deep Q network structure is constructed. First, the input layer is determined. The input layer receives the combined feature representation of the fusion feature vector and the historical data. The dimension of the fusion feature vector is d f , and the dimension of the historical data after feature engineering processing is d h . Therefore, the dimension of the combined feature representation is d = d f +d h . The number of input layer nodes is d, which can comprehensively input the information of multi-modal data fusion and the related information of historical data into the network. The fusion feature vector contains comprehensive information of the current multi-modal data, and the historical data can provide trend and regularity information of the data. The combination of the two can provide a more comprehensive basis for calibration model parameter adjustment, so that the network can make full use of multi-source data information for decision-making, and improve the accuracy and adaptability of the parameter adjustment strategy.

[0096] The hidden layer is set, a plurality of fully connected layers are constructed, three hidden layers are set, the first hidden layer node number is set as n1=128, the second hidden layer node number n2=64, and the third hidden layer node number n3=32, and ReLU is used as an activation function after each hidden layer, that is, ReLU(x)=max(0,x); The role of the hidden layer is to abstract and extract features from the input data, multiple hidden layers can gradually extract higher-level and more abstract features, with the increase of network depth, more complex relationships and patterns in the data can be learned, the expression ability of the network is enhanced, the network can better understand the mapping relationship between the input data and the calibration model parameter adjustment strategy, and the ability of the network to develop appropriate parameter adjustment strategies for different data conditions is improved.

[0097] The output layer is determined, the output layer corresponds to the calibration model parameter adjustment strategy, the output layer node number is the same as the number of adjustable parameter types, the output of the network is directly corresponding to the parameter adjustment of the calibration model, which is convenient for directly operating the calibration model according to the decision of the network, realizes the direct mapping from data features to parameter adjustment strategy, improves the efficiency and accuracy of adjustment, and can quickly and accurately adjust the calibration model parameters according to the data situation.

[0098] The actual situation and calibration effect of the current data are quantified as the reward signal of the deep Q network. First, the index for measuring the calibration effect is determined by using the mean square error to measure the error between the calibrated data and the real crop growth data, and the calibrated data is set as The real data is y, then Where m is the number of data samples. If the mean square error is reduced compared with the last calibration, it means that the calibration effect is improved, and then a positive reward is given, otherwise, a negative reward is given. The quantified reward signal is to let the deep Q network know the good and bad of the parameter adjustment strategy it takes, so as to guide the network to learn a better strategy, so that the network can continuously optimize its decision-making process according to the reward feedback, and improve the effectiveness of the network in adjusting the calibration model parameters.

[0099] The deep Q network is trained through the experience replay mechanism. First, an experience replay buffer is created to store historical state, action and reward information. After each network parameter adjustment decision, the current state, action and reward obtained are stored in the experience replay buffer. The state is set as s, the action a, and the reward r, and the tuple (s, a, r) is stored.

[0100] In the training process, a batch of data is randomly sampled from the experience replay buffer, the sampling number is set as b, and for each data tuple (s i , a i , r i), the target Q value is calculated, and the double Q learning strategy is adopted for the calculation of the target Q value, and the formula is where Q(s i , a i ) is the Q value estimation of the current network for state s i and action a i , α is the learning rate, γ is the discount factor, which is used to weigh the importance of future rewards, Q tar (s i+1 , a') is the maximum Q value estimation of the target network for the next state s i+1 and different actions a', and the target network regularly copies the parameters from the main network to stabilize the training process. The experience replay mechanism can break the correlation between data, improve the stability and generalization ability of network training, and the double Q learning strategy can reduce the overestimation problem of Q value estimation, so that the network can learn the optimal parameter adjustment strategy more effectively, and improve the decision-making ability of the network under different data conditions.

[0101] According to the calculated target Q value, the gradient descent algorithm is used to update the parameters of the main network, and the parameters of the network are θ, and the loss function adopts the mean square error loss function The gradient of the loss function with respect to the parameter θ is calculated θ L, and then the update rule of the gradient descent algorithm θ = θ-η▽ θ L is used to update the parameters of the main network. The gradient descent algorithm can adjust the network parameters in the direction of reducing the loss function according to the gradient information of the loss function, so that the Q value estimation of the network gradually approaches the target Q value, which can effectively optimize the network parameters, make the network learn a better parameter adjustment strategy, and continuously improve the decision-making ability of the network, so that it can more accurately adjust the calibration model parameters according to the data situation, and improve the quality of data calibration.

[0102] The calibration evaluation feedback unit 4 sets the calibration effect quantitative index, evaluates the calibrated data according to the calibration effect quantitative index, and if the calibration effect is not as expected, feeds back to the multi-modal data fusion unit 2 to adjust the feature extraction strategy and the feature weight distribution strategy, and feeds back to the adaptive calibration unit 3 to optimize the parameter adjustment strategy, and at the same time, provides decision support for agricultural production based on the calibrated data.

[0103] In the application, the multi-modal data is collected and associated by the data collection association unit 1, the multi-modal data fusion unit 2 extracts features by using a convolutional neural network, the precise fusion feature vector is obtained by deeply fusing the data through a multi-modal feature fusion model and various optimization technologies, the self-adaptive calibration unit 3 dynamically adjusts the calibration model parameters by combining historical data with the help of a reinforcement learning algorithm and a deep Q network, the calibration effect is evaluated by the calibration evaluation feedback unit 4, and if the effect is not expected, feedback optimization is provided, and meanwhile, decision support is provided based on the calibrated data, so that the problem of data accuracy caused by the complex agricultural environment is effectively solved, and the data auditing reliability is improved.

[0104] The basic principles, main features and advantages of the present application are shown and described above. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A non-corresponding audit system for Internet of Things data sharing, characterized in that: It includes a data acquisition and association unit (1), a multimodal data fusion unit (2), an adaptive calibration unit (3), and a calibration evaluation feedback unit (4); The data collection and association unit (1) deploys sensors and cameras in farmland and greenhouses respectively, each sensor collects data at a predetermined time interval and attaches a timestamp and device ID, and records the operating status data of the sensor, and then uses a distributed data association network to associate multimodal data within the same time window and located in the same farmland area and greenhouse to form an associated data set; The multimodal data fusion unit (2) directly extracts features from numerical data, and extracts texture, color distribution, and leaf morphology features from crop image data through a feature conversion module (21) using a convolutional neural network and a multi-branch parallel architecture, and encodes them into feature vectors. The sensor state data is converted into feature representations according to a preset encoding rule, and a multimodal feature fusion model of a feature enhancement module (22) is introduced. The multimodal feature fusion model adopts a feature enhancement method and an attention mechanism based on a generative adversarial network, and dynamically allocates weights based on the importance of different modal data at different stages of crop growth to the judgment of growth status. The weight allocation result is input into a weight optimization module (23). When fusing multimodal data, the weight optimization module (23) adopts an attention mechanism weight optimization method based on quantum particle swarm optimization to deeply fuse the multimodal data. Finally, a feature pre-training method based on a deep belief network is adopted by a feature fusion module (24) to obtain a fusion feature vector that comprehensively reflects the accuracy of crop growth environment data. The adaptive calibration unit (3) constructs a data calibration model, combines the fusion feature vector with the historical data, and then dynamically adjusts the calibration model parameters based on the actual situation of the current data and the calibration effect with the help of a reinforcement learning algorithm; The calibration evaluation feedback unit (4) sets a calibration effect quantitative index, and evaluates the calibrated data based on the calibration effect quantitative index. If the data does not meet expectations, the calibration evaluation feedback unit (4) provides feedback to the multimodal data fusion unit (2) to adjust the feature extraction strategy and the feature weight distribution strategy, and provides feedback to the adaptive calibration unit (3) to optimize the parameter adjustment strategy, while providing decision support for agricultural production based on the calibrated data.

2. The non-corresponding audit system for Internet of Things data sharing according to claim 1 is characterized by: The feature conversion module (21) specifically operates as follows: Three parallel branches are constructed. The first branch uses a convolutional neural network to extract multi-scale texture features by setting the convolution kernel size. After the features are extracted, they are input into the normalization layer and activation function, and the dimension is reduced by the maximum pooling layer. The second branch uses a residual network to extract image semantic features and sets residual blocks. The residual blocks are used to solve the gradient vanishing problem in deep neural networks. Each residual block contains two convolutional layers and a skip connection. The skip connection is used to directly connect the output of a layer to the input of a subsequent layer. The third branch uses a deep learning network to extract the overall color distribution and rough shape features of the image. The features extracted by the three branches are spliced ​​and fused at the end of the network to form crop image features.

3. The non-corresponding audit system for Internet of Things data sharing according to claim 2 is characterized by: After converting the sensor state data into feature representation, the feature conversion module (21) adopts a feature dimension reduction method based on kernel principal component analysis, specifically as follows: First, calculate the kernel matrix of the sensor state data feature matrix, select the Gaussian kernel function, and then solve the eigenvalues ​​and eigenvectors of the kernel matrix; The eigenvalues ​​corresponding to the first several eigenvectors whose cumulative contribution rates reach a predetermined threshold are selected to construct a projection matrix, and the original sensor state data features are projected onto the projection matrix to obtain the feature representation after dimensionality reduction.

4. The non-corresponding audit system for Internet of Things data sharing according to claim 3 is characterized by: The feature enhancement module (22) specifically operates as follows: Constructing a generator network and a discriminator network, wherein the generator network takes a random noise vector and a fused feature vector as input, wherein the random noise vector is used to generate a new data sample, and a new feature vector is generated through a deconvolution layer and an upsampling operation; The discriminator network receives the true fused feature vector and the feature vector generated by the generator, performs feature extraction and discrimination through a convolutional network, and outputs a discrimination result representing the authenticity probability of the input feature vector. The true fused feature vector refers to a new feature vector formed by integrating multiple feature vectors. During the training process, the generator and the discriminator perform adversarial training. Through this adversarial training mechanism, the generator generates a secondary feature vector, which is then concatenated with the original fusion feature vector and input into the fusion model again for deep fusion.

5. The non-corresponding audit system for Internet of Things data sharing according to claim 1 is characterized in that: The multimodal data fusion unit (2) adopts an adaptive learning rate adjustment strategy during the multimodal feature fusion model training process, which is as follows: Set the initial learning rate η0. During the training process, after every t training rounds, calculate the loss function value L of the current model on the validation set. t The moving average of the loss function values ​​of the previous k training batches Ratio If r t <θ1, then the learning rate increases by α%, if r t >θ2, the learning rate decreases by β%, if θ1≤r t ≤θ2, the learning rate remains unchanged, and θ1 and θ2 are preset thresholds.

6. The non-corresponding audit system for Internet of Things data sharing according to claim 4 is characterized in that: The weight optimization module (23) specifically operates as follows: The weight parameters in the attention mechanism are encoded into quantum bits to construct a quantum particle swarm. Each quantum particle represents a set of possible weight parameter combinations, and the position and velocity of the particle correspond to the value and update direction of the weight parameter respectively. During the quantum particle swarm optimization process, the evaluation index of the multimodal data fusion model is calculated based on the current position of the particle. The evaluation index is the similarity between the fusion feature vector and the real crop growth status label; The individual optimal position and group optimal position of the particles are updated according to the evaluation indicators, and the position and velocity of the particles are updated through quantum rotating gate operation. After multiple iterations, the optimal weight parameter combination is obtained.

7. The non-corresponding audit system for Internet of Things data sharing according to claim 6, characterized in that: The feature fusion module (24) specifically operates as follows: Constructing a deep belief network, which includes a restricted Boltzmann machine layer. The features extracted from numerical data, crop image data, and sensor state data are input into corresponding restricted Boltzmann machine layers for pre-training. During the pre-training process, the contrastive divergence algorithm is used to train the restricted Boltzmann machine layer. The input data is reconstructed by adjusting the network parameters to obtain the quadratic parameters, which are used as the initial parameters of the multimodal feature fusion model. Then, labeled crop growth status data is used for training to obtain a fusion feature vector that comprehensively reflects the accuracy of crop growth environment data.

8. The non-corresponding audit system for Internet of Things data sharing according to claim 1, characterized in that: The reinforcement learning algorithm in the adaptive calibration unit (3) uses a deep Q network to dynamically adjust the calibration model parameters, as follows: Construct a deep Q network structure, including an input layer, a hidden layer, and an output layer. The input layer receives the combined feature representation of the fused feature vector and the historical data, and the output layer corresponds to the calibration model parameter adjustment strategy. The number of nodes in the output layer is the same as the number of adjustable parameters. The actual situation of the current data and the calibration effect are quantified and used as the reward signal of the deep Q network. The deep Q network is trained through the experience replay mechanism, storing historical state, action and reward information. During the training process, a batch of data is randomly sampled from the experience replay buffer to calculate the target Q value; According to the calculated target Q value, the parameters of the main network are updated using the gradient descent algorithm.

Images