Farmland disease remote sensing image prediction generation method and device based on stable diffusion model and processing method

Abstract

The invention discloses a farmland disease remote sensing image prediction generation method and device based on a stable diffusion model and a processing method. The prediction generation method comprises the following steps: firstly, obtaining a model image data set from remote sensing image data, and fusing the model image data set to obtain a fusion feature; according to the fusion features, generating data I'm; then evaluating the difference between the image data Im in the mth time period and the predicted image data I'm in the mth time period to obtain an image evaluation index, and if the image evaluation index can meet a preset value requirement, setting the stable diffusion model as a target stable diffusion model; and according to the target stable diffusion model, predicting prediction image data Im + 1 of the farmland disease in the (m + 1) th time period, and finally carrying out semantic segmentation on the prediction data to obtain potential disease data. The prediction generation method can combine timing and color information. According to the processing method, the generated future data can be analyzed and processed to obtain potential disease data which can be visually displayed.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method, device and processing method for predicting and generating farmland disease remote sensing images based on a stable diffusion model. Background Art

[0002] When using remote sensing images to analyze farmland diseases, the existing technology is to observe and analyze the current images to predict the type of disease. However, this method mainly relies on the experience of the observer or analyst, and cannot capture the dynamic changes of the disease, and cannot accurately and reliably predict farmland diseases. Summary of the invention

[0003] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies in the prior art and propose a method, device and processing method for predicting and generating farmland disease remote sensing images based on a stable diffusion model. The prediction and generation method can automatically generate high-quality prediction images, thereby improving the accuracy and reliability of potential farmland disease prediction.

[0004] In a first aspect, the present invention provides a method for predicting and generating farmland disease remote sensing images based on a stable diffusion model, the method specifically comprising the following steps:

[0005] Step S1: Acquire N groups of remote sensing image data of farmland disease areas;

[0006] Step S2: randomly extracting one group from the N groups of remote sensing image data of the farmland disease area as a model image data set; the model image data set includes the first time period image data I1, the second time period image data I2, ..., the m-1th time period image data I m-1 , image data I for the mth time period m ;

[0007] Step S3: The first time period image data I1, the second time period image data I2, ..., the m-1th time period image data I m-1 Perform fusion to obtain fusion features;

[0008] Step S4: Input the fusion feature into the stable diffusion model, so that the stable diffusion model outputs the predicted image data I' of the mth time period of the farmland disease m ;

[0009] Step S5: Evaluate the image data I of the mth time period m and the predicted image data I' for the mth time period m The difference between them is used to obtain the image evaluation index;

[0010] Step S6: Determine whether the image evaluation index meets the preset requirements:

[0011] If the image evaluation index can meet the preset value requirement, the stable diffusion model is set as the target stable diffusion model; if not, another set of data is randomly selected from N as the model data set, and steps S2 to S6 are repeated until the image evaluation index can meet the preset requirement;

[0012] Step S7: According to the target stable diffusion model, predict the image data I of the m+1th time period of farmland disease m+1 , in order to complete the prediction and generation of farmland disease remote sensing images based on the stable diffusion model;

[0013] Wherein, N and m are both natural numbers greater than 1.

[0014] Furthermore, the step S3 specifically includes the following steps:

[0015] Step S31: from the first time period image data I1, the second time period image data I2, ..., the m-1th time period image data I m-1 Extracting time series features; and, from the image data I of the m-1th time period m-1 Extract color features;

[0016] Step S32: Fusing the time series feature and the color feature to obtain a fusion feature X θ ;

[0017] The calculation formula of the fusion feature is as follows:

[0018]

[0019] in,

[0020] X t Represents the time series characteristics, X c Represents color characteristics, w t represents the time series feature weight, w c represents the weight of color features;

[0021] Represents matrix concatenation.

[0022] Furthermore, the step S5 specifically includes the following steps:

[0023] The FID is used to measure the image data I in the mth time period. m and the predicted image data I' for the mth time period m The similarity between them is used to obtain the FID score;

[0024] The calculation formula of the FID is as follows:

[0025]

[0026] Among them, g represents the characteristics of Im;

[0027] r represents the characteristics of I'm;

[0028] μ g represents the mean of the features of Im;

[0029] Σ g represents the covariance matrix of Im;

[0030] μ r Indicates I' m The mean of the features, Σ r Indicates I' m The covariance matrix of the features, is μ g and μ r The L2 norm of the squared difference, Tr represents the trace of the matrix.

[0031] Furthermore, the step S4 specifically includes the following steps:

[0032] Step S41: Fusing the fused features into the denoising network of the stable diffusion model in a cross-attention mechanism to obtain an attention output Z;

[0033] The calculation formula of the attention output Z is as follows:

[0034]

[0035] in,

[0036] CrossAttention represents the cross attention mechanism function;

[0037] Softmax() represents the normalization function;

[0038] Q represents the query matrix;

[0039] K represents the bond matrix;

[0040] K T represents the transpose operation of K;

[0041] V represents the value matrix;

[0042] d represents the scaling factor;

[0043] The calculation formula of the query matrix Q is as follows:

[0044] Q=W Q ·φ(z t ) (3)

[0045] Among them, z t represents the input feature at time t, φ(z t ) represents z t The characteristic conversion function, W Q represents the query matrix weight;

[0046] The calculation formula of the key matrix K is as follows:

[0047] K=W K ·τ(X θ ) (4)

[0048] Among them, X θ represents the fusion feature, τ(X θ ) indicates X θ The characteristic conversion function, W K represents the bond matrix weight;

[0049] The calculation formula of the key matrix K is as follows:

[0050] V=W V ·τ(X θ ) (5)

[0051] Among them, W v represents the value matrix weight;

[0052] Step S42: Decode and transform the attention output Z to obtain the predicted image data I' of the mth time period of farmland disease m , which is expressed as formula (6):

[0053] I' m =D(Z) (6)

[0054] Among them, the D(Z) function represents the decoding output of the attention output.

[0055] Furthermore, the denoising network of the stable diffusion model in step S41 is trained by a loss function;

[0056] The loss function for training the denoising network is as follows:

[0057]

[0058] in,

[0059] E represents the expected value, x represents the input sample, δ represents the parameter sampled from the standard normal distribution N(0,1), and t represents time;

[0060] δ σ (x t ,t) represents the output after cross attention mechanism and time-color processing;

[0061] The squared error defined for the 2-norm.

[0062] In a second aspect, the present invention provides a method for treating potential farmland diseases, the method comprising the following steps:

[0063] Step K1: Using the method for predicting and generating farmland disease remote sensing images based on the stable diffusion model described in the first aspect, obtain the predicted image data I of the farmland disease in the m+1th time period m+1 ;

[0064] Step K2: generating the data I m+1 Perform semantic segmentation of the diseased area to obtain the potential damage area;

[0065] Step K3: Based on the image data I1 of the first time period, the image data I2 of the second time period, ..., the image data I of the m-1th time period m-1 , image data I for the mth time period m , the predicted image data I m+1 , and combining the potential damage areas to obtain potential disease data;

[0066] Step K4: Processing potential farmland diseases according to the potential disease data.

[0067] Furthermore, the potential disease data includes potential disease area, potential disease type and potential disease trend;

[0068] After step K4, the method further includes step K5;

[0069] Step K5: Visually display the potential disease area, the potential disease type and the potential disease trend.

[0070] In a third aspect, the present invention provides a device for predicting and generating farmland disease remote sensing images based on a stable diffusion model, the device comprising:

[0071] An acquisition unit, used for acquiring remote sensing image data of N groups of farmland disease areas;

[0072] The first processing unit is connected to the acquisition unit and is used to randomly extract a group of remote sensing image data of the N groups of farmland disease areas as a model image data set; the model image data set includes the first time period image data I1, the second time period image data I2, ..., the m-1th time period image data I m-1 , image data I for the mth time period m ;

[0073] The second processing unit is connected to the first processing unit and is used to process the first time period image data I1, the second time period image data I2, ..., the m-1th time period image data I m-1 Perform fusion to obtain fusion features;

[0074] The third processing unit is connected to the second processing unit and is used to input the fusion feature into the stable diffusion model so that the stable diffusion model outputs the predicted image data I' of the mth time period of the farmland disease. m ;

[0075] an evaluation unit connected to the third processing unit, for evaluating the image data I of the mth time period m and the predicted image data I' for the mth time period m The difference between them is used to obtain the image evaluation index;

[0076] A determination unit, connected to the evaluation unit, for determining whether the image evaluation index can meet the preset requirements;

[0077] a fourth processing unit connected to the determination unit, and configured to set the stable diffusion model as a target stable diffusion model if the determination unit determines that the image evaluation index can meet the preset value requirement; and to randomly extract another set of data from N as a model image data set if the determination unit determines that the image evaluation index cannot meet the preset value requirement;

[0078] A prediction unit connected to the fourth processing unit is used to predict the farmland disease in the m+1th time period prediction image data I according to the target stable diffusion model. m+1 , in order to complete the prediction and generation of farmland disease remote sensing images based on the stable diffusion model;

[0079] Wherein, N and m are both natural numbers greater than 1.

[0080] Furthermore, the second processing unit includes:

[0081] The extraction module is used to extract the image data I1 of the first time period, the image data I2 of the second time period, ..., the image data I of the m-1th time period, m-1 Extracting time series features; and, from the image data I of the m-1th time period m-1 Extract color features;

[0082] A fusion module, connected to the extraction module, is used to fuse the time series feature and the color feature to obtain a fusion feature X θ ;

[0083] The calculation formula of the fusion feature is as follows:

[0084]

[0085] in,

[0086] X t Represents the time series characteristics, X c Represents color characteristics, w t represents the time series feature weight, w c represents the weight of color features;

[0087] Represents matrix concatenation.

[0088] Furthermore, the third processing unit includes:

[0089] A processing module, used for fusing the fused features into a denoising network of a stable diffusion model in a cross-attention mechanism to obtain an attention output Z;

[0090] The calculation formula of the attention output Z is as follows:

[0091]

[0092] in,

[0093] CrossAttention represents the cross attention mechanism function;

[0094] Softmax() represents the normalization function;

[0095] Q represents the query matrix;

[0096] K represents the bond matrix;

[0097] K T represents the transpose operation of K;

[0098] V represents the value matrix;

[0099] d represents the scaling factor;

[0100] The calculation formula of the query matrix Q is as follows:

[0101] Q=W Q ·φ(z t );

[0102] Among them, z t represents the input features at time t, φ(z t ) represents z t The characteristic conversion function, W Q represents the query matrix weight;

[0103] The calculation formula of the key matrix K is as follows:

[0104] K=W K ·τ(X θ );

[0105] Among them, X θ represents the fusion feature, τ(X θ ) indicates X θ The characteristic conversion function, W K represents the bond matrix weight;

[0106] The calculation formula of the key matrix K is as follows:

[0107] V=W V ·τ(X θ );

[0108] Among them, W v represents the value matrix weight;

[0109] A decoding module is connected to the processing module and is used to decode and convert the attention output Z to obtain the predicted image data I' of the mth time period of the farmland disease. m , which is expressed as follows:

[0110] I' m =D(Z);

[0111] Among them, the D(Z) function represents the decoding output of the attention output.

[0112] Beneficial effects of the present invention:

[0113] 1. The present invention can automatically generate high-quality prediction images based on existing image data, thereby improving the accuracy and reliability of potential farmland disease prediction.

[0114] 2. The present invention integrates time series features and color features, so as to more comprehensively reflect the growth status and disease evolution of farmland and improve the prediction accuracy of the neural network model.

[0115] 3. The present invention performs quantitative evaluation on the generated images based on image evaluation indicators, which helps to objectively analyze the model performance, facilitates optimization and adjustment, and improves the reliability of the entire neural network model.

[0116] 4. The present invention can treat potential farmland diseases and help formulate more targeted prevention and control strategies.

[0117] 5. The present invention can accurately identify potential disease areas, thereby enabling the rational allocation of prevention and control resources.

[0118] 6. The present invention can effectively reduce the impact of potential farmland diseases on agricultural production, improve crop yield and quality, and promote the development of sustainable agriculture through early identification, precise intervention and data-driven decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 It is a schematic diagram of a method for predicting and generating farmland disease remote sensing images based on a stable diffusion model in an embodiment of the present invention;

[0120] Figure 2 Schematic diagram of the process of processing potential farmland diseases in an embodiment of the present invention;

[0121] Figure 3 Schematic diagram of the neural network model structure in an embodiment of the present invention;

[0122] Figure 4 Schematic diagram of a device for predicting and generating farmland disease remote sensing images based on a stable diffusion model in an embodiment of the present invention;

[0123] Among them, the figure numbers are: 10, acquisition unit, 20, first processing unit, 30, second processing unit, 40, third processing unit, 50, evaluation unit, 60, determination unit, 70, fourth processing unit, 80, prediction unit. DETAILED DESCRIPTION

[0124] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0125] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0126] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0127] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0128] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0129] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0130] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0131] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0132] Embodiment 1:

[0133] like Figure 1 and Figure 2 As shown, this embodiment provides a method for predicting and generating farmland disease remote sensing images based on a stable diffusion model, and the method specifically includes the following steps:

[0134] Step S1: Obtain N groups of remote sensing image data of farmland disease areas.

[0135] Specifically, we must first clarify the requirements such as farmland disease types, time range and image resolution; then select the appropriate remote sensing data source, such as public satellite data (NASA, USGS), commercial satellite images (such as Planet Labs) or drone images; finally, obtain images through remote sensing satellites or drones.

[0136] Step S2: randomly extracting one group from the N groups of remote sensing image data of the farmland disease area as a model image data set; the model image data set includes the first time period image data I1, the second time period image data I2, ..., the m-1th time period image data I m-1 , image data I for the mth time period m .

[0137] This implementation uses a random sampling method to extract a set of data from N sets of remote sensing image data. In addition, it can also be achieved by writing a random number generation program or using tools (such as the random library in Python).

[0138] Step S3: The first time period image data I1, the second time period image data I2, ..., the m-1th time period image data I m-1 The fusion is performed to obtain the fusion features.

[0139] There are many image fusion methods, such as mean fusion and weighted fusion, but none of them is as good as neural network. Neural network is used in this embodiment because of its self-learning ability, nonlinear modeling advantages, robustness and high output quality. As a specific implementation method, step S3 specifically includes the following steps:

[0140] Step S31: from the first time period image data I1, the second time period image data I2, ..., the m-1th time period image data I m-1 Extracting time series features; and, from the image data I of the m-1th time period m-1 Extract color features;

[0141] Step S32: Fusing the time series feature and the color feature to obtain a fusion feature X θ ;

[0142] The calculation formula of the fusion feature is as follows:

[0143]

[0144] in,

[0145] X t Represents the time series characteristics, X c Represents color characteristics, w t represents the time series feature weight, w c represents the weight of color features;

[0146] Represents matrix concatenation.

[0147] Step S4: Input the fusion feature into the stable diffusion model, so that the stable diffusion model outputs the predicted image data I' of the mth time period of the farmland disease m .

[0148] As a specific implementation, step S4 specifically includes the following steps:

[0149] Step S41: Fusing the fused features into the denoising network of the stable diffusion model in a cross-attention mechanism to obtain an attention output Z;

[0150] The calculation formula of the attention output Z is as follows:

[0151]

[0152] in,

[0153] CrossAttention represents the cross attention mechanism function;

[0154] Softmax() represents the normalization function;

[0155] Q represents the query matrix;

[0156] K represents the bond matrix;

[0157] K T represents the transpose operation of K;

[0158] V represents the value matrix;

[0159] d represents the scaling factor;

[0160] The calculation formula of the query matrix Q is as follows:

[0161] Q=W Q ·φ(z t )(3)

[0162] Among them, z t represents the input features at time t, φ(z t ) represents z t The characteristic conversion function, W Q represents the query matrix weight;

[0163] The calculation formula of the key matrix K is as follows:

[0164] K=W K ·τ(X θ )(4)

[0165] Among them, X θ represents the fusion feature, τ(X θ ) indicates X θ The characteristic conversion function, W K represents the bond matrix weight;

[0166] The calculation formula of the key matrix K is as follows:

[0167] V=W V ·τ(X θ )(5)

[0168] Among them, W v represents the value matrix weight;

[0169] Step S42: Decode and transform the attention output Z to obtain the predicted image data I' of the mth time period of farmland disease m , which is expressed as formula (6):

[0170] I' m =D(Z) (6)

[0171] Among them, the D(Z) function represents the decoding output of the attention output.

[0172] Furthermore, the denoising network of the stable diffusion model in step S41 is trained by a loss function;

[0173] The loss function for training the denoising network is as follows:

[0174]

[0175] in,

[0176] E represents the expected value, x represents the input sample, δ represents the parameter sampled from the standard normal distribution N(0,1), and t represents time;

[0177] δ σ (x t ,t) represents the output after cross attention mechanism and time-color processing;

[0178] The squared error defined for the 2-norm.

[0179] Step S5: Evaluate the image data I of the mth time period m and the predicted image data I' for the mth time period m The difference between them is used to obtain the image evaluation index.

[0180] As a specific implementation, step S5 specifically includes the following steps:

[0181] The FID is used to measure the image data I in the mth time period. m and the predicted image data I' for the mth time period m The similarity between them is used to obtain the FID score;

[0182] The calculation formula of the FID is as follows:

[0183]

[0184] Among them, g represents the characteristics of Im;

[0185] r represents the characteristics of I'm;

[0186] μ g represents the mean of the features of Im;

[0187] Σ g represents the covariance matrix of Im;

[0188] μ r Indicates I' m The mean of the features, Σ r Indicates I' m The covariance matrix of the features, is μg and μ r The L2 norm of the squared difference, Tr represents the trace of the matrix.

[0189] In addition to Fréchet Inception Distance (FID), there are many other evaluation indicators that can evaluate the image data I in the mth time period. m and the predicted image data I' for the mth time period m The differences between them mainly include the following:

[0190] 1. Structural Similarity Index (SSIM)

[0191] SSIM is a metric used to measure the difference in visual quality between two images, taking into account factors such as brightness, contrast, and structure. The value of SSIM ranges from 0 to 1, with 1 indicating that they are exactly the same and 0 indicating that they are completely different.

[0192] 2. Peak Signal-to-Noise Ratio (PSNR)

[0193] Note: PSNR is a commonly used image quality evaluation indicator, which expresses the signal-to-noise ratio between the original image and the generated image. The PSNR value is usually expressed in decibels (dB), and the higher the value, the better the quality.

[0194] 3. Perceptual Loss for Generative Adversarial Networks

[0195] By calculating the difference between the generated image and the real image in the feature space, a pre-trained convolutional neural network (such as VGG) is usually used to extract features and compare the differences in high-level features.

[0196] 4. Mean Absolute Error (MAE)

[0197] Calculate the average absolute difference between the corresponding pixel values ​​of the two images. The smaller the MAE, the closer the generated image is to the real image.

[0198] 5. Mean Square Error (MSE)

[0199] The MSE is the squared average of the differences between the image pixel values. The smaller the MSE, the closer the generated image is to the real image. Although MSE is simple, it has limited reflection on the perceived quality.

[0200] This embodiment adopts the FID method, but other methods may also be adopted.

[0201] Step S6: Determine whether the image evaluation index meets the preset requirements:

[0202] If the image evaluation index can meet the preset value requirements, the stable diffusion model is set as the target stable diffusion model; if not, another set of data is randomly selected from N as the model data set, and steps S2 to S6 are repeated until the image evaluation index can meet the preset requirements.

[0203] Step S7: According to the target stable diffusion model, predict the image data I of the m+1th time period of farmland disease m+1 , in order to complete the prediction and generation of remote sensing images of farmland diseases based on the stable diffusion model.

[0204] Wherein, N and m are both natural numbers greater than 1.

[0205] Embodiment 2:

[0206] like Figure 1 As shown, this embodiment provides a method for treating potential farmland diseases, the method comprising the following steps:

[0207] Step K1: Using the method for predicting and generating farmland disease remote sensing images based on the stable diffusion model described in Example 1, obtain the predicted image data I of the farmland disease in the m+1th time period m+1 .

[0208] Step K2: generating the data I m+1 Perform semantic segmentation of the diseased area to obtain the area where potential damage may occur.

[0209] Specifically, the optional semantic segmentation models in this embodiment include U-Net, DeepLab or FCN. However, U-Net is suitable for image segmentation tasks, and this embodiment combines the U-Net denoising network, cross attention mechanism and time-color module in the stable diffusion model.

[0210] Step K3: Obtain potential disease data based on the first time period image data I1, the second time period image data I2, ..., the m-1th time period image data Im-1, the mth time period image data Im, the predicted image data Im+1, and the potential damage area. The potential disease data includes potential disease area, potential disease type, and potential disease trend.

[0211] Potential disease area refers to the total area of ​​affected areas identified in the potential damage area; the process of obtaining the potential disease area is to first extract all disease areas from the binary mask of the potential damage area, and then convert them into actual areas based on the number of pixels in these areas and the spatial resolution of the image. Potential disease types are different types of potential diseases, referring to the different types of identified diseases; the process of obtaining potential disease types is to first use machine learning or deep learning methods (such as convolutional neural networks) to extract features from potential disease areas, and then compare the extracted features with the existing disease type database for classification and identification. Potential disease trend refers to the trend of disease development over time; the process of obtaining potential disease trend is to use regression models or time series prediction algorithms (such as LSTM) based on existing data to predict the development trend of diseases in the future time period.

[0212] Step K4: Processing potential farmland diseases according to the potential disease data;

[0213] Step K5: Visually display the potential disease area, the potential disease type and the potential disease trend to complete the treatment of potential farmland diseases.

[0214] Embodiment 3:

[0215] This embodiment is a combination of Embodiment 1 and Embodiment 2, which generates future farmland disease remote sensing images through existing time-series remote sensing images, and combines the semantic segmentation model to identify the diseased areas in the future images, thereby achieving an intuitive and detailed prediction of the future farmland disease range.

[0216] Combine the following Figure 1 The specific implementation process of this implementation is introduced as follows:

[0217] 1. Construct a data set: Use a drone to take remote sensing images of diseased farmland areas, and group them according to farmland areas and time series relationships. This embodiment sets the time span to 1 day. When the weather is clear and the light is sufficient at noon every day, the drone takes photos of the same diseased farmland. The drone is at the same height during shooting, and the lens position and zoom are as similar as possible. The remote sensing images taken from the first day to the fifth day are grouped as a group of 5 images, denoted as I1, I2, I3, I4, and I5. You can shoot separately under different farmland, different diseases, different time periods, and different time spans to obtain N groups of data, each with 5 images, to construct a data set D, that is, Here, k represents a data group. In this embodiment, it is assumed that {I1, I2, I3, I4, I5} is a group of data randomly extracted from the data set D, and the following description is made under this premise.

[0218] 2. Model construction: The structure of the stable diffusion model includes three separately trained parts, namely, perceptual image compression based on variational autoencoder (VAE) and decoder, latent diffusion model based on denoising network, and conditional mechanism based on contrastive language-image pre-training (CLIP) multimodal model. On this basis, the time-color module proposed in this embodiment can replace the original conditional mechanism, integrate time-series information and image color information, and generate images through the cross-attention mechanism combined with the denoising network of the diffusion model. The model structure is as follows: Figure 3 shown.

[0219] like Figure 3 As shown, in the timing section, They represent the results of converting I1, I2, I3, and I4 into grayscale images. Denote the parameter as θ t Multi-layer convolutional neural network, X t Indicated by The timing characteristics of the output. In the color section, represents the three-channel matrix obtained by digitizing I4, Denote the parameter as θ c Multi-layer convolutional neural network, X c Indicated by The color features of the output. In the following feature fusion process, w t and w c Respectively represent the weight matrices corresponding to the fusion of temporal features and color features, X θ represents the result of fusion, θ represents the parameters of the convolutional neural network in the module, that is, θ t and θ c , Represents splicing The calculation formula (1) for the left and right matrices is as follows:

[0220]

[0221] In this embodiment, the resolution of a group of data {I1, I2, I3, I4, I5} in the data set D is 512x512. In the time series part, {I1, I2, I3, I4} is converted into a grayscale image and normalized by z-score. The size of is 512x512x4, that is In the next multi-layer convolutional neural network In , the parameters of the first convolution layer are (4,64,3,1), which represent the number of input channels, the number of output channels, the convolution kernel size, and the stride. The parameters of the second convolution layer are (64,128,3,2), and repeated padding is used for padding. The parameters of the third convolution layer are (128,128,3,2), the parameters of the fourth convolution layer are (128,64,3,2), and the parameters of the fifth convolution layer are (64,1,3,2). The output of each convolution layer is followed by batch normalization and activation function ReLU, and the final output feature map X is obtained. t The size of is 31×31×1, where the length and width are 31 and the number of channels is 1, that is,

[0222] In the color part, the input I4 is first converted into three channels of R, G, and B and z-score normalized to obtain The size is 512×512×3, that is In the next In the example, the parameters of the first convolution layer are (3, 64, 3, 1), and the parameters of the remaining four convolution layers are the same as those of the timing part. A normalization layer and an activation function are also connected after each convolution layer, and the final output feature map X c The size of is 31×31×1, where the length and width are 31 and the number of channels is 1, that is,

[0223] In the feature fusion stage, w t and w c are two randomly initialized trainable weight matrices, weights w t With the timing characteristic diagram X t Multiply them together, the size is 31×31, that is And w t The element a of the matrix i,j The value range of (1≤i≤31,1≤j≤31) is a i,j ∈(-1,1). On the other hand, the weight w c With color feature map X c Multiplication, w c The size is also 31×31, that is And w t The matrix element b i,j The value range is b i,j ∈(-1,1). Through formula (1), the two features are multiplied by the corresponding weight matrix respectively, and the two features can be mapped to the same feature space, and then spliced ​​together to complete the fusion, and the fusion result X with a size of 31×62 is obtained. θ ,Right now

[0224] Next, this embodiment will X θ It is fused into the denoising network of the stable diffusion model in the form of a cross-attention mechanism, and the fusion calculation is shown in the following formula:

[0225]

[0226] K=W K ·τ(X θ ) (3)

[0227] V=W V ·τ(X θ ) (4)

[0228]

[0229] The stable diffusion model will randomly generate a noisy image when predicting, and then gradually predict the noise through the model, gradually remove the noise, and generate an image. During training, the stable diffusion model will gradually superimpose random noise on the target image, and then gradually predict the noise through the model to update the model parameters. In the above formula, z t represents the prediction result of the t-th step of the stable diffusion model denoising network, and τ(X θ )=w τ X θ Respectively represent z t and X θ The linear transformation function of z t and X θ Mapped to the same feature space, where and w τ is a randomly initialized, trainable weight matrix.

[0230] In the cross-attention mechanism, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively. According to formulas (2), (3), and (4), and τ(X θ ) is linearly transformed again to obtain Q, K and V, where W Q , W K and W V is a randomly initialized, trainable weight matrix. According to formula (5), Q and K T By doing the inner product, we can get the similarity between the query matrix Q and the key matrix K. After normalization, we can do the inner product with the value to get z. t With X θ The fusion result containing attention information is used to denoise the output of the denoising network in the step-by-step denoising process of the stable diffusion model. t With X θCross-attention fusion can gradually inject timing and color information into the denoising network, making the denoised image close to I5.

[0231] In this embodiment, d = 64, then the calculation result of CrossAttention(Q, K, V) is

[0234] 3. Training model: First, this embodiment uses the default method of the stable diffusion model to train the perceptual image compression part. During training, I5 is used as the input of the perceptual image compression. According to the formula Using the Encoder Compress I5 into a latent vector x, and when generating, according to the formula Using a decoder The latent vector x is restored to the remote sensing image I'5.

[0235] Secondly, this embodiment combines the U-Net denoising network, the cross attention mechanism and the time-color module in the stable diffusion model, and uses δ σ Represents, where σ represents the weight parameter of δ, including the weight parameter of the denoising network U-Net, the weight parameter of the cross attention mechanism w τ , W Q , W K , W V , Convolutional Neural Network in Time-Color Module The weight parameter θ t , Convolutional Neural Network The weight parameter θ c and w which integrates temporal color information t and w c In this embodiment, the square error defined by the two-norm is used to define the loss function L and train σ, as shown in the following formula (6):

[0236]

[0237] Among them, x and δ both obey the standard normal distribution, and δ represents the Gaussian noise added in the noise addition stage. In the diffusion model, noise addition and denoising are gradually operated at each time step. Therefore, in formula (6), t represents the time step of noise addition and denoising, and x t represents the latent vector that needs to be denoised in the tth step, δ σ (x t ,t) means that at time step t, the latent vector x t The training uses the adam optimizer and the stochastic gradient descent method to update the network weight parameters.

[0238] Next, this embodiment uses I5 as the real image and I'5 as the generated image, and uses FID (Fréchet Inception Distance) as the evaluation index of the image generation effect. FID is an index that can be used to measure the similarity between image data distributions and is often used to evaluate image generation models. FID uses the Inception V3 model to calculate the feature representation of the real data set and the generated data set and score them, and the calculation is as follows:

[0239]

[0240] Among them, I5 and I'5 are represented by the Inception V3 model, μ g represents the mean of the features of I5, Σ g represents the covariance matrix of the features of I5, μ r represents the mean of the features of I'5, Σ r represents the covariance matrix of the features of I'5, is the L2 norm of the square of the mean vector difference, and Tr represents the trace of the matrix.

[0241] When the FID score is lower, it means that the distribution of the generated image is closer to the real image, the quality of the generated image is higher, and the training model effect is better.

[0242] 4. Generate and analyze predicted images: First, use drones to capture time-series remote sensing image data of farmland that meets the same conditions as the dataset. Secondly, input 4 time-series remote sensing images into the system, and use the model to generate future farmland remote sensing images. Next, use existing semantic segmentation technology solutions such as FCN, PSPNet or DeepLab to analyze the generated remote sensing images, divide the areas where diseases will occur, mark them, and display them in the system. Finally, based on the generated images and segmentation results, calculations are performed, and the result data such as the area, trend, type and suggestions of future farmland diseases are displayed in the system.

[0243] Combined with the proposed time-color module, improvements are made on the existing stable diffusion model, allowing the model to learn the temporal information of images.

[0244] By generating future remote sensing data from historical remote sensing data of farmland and conducting disease analysis on it, the results obtained can be more intuitive and specific than existing farmland disease prediction schemes.

[0245] Embodiment 4:

[0246] like Figure 4 As shown, this embodiment provides a device for predicting and generating farmland disease remote sensing images based on a stable diffusion model, the device comprising:

[0247] An acquisition unit 10 is used to acquire remote sensing image data of N groups of farmland disease areas;

[0248] The first processing unit 20 is connected to the acquisition unit 10, and is used to randomly extract a group of remote sensing image data of the N groups of farmland disease areas as a model image data set; the model image data set includes the first time period image data I1, the second time period image data I2, ..., the m-1th time period image data I m-1 , image data I for the mth time period m ;

[0249] The second processing unit 30 is connected to the first processing unit 20 and is used to process the first time period image data I1, the second time period image data I2, ..., the m-1th time period image data I m-1 Perform fusion to obtain fusion features;

[0250] The third processing unit 40 is connected to the second processing unit 30 and is used to input the fusion feature into the stable diffusion model so that the stable diffusion model outputs the predicted image data I′ of the mth time period of the farmland disease. m ;

[0251] The evaluation unit 50 is connected to the third processing unit 40 and is used to evaluate the image data I of the mth time period. m and the predicted image data I' for the mth time period m The difference between them is used to obtain the image evaluation index;

[0252] The determination unit 60 is connected to the evaluation unit 50 and is used to determine whether the image evaluation index meets the preset requirements;

[0253] The fourth processing unit 70 is connected to the determination unit 60, and is used to set the stable diffusion model as the target stable diffusion model if the determination unit determines that the image evaluation index can meet the preset value requirement; and is also used to randomly extract another set of data from N as the model data set if the determination unit determines that the image evaluation index cannot meet the preset value requirement;

[0254] The prediction unit 80 is connected to the fourth processing unit 70 and is used to predict the farmland disease in the m+1th time period prediction image data I according to the target stable diffusion model. m+1 , in order to complete the prediction and generation of farmland disease remote sensing images based on the stable diffusion model;

[0255] Wherein, N and m are both natural numbers greater than 1.

[0256] As a specific implementation, the second processing unit 30 includes:

[0257] The extraction module is used to extract the image data I1 of the first time period, the image data I2 of the second time period, ..., the image data I of the m-1th time period, m-1 Extracting time series features; and, from the image data I of the m-1th time period m-1 Extract color features;

[0258] A fusion module, connected to the extraction module, is used to fuse the time series feature and the color feature to obtain a fusion feature X θ ;

[0259] The calculation formula of the fusion feature is as follows:

[0260]

[0261] in,

[0262] X t Represents the time series characteristics, X c Represents color characteristics, w t represents the time series feature weight, w c represents the weight of color features;

[0263] Represents matrix concatenation.

[0264] As a specific implementation, the third processing unit 40 includes:

[0265] A processing module, used for fusing the fused features into a denoising network of a stable diffusion model in a cross-attention mechanism to obtain an attention output Z;

[0266] The calculation formula of the attention output Z is as follows:

[0267]

[0268] in,

[0269] CrossAttention represents the cross attention mechanism function;

[0270] Softmax() represents the normalization function;

[0271] Q represents the query matrix;

[0272] K represents the bond matrix;

[0273] K T represents the transpose operation of K;

[0274] V represents the value matrix;

[0275] d represents the scaling factor;

[0276] The calculation formula of the query matrix Q is as follows:

[0277] Q=W Q ·φ(z t );

[0278] Among them, z t represents the input feature at time t, φ(z t ) represents z t The characteristic conversion function, W Q represents the query matrix weight;

[0279] The calculation formula of the key matrix K is as follows:

[0280] K=W K ·τ(X θ );

[0281] Among them, X θ represents the fusion feature, τ(X θ ) indicates X θ The characteristic conversion function, W K represents the bond matrix weight;

[0282] The calculation formula of the key matrix K is as follows:

[0283] V=W V ·τ(X θ );

[0284] Among them, W v represents the value matrix weight;

[0285] A decoding module is connected to the processing module and is used to decode and convert the attention output Z to obtain the predicted image data I' of the mth time period of the farmland disease. m , which is expressed as follows:

[0286] I' m =D(Z);

[0287] Among them, the D(Z) function represents the decoding output of the attention output.

[0288] The device in this embodiment can execute the method in embodiment 1.

[0289] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for predicting and generating farmland disease remote sensing images based on a stable diffusion model. It is characterized in that The method comprises the following steps: Step S1: Acquire N groups of remote sensing image data of farmland disease areas; Step S2: randomly extracting one group from the N groups of remote sensing image data of farmland disease areas as a model image data set; the model image data set includes image data I of the first time period 1 , the second time period image data I 2 , ..., image data I of the m-1th time period m-1 , image data I for the mth time period m ; Step S3: The first time period image data I 1 , the second time period image data I 2 , ..., the image data I of the m-1th time period m-1 Perform fusion to obtain fusion features; Step S4: Input the fusion feature into the stable diffusion model, so that the stable diffusion model outputs the predicted image data I' of the mth time period of the farmland disease m ; Step S5: Evaluate the image data I of the mth time period m and the predicted image data I' for the mth time period m The difference between them is used to obtain the image evaluation index; Step S6: Determine whether the image evaluation index can meet the preset requirements: If the image evaluation index can meet the preset value requirement, the stable diffusion model is set as the target stable diffusion model; if not, another set of data is randomly selected from N as the model data set, and steps S2 to S6 are repeated until the image evaluation index can meet the preset requirement; Step S7: According to the target stable diffusion model, predict the image data I of the m+1th time period of farmland disease m+1 , in order to complete the prediction and generation of farmland disease remote sensing images based on the stable diffusion model; Wherein, N and m are both natural numbers greater than 1.

2. The method for predicting and generating farmland disease remote sensing images based on a stable diffusion model according to claim 1, It is characterized in that The step S3 specifically includes the following steps: Step S31: from the first time period image data I 1 , the second time period image data I 2 , ..., the image data I of the m-1th time period m-1 Extracting time series features; and, from the image data I of the m-1th time period m-1 Extract color features; Step S32: Fusing the time series feature and the color feature to obtain a fusion feature X θ ; The calculation formula of the fusion feature is as follows: in, X t represents the time series characteristics, X c Represents color characteristics, w t represents the time series feature weight, w c represents the weight of color features; Represents matrix concatenation.

3. The method for predicting and generating farmland disease remote sensing images based on a stable diffusion model according to claim 1, It is characterized in that The step S5 specifically includes the following steps: The FID is used to measure the image data I in the mth time period. m and the predicted image data I' for the mth time period m The similarity between them is used to obtain the FID score; The calculation formula of the FID is as follows: Among them, g represents the characteristics of Im; r represents the characteristics of I'm; μ g represents the mean of the features of Im; Σ g represents the covariance matrix of Im; μ r Indicates I' m The mean of the features, Σ r Indicates I' m The covariance matrix of the features, is μ g and μ r The L2 norm of the squared difference, Tr represents the trace of the matrix.

4. The method for predicting and generating farmland disease remote sensing images based on a stable diffusion model according to any one of claims 1 to 3, It is characterized in that The step S4 specifically includes the following steps: Step S41: Fusing the fused features into the denoising network of the stable diffusion model in a cross-attention mechanism to obtain an attention output Z; The calculation formula of the attention output Z is as follows: in, CrossAttention represents the cross attention mechanism function; Softmax() represents the normalization function; Q represents the query matrix; K represents the bond matrix; K T represents the transpose operation of K; V represents the value matrix; d represents the scaling factor; The calculation formula of the query matrix Q is as follows: Q=W Q ·φ(z t ) (3) Among them, z t represents the input features at time t, φ(z t ) represents z t The characteristic conversion function, W Q represents the query matrix weight; The calculation formula of the key matrix K is as follows: K=W K ·τ(X θ ) (4) Among them, X θ represents the fusion feature, τ(X θ ) indicates X θ The characteristic conversion function, W K represents the bond matrix weight; The calculation formula of the key matrix K is as follows: V=W V ·τ(X θ ) (5) Among them, W v represents the value matrix weight; Step S42: Decode and transform the attention output Z to obtain the predicted image data I' of the mth time period of the farmland disease m , which is expressed as formula (6): I’ m =D(Z) (6) Among them, the D(Z) function represents the decoding output of the attention output.

5. The method for predicting and generating farmland disease remote sensing images based on a stable diffusion model according to claim 4, It is characterized in that The denoising network of the stable diffusion model in step S41 is trained by a loss function; The loss function for training the denoising network is as follows: in, E represents the expected value, x represents the input sample, δ represents the parameter sampled from the standard normal distribution N(0,1), and t represents time; δ σ (x t ,t) represents the output after cross attention mechanism and time-color processing; The squared error defined for the 2-norm.

6. A method for treating potential farmland diseases. It is characterized in that The method comprises the following steps: Step K1: Using the method for predicting and generating farmland disease remote sensing images based on a stable diffusion model as described in any one of claims 1 to 5, obtaining the predicted image data I of the farmland disease in the m+1th time period m+1 ; Step K2: The predicted image data I m+1 Perform semantic segmentation of the diseased area to obtain the potential damage area; Step K3: Based on the image data I of the first time period 1 , the second time period image data I 2 , ..., image data I of the m-1th time period m-1 , image data I for the mth time period m , the predicted image data I m+1 , combining the potential damage areas to obtain potential disease data; Step K4: Processing potential farmland diseases according to the potential disease data.

7. The method for treating potential farmland diseases according to claim 6, It is characterized in that The potential disease data include potential disease area, potential disease type and potential disease trend; After step K4, the method further includes step K5; Step K5: Visually display the potential disease area, the potential disease type and the potential disease trend.

8. A device for predicting and generating farmland disease remote sensing images based on a stable diffusion model. It is characterized in that The device comprises: An acquisition unit, used for acquiring remote sensing image data of N groups of farmland disease areas; The first processing unit is connected to the acquisition unit and is used to randomly select a group from the N groups of remote sensing image data of the farmland disease area as a model image data set; the model image data set includes image data I of the first time period 1 , the second time period image data I 2 , ..., image data I of the m-1th time period m-1 , image data I for the mth time period m ; A second processing unit is connected to the first processing unit and is used to convert the image data of the first time period I 1 , the second time period image data I 2 , ..., the image data I of the m-1th time period m-1 Perform fusion to obtain fusion features; The third processing unit is connected to the second processing unit and is used to input the fusion feature into the stable diffusion model so that the stable diffusion model outputs the predicted image data I' of the mth time period of the farmland disease. m ; an evaluation unit connected to the third processing unit, for evaluating the image data I of the mth time period m and the predicted image data I' for the mth time period m The difference between them is used to obtain the image evaluation index; A determination unit, connected to the evaluation unit, for determining whether the image evaluation index can meet the preset requirements; a fourth processing unit connected to the determination unit, and configured to set the stable diffusion model as a target stable diffusion model if the determination unit determines that the image evaluation index can meet the preset value requirement; and to randomly extract another set of data from N as a model image data set if the determination unit determines that the image evaluation index cannot meet the preset value requirement; A prediction unit connected to the fourth processing unit is used to predict the farmland disease in the m+1th time period prediction image data I according to the target stable diffusion model. m+1 , in order to complete the prediction and generation of farmland disease remote sensing images based on the stable diffusion model; Wherein, N and m are both natural numbers greater than 1.

9. The device for predicting and generating farmland disease remote sensing images based on a stable diffusion model according to claim 8, It is characterized in that The second processing unit comprises: An extraction module is used to extract the image data I from the first time period 1 , the second time period image data I 2 , ..., the image data I of the m-1th time period m-1 Extracting time series features; and, from the image data I of the m-1th time period m-1 Extract color features; A fusion module, connected to the extraction module, is used to fuse the time series feature and the color feature to obtain a fusion feature X θ ; The calculation formula of the fusion feature is as follows: in, X t represents the time series characteristics, X c Represents color characteristics, w t represents the time series feature weight, w c represents the weight of color features; Represents matrix concatenation.

10. The device for predicting and generating farmland disease remote sensing images based on a stable diffusion model according to claim 8 or 9, It is characterized in that The third processing unit comprises: A processing module, used for fusing the fused features into a denoising network of a stable diffusion model in a cross-attention mechanism to obtain an attention output Z; The calculation formula of the attention output Z is as follows: in, CrossAttention represents the cross attention mechanism function; Softmax() represents the normalization function; Q represents the query matrix; K represents the bond matrix; K T represents the transpose operation of K; V represents the value matrix; d represents the scaling factor; The calculation formula of the query matrix Q is as follows: Q=W Q ·φ(z t ); Among them, z t represents the input features at time t, φ(z t ) represents z t The characteristic conversion function, W Q represents the query matrix weight; The calculation formula of the key matrix K is as follows: K=W K ·τ(X θ ) Among them, X θ represents the fusion feature, τ(X θ ) indicates X θ The characteristic conversion function, W K represents the bond matrix weight; The calculation formula of the key matrix K is as follows: V=W V ·τ(X θ ); Among them, W v represents the value matrix weight; A decoding module is connected to the processing module and is used to decode and convert the attention output Z to obtain the predicted image data I' of the mth time period of the farmland disease. m , which is expressed as follows: I’ m =D(Z); Among them, the D(Z) function represents the decoding output of the attention output.

Images