Generative AI-based pest and disease identification evaluation and multi-modal question and answer method

Abstract

The invention discloses a generative AI-based disease and insect pest identification evaluation and multi-modal question and answer method, and relates to the technical field of crop disease and insect pest prevention, and the method comprises the steps: obtaining multi-modal data containing a standardized image and a structured text feature; utilizing a pre-trained pest and disease target detection model to identify the spatial position of a pest and disease target in the image, and cutting to obtain a target area image; a color feature vector and a shape feature vector of the target area are extracted, text features are combined, semantic enhancement and cross-modal alignment are performed through generative AI, and an explicit feature vector of diseases and insect pests is constructed; matching and analyzing with a preset disease and pest knowledge base, and identifying disease and pest types; quantifying the damage degree of the diseases and pests according to the characteristic parameter change characteristics to obtain a damage grade judgment result; generating a question-answer reply based on the type identification result and the hazard level; according to the method, the problems that the disease and pest recognition accuracy is limited, the harm degree is difficult to quantify and intelligent multi-modal decision question and answer support cannot be provided are solved.

Description

Technical Field

[0001] This invention relates to the field of crop pest and disease prevention technology, and more specifically, to a pest and disease identification, assessment, and multimodal question-answering method based on generative AI. Background Technology

[0002] Crop diseases and pests are a significant factor affecting agricultural production. Timely and accurate identification of disease and pest types and assessment of their severity are crucial for ensuring food security and increasing crop yields. Traditional crop disease and pest detection mainly relies on manual field inspections or experience-based judgment. This method suffers from problems such as high labor intensity, low efficiency, strong subjectivity in judgment, and difficulty in quantifying the degree of damage caused by diseases and pests, making it difficult to meet the needs of modern agriculture for refined and intelligent management.

[0003] In recent years, with the development of computer vision and artificial intelligence technologies, image-based identification of crop diseases and pests has gradually become a research hotspot. Existing technologies mainly employ convolutional neural networks (CNNs) or object detection models to classify or locate disease and pest images, thereby achieving disease and pest identification. However, existing technologies still have the following shortcomings: 1. Existing technologies often use independent processing modes for data related to pests and diseases (such as image data, crop varieties, disease occurrence time, geographical location, and other text data). Image features and text features lack effective semantic alignment and correlation mining, and different information cannot form synergistic support. As a result, in complex scenarios (such as different varieties of the same crop being infected with the same pest or disease, or similar pests and diseases exhibiting feature variations in different environments), the accuracy of pest and disease identification is limited, making it difficult to cope with the challenges brought about by the multi-source and heterogeneous nature of agricultural data.

[0004] 2. Existing hazard assessment methods mostly rely on simple comparisons of single characteristic parameters, without fully considering the dynamic changes of pest and disease characteristic parameters, and without integrating related information such as crop resistance and environmental adaptability. They also lack quantitative analysis models based on probabilistic reasoning, which leads to subjective and crude determination of hazard levels and fails to provide farmers with accurate basis for prevention and control priorities and measures selection.

[0005] 3. Existing methods are difficult to combine with generative artificial intelligence to provide multimodal question-answering support for agricultural decision-making, such as combining meteorological and crop growth information to provide farmers with comprehensive planting and management suggestions. Summary of the Invention

[0006] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a method for pest and disease identification, assessment, and multimodal question answering based on generative AI, which solves the problems of fragmented pest and disease image features and textual information, limited identification accuracy, difficulty in quantifying the degree of harm, and inability to provide intelligent multimodal decision-making and question-answering support.

[0007] To achieve the above objectives, the present invention provides the following technical solution: Firstly, this application provides a method for pest and disease identification, assessment, and multimodal question answering based on generative AI. The method includes: acquiring multimodal data containing standardized image and structured text features; using a pre-trained pest and disease target detection model to identify the spatial location of pest and disease targets in the image and cropping the target region image; extracting the color and shape feature vectors of the target region, combining them with text features, and performing semantic enhancement and cross-modal alignment using generative AI to construct explicit feature vectors for pests and diseases; matching and analyzing the explicit feature vectors with a pre-set pest and disease knowledge base to identify the types of pests and diseases; quantifying the degree of harm caused by pests and diseases based on changes in feature parameters to obtain a harm level determination result; and generating question-and-answer responses based on the type identification result and the harm level.

[0008] In one embodiment, identifying the spatial location of pests in an image using a pre-trained pest detection model includes: fusing multi-scale features of a standardized image through the detection model to obtain a fused feature map; predicting candidate bounding boxes and their confidence scores based on the fused feature map; filtering candidate bounding boxes based on a confidence score threshold and performing non-maximum suppression processing on the filtering results to obtain the pest target bounding boxes and spatial location information.

[0009] In one embodiment, obtaining the target region image includes: acquiring the width and height values ​​of the target bounding box of pests and diseases, and calculating the aspect ratio; determining the horizontal and vertical expansion ratios based on the aspect ratio, and calculating the expanded cropping size; generating a cropping bounding box based on the center of the bounding box and the cropping size; performing boundary compliance verification and correction on the cropping bounding box, and cropping the target region image of pests and diseases from the original image.

[0010] In one embodiment, the extraction of color feature vectors includes: performing illumination normalization processing on the image of the target area of ​​pests and diseases; dividing the image into several local blocks based on pixel saliency index; extracting the color features of each local block and comparing them with a preset disease color template; and spatially fusing and normalizing the compared local color features to generate a color feature vector.

[0011] In one embodiment, the extraction of shape feature vectors includes: constructing an image pyramid based on the image of the target area of ​​pests and diseases to obtain a multi-scale image sequence; extracting contours from the images at each scale and performing optimization processing; weighting and fusing the multi-scale contours to generate an overall target contour; and extracting shape features based on the overall target contour to generate a shape feature vector.

[0012] In one embodiment, the color feature vector and shape feature vector of the target region are extracted, combined with text features, and semantic enhancement and cross-modal alignment are performed by generative AI to construct an explicit feature vector of pests and diseases. This includes: standardizing and concatenating the color feature vector, shape feature vector, and structured text features to generate a multimodal initial feature vector; inputting the multimodal initial feature vector into a pre-trained generative AI model; performing semantic association mining and completion through the cross-modal attention mechanism of the model, and enhancing and optimizing the features; and normalizing the optimized features to generate an explicit feature vector of pests and diseases.

[0013] In one embodiment, the explicit feature vector is matched and analyzed with a preset pest and disease knowledge base to identify the types of pests and diseases. This includes: calculating the similarity between the explicit feature vector and each standard feature vector in the knowledge base; selecting the top few categories with similarity higher than a preset threshold to form a candidate set; calculating a comprehensive matching score for the categories in the candidate set through weighted voting; and determining the category with the highest score as the type of pest or disease.

[0014] In one embodiment, the degree of harm caused by pests and diseases is quantified based on the changes in feature parameters to obtain a hazard level determination result, including: calculating the deviation value of the feature parameters in the explicit feature vector relative to the corresponding standard feature vector in the knowledge base; calculating the comprehensive hazard index by passing the deviation value through a preset Bayesian network model; and determining the hazard level of pests and diseases based on the comprehensive hazard index.

[0015] In one embodiment, determining the hazard level of pests and diseases based on a comprehensive hazard index includes: establishing a Bayesian network structure based on the conditional dependencies between pest and disease characteristic parameters and historical data, and constructing a conditional probability table for each node; inputting the deviation values ​​of the characteristic parameters as observational evidence into the network; calculating the posterior probability distribution of the hazard level nodes through Bayesian inference, and obtaining the comprehensive hazard index accordingly; and comparing the comprehensive hazard index with a preset threshold to determine the final hazard level.

[0016] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: By organically integrating pest and disease target detection, orientation-adaptive cropping, explicit color and multi-scale shape feature extraction, generative AI-driven cross-modal semantic enhancement and alignment, and Bayesian network-based probabilistic assessment of hazard levels, an end-to-end pest and disease identification and assessment system has been constructed, encompassing multi-modal data acquisition and intelligent question-answering output. Based on multi-scale target localization and refined cropping, it extracts complementary features from multi-dimensional information such as color, shape, and structured text, and leverages the cross-modal attention mechanism of generative AI to achieve semantic enhancement and alignment, thereby improving the completeness and robustness of pest and disease feature representation. On the one hand, it improves the accuracy and interpretability of pest and disease identification by introducing a knowledge base similarity matching and weighted decision-making mechanism in the identification stage; on the other hand, in the hazard assessment stage, it uses Bayesian network modeling to model the conditional dependencies between features and obtains quantifiable comprehensive hazard indicators through probabilistic reasoning, making the hazard level determination both data-driven and interpretable; overall, the scheme realizes a closed-loop processing of "accurate identification - quantitative assessment - intelligent question answering", which has high precision, strong robustness and good interpretability, and is suitable for automated pest and disease analysis and intelligent decision support in complex agricultural scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a generative AI-based pest and disease identification, assessment, and multimodal question-answering method provided in an embodiment of this application.

[0019] Figure 2 A comparison diagram of the similarity between the explicit feature vectors provided in the embodiments of this application and the standard features of the pest and disease knowledge base. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.

[0022] It should also be noted that, in this document, terms such as “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the article or device that includes the aforementioned element.

[0023] Reference Figure 1 As shown in the diagram, this invention provides a flowchart of a generative AI-based method for pest and disease identification, assessment, and multimodal question answering, which includes the following steps: S1. Acquire multimodal data of crop diseases and pests to be diagnosed. The multimodal data includes image data of crop diseases and pests, as well as textual description data including the occurrence time, geographical location, and crop variety of diseases and pests. Preprocess the image data to obtain standardized image data. The preprocessing includes filtering images for validity, removing invalid images with severe occlusion, blurriness, or abnormal exposure; uniformly adjusting the image size to meet the input size requirements of the disease and pest target detection model; and correcting the image brightness and contrast to reduce the interference of different lighting conditions on the expression of disease and pest target features. The preprocessed image data is labeled with bounding boxes for "pest and disease areas" (such as lesions and pests) using the roLabelImg annotation tool. After annotation, an XML file is generated (recording the location and category label of the pest and disease area). The XML is converted into a TXT format annotation file (the TXT format annotation file is used to record the category information and spatial location information of the pest and disease target in the image) to obtain standardized image data. The training set / validation set is divided to construct a pest and disease training sample dataset. At the same time, the text description data is semantically parsed and normalized to generate structured text features.

[0024] S2, based on standardized image data, uses a pre-trained pest and disease target detection model to detect pests and diseases in the image, obtaining spatial location information of the pests and diseases in the image, including: Load the pest and disease target detection model based on the YOLOv5s network structure that has been trained. The pest and disease target detection model includes a backbone network for feature extraction, a multi-scale feature fusion structure, and a target detection head for target localization and confidence prediction. Load the corresponding optimal weight parameters to put the model into the inference working state. The standardized image data is input into the pest and disease target detection model. The image is subjected to multi-layer convolution operation through the backbone network of the model to extract image features at different levels. The image features include lesion texture features, pest morphology features and background semantic information. The image features at different scales are fused using a multi-scale feature fusion structure to form a fused feature map containing information at different spatial resolutions, which is used to adapt to the detection needs of pest and disease targets at different scales. Based on the fused feature map, the target detection head performs target prediction on each scale feature map and outputs multiple pest and disease candidate target prediction results. The prediction results include the center coordinates, width and height information of the candidate bounding box, as well as the corresponding target confidence, which are used to represent the potential location and probability of existence of pest and disease targets in the image. The candidate target prediction results are filtered by confidence level. Candidate bounding boxes with target confidence levels lower than a preset confidence threshold are removed, and the remaining candidate bounding boxes are retained to reduce false detection results in the background area. Non-maximum suppression is performed on the candidate bounding boxes after confidence screening. The intersection-union ratio (IoU) between any two candidate bounding boxes is calculated. Specifically, the area of ​​the overlapping region of the two candidate bounding boxes is calculated and the ratio of the union of the areas of the two candidate bounding boxes is calculated to obtain the IoU value. When the IoU is greater than a preset overlap threshold, the candidate bounding boxes with high target confidence are retained and the remaining candidate bounding boxes are suppressed, thereby obtaining the final set of target bounding boxes for pests and diseases. The spatial location information of pests and diseases is generated, which includes the center coordinates, width and height of the bounding boxes.

[0025] The pest and disease target detection model is a single-stage target detection model built on the YOLOv5s network structure, and its construction and training steps include: Based on the training sample dataset of pests and diseases, the pre-trained weight parameters corresponding to the YOLOv5s network structure are loaded to introduce the feature representation capabilities already learned in the general object detection task, thus forming an initialized pest and disease target detection model. For the initialized pest and disease target detection model, set model training parameters, including learning rate, number of iterations and batch size, to control the update method and update frequency of model parameters during training, so as to adapt to the training requirements of pest and disease target detection task. Based on the model training parameters, the initialized pest and disease target detection model is iteratively trained using a joint loss function that includes bounding box regression loss and target confidence loss, so that the model gradually learns the texture features, morphological contour features and spatial distribution features of the pest and disease target area. The joint loss function is calculated using the following formula:

[0026] In the formula, For the joint loss function, The bounding box regression loss measures the deviation between the predicted bounding box and the actual pest and disease bounding box in terms of location, size, and shape, thereby guiding the model to accurately locate the pest and disease area. The target confidence loss is used to evaluate whether the bounding boxes predicted by the model truly contain the pest and disease targets, thereby suppressing false detections in the background region and improving detection reliability. , These are the weighting coefficients, used to balance the impact of each loss term on model training.

[0027] During iterative training, the trained model is applied to the validation dataset, and the training process is adjusted based on the model's detection performance on the validation dataset. During training, the model is applied to a validation dataset after each iteration. The model's detection performance is evaluated by calculating the overlap between the predicted bounding boxes and the ground truth labels. The overlap between the predicted and ground truth bounding boxes is obtained through the intersection-overall ratio (IoU). Specifically, the center coordinates, width, and height of the ground truth and predicted bounding boxes are converted into the coordinates of the corner points of rectangles. The area of ​​the overlapping region of the two rectangles is calculated and the ratio of the union of the areas of the two rectangles is obtained to obtain the IoU value. When the detection performance does not reach the preset threshold, the training process can be adjusted by adjusting training parameters (such as learning rate, number of iterations, or batch size) or by introducing an early stopping strategy to continuously improve the model's performance in target localization accuracy until the model performance is stable and meets the preset requirements.

[0028] When the model's detection performance on the validation dataset reaches a preset threshold, the model training is considered complete, and the model parameters at this point are saved as the optimal weight parameters for pest and disease target detection.

[0029] It should be noted that building a pest and disease target detection model based on the YOLOv5s network structure can achieve end-to-end rapid localization and identification of pest and disease targets while maintaining the efficient inference characteristics of single-stage target detection. Among them, the YOLOv5s network has the characteristics of lightweight structure, strong feature extraction and multi-scale fusion capabilities, which can effectively capture the texture and contour features of lesions and pests at different scales, while reducing the computational complexity and deployment cost of the model. Thus, while meeting the accuracy requirements of pest and disease detection, it improves the real-time performance and practicality of the model in agricultural field applications.

[0030] S3. Based on the spatial location information of pests and diseases, the corresponding target area image of pests and diseases is cropped, and based on the target area image of pests and diseases, feature extraction is performed on the target area of ​​pests and diseases. The feature extraction includes explicit feature extraction and shape feature extraction, and color feature vector and shape feature vector representing the characteristics of pests and diseases are obtained.

[0031] In this embodiment, based on the spatial location information of pests and diseases, the corresponding target area image of pests and diseases is cropped, including: Based on the spatial location information of the pests and diseases, the width and height values ​​of the target bounding box are obtained by parsing, and the width and height values ​​are used as the basic parameters for subsequent calculation of the clipping area; Based on the width and height values, the aspect ratio of the pest target is calculated. The aspect ratio is defined as the ratio of the width to the height of the target bounding box, and is used to characterize the morphological features of the pest target in the horizontal and vertical directions. Based on the comparison result between the aspect ratio and the preset aspect ratio threshold, the horizontal expansion ratio and the vertical expansion ratio are determined. If the aspect ratio is greater than the aspect ratio threshold, the horizontal expansion ratio is set to be greater than the vertical expansion ratio to enhance the horizontal expansion. The specific calculation formula for the expansion ratio is as follows:

[0032]

[0033] In the formula, The horizontal expansion ratio, The vertical expansion ratio, This is a preset directional enhancement amount, used to increase the expansion ratio in the main direction. Based on the expansion coefficient, satisfying This is used to ensure that the clipped area has a minimum expansion based on the original bounding box.

[0034] If the aspect ratio does not exceed the aspect ratio threshold, the horizontal expansion ratio is set to be less than and equal to the vertical expansion ratio to enhance the vertical expansion. The specific calculation formula for the expansion ratio is as follows:

[0035]

[0036] Based on the determined horizontal and vertical expansion ratios, calculate the expanded cutting width and cutting height respectively;

[0037]

[0038] In the formula, For the expanded cropping width, For the extended cutting height, This is the initial cropping width. This is the initial cutting height.

[0039] Using the center coordinates of the target bounding box of the pests and diseases as the center of symmetry, and combining the expanded clipping width and clipping height, an adaptively expanded clipping bounding box is generated, and the center coordinates, width, and height of the clipping bounding box are converted into the coordinates of the upper left corner and the lower right corner of the bounding box in the image pixel coordinate system. The cropping bounding box is subjected to boundary compliance verification to determine whether the coordinates of the cropping bounding box exceed the pixel range of the original image. When the cropping bounding box exceeds the boundary of the original image, the coordinates of the excess part are corrected to the corresponding image boundary pixel coordinates. Boundary compliance verification of the cropping bounding box refers to comparing the coordinates of its top-left and bottom-right corners with the pixel coordinate range of the original image after generating the expanded cropping bounding box. The effective pixel range of the original image is defined by its width W and height H, and its horizontal pixel coordinate range is [0, W]. 1], the vertical pixel coordinate range is [0, H [1] During the verification process, if any coordinate value of the cropping bounding box is less than zero or greater than the maximum pixel index in the corresponding direction, it is determined that the cropping bounding box has exceeded the boundary. The out-of-bounds coordinates are corrected according to the principle of "mapping to the nearest image boundary". That is, coordinate values ​​less than zero are corrected to zero, and coordinate values ​​greater than the maximum pixel index are corrected to the maximum pixel index. In this way, without changing the calculation logic of the center position of the cropping bounding box, it is ensured that the corrected cropping bounding box falls completely within the effective pixel area of ​​the original image, avoiding invalid pixel access or image out-of-bounds errors during the cropping process, and ensuring the stability and feasibility of subsequent cropping operations of pest and disease target areas.

[0040] Based on the cropped bounding box that has been verified and corrected for compliance, the corresponding pixel region is extracted from the pixel matrix of the original image to obtain an image of the pest and disease target region that is adaptively expanded in the direction of the pest and disease target.

[0041] It should be noted that when cropping images of pest and disease target areas, a fixed-ratio isotropic expansion method is no longer used. Instead, the expansion ratio of the cropping area in different directions is dynamically adjusted based on the morphological differences of the pest and disease target in the horizontal and vertical directions. This allows the cropping area to better match the true shape characteristics of the pest and disease target. This method ensures that the pest and disease target is completely contained within the cropping area while effectively reducing the introduction of irrelevant background areas, increasing the proportion of effective information and the concentration of feature expression in the target area image. At the same time, by strictly verifying the boundary compliance and correcting the coordinates of the cropping bounding box, it is ensured that the cropping area is always within the effective pixel range of the original image, avoiding cropping anomalies or invalid pixel interference caused by exceeding the boundary, thereby improving the stability and robustness of the cropping process. The resulting directionally adaptively expanded cropped area image is superior to the traditional fixed-ratio cropping method in terms of scale rationality, target integrity, and background interference control. This is beneficial for subsequent refined identification, classification, and grade evaluation of pest and disease targets, improving the accuracy and practicality of the overall intelligent pest and disease identification system.

[0042] Furthermore, the explicit feature extraction includes: The target area image of the pests and diseases is subjected to illumination normalization processing to reduce the influence of illumination changes on color features and generate an illumination-adaptive target area image. The illumination normalization processing of the target area image for pests and diseases involves first converting the target area image from the original RGB color space to a normalized color space, such as normalized RGB. In the normalized RGB case, the red, green, and blue channel values ​​R, G, and B of each pixel are normalized, and the normalized values ​​r, g, and b of each channel are calculated. The specific calculation formula for each channel normalized value is as follows:

[0043]

[0044]

[0045] In the formula, r, g, b are the normalized values ​​of the red, green, and blue channels, respectively, and R, G, B are the channel values ​​of the red, green, and blue channels, respectively.

[0046] This processing eliminates the influence of light intensity on the absolute values ​​of each channel, so that the color features mainly reflect the reflective properties of pests and diseases rather than changes in light.

[0047] Based on the illumination-adaptive target region image, a saliency index for each pixel is calculated, which is determined by the local color mean and the local color variance. The specific calculation formula for the significance index is as follows:

[0048]

[0049]

[0050] In the formula, This is the local color average. This item reflects the degree to which the color of a local patch deviates from the overall diseased area. The greater the color difference, the more likely it is a lesion area. Let K be the set of pixels contained in the i-th local block, and K be the total number of pixels in the local block. Let be the normalized color vector of pixel p, where , These are the normalized values ​​for the red, green, and blue channels of pixel p, respectively. This is the global average color vector of the target region image. Euclidean distance is used to measure the difference between local block color and global color. The local color variance characterizes the intensity of color variation within a local area. The larger the variance, the more uneven the local color, which may indicate the edge of a lesion or a complex area. For local blocks The average color vector of the inner pixel, As a significance indicator, , These are weighting coefficients, used to balance the contribution of local block and global color differences and local internal variance to the significance index. They can be set based on experience with pest and disease scenarios.

[0051] According to the saliency index, the target region image is divided into several local blocks. If the saliency index exceeds the preset saliency threshold, the local block is determined to be a salient region and retained for subsequent feature extraction. Otherwise, it is regarded as a non-salient region, thereby realizing adaptive spatial segmentation of lesions or salient color regions. Each local block covers part of the lesions or salient color regions of the target region to form a spatially adaptive local processing unit. Color features are extracted from the pixel data of each local processing unit. The color features include the mean, variance and color histogram of the color channels of each local processing unit. The color histogram adopts a preset quantization level or interval. The color features of the local processing unit are used to characterize the color characteristics of the lesion. Based on the preset disease types and their typical color distribution, the color features of local processing units are compared with the color templates of disease types. By calculating the similarity, it is dynamically determined which color channels, statistics, or histogram intervals best reflect the disease characteristics of the current local block (i.e., the similarity is higher than the preset threshold), and only these most discriminative features are retained, thereby forming local color features guided by disease types, enhancing the ability of color features to identify and discriminate specific disease types. The local color features selected by disease type are spatially fused and normalized to generate a color feature vector of the overall disease and pest target area. The color feature vector is used to characterize the characteristics of the disease and pest target in terms of color distribution, color difference and color salience.

[0052] It should be noted that the explicit feature extraction method described above can effectively eliminate the influence of light changes on color features while ensuring the integrity of the pest and disease target, so that the color features mainly reflect the reflective properties of the target itself. At the same time, through adaptive block segmentation based on local saliency indicators, lesions or significant color areas are accurately located, and the color mean, variance, and histogram features of local blocks are extracted. These features are then dynamically selected and filtered in combination with preset disease type color templates to form disease type-guided local color features. Finally, these local features are spatially fused and normalized to obtain the overall color feature vector. This makes the extracted features both locally sensitive and globally expressive, which can significantly enhance the robustness, discriminative power, and identification ability of pest and disease color features under different lighting conditions and complex scenarios.

[0053] Furthermore, the shape feature extraction includes: Based on the images of the target areas of pests and diseases, an image pyramid is constructed, which includes a Gaussian pyramid or a Laplace pyramid, to generate image sequences of different scales. The size of each scale image is gradually reduced or smoothed to capture both large-scale and small-scale lesion features simultaneously. For each scale of the image, contour extraction is performed, and Sobel, Canny, Laplacian or other gradient operators are used to generate a scale-specific contour set to ensure that the contour information of the target can be detected at different scales. Closure processing and noise filtering are performed on the contour sets at each scale. This sequential closure and noise filtering operation enhances the structural integrity and reliability of the contours. Specifically, morphological dilation is first applied to the binary mask or edge image corresponding to the contour, connecting adjacent contour regions with minor breaks. Then, erosion is performed to restore the original boundary positions of the contours and remove redundant pixels introduced by dilation, achieving a "connect first, then shrink" closure effect. For contours at smaller scales or with local gaps, contour interpolation can be used to fill in intermediate points between adjacent contour points according to linear or curve fitting rules, thus filling in discontinuous regions. Simultaneously, by setting minimum area or minimum length thresholds, isolated small contours or discrete noise points are filtered out, eliminating interfering structures that do not conform to the target features. Through this joint processing, more continuous, smooth, and noise-resistant contour representations can be obtained at different scales, providing stable input for subsequent target analysis and cropping. The processed contour sets at all scales are merged according to preset scale weights to generate the overall target contour; The overall target contour is calculated using the following formula:

[0054] In the formula, For the overall target outline, To enhance the ability to identify major lesions and partially occluded areas, a preset scale weight is used. Let N be the set of contours at scale n, where N represents all scales.

[0055] Based on the overall target outline, shape features of the pests and diseases are extracted. These shape features include area, perimeter, aspect ratio, roundness, and convexity, forming a multi-scale fused shape feature vector of the pests and diseases. This shape feature vector is used to characterize the feature information of the pests and diseases in terms of geometric shape, outline structure, and scale characteristics.

[0056] It should be noted that when extracting shape features from target areas of pests and diseases, we are no longer limited to contour information at a single scale. Instead, we construct multi-scale images and extract and fuse contour features at each scale. This allows the overall morphological features of large-area lesions to be effectively captured simultaneously, along with the features of small-scale lesions and edge details. At the same time, by assigning weights to contours at different scales and fusing them, we can highlight key scale information that is significant for the discrimination of pest and disease targets and suppress invalid contours introduced by noise, texture interference, or non-target areas. This ensures high contour integrity and shape description accuracy even in complex scenes where lesions are occluded, have blurred boundaries, or have irregular shapes.

[0057] S4, based on color feature vectors, shape feature vectors, and structured text features, uses generative AI to perform semantic enhancement and cross-modal alignment processing to construct explicit feature vectors of pests and diseases, and performs matching analysis with standard features in a pre-established pest and disease knowledge base to identify the types of pests and diseases.

[0058] In this embodiment, based on color feature vectors, shape feature vectors, and structured text features, semantic enhancement and cross-modal alignment are performed using generative AI to construct explicit feature vectors for pests and diseases, including: Standardize the color feature vector, shape feature vector, and structured text features respectively to obtain standardized unimodal features: Based on the dimensional distribution characteristics of each feature, the Z-score normalization algorithm is used to normalize and transform the color feature vector and shape feature vector to eliminate the difference in dimensions; the structured text features (including the occurrence time of pests and diseases, geographical location, and semantic coding results of crop varieties) are compressed to the [0,1] interval using the Min-Max normalization algorithm to obtain 3 sets of standardized single-modal features. Based on standardized single-modal features, an initial feature vector for multimodal pest and disease control is generated through feature concatenation: Following a fixed dimensional order of "color feature vector → shape feature vector → structured text feature", the three sets of standardized single-modal features are concatenated and spliced ​​together to form an initial feature vector of pest and disease multimodality with a dimension equal to the sum of the dimensions of the three features, thus preserving the integrity of the original feature information of each modality. The initial feature vectors of the multimodal model are input into a pre-trained generative AI model, and the semantic associations between modalities are mined through a cross-modal attention mechanism. The generative AI model described is a large language model fine-tuned from an agricultural pest and disease image-text pairing dataset. Its underlying structure adopts a layered architecture of "visual encoder + text encoder + cross-modal attention fusion module + feature generation and optimization module". The model calculates the semantic similarity matrix between image modal features (color + shape) and text modal features through the cross-modal attention layer, locates cross-modal association targets such as "lesion color - crop variety" and "outline shape - onset time", and establishes semantic mapping relationships between modalities. The fine-tuning process of the generative AI model includes: using a dataset containing 100,000+ images-text pairing samples of pests and diseases (each sample contains an image of the pest and disease area, corresponding text description and feature label) as the training set, setting a joint loss function of cross-modal semantic alignment loss and feature reconstruction loss, and iteratively training for 50-100 rounds to enable the model to learn the cross-modal semantic association rules in the agricultural pest and disease scenario, and ensure the accuracy of semantic association mining; Among them, cross-modal semantic alignment loss is used to constrain the image representation and text representation of the same pest sample to be closer in the shared semantic space and farther apart between different samples, thereby achieving consistent alignment of image and text semantics.

[0059] Feature reconstruction loss: used to constrain the model to accurately reconstruct the original image or text features from compressed or fused cross-modal features, so as to ensure that key information is not lost and improve the integrity of representation.

[0060] Based on cross-modal semantic association results, semantic completion and cross-modal bias correction are performed using a generative AI model; In this process, the generative AI model calls upon a pre-defined multimodal semantic mapping rule library for agricultural pests and diseases to semantically complete missing related features in the initial multimodal feature vector (such as disease development stage information not mentioned in the text but implied in the image features, and crop resistance information not shown in the image but clearly stated in the text). At the same time, based on the cross-modal bias correction coefficient learned during the model fine-tuning process, the semantic misalignment problem between image features and text features is corrected to obtain the semantically completed multimodal feature vector. The agricultural pest and disease multimodal semantic mapping rule library is generated by semantic mining of historical data from the Anhui Agricultural Network's agricultural knowledge base, pest and disease image library, and science and technology commissioner service platform through a generative AI model. It includes 87 fixed association rules such as "crop variety - typical lesion color" and "disease season - characteristic shape", providing a basis for semantic completion.

[0061] The cross-modal bias correction coefficients are a set of parameters obtained during the fine-tuning phase of the generative AI model by statistically learning the alignment errors between pest and disease image features and corresponding text features in a unified semantic space. Specifically, they represent the offset weights and correction factors of each modality feature in different semantic dimensions. During training, the model continuously calculates the semantic distance between image modality embedding vectors and text modality embedding vectors, constrained by cross-modal semantic alignment loss. When a systematic offset occurs between the two in the same pest and disease sample, the model automatically adjusts the bias correction coefficients of the corresponding dimensions through backpropagation. This approach aims to ensure that the perceptual information in image features, such as the color and morphological evolution of lesions, is semantically consistent with the descriptive information in text features, such as the disease stage and crop variety. In the actual inference stage, the generative AI model uses the learned cross-modal bias correction coefficient to perform weighted translation and scale correction on each modal sub-vector in the initial multimodal feature vector. This compensates for semantic misalignment caused by differences in acquisition conditions, inconsistent descriptive granularity, or modal missingness, thereby ensuring that the corrected multimodal features maintain semantic consistency and comparability within a unified semantic space. Finally, a semantically complete multimodal feature vector is obtained.

[0062] Based on the semantically completed multimodal feature vectors, feature enhancement and dimensionality optimization are performed using a generative AI model: Generative AI models use multilayer perceptrons to perform nonlinear transformations on the completed multimodal feature vectors, strengthening the weights of key features that are strongly correlated with the types and severity of pests and diseases. At the same time, with the goal of minimizing the feature reconstruction loss of the model, that is, requiring the compressed and transformed feature vectors to restore the original semantic completion feature information to the greatest extent in the reconstruction network, the effectiveness of each dimension is evaluated through the backpropagation mechanism, and feature dimensions that have been in a low contribution or high redundancy state for a long time are reduced or merged to obtain cross-modal feature vectors with optimized dimensions. The cross-modal feature vectors after dimensionality optimization are normalized to generate the final explicit feature vectors of pests and diseases. The L2 normalization algorithm is used to convert the optimized cross-modal feature vectors into unit vectors to ensure that the feature vectors have uniform modulus. This results in explicit feature vectors of pests and diseases that combine image feature recognition, text semantic relevance, and modal fusion consistency. These vectors are then used for subsequent matching analysis with standard features of the pest and disease knowledge base.

[0063] Furthermore, matching analysis is performed with standard features in a pre-established pest and disease knowledge base to identify the types of pests and diseases, including: Obtain a pre-established knowledge base of pests and diseases, which stores standard explicit feature vectors and their category labels corresponding to various pest and disease categories. The standard explicit feature vectors are obtained by performing the same feature extraction and processing process as the optimized explicit feature vectors on typical pest and disease samples. The explicit feature vector is compared with each standard explicit feature vector in the knowledge base. The similarity calculation includes cosine similarity, which is used to quantify the degree of matching between the explicit feature vector and the standard features of each pest and disease category. like Figure 2 As shown, based on the similarity measurement results, the top several pest and disease categories with similarity higher than the preset similarity threshold are selected as candidate categories to form a multi-candidate matching set; For each candidate category in the multi-candidate matching set, a weight coefficient is assigned based on its similarity value, and the comprehensive matching score of the candidate set is calculated through a weighted voting method to quantify the importance of each candidate category in the final judgment; First, the similarity value between each candidate category and the explicit feature vector is converted into a weight coefficient. For example, the similarity value is mapped to [0,1] through normalization, so that the larger the weight, the higher the degree of matching. Then, the weight coefficients of all candidate categories are weighted and voted on with their corresponding category labels to calculate the comprehensive matching score of the candidate set, thereby quantifying the importance of each candidate category in the final judgment and providing a basis for determining the final pest and disease category.

[0064] Based on the comprehensive matching score, the candidate category with the highest score is selected as the final identification result of the current pest and disease target, thereby determining the type of pest and disease.

[0065] It should be noted that the above method not only considers multiple highly similar candidate categories, improving the recognition flexibility in cases of ambiguous boundaries or similar categories, but also achieves refined judgment of candidate matching through threshold screening and weighted analysis, significantly improving the robustness and reliability of pest and disease identification, while adapting to the identification needs in the context of multiple coexisting diseases or complex environments.

[0066] S5, based on the type of pests and diseases, uses the characteristic parameter changes of explicit feature vectors to quantitatively assess the degree of harm caused by pests and diseases, and obtains the result of pest and disease harm level determination.

[0067] In this embodiment, based on the type of pest or disease, the degree of harm caused by the pest or disease is quantitatively assessed through the change characteristics of the feature parameters of the explicit feature vector, resulting in a pest or disease harm level determination result, including: Obtain the explicit feature vector of the corresponding pest or disease target; Based on the explicit feature vector and the standard feature vector of the pest in the knowledge base, the deviation value of the feature parameters is calculated. The deviation value includes color channel deviation, shape and size change, lesion area ratio and texture difference, which are used to quantify the growth stage, spread range and lesion degree of the pest in the target area. Based on the deviation values ​​of the feature parameters, a comprehensive hazard index is generated by mapping through a preset Bayesian network model, and the hazard level is determined to obtain the hazard level of the pests and diseases.

[0068] Furthermore, based on the deviation values ​​of the aforementioned feature parameters, a comprehensive hazard index is generated through mapping using a preset Bayesian network model, and the hazard level is determined to obtain the pest and disease hazard level, including: Based on the conditional dependencies between the characteristic parameters of pests and diseases and historical damage data, a Bayesian network structure is established. The network nodes include characteristic parameter deviation value nodes and damage level nodes. The directed edges between nodes represent the direct influence of the features on the damage level. In establishing the Bayesian network structure, the causal or conditional dependencies between various pest and disease characteristic parameters are first analyzed based on agricultural expert knowledge and existing research experience. This preliminarily determines which features might directly or indirectly affect the hazard level. If expert knowledge clearly indicates that feature parameter A has a causal impact on feature parameter B or the hazard level, directed edges are set from feature parameter A to feature parameter B or from feature parameter A to the hazard level. If only a statistical correlation exists without a clear causal direction, undirected connections are established initially, with the direction to be determined later. If two feature parameters are determined to be independent under a given hazard level condition through a conditional independence test, no direct connection is established, and this forms the initial network topology. Subsequently, the structure is validated and optimized using historical pest and disease datasets. First, the Pearson correlation coefficient and mutual information between each feature parameter are calculated. If the absolute value of the Pearson correlation coefficient is less than 0.2... If the conditional mutual information is less than a preset threshold, the corresponding connection is deleted. Secondly, a structural learning algorithm is used to automatically optimize the network, preferably using the K2 algorithm or the Hill-Climbing search algorithm, limiting the maximum number of parent nodes of each node to no more than 3, and outputting the optimal structure. Based on this, the feature parameter deviation value is used as the input node and the hazard level is used as the target node, and finally a Bayesian network model that can reflect the causal relationship between features and hazards is formed for subsequent probabilistic reasoning and risk assessment.

[0069] For each node in the Bayesian network, construct a conditional probability table to represent the probability distribution of the current child node (such as the hazard level) under the condition of the parent node's feature value; The deviation values ​​of the feature parameters actually measured in the target area are used as observation evidence and input into the Bayesian network. The posterior probability distribution of the hazard level nodes is calculated using Bayesian inference algorithms (such as belief propagation, variable elimination, or sampling inference). In a Bayesian network, the state of each node is defined by a conditional probability table (CPT), which represents the probability distribution of a child node occurring when the parent node takes a specific value. After inputting the actual measured deviation values ​​of the feature parameters of the target region as observational evidence into the Bayesian network, the states of the corresponding feature nodes are first fixed to the observed values. Then, based on the network size and structure, an appropriate Bayesian inference algorithm is selected to calculate the posterior probability: when the network structure is a sparse graph and there are no large loops, the belief propagation algorithm is preferred, iteratively transmitting "messages" between nodes until the maximum change between two adjacent iterations is less than a preset threshold; when the network size is small or the structure is simple, a variable elimination algorithm can be used, sequentially integrating and eliminating non-target variables according to a pre-set elimination order to finally obtain the conditional distribution of the hazard level nodes; when the network size is large or there are complex dependencies, Markov chain Monte Carlo (MCMC) sampling inference can be used, approximating the true posterior distribution through multiple rounds of random sampling, and statistically obtaining the posterior probability of the hazard level after sample convergence. The above reasoning process is based on the conditional probability table (CPT) of each node, propagating observational evidence step by step to parent and child nodes, ultimately outputting the posterior probability distribution of each hazard level under the current observation conditions, used to quantitatively assess the severity of pest and disease damage. In this way, the probability of occurrence of each hazard level is quantified, reflecting the actual impact of changes in characteristic parameters on the overall disease damage. The role of the CPT is to provide a probability mapping basis for the network, enabling the precise quantification and calculation of the relationship between characteristic bias values ​​and hazard levels, thus achieving a scientific and interpretable hazard assessment.

[0070] Calculate the comprehensive hazard index based on the posterior probability distribution of the hazard level; The comprehensive hazard index is calculated using the following formula:

[0071] In the formula, To provide comprehensive hazard indicators, For the posterior probability distribution, The rating is based on the grade score. According to the hazard level, This represents the deviation value of the characteristic parameter.

[0072] The comprehensive hazard index is compared with a preset level threshold to determine the hazard level of pests and diseases in the target area, thereby generating a hazard level determination result.

[0073] The calculated comprehensive hazard index U is compared one by one with a pre-set set of hazard level thresholds, such as mild threshold T1, moderate threshold T2, and severe threshold T3. The corresponding hazard level is determined based on the position of U within each threshold interval. If If it is determined to be a minor hazard, then... Then it is judged as a moderate hazard. If the damage is severe, a clear level of pest and disease severity will be generated for the target area, which will be used to guide prevention and control decisions or subsequent statistical analysis.

[0074] It should be noted that by using the feature parameter deviation values ​​of the target area of ​​pests and diseases as input, and using Bayesian networks to infer the conditional dependencies between each feature and the hazard level, the contribution of different features to the hazard of the disease can be scientifically quantified, a comprehensive hazard index can be generated, and a specific hazard level can be determined by combining it with a preset threshold. This not only realizes the quantification and interpretable assessment of the degree of pest and disease damage, but also handles the interaction and uncertainty between features, improves the accuracy, reliability, and practicality of hazard level determination, and provides a scientific basis for precision agricultural prevention and control and decision-making.

[0075] S6, based on the recognition and judgment results, responds to user multimodal queries and generates accurate question-and-answer responses, including: based on the aforementioned determined pest and disease types and damage levels, receiving user multimodal queries (including pest and disease image uploads, natural language text questions, and image-text combined consultations, etc.), first clarifying the user's core query requirements through semantic parsing and modality recognition technology (such as pest and disease control methods, damage mitigation measures, and meteorological adaptation suggestions); then calling upon cross-media agricultural meteorological knowledge graphs and vertical domain knowledge bases, combined with RAG retrieval enhancement technology, to accurately locate the pest and disease types, damage levels, and query requirements. The system matches core knowledge (including targeted prevention and control plans, crop resistance management measures, meteorological condition control suggestions, historical case references, etc.); then, it takes the user's query intent, pest and disease identification results, hazard level data, and retrieved core knowledge as joint inputs and feeds them into a generative AI model that has been fine-tuned with data from the agricultural meteorology field. The model generates a structured response containing "pest and disease type confirmation - hazard level interpretation - core influencing factor analysis - step-by-step implementation suggestions - meteorological risk warning" through multimodal information semantic alignment and logical integration. At the same time, it supports multiple rounds of interactive optimization to ensure that the question and answer response is both professional, targeted, and operable.

[0076] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0077] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0078] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0079] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0080] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for pest and disease identification, assessment, and multimodal question answering based on generative AI, characterized in that, include: Acquire multimodal data containing standardized image and structured text features; A pre-trained pest and disease target detection model is used to identify the spatial location of pest and disease targets in an image, and the target area image is then cropped. Extract the color and shape feature vectors of the target area, combine them with text features, and use generative AI to perform semantic enhancement and cross-modal alignment to construct explicit feature vectors for pests and diseases. The explicit feature vectors are matched and analyzed with a pre-defined pest and disease knowledge base to identify the types of pests and diseases. The degree of damage caused by pests and diseases is quantified based on the changes in characteristic parameters, and the result of the damage level determination is obtained. Questions and answers are generated based on the species identification results and hazard levels.

2. The method for pest and disease identification, assessment, and multimodal question answering based on generative AI according to claim 1, characterized in that, The method of identifying the spatial location of pests and diseases in an image using a pre-trained pest and disease target detection model includes: The standardized image is subjected to multi-scale feature fusion through the detection model to obtain a fused feature map; Predict candidate bounding boxes and their confidence scores based on fused feature maps; Candidate bounding boxes are selected based on confidence thresholds, and non-maximum suppression is applied to the selection results to obtain the bounding boxes and spatial location information of pest and disease targets.

3. The method for pest and disease identification, assessment, and multimodal question answering based on generative AI according to claim 1, characterized in that, The process of obtaining the target region image includes: Obtain the width and height values ​​of the bounding box of the pest and disease target, and calculate the aspect ratio; Determine the horizontal and vertical expansion ratios based on the aspect ratio, and calculate the expanded cutting dimensions; Using the center of the bounding box as a reference, generate a clipping bounding box according to the clipping dimensions; The boundary compliance of the cropping bounding box is checked and corrected, and the target area of ​​the pests and diseases is cropped from the original image.

4. The method for pest and disease identification, assessment, and multimodal question answering based on generative AI according to claim 1, characterized in that, The extraction of the color feature vector includes: Illumination normalization processing is performed on images of target areas affected by pests and diseases; The image is divided into several local blocks based on pixel saliency index; Extract the color features of each local block and compare them with the preset disease color template; The local color features after comparison are spatially fused and normalized to generate a color feature vector.

5. The method for pest and disease identification, assessment, and multimodal question answering based on generative AI according to claim 1, characterized in that, The extraction of the shape feature vector includes: An image pyramid is constructed based on images of target areas affected by pests and diseases to obtain multi-scale image sequences. Contours are extracted from images at various scales and then optimized. The multi-scale contours are weighted and fused to generate the overall target contour. Shape features are extracted based on the overall target contour, and a shape feature vector is generated.

6. The method for pest and disease identification, assessment, and multimodal question answering based on generative AI according to claim 1, characterized in that, The extracted color and shape feature vectors of the target region are combined with text features and semantically enhanced and cross-modal aligned using generative AI to construct explicit feature vectors for pests and diseases, including: The color feature vector, shape feature vector, and structured text features are standardized and concatenated to generate a multimodal initial feature vector; Input the multimodal initial feature vectors into the pre-trained generative AI model; The model utilizes a cross-modal attention mechanism to mine and complete semantic associations, and enhances and optimizes features. The optimized features are normalized to generate explicit feature vectors for pests and diseases.

7. The method for pest and disease identification, assessment, and multimodal question answering based on generative AI according to claim 1, characterized in that, The step of matching and analyzing explicit feature vectors with a pre-defined pest and disease knowledge base to identify the types of pests and diseases includes: Calculate the similarity between the explicit feature vector and each standard feature vector in the knowledge base; select the top few categories with similarity higher than a preset threshold to form a candidate set; calculate the comprehensive matching score of the categories in the candidate set through weighted voting, and determine the category with the highest score as the pest / disease type.

8. The method for pest and disease identification, assessment, and multimodal question answering based on generative AI according to claim 1, characterized in that, The process of quantifying the degree of damage caused by pests and diseases based on changes in characteristic parameters to obtain a damage level determination result includes: Calculate the deviation of the feature parameters in the explicit feature vector relative to the corresponding standard feature vector in the knowledge base; calculate the comprehensive hazard index by passing the deviation value through a preset Bayesian network model; and determine the hazard level of pests and diseases based on the comprehensive hazard index.

9. The method for pest and disease identification, assessment, and multimodal question answering based on generative AI according to claim 8, characterized in that, The determination of pest and disease hazard levels based on comprehensive hazard indicators includes: A Bayesian network structure is established based on the conditional dependencies between pest and disease characteristic parameters and historical data, and a conditional probability table is constructed for each node. The deviation values ​​of the characteristic parameters are input into the network as observational evidence. The posterior probability distribution of the hazard level nodes is calculated through Bayesian inference, and the comprehensive hazard index is obtained accordingly. The comprehensive hazard index is compared with a preset threshold to determine the final hazard level.

Images