Plant state evaluation method and system based on multi-modal thinking chain technology

Abstract

The invention relates to the technical field of data analysis, in particular to a plant state evaluation method and system based on a multi-modal thinking chain technology, and the method comprises the steps: firstly obtaining a plant disease image, a disease description text, an environment parameter and a target crop type, and then associating the target crop type with the disease description text; the method comprises the following steps: acquiring a plant disease image, inputting associated data into a pre-trained language model, calculating semantic features of a disease description text, inputting the plant disease image and environmental parameters into a preset reinforcement learning and interpretability enhancement model, respectively calculating to obtain an image quality score and a pathological constraint reward parameter of the plant disease image, and finally, obtaining the image quality score and the pathological constraint reward parameter of the plant disease image. And inputting the pathological constraint reward parameters, the image quality score and the semantic features into a preset multi-modal thinking chain reasoning model, and calculating to obtain an evaluation result of the plant disease image. Compared with the prior art, the evaluation result of the method is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology. More specifically, the present invention relates to a plant status assessment method and system based on multimodal thinking chain technology. Background Art

[0002] Plant diseases are characterized by persistent disturbances in plant growth and development caused by biotic or abiotic factors, leading to abnormalities in physiological functions, tissue structure, or external morphology, which in turn affect normal growth and development or cause plant death. Plant diseases exhibit a high degree of heterogeneity in their visual manifestations, manifested by the superposition of multiple symptoms, cross-organ and cross-growth stage variation, and uncontrollable dynamic evolution. Multi-dimensional symptom superposition refers to the combination of multiple features, including color abnormalities (yellowing or browning), texture changes (necrotic spots or mildew), and morphological distortions (leaf curling or wilting). Cross-organ and cross-growth stage variation refers to significant differences in the visual manifestations of the same disease across different organs (leaves, stems, or fruits) or production stages (seedling, flowering, or fruiting) within the same plant. Uncontrollable dynamic evolution refers to the morphological or color changes of symptoms over time from infection to outbreak.

[0003] Regarding the aforementioned visual representation processing and analysis of plant diseases, a Chinese patent application, publication number CN119066527A, discloses a plant health assessment method, system, and medium based on an AI model. This method directly processes and analyzes plant image data using a "black box" AI model. This approach fails to effectively integrate plant pathology knowledge and environmental factors into the model's data analysis, resulting in inaccurate plant disease assessment results.

[0004] Therefore, the plant disease assessment results obtained by using existing technical analysis and processing are inaccurate. Summary of the Invention

[0005] In order to solve the above technical problem of inaccurate evaluation results, the present invention provides solutions in the following aspects.

[0006] In a first aspect, the present invention discloses a plant status assessment method based on multimodal thinking chain technology, comprising: S10: Obtain plant disease images, disease description text, environmental parameters and target crop species; S20: Associating the target crop species with the disease description text, and inputting the associated data into a pre-trained language model to calculate the semantic features of the disease description text; S30: Inputting the plant disease image and environmental parameters into a preset reinforcement learning and interpretability enhancement model to respectively calculate the image quality score and pathology constraint reward parameter of the plant disease image; S40: Input the pathology constraint reward parameters, image quality score and semantic features into the preset multimodal thinking chain reasoning model to calculate the evaluation results of the plant disease image.

[0007] Beneficial effects: When evaluating plant disease images, the method of the present invention first obtains plant disease images, disease description text, environmental parameters, and target crop species as multi-dimensional analysis and processing data; then, the associated data of the target crop species and disease description text are input into a pre-trained language model to obtain more accurate semantic features; finally, the image quality score, pathology constraint reward parameters, and semantic features are input into a multimodal thinking chain reasoning model for multi-dimensional analysis and calculation, thereby obtaining more reliable and accurate evaluation results. Compared with the existing technology, the method of the present invention fully considers the impact of different plant species and different environments on plant diseases, and integrates plant pathology knowledge into the model analysis and calculation, fully considering the impact of multiple factors on plant diseases. The evaluation results calculated using the present invention are more accurate.

[0008] Preferably, the language model adopts BERT (Bidirectional Encoder Representations from Transformer), and step S20 includes: Insert the explicitly labeled target crop species before the disease description text to obtain associated data; The associated data is input into BERT for multimodal fusion and feature extraction, and semantic features are calculated.

[0009] Beneficial effects: Relying on BERT, the method of the present invention has the ability to extract semantic features with the knowledge perception ability of plant pathology images. The semantic features obtained in this way are more accurate and can optimize the alignment between text and visual images.

[0010] Preferably, the calculation formula for image quality score in the reinforcement learning and interpretability enhancement model is:

[0011] Where, represents the image quality score, Indicates the image clarity score, represents the texture significance score of the lesion area, Indicates the color contrast at the boundary between healthy and diseased areas.

[0012] Beneficial effects: For image quality scoring, the method of the present invention incorporates image clarity scoring, texture significance scoring of the diseased area, and color contrast at the boundary between the diseased and healthy areas for comprehensive calculation, fully taking into account the visual representation of plant diseases. Compared with the existing technology, the method of the present invention can perform data analysis on plant disease images more comprehensively, and has higher universality and accuracy.

[0013] Preferably, the method for confirming the reward parameters for pathological constraints in the reinforcement learning and interpretability enhancement model is: In response to the plant disease image and the environmental parameters satisfying the conditions of the disease triangle theory, setting the pathology constraint reward parameter to a first set value; or In response to a logical contradiction between the plant disease image and the environmental parameter, the pathology constraint reward parameter is set to a second setting value, wherein an absolute value of the first setting value is greater than an absolute value of the second setting value, and the second setting value is a negative value.

[0014] Preferably, the conditions for determining whether the disease triangle theory is satisfied are: Condition 1: The plant disease image contains pathogens; Condition 2: The plant disease image has a host susceptibility match; Condition 3: The environmental parameters are within the preset environmental parameter judgment range; If any one of the conditions 1, 2 or 3 is met, the disease triangle theory is determined to be satisfied; otherwise, it is determined to be the logical contradiction.

[0015] Beneficial effects: The method of the present invention sets up a calculation mechanism for pathological constraint reward parameters, which can more accurately calculate the pathological constraint reward parameters, so that the multimodal thinking chain reasoning model can obtain a more reliable and accurate data source, thereby improving the accuracy of the evaluation results.

[0016] Preferably, the multimodal thinking chain reasoning model includes an uncertainty estimation module and a U-Net architecture, a probability prediction model and a selection model, and step S40 includes: Based on the preset uncertainty estimation module, the diseased area of ​​the plant disease image is located to obtain a list of disease detection frame areas; The U-Net architecture is used to predict the segmentation mask of the diseased area in the detection box area list; the cross-attention mechanism is used to fuse the segmentation mask and semantic features to obtain the symptom area; Extract local features of the symptom area and match the local features with the preset knowledge base to obtain matching results; The matching results, pathology constraint reward parameters and image quality scores are input into the preset probability prediction model and selection model to calculate the evaluation results.

[0017] Preferably, before step S30, the method of the present invention further comprises: Input the plant disease image into the preset neural network model to calculate the image features of each organ in the plant disease image; Use the preset multi-layer perceptron to encode the environmental parameters and obtain the environmental features; The image features, semantic features and environmental features are spliced ​​together to obtain a multimodal feature vector; Multimodal feature vectors are used to enhance the representation and interpretability of features.

[0018] Preferably, the neural network model includes at least YOLOv8-tiny and CNN. The plant disease image is input into the preset neural network model to calculate the image features of each organ in the plant disease image, specifically: Input the plant disease image into YOLOv8-tiny to generate candidate disease frames; The preset organ type weight parameters and candidate disease frames are input into CNN to calculate the image features.

[0019] Preferably, the environmental parameters include at least temperature, humidity and light intensity; a preset multi-layer perceptron is used to encode the environmental parameters to obtain environmental characteristics, specifically: Temperature, humidity and light intensity are input into the multi-layer perceptron for encoding to obtain a low-dimensional vector as the environmental feature.

[0020] In the second aspect, the present invention also discloses a plant status assessment system based on multimodal thinking chain technology, including a processor and a memory, the memory storing computer program instructions, and when the computer program instructions are executed by the processor, the plant status assessment method based on multimodal thinking chain technology recorded in the first aspect is implemented.

[0021] The beneficial effects of the present invention are: (1) Compared with the existing technology, the method of the present invention fully considers the impact of different plant species and different environments on plant diseases, and integrates plant pathology knowledge into the model analysis and calculation, fully considering the impact of multiple factors on plant diseases. Therefore, the evaluation results calculated by the present invention are more accurate.

[0022] (2) Compared with the existing technology, the method of the present invention has the ability to extract semantic features of plant pathology images, and the extracted semantic features are more reliable.

[0023] (3) Compared with the existing technology, the method of the present invention sets a pathological constraint reward parameter operation mechanism, which can more accurately calculate the pathological constraint reward parameter, so that the multimodal thinking chain reasoning model can obtain a more reliable and accurate data source, thereby improving the accuracy of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Flowchart of a plant status assessment method based on multimodal thinking chain technology in Embodiment 1 of the present invention; Figure 2 This is a structural block diagram of the CNN model in Example 1 of the present invention; Figure 3 It is a structural diagram of a plant status assessment system based on multimodal thinking chain technology in Example 3 of the present invention. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0026] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0027] Example 1 like Figure 1 As shown, this embodiment discloses a plant status assessment method based on multimodal thinking chain technology, including: S10: Obtain plant disease images, disease description text, environmental parameters and target crop species.

[0028] In this embodiment, plant disease images can be captured using a camera. These images can be of the entire plant or of specific parts, such as the stem, stalk, root, leaf, or fruit (or ear). Disease descriptions include text such as "leaf yellowing and browning," "necrotic spots and mildew on leaves," "leaf curling and wilting," or "brown spots on ear stems." These descriptions vary for different crops, as disease symptoms manifest differently. Disease descriptions and target crop types can be input through manual observation or derived from existing visual recognition models. Environmental parameters primarily include temperature, humidity, and light intensity.

[0029] S20: Associating the target crop species with the disease description text, and inputting the associated data into a pre-trained language model to calculate the semantic features of the disease description text.

[0030] Specifically, the language model adopts BERT, and step S20 includes: S21: Insert the explicitly labeled target crop species before the disease description text to obtain associated data; S22: Input the associated data into BERT for multimodal fusion and feature extraction, and calculate the semantic features.

[0031] S30: Input the plant disease image and environmental parameters into the preset reinforcement learning and interpretability enhancement model to calculate the image quality score and pathology constraint reward parameter of the plant disease image respectively.

[0032] It's important to explain that the image quality score encourages the model to prioritize clear, disease-prone images, suppressing misjudgments caused by low-quality input. The pathology constraint reward parameter, on the other hand, quantifies whether the model's reasoning process aligns with domain knowledge, guiding the model to generate logical evaluation results.

[0033] S40: Input the pathology constraint reward parameters, image quality score and semantic features into the preset multimodal thinking chain reasoning model to calculate the evaluation results of the plant disease image.

[0034] Through steps S10-S40, the method of the present invention fully considers the impact of different plant species, ambient temperature, humidity, and light intensity on plant diseases, making the method of the present invention more universal. Regarding semantic feature extraction, the semantic features calculated by the BERT model are more accurate. The method of the present invention also calculates image quality scores and pathology constraint reward parameters to guide the multimodal thinking chain reasoning model to obtain more accurate evaluation results.

[0035] Example 2 Based on the first embodiment, this embodiment discloses a plant status assessment method based on multimodal thinking chain technology. Before step S30, the method of the present invention includes: S200: Inputting the plant disease image into a preset neural network model to calculate the image features of each organ in the plant disease image.

[0036] S201: Using a preset multi-layer perceptron, the environmental parameters are encoded to obtain environmental features.

[0037] S202: Concatenate image features, semantic features, and environmental features to obtain a multimodal feature vector.

[0038] Multimodal feature vectors are used to enhance the representation and interpretability of features. For example, the inference path of "high humidity + lesion spread (modal feature vector) → high disease risk" is obtained.

[0039] More specifically, the neural network model includes at least YOLOv8-tiny and CNN, and the above step S200 is specifically as follows: Input the plant disease image into YOLOv8-tiny to generate candidate disease boxes.

[0040] The preset organ type weight parameters and candidate disease frames are input into CNN to calculate the image features.

[0041] It should be explained that the above-mentioned YOLOv8-tiny is a lightweight target detection model that can be applied to data processing in various plant disease image formats (such as JPEG, PNG, etc.). This model requires less memory and has more beneficial performance.

[0042] In this embodiment, after the candidate disease frames are generated, a candidate set is automatically generated to facilitate data processing, and its parameters are defined as ,in, represents the disease candidate set, After generating the candidate disease set, the method of the present invention further filters non-disease areas such as background noise based on the saliency heat map (Grad-CAM++).

[0043] The preset organ type weight parameters and candidate disease frames are input into CNN to calculate the image features, specifically: An organ-specific branch network is introduced to dynamically adjust the feature response weights of different organs through the attention mechanism to calculate the image features, which are expressed as follows:

[0044] in, A vector representing image features, Indicates the target plant The weight parameter of each organ is determined by the organ type (leaf weight 0.6, stem weight 0.3). Represents the CNN model function 。

[0045] like Figure 2 As shown, it needs to be explained that the above CNN is a cross-organ feature adaptation neural network, and its more specific structure is as follows: Backbone layer (backbone network): uses ResNet-18 (removing the last fully connected layer) to extract image features, with an output shape of [Batch, 512, H, W]; Organ-Specific Branch layer: It consists of multiple (equal to the number of organs) parallel CNN branches, each of which contains 4 layers of convolution with different expansion rates, and finally outputs a 64-channel feature map; Dynamic Attention Fusion layer: includes channel attention and spatial attention, which are used to adjust the weights and fuse the outputs of multiple branches; Weight calculation layer: Softmax is used to calculate dynamic weights and fuse the feature outputs of specific branches of each organ.

[0046] Preferably, the specific algorithm of step S21 may be:

[0047] in, Represents rice, Indicates circular lesions on the leaves. Indicates the target crop type, Indicates the disease description text, Represents linked data.

[0048] It should be noted that the above algorithm is only an example algorithm for circular lesions on rice leaves. For other types of plants and lesion characteristics, and Perform adaptive replacement.

[0049] Furthermore, in step S22, the average pooling result of the last hidden layer of the BERT model is taken as the semantic vector to extract the crop-specific semantic vector (i.e. semantic features).

[0050] Furthermore, the specific process of step S22 is as follows: S221: Use BERT to encode contextual data and obtain context-sensitive word vector representations :

[0051] Where, represents the pre-trained language model BERT, is the sequence length, is the hidden layer dimension, represents the set of real numbers.

[0052] S222: Based on the disease symptom-specific attention mechanism, symptom-related words are given higher weights, specifically: A. Build a domain dictionary containing 5000+ plant disease symptom terms ; B. Calculate the matching score between each word in the disease description text and the symptom dictionary. The specific formula is:

[0053] in, It is a calculation word Word in the dictionary with symptoms The semantic similarity function between Indicates taking the maximum value of all similarities as the matching score of the word .

[0054] C. Based on the matching score, construct the symptom attention mask. The specific formula is:

[0055] in, represents the attention enhancement coefficient, which is 0.5 in this embodiment. represents the symptom attention mask, .

[0056] D. Using the symptom attention mask, we can calculate the semantic representation of symptom enhancement:

[0057] S223: Construct a multi-level crop-specific knowledge graph to dynamically align text symptoms with graph entities. Specifically: E. Build a triple relationship network for each target crop type ; The entity types are divided into: Disease, Symptom, Pathogen, OrganType, and GrowthStage; the relationship type is defined as: i. is_symptom_of (Symptom→Disease): For example, “brown spots→rice blast” ii. caused_by(Disease→Pathogen): such as "rice blast→Pyricularia oryzae" iii. manifests_as(Pathogen→Symptom): such as “Pyricularia oryzae→brown lesions” iv. appears_on(Symptom→OrganType): such as "brown spots→leaves" v. occurs_at(Symptom→GrowthStage): such as "brown lesions→tillering stage" F. Use the standard relationship model between entity link and symptom type to calculate the entity link result. The specific formula is:

[0058] in, Represents the entity linking result, represents the embedding representation of all entities in the knowledge graph, Represents the bidirectional interactive attention network model (Bi-Directional Attention Flow).

[0059] G. Build associated subgraphs based on entity linking results ; H. Using Graph Attention Network (GAT) to deal with related subgraphs Perform message passing to obtain knowledge-enhanced semantic representation :

[0060] I. Fuse the knowledge-enhanced semantic representation with the original text features to obtain the final semantic vector :

[0061] Where, and is a learnable fusion parameter.

[0062] To achieve crop-specific contrastive learning and cross-modal alignment, a multi-level contrastive learning framework is designed in the BERT model. Specifically, a contrastive loss function with crop weight adjustment is introduced:

[0063] Where, represents the temperature parameter, which is 0.07 in this embodiment; Represents the computational semantic vector and the vector of image features In this embodiment, the similarity calculation function Use cosine similarity calculation function; Indicates the Visual feature vectors in the batch of training data.

[0064] Example 2 Different from the first embodiment, this embodiment improves the traditional BERT model framework and conducts comparative training on image features and semantic features, especially the A to I process in step S222-step S223, which can further enhance the semantic vector of the final output. accuracy.

[0065] Furthermore, in step S201, a preset multi-layer perceptron is used to encode the environmental parameters to obtain environmental features, specifically: The temperature, humidity and light intensity are input into the multi-layer perceptron (MLP) for encoding to obtain a low-dimensional vector As an environmental feature.

[0066] Furthermore, the calculation formula for combining the image features, semantic features, and environmental features in step S202 is:

[0067] Where, represents the multimodal feature vector, Represents the concatenation function.

[0068] Furthermore, in step S30, the pathology constraint reward parameter in the reinforcement learning and interpretability enhancement model is confirmed as follows: In response to the plant disease image and the environmental parameters satisfying the conditions of the disease triangle theory, setting the pathology constraint reward parameter to a first set value; or In response to a logical contradiction between the plant disease image and the environmental parameter, the pathology constraint reward parameter is set to a second setting value, wherein an absolute value of the first setting value is greater than an absolute value of the second setting value, and the second setting value is a negative value.

[0069] More specifically, the calculation formula for the pathological constraint reward parameter in the reinforcement learning and interpretability enhancement model is:

[0070] Where, represents the pathological constraint reward parameter, Indicates that the conditions of the disease triangle theory are met The value of Indicates a logical contradiction The value of and are all real numbers, Greater than ,and Take a negative value.

[0071] In this embodiment, Take 1, and Take -0.5.

[0072] Among them, the conditions for judging whether the disease triangle theory is satisfied are: The presence of pathogens in plant disease images; There is host susceptibility matching in plant disease images; The environmental parameters are within the preset environmental parameter judgment range; If any of the above conditions is met, the disease triangle theory is judged to be satisfied; otherwise, it is judged to be a logical contradiction.

[0073] More specifically, for pathogen identification, candidate disease frames can be located within the diseased area to determine the pathogen's location. If the number of candidate disease frames meeting the confidence threshold is greater than 0, the pathogen is confirmed to be present. For host susceptibility matching, a database of crop variety resistance (e.g., rice blast resistance levels) can be integrated. If the disease type inferred by the model (reinforcement learning and interpretability enhancement model) conflicts with host resistance (e.g., a resistant variety is identified as highly susceptible), a penalty is triggered (i.e., a match). For environmental parameter identification, sensor-collected weather data (e.g., humidity >85%, temperature 20-25°C) can be used to determine whether pathogen activity conditions are met (e.g., late blight requires high humidity).

[0074] It should be explained that the significance of calculating the above-mentioned pathological constraint reward parameters is to quantify whether the reasoning process of the model (reinforcement learning and interpretability enhancement model) conforms to domain knowledge and guide the model to generate logical evaluation results.

[0075] Furthermore, the calculation formula for image quality score in the reinforcement learning and interpretability enhancement model is:

[0076] Where, represents the image quality score, Indicates the image clarity score, represents the texture significance score of the lesion area, Indicates the color contrast at the boundary between healthy and diseased areas.

[0077] More specifically, the above image clarity score can be calculated by a super-resolution network (ESRGAN).

[0078] More specifically, the texture saliency score of the lesion area needs to be calculated based on the Gray-Level Co-occurrence Matrix. The specific method is: In the symptomatic area after segmentation Count the pixel pairs separated by a certain distance in each direction The co-occurrence frequency of , generates the GLCM matrix , and then the GLCM matrix Normalize to get , the normalized calculation formula is:

[0079] Where, Indicates the number of gray levels, The horizontal axis is The vertical axis is The above and The definition of is limited to this formula. If there is a definition overlap in actual applications, Can be replaced by x, Can be replaced by y.

[0080] Then, according to , the texture significance score of the lesion area is calculated. The specific calculation process is:

[0081] More specifically, the color contrast at the healthy-disease boundary needs to be calculated based on the HSV space. The specific method is: Input original image RGB information , and perform the following operations on the image in sequence: 1. First, grayscale the RGB information of the original image (used to convert the RGB image into a grayscale image to reduce the amount of calculation), then perform Gaussian filtering denoising (used to smooth the image and reduce noise interference), Canny edge detection (used to extract strong edges at the junction of disease and health), calculate the distance matrix (used to perform distance transformation on edge pixels and quantify the distance from each boundary point to the nearest background pixel), adaptive threshold processing (used to mark the area with larger distance as the disease side and the area with smaller distance as the healthy side based on the distance distribution), morphological processing optimization (specifically the closing operation is used to fill small holes in the disease area and enhance the continuity of the mask), and neighborhood classification (by traversing the junction pixels, for each edge pixel, check whether there is a disease mask pixel in its neighborhood), to obtain the disease side feature mask (mask_disease) and the healthy side feature mask (mask_healthy).

[0082] 2. Convert the image from RGB space to HSV space and perform channel separation to obtain the H channel (h_channel), S channel (s_channel), and V channel (v_channel). Calculate the histograms of the diseased side and healthy side components, respectively, to obtain the diseased side histograms (hist_h_disease, hist_s_disease, hist_v_disease) and healthy side histograms (hist_h_healthy, hist_s_healthy, hist_v_healthy). The specific method is to use the OpenCV calcHist function. The specific parameters of the input image images are h_channel, s_channel, and v_channel respectively. The channels for calculating the histogram are specified as [0]. The masks are mask_disease and mask_healthy respectively. The number of BINs histSize is

[180] (H channel),

[256] (S channel), and

[256] (V channel). The pixel value ranges are [0,180] (H channel), [0,256] (S channel), and [0,256] (V channel).

[0083] 3. Quantify the color contrast and compare the HSV component distribution differences between the diseased side and the healthy side. The specific method is as follows: Calculate the histogram intersection distance. The specific calculation formula is:

[0084] in, is the number of BINs in the histogram, and Substituting (hist_h_disease, hist_h_healthy), (hist_s_disease, hist_s_healthy), and (hist_v_disease, hist_v_healthy) into the calculation in pairs, we can get 、 and Based on the relationship between hue difference and disease recognition sensitivity, saturation reflects the degree of disease, and brightness is greatly affected by light, the histogram results of HSV are given. In this embodiment, the weight parameters of the three channels can be set as 、 and Finally, the calculation formula for the color contrast at the healthy-disease junction is:

[0085] Through the above technical solution, for image quality scoring, the method of the present invention incorporates image clarity scoring, texture significance scoring of the diseased area, and color contrast at the boundary between diseased and healthy areas for comprehensive calculation, fully taking into account the visual representation of plant diseases. Compared with the existing technology, the method of the present invention can perform data analysis on plant disease images more comprehensively, and has higher universality and accuracy.

[0086] Furthermore, step S40 includes: S41: Based on a preset uncertainty estimation module, the diseased area of ​​the plant disease image is located to obtain a list of disease detection frame areas.

[0087] Specifically, based on the random initial detection box , generate multi-scale feature maps through iterative ROI Align, and optimize the bounding box coordinates by combining the gradient features of the disease edge. Introduce the uncertainty estimation module (Monte Carlo Dropout) for confidence calibration, filter low-confidence candidate boxes (confidence < 0.7), and obtain the final disease detection box area list .

[0088] S42: The U-Net architecture is used to predict the segmentation mask of the diseased area in the detection box area list; at the same time, the cross-attention mechanism is used to fuse the segmentation mask and semantic features to obtain the symptom area.

[0089] Specifically, traverse the list of disease detection frame areas , using the U-Net architecture to predict the segmentation mask of the diseased area. By fusing the segmentation mask with the text semantic features through the cross-attention mechanism, the interference of irrelevant areas is suppressed, and the segmented symptom area is obtained. , to achieve cross-modal interaction, the specific calculation formula to achieve this process is:

[0090] Where, and represents the modal interaction matrix that can be obtained through learning (training), Represents the Softmax activation function.

[0091] S43: Extract local features of the symptom area, and match the local features with a preset knowledge base to obtain a matching result.

[0092] Specifically, after extracting the local features of the symptom area, the spatial relationship between multiple regions is captured through the Transformer encoder to achieve global context modeling. Then, the similarity (cosine similarity) between the local features and the typical cases in the disease knowledge base is calculated to generate a list of candidate diseases. And the corresponding similarity , the above candidate disease list and the corresponding similarity are taken as the matching results.

[0093] S44: Input the matching results, pathology constraint reward parameters and image quality scores into the preset probability prediction model and selection model to calculate the evaluation results.

[0094] Specifically, the above probability prediction model is as follows:

[0095] The above probability prediction model uses pathological constraint reward parameter For the i-th candidate disease The predicted probability Make adjustments and get the adjusted probability .

[0096] Specifically, the above selection model is as follows:

[0097] Where, Indicates the evaluation results, It means to find the disease type with the highest probability from the disease types that meet the conditions. represents the confidence threshold, Represents a list of candidate diseases.

[0098] The calculation formula of the confidence threshold is:

[0099] Where, represents the basic confidence threshold. In this embodiment, Take 0.7; represents the quality adjustment coefficient. In this embodiment, When the image quality is low, the basic confidence threshold can be increased to reduce the risk of misdiagnosis.

[0100] Example 3 like Figure 3 As shown, this embodiment discloses a plant status assessment system based on multimodal thinking chain technology, which includes a processor and a memory, and the memory stores computer program instructions. When the computer program instructions are executed by the processor, the plant status assessment method based on multimodal thinking chain technology recorded in embodiment one or two is implemented.

Claims

1. A plant status assessment method based on multimodal thinking chain technology, characterized in that: include: S10: Obtain plant disease images, disease description text, environmental parameters and target crop species; S20: Associating the target crop species with the disease description text, inputting the associated data into a pre-trained language model, and calculating semantic features of the disease description text; S30: Inputting the plant disease image and the environmental parameters into a preset reinforcement learning and interpretability enhancement model to respectively calculate an image quality score and a pathology constraint reward parameter of the plant disease image; S40: Inputting the pathology constraint reward parameter, the image quality score and the semantic feature into a preset multimodal thinking chain reasoning model to calculate the evaluation result of the plant disease image.

2. The plant status assessment method based on multimodal thinking chain technology according to claim 1 is characterized in that: The language model adopts BERT, and step S20 includes: Inserting the explicitly marked target crop species before the disease description text to obtain associated data; The associated data is input into the BERT for multimodal fusion and feature extraction, and the semantic features are calculated.

3. The plant status assessment method based on multimodal thinking chain technology according to claim 1 is characterized in that: The calculation formula for the image quality score in the reinforcement learning and interpretability enhancement model is: Where, represents the image quality score, Indicates the image clarity score, represents the texture significance score of the lesion area, Indicates the color contrast at the boundary between healthy and diseased areas.

4. The plant status assessment method based on multimodal thinking chain technology according to claim 1 is characterized in that: The method for confirming the pathological constraint reward parameters in the reinforcement learning and interpretability enhancement model is as follows: In response to the plant disease image and the environmental parameters satisfying the conditions of the disease triangle theory, setting the pathology constraint reward parameter to a first set value; or In response to a logical contradiction between the plant disease image and the environmental parameter, the pathology constraint reward parameter is set to a second setting value, wherein an absolute value of the first setting value is greater than an absolute value of the second setting value, and the second setting value is a negative value.

5. The plant status assessment method based on multimodal thinking chain technology according to claim 4 is characterized in that: The conditions for judging whether the disease triangle theory is satisfied are: Condition 1: The plant disease image contains pathogens; Condition 2: The plant disease image has a host susceptibility match; Condition 3: The environmental parameters are within the preset environmental parameter judgment range; If any one of the conditions 1, 2 or 3 is met, the disease triangle theory is determined to be satisfied; otherwise, it is determined to be the logical contradiction.

6. The plant status assessment method based on multimodal thinking chain technology according to claim 1 is characterized in that: The multimodal thinking chain reasoning model includes an uncertainty estimation module, a U-Net architecture, a probability prediction model and a selection model. Step S40 includes: based on a preset uncertainty estimation module, locating the diseased area of ​​the plant disease image to obtain a disease detection frame area list; using the U-Net architecture to predict the segmentation mask of the diseased area in the detection frame area list; using a cross-attention mechanism to fuse the segmentation mask and the semantic features to obtain a symptom area; extracting local features of the symptom area, and matching the local features with a preset knowledge base to obtain a matching result; inputting the matching result, the pathology constraint reward parameter and the image quality score into a preset probability prediction model and selection model to calculate the evaluation result.

7. The plant status assessment method based on multimodal thinking chain technology according to claim 1 is characterized in that: Before step S30, the method further includes: Inputting the plant disease image into a preset neural network model to calculate image features of each organ in the plant disease image; Using a preset multi-layer perceptron to encode the environmental parameters to obtain environmental features; The image features, the semantic features and the environmental features are spliced ​​together to obtain a multimodal feature vector; the multimodal feature vector is used to enhance the representation ability and interpretability of the features.

8. The plant status assessment method based on multimodal thinking chain technology according to claim 7 is characterized in that: The neural network model includes at least YOLOv8-tiny and CNN. The plant disease image is input into a preset neural network model to calculate the image features of each organ in the plant disease image, specifically: Inputting the plant disease image into the YOLOv8-tiny to generate candidate disease frames; The preset organ type weight parameters and the candidate disease frame are input into the CNN to calculate the image features.

9. The plant status assessment method based on multimodal thinking chain technology according to claim 7 is characterized in that: The environmental parameters include at least temperature, humidity and light intensity. A preset multi-layer perceptron is used to encode the environmental parameters to obtain environmental characteristics, specifically: The temperature, humidity and light intensity are input into a multi-layer perceptron for encoding to obtain a low-dimensional vector as the environmental feature.

10. A plant status assessment system based on multimodal thinking chain technology, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the plant state assessment method based on the multimodal thinking chain technology according to any one of claims 1 to 9 is implemented.

Images