Intelligent diagnosis and decision method for crop growth state based on multi-modal data fusion

Abstract

The present application relates to the field of crop growth state intelligent diagnosis and decision-making, and particularly relates to a crop growth state intelligent diagnosis and decision-making method based on multi-modal data fusion. The content includes: obtaining disease and pest image evidence, growth image evidence and color image evidence based on image data of target crops, thereby obtaining image abnormality overall evidence; calculating stage environment deviation evidence based on pre-processed environmental sensor data, stage suitable environment central vector and stage tolerance scale vector, and calculating the consistency coefficient between image abnormality overall evidence and stage environment deviation evidence to generate a comprehensive diagnostic risk vector; calculating future weather forward-looking deviation value to generate the minimum necessary farming action; after the action is executed, the above steps are continued to collect data to execute the farming decision-making. The present application solves the problems of single image detection dimension, lack of credibility calibration when image evidence and environmental evidence conflict, and inaccurate farming regulation action in the existing crop growth state monitoring method.

Description

Technical Field

[0001] This invention relates to the field of intelligent diagnosis and decision-making for crop growth status, and particularly to a method for intelligent diagnosis and decision-making for crop growth status based on multimodal data fusion. Background Technology

[0002] With the development of facility agriculture, IoT sensors, image recognition, and edge computing technologies, image data, environmental data, and weather forecast data during crop cultivation can be continuously collected and used to assist agricultural management. Existing methods for monitoring crop growth status typically use single image recognition to identify pests and diseases, or use fixed environmental thresholds to determine whether ventilation, irrigation, supplemental lighting, or other control measures are needed. While these methods can achieve automated monitoring to some extent, they still have significant technical shortcomings: First, image recognition results typically focus only on pest and disease categories, failing to model pests and diseases, growth status, and color anomalies as different dimensions. This leads to the easy misjudgment or underjudgment of atypical anomalies such as yellowing leaves, wilting, chlorosis, and delayed growth. Second, environmental control usually uses a uniform threshold, failing to fully consider the different tolerance ranges of the target crop at different growth stages for temperature, humidity, soil moisture content, and light intensity. Third, existing multimodal fusion methods mostly employ simple stitching, weighted summation, or ordinary attention mechanisms, which cannot determine whether image anomalies are truly supported by the current growth stage and environmental deviations. When image evidence conflicts with environmental evidence, false alarms are easily generated. Fourth, existing agricultural decision-making is mostly post-event response control, only triggering alarms or executing equipment control based on whether the current monitoring value exceeds the threshold. It cannot combine future weather changes and current risk status to predict the impact of candidate actions on future risks, and it is also difficult to achieve minimal necessary intervention. Summary of the Invention

[0003] This invention provides an intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion, in order to solve the problems existing in crop growth status monitoring methods, such as single image detection dimension, underutilization of environmental threshold differences at different growth stages, lack of credibility calibration when image evidence and environmental evidence conflict, and easy over- or delayed agricultural regulation actions.

[0004] The intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion of the present invention includes the following steps: S1. Simultaneously acquire image data of the target crop, environmental sensor data, future weather forecast data, and current growth stage, and form a multimodal synchronous data set; preprocess, quality judge, effective organ extraction, and three-channel recognition of the image data of the target crop to obtain image evidence of pests and diseases, image evidence of growth, and color image evidence. S2. Based on the current growth stage of the target crop, retrieve the suitable environmental parameters for that stage to obtain the stage-suitable environment center vector and the stage-tolerance scale vector; preprocess the environmental sensor data, and calculate the stage-environment deviation evidence based on the preprocessed environmental sensor data, the stage-suitable environment center vector, and the stage-tolerance scale vector; calculate the overall image anomaly evidence based on pest and disease image evidence, growth image evidence, and color image evidence, and then calculate the consistency coefficient between the overall image anomaly evidence and the stage-environment deviation evidence; generate a comprehensive diagnostic risk vector based on the pest and disease image evidence, growth image evidence, color image evidence, stage-environment deviation evidence, and the consistency coefficient. S3. Calculate the forward deviation value of future weather based on future weather forecast data and the current growth stage, and determine the set of candidate agricultural actions based on the comprehensive diagnostic risk vector; establish an action-risk coverage vector based on the set of candidate agricultural actions, and generate the minimum necessary agricultural actions by combining evidence of pests and diseases in images, evidence of growth in images, evidence of color in images, evidence of stage environmental deviation, evidence of overall image anomalies, and forward deviation value of future weather; after the action is executed, continue to collect target crop image data, environmental sensor data, and future weather forecast data, and recalculate the comprehensive diagnostic risk vector according to S1 and S2, and adjust agricultural decisions by updating the action-risk coverage vector.

[0005] Preferably, the environmental sensor data includes air temperature, air humidity, soil moisture content, and light intensity; the current growth stage of the target crop is determined based on the number of days between the sowing date or planting date and the current date; during data collection, the clock of the edge computing node is used as a unified time reference, and the image data of the target crop, environmental sensor data, future weather forecast data, and current growth stage within the same time window are timestamped to form a multimodal synchronous data set at any given time.

[0006] Preferably, the image data of the target crop is subjected to distortion correction, and the image data after distortion correction is subjected to color space conversion and brightness equalization, and then denoising processing to obtain a preprocessed image. The preprocessed image is then judged in terms of quality, and the judgment results are divided into overexposed images, blurred images, and images without crop targets or severely occluded images. Images that meet any of the above quality judgment results are marked as low-quality images, and the remaining images are qualified preprocessed images. The qualified preprocessed images are input into the effective organ detection model to obtain effective organ image blocks. The effective organ image blocks are input into the joint recognition model, and the model outputs disease and pest image evidence, growth image evidence, and color image evidence through three parallel classification output layers.

[0007] The preferred formula for calculating evidence of deviation from the stage environment is: in, Evidence of deviation from the phased environment; The temperature of the air after pretreatment; The humidity of the air after pretreatment; The soil moisture content after pretreatment; The pre-processed light intensity; the suitable environmental center vector for the stage. ; These represent the center values ​​of the suitable ranges for pretreated air temperature, air humidity, soil moisture content, and light intensity at the current growth stage; the stage tolerance scale vector. ; These represent half the width of the suitable range for pretreated air temperature, air humidity, soil moisture content, and light intensity at the current growth stage. Used to limit environmental deviation results to a range of 0 to 1.

[0008] Preferably, the overall evidence of image anomalies is the average of the evidence of pests and diseases, the evidence of growth, and the evidence of color.

[0009] Preferably, the absolute value of the difference between the overall evidence of image anomalies and the evidence of deviation from the stage environment is first calculated, then the minimum value is taken by 1, and then the minimum value is subtracted from 1 to obtain the consistency coefficient between the overall evidence of image anomalies and the evidence of deviation from the stage environment.

[0010] Preferably, the comprehensive diagnostic risk vector includes pest and disease risk, abnormal growth risk, abnormal color risk, and environmental stress risk; and evidence of stage environmental deviation is used as environmental stress risk.

[0011] Preferably, deviation judgment is made on the future meteorological forecast data composed of air temperature forecast, air humidity forecast and rainfall forecast, and the deviation of temperature, humidity and rainfall is calculated hourly. The deviation of temperature, humidity and rainfall within the same hour is averaged to obtain the meteorological deviation for the corresponding hour. Based on the meteorological deviation of each hour, the forward deviation value of future meteorology is obtained.

[0012] The preferred formula for calculating the minimum necessary agricultural actions is: in, Indicates the time of diagnosis From the set of candidate agricultural actions The single least necessary agricultural action selected from the list; This is a comprehensive diagnostic risk vector; A cross-modal evidence matrix; Image evidence of pests and diseases; Evidence of growth through images; Evidence of color images; This is overall evidence of image anomalies; Evidence of deviation from the phased environment; This represents the forward-looking deviation value for future weather conditions; Indicates element-wise multiplication; It is a four-dimensional row vector of all 1s; It is a three-dimensional column vector of all 1s; Indicates transpose; Action-risk coverage vector; This represents the sum of four uncovered risk components, used to derive candidate agricultural actions. Total amount of uncovered risk after execution; Preset fixed action costs; This indicates that the operation will be temporarily suspended. This is an indicator function.

[0013] The beneficial effects of the technical solution of the present invention are: 1. The abnormality of the target crop image is divided into three image evidence channels: pests and diseases, growth status and color. This enables the diagnostic results to distinguish different sources of abnormality, such as pests and diseases, wilting or growth retardation, yellowing or fading, thereby improving the precision of crop status identification.

[0014] 2. By calculating the environmental deviation evidence based on the current growth stage, the environmental risk assessment no longer relies on a uniform threshold across all stages, but rather matches the suitable environmental range corresponding to the current growth stage of the target crop.

[0015] 3. By using the consistency coefficient between overall evidence of image anomalies and stage-specific environmental deviation evidence, we can determine whether image anomalies and environmental anomalies support each other, thereby reducing false alarms caused by a single image anomaly or a single environmental anomaly.

[0016] 4. By integrating cross-modal attention fusion, the comprehensive diagnostic risk vector, three types of image evidence, overall image anomaly evidence, stage environmental deviation evidence, and future weather forecast deviation value are uniformly incorporated into agricultural decision-making, so that agricultural recommendations can simultaneously reflect the current crop status, current environmental deviation, and future weather changes.

[0017] 5. By using the action-risk coverage vector, select the single action with the smallest total uncovered risk from the candidate agricultural actions, and recalculate it at the next diagnostic time to achieve rolling treatment, so as to avoid excessive intervention or control conflicts caused by multiple devices triggering at the same time. Attached Figure Description

[0018] Figure 1 This is a flowchart of the intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion as described in this invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] The following description, in conjunction with the accompanying drawings, details the specific scheme of the intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion provided by this invention.

[0022] See attached document Figure 1 The diagram illustrates a flowchart of an intelligent crop growth status diagnosis and decision-making method based on multimodal data fusion, provided by an embodiment of the present invention. The method includes the following steps: S1. Simultaneously acquire image data of the target crop, environmental sensor data, future weather forecast data, and current growth stage, and form a multimodal synchronous data set; preprocess, quality judge, extract effective organs, and perform three-channel recognition on the image data of the target crop to obtain image evidence of pests and diseases, image evidence of growth, and color image evidence.

[0023] First, based on fixed RGB cameras, air temperature and humidity sensors, soil moisture sensors, light intensity sensors, weather forecasting interfaces, and edge computing nodes deployed within the target crop planting area, multimodal raw data during the target crop growth process are collected synchronously.

[0024] The fixed RGB camera is installed above the target crop planting area to capture canopy images of the target crop and use these canopy images as image data of the target crop. Image data of the target crop It includes information on the target crop's leaf morphology, leaf texture, canopy coverage, and color distribution.

[0025] Air temperature and humidity sensors are installed above the target crop canopy to collect air temperature and humidity data. Soil moisture sensors are buried in the main root distribution area of ​​the target crop to collect soil moisture data. Light intensity sensors are placed near the canopy height to collect light intensity data. Air temperature, air humidity, soil moisture content, and light intensity together constitute environmental sensor data. .

[0026] The weather forecasting interface is used to obtain air temperature forecast sequences, air humidity forecast sequences, and rainfall forecast sequences for the next few hours. As a specific example, it obtains air temperature forecast sequences, air humidity forecast sequences, and rainfall forecast sequences for the next 6 hours, and combines these sequences to form future weather forecast data. .

[0027] Current growth stage of the target crop The current growth stage of the target crop is determined based on the number of days between its sowing or transplanting date and the current date. The intervals for each growth stage are determined according to the growth period divisions recorded in the current standard cultivation technical regulations for the target crop. When the interval falls within the time range of a certain growth stage of the target crop, the corresponding growth stage is determined as the current growth stage of the target crop. .

[0028] During data acquisition, the clocks of the edge computing nodes are used as a unified time reference to collect image data of the target crop within the same time window. Environmental sensor data Future weather forecast data and current growth stage Perform timestamp binding to form a time. Multimodal synchronization data set ,in, The time index for diagnosing the growth status of the target crop indicates the time point at which a multimodal raw data synchronous acquisition and diagnostic calculation is completed. The time window is centered on the image data acquisition time, with a length of 2.5 minutes before and after the image data acquisition time.

[0029] After completing the acquisition of multimodal raw data, the image data of the target crop Preprocessing, quality assessment, effective organ extraction, and three-channel recognition are performed. Specifically, the intrinsic parameter matrix and distortion coefficients of the fixed RGB camera are first obtained using Zhang Zhengyou's checkerboard camera calibration method, and then the image data of the target crop are analyzed accordingly. Radial and tangential distortion corrections were performed. The corrected image data was then converted from the RGB color space to the Lab color space, where luminance and color are separated. In this color space, only the luminance channel underwent contrast-limited adaptive histogram equalization, while the color channels were left unprocessed. The processed image was then converted back to the RGB color space to reduce the impact of uneven illumination on image recognition and avoid tone shift caused by separately enhancing the RGB channels. Bilateral filtering was used for image denoising. The neighborhood window diameter for bilateral filtering was determined based on the normalized input size of 640×640 pixels, ranging from 5 to 9 pixels. The spatial domain standard deviation was determined based on the spatial continuity of leaf edges and vein textures in the image data, ranging from 2 to 5. The value domain standard deviation was determined based on the color or grayscale differences between the leaf area, background area, and lesion area, ranging from 25 to 75. This reduced sensor noise and fine texture noise while preventing excessive smoothing of leaf edges, vein textures, and lesion boundaries. Finally, a letterbox method with proportional scaling and edge filling was used to normalize the image. The pixel dimensions are used to obtain the preprocessed image.

[0030] Subsequently, the preprocessed images are quality-assessed as follows: when the grayscale values ​​of all three RGB channels are greater than 250, they are considered overexposed pixels; when the proportion of overexposed pixels in the image is greater than 20%, the image is considered overexposed; when the Laplacian variance of the grayscale image is less than 100, it is considered a blurred image (the grayscale image is obtained by converting the RGB color image using a psychological formula); when the YOLOv8 effective organ detection model does not output effective organ detection boxes with a confidence score greater than 0.25, or when the total area of ​​the effective organ detection boxes accounts for less than 5% of the image area, it is considered an image without crop targets or a severely occluded image. Images that meet any of the above quality assessment conditions are marked as low-quality images, and a manual review prompt is output; the remaining images are considered qualified preprocessed images and enter the effective organ extraction process.

[0031] Next, the qualified preprocessed images are input into the YOLOv8 effective organ detection model. Effective organ detection boxes with a confidence score greater than 0.25 are retained, and effective organ image patches are cropped based on these boxes, while background, soil, and support areas outside the effective organ detection boxes are discarded. Effective organ image patches include the main areas of the target crop: leaves, stems, flowers, fruits, and canopy. Finally, the effective organ image patches are input into a joint recognition model. This model uses the YOLOv8 backbone network to extract shared image features from the effective organ image patches and sets up three parallel classification output layers based on existing multi-task learning structures. Each classification output layer consists of a global average pooling layer, a fully connected layer, and a sigmoid output layer connected sequentially. The three parallel classification output layers have the same input but different output tasks: a pest and disease classification output layer, a growth status classification output layer, and a color classification output layer. The three parallel classification output layers have the same hierarchical structure, but the learnable parameters of each classification output layer are independent. Specifically, the global average pooling layer performs spatial dimensional average pooling on the 20×20×512 feature map output by the YOLOv8 backbone network, resulting in a 512-dimensional feature vector. The fully connected layers in each classification output layer map the 512-dimensional feature vector to a 128-dimensional latent feature representation, and generate corresponding task-specific image evidence through their respective independent Sigmoid output layers. The pest and disease classification output layer maps shared image features to pest and disease image evidence. The growth classification output layer maps shared image features to growth image evidence. The color classification output layer maps shared image features to color image evidence. Image evidence of pests and diseases Used to characterize the probability or intensity of abnormalities in corresponding diseases, pests, and insect damage in target crops; growth image evidence Used to characterize wilting, growth retardation, abnormal plant architecture, insufficient canopy cover, and abnormal organ development; color image evidence These three types of image evidence are used to characterize yellowing, fading, darkening, uneven coloring, and localized abnormal color spots. Together, they form the image evidence set. Thus, after one feature extraction, the same effective organ image block yields three types of image evidence: pests and diseases, growth, and color.

[0032] S2. Based on the current growth stage of the target crop, retrieve the suitable environmental parameters for that stage to obtain the stage-suitable environment center vector and the stage-tolerance scale vector; preprocess the environmental sensor data, and calculate the stage-environment deviation evidence based on the preprocessed environmental sensor data, the stage-suitable environment center vector, and the stage-tolerance scale vector; calculate the overall image anomaly evidence based on pest and disease image evidence, growth image evidence, and color image evidence, and then calculate the consistency coefficient between the overall image anomaly evidence and the stage-environment deviation evidence; generate a comprehensive diagnostic risk vector based on the pest and disease image evidence, growth image evidence, color image evidence, stage-environment deviation evidence, and the consistency coefficient.

[0033] Based on the current growth stage of the target crop To retrieve suitable environmental parameters for the target crop stage, the following steps are taken: First, establish a target crop stage environmental knowledge base, determined by the target crop's standard agronomic management practices. This knowledge base includes suitable temperature, humidity, soil moisture content, light intensity, and rainfall impact thresholds for each growth stage of the target crop. Then, based on the target crop's current growth stage... The calculation stage is suitable for the center vector of the environment. and stage tolerance scale vector ,in, This represents the average of the upper and lower limits of the suitable range for temperature, humidity, soil moisture content, and light intensity at the current growth stage. This is half the width of the suitable range for temperature, humidity, soil moisture content, and light intensity at the current growth stage. Then, based on the current environmental sensor data... Preprocessing is performed first, based on the suitable environmental ranges for each growth stage as recorded in the standard agronomic management procedures for the target crop, and combined with the measurement ranges of environmental sensors within the target crop planting area, to determine the physically reasonable range of environmental variables. Environmental sensor data whose collected values ​​exceed the physically reasonable range are truncated to avoid abnormal sampling values ​​affecting subsequent calculations. Environmental sensors include, but are not limited to, air temperature and humidity sensors, soil moisture sensors, and light intensity sensors. Then, based on the time reference of the edge computing nodes, multiple sampling results within the same time window are time-aligned and averaged to reduce instantaneous fluctuations, ultimately obtaining the preprocessed environmental sensor data. The preprocessed environmental sensor data is then compared with the stage-suitable environment center vector. and stage tolerance scale vector By comparison, evidence of deviation from the stage environment is obtained. The calculation formula is: , in, The temperature of the air after pretreatment; The humidity of the air after pretreatment; The soil moisture content after pretreatment; The light intensity after preprocessing; These are the center values ​​of the suitable ranges for air temperature, air humidity, soil moisture content, and light intensity after pretreatment, respectively, representing the average values ​​of the upper and lower limits of each suitable environmental range under the current growth stage. These represent half the width of the suitable range for pretreated air temperature, air humidity, soil moisture content, and light intensity at the current growth stage. This is used to limit environmental deviation results to a range of 0 to 1. Therefore, the closer the preprocessed environmental sensor data is to the suitable environmental center for the current growth stage, the better. The closer to 0, the more significantly the preprocessed environmental sensor data deviates from the suitable environmental center for the current growth stage. The closer it is to 1.

[0034] Subsequently, to avoid directly outputting high-risk results when image evidence and environmental deviation evidence contradict each other, the overall evidence of image anomalies is calculated. Evidence of deviation from stage environment Consistency coefficient between Overall evidence of image anomalies The average of the three types of image evidence, the consistency coefficient Calculate using the following formula: , , in, This represents the overall evidence of image anomalies, indicating the overall intensity of three types of image anomalies: pests and diseases, growth status, and color. This represents the consistency coefficient between overall evidence of image anomalies and evidence of deviation from the stage environment. A coefficient indicating similarity between overall evidence of image anomalies and evidence of deviation from the stage environment is considered auspicious. The closer the value is to 1, the more mutually supportive the image anomaly and the deviation from the current environment become. When the overall evidence of image anomalies differs significantly from the evidence of deviation from the current environment, the more supportive the image anomaly becomes. The closer it is to 0, the more likely the image is abnormal and deviates from the current environment.

[0035] Finally, based on image evidence of pests and diseases Growth image evidence Color image evidence Evidence of deviation from stage environment Consistency coefficient Generate a comprehensive diagnostic risk vector Specifically ,in, The risk of pests and diseases is indicated and calculated as follows: ; This indicates the risk of abnormal growth, calculated as follows: ; The risk of color anomalies is indicated and calculated as follows: ; Indicates environmental stress risk. .in, Environmental risks verified by image evidence; , and These represent the risk supplementation space that is not yet fully occupied by the corresponding image evidence, so that environmental evidence can supplement risks only when the corresponding image evidence is not saturated, without diluting the existing high-confidence image anomaly evidence.

[0036] S3. Calculate the forward deviation value of future weather based on future weather forecast data and the current growth stage, and determine the set of candidate agricultural actions based on the comprehensive diagnostic risk vector; establish an action-risk coverage vector based on the set of candidate agricultural actions, and generate the minimum necessary agricultural actions by combining evidence of pests and diseases in images, evidence of growth in images, evidence of color in images, evidence of stage environmental deviation, evidence of overall image anomalies, and forward deviation value of future weather; after the action is executed, continue to collect target crop image data, environmental sensor data, and future weather forecast data, and recalculate the comprehensive diagnostic risk vector according to S1 and S2, and adjust agricultural decisions by updating the action-risk coverage vector.

[0037] Based on future weather forecast data and current growth stage Calculate the forward deviation of future weather forecasts This is used to characterize the degree of deviation of meteorological conditions in the next 6 hours from the suitable environmental center of the current growth stage. Specifically, deviation judgments are made for the predicted air temperature, air humidity, and rainfall values ​​for the next 1 to 6 hours. For any hour within the next 6 hours, if the predicted air temperature is within the suitable temperature range of the current growth stage, the corresponding hourly temperature deviation is recorded as 0. If the predicted air temperature is higher than the upper limit of the suitable temperature range or lower than the lower limit, a two-sided deviation calculation method is used to quantify the excess portion. When it is higher than the upper limit, the difference between the predicted air temperature and the upper limit is taken; when it is lower than the lower limit, the difference between the lower limit and the predicted air temperature is taken. Both are expressed as positive numbers, and the temperature deviation is calculated as a proportion of the width of the suitable temperature range. Finally, an upper limit cutoff function is used to limit the temperature deviation to the range of 0 to 1. For predicted air humidity, when it falls within the suitable humidity range for the current growth stage, the corresponding one-hour humidity deviation is recorded as 0. When the predicted air humidity exceeds the upper limit of the suitable humidity range or falls below the lower limit, a two-sided deviation calculation method is used to quantify the excess portion. Specifically, when it exceeds the upper limit, the difference between the predicted air humidity and the upper limit is used; when it falls below the lower limit, the difference between the lower limit and the predicted air humidity is used. This difference is then used to calculate the humidity deviation in proportion to the width of the suitable humidity range, and an upper limit cutoff function is applied to limit the humidity deviation to the range of 0 to 1. For predicted rainfall, when it is not higher than the rainfall impact threshold corresponding to the current growth stage, the corresponding one-hour rainfall deviation is recorded as 0. When it exceeds the rainfall impact threshold, the rainfall deviation is first calculated based on the ratio of the excess portion to the rainfall impact threshold, and then an upper limit cutoff function is applied to limit the rainfall deviation to the range of 0 to 1. Then, the temperature deviation, humidity deviation, and rainfall deviation within the same hour are averaged to obtain the corresponding one-hour meteorological deviation. Finally, the meteorological deviation over the next 6 hours is averaged to obtain the future meteorological forward deviation value. The range of values ​​is , The larger the value, the more significant the deviation of the meteorological conditions in the next 6 hours from the suitable environmental range of the current growth stage. Among them, the suitable temperature range, suitable humidity range, and rainfall impact threshold are all derived from the target crop stage environmental knowledge base.

[0038] Next, based on the current standard cultivation technical regulations, standard agronomic management rules, feasible equipment already deployed in the target crop planting area, and comprehensive diagnostic risk vectors, a set of candidate agricultural actions is determined. The existing standard cultivation technical regulations and standard agronomic management rules for the target crop are used to determine the permissible temperature and humidity control, irrigation, supplemental lighting, and pest and disease control measures for the target crop at different growth stages; the executable equipment already deployed in the target crop planting area, such as fans, supplemental lighting, drip irrigation devices, and management terminals, serve as a set of candidate agricultural actions. The actual execution target; the comprehensive diagnostic risk vector, including pest and disease risk, abnormal growth risk, abnormal color risk, and environmental stress risk, is used to limit the types of risks that candidate agricultural actions must be able to cover. Specifically, the set of candidate agricultural actions... This includes ventilation and dehumidification, supplemental lighting, irrigation, pest and disease early warning, and temporary suspension of operations. Ventilation and dehumidification are performed by fans to reduce the environmental stress risks caused by high humidity or poor ventilation, and to help reduce the risk of pests and diseases under high humidity conditions. Supplemental lighting is performed by supplemental lighting lamps to reduce the risk of abnormal growth and color under low light conditions. Irrigation is performed by drip irrigation devices to reduce the risk of abnormal growth, color, and environmental stress caused by insufficient soil moisture. Pest and disease early warning is performed by the management terminal to push abnormal images, main risk categories, and risk causes when the risk of pests and diseases is high, triggering manual review or manual processing. The management terminal includes, but is not limited to, agricultural IoT management platform servers, farm-local edge computing servers, or agricultural mobile monitoring applications. Temporary suspension of operations is used when candidate agricultural actions are unnecessary or the corresponding uncovered risks are no greater than the uncovered risks corresponding to any candidate agricultural action. In this case, equipment control is not triggered, and data is only collected for the next diagnostic moment, thus avoiding excessive intervention.

[0039] As a specific embodiment, each type of candidate agricultural action is further defined as a single basic action unit: ventilation and dehumidification corresponds to the fan being turned on for 20 minutes, supplemental lighting corresponds to the supplemental lighting being turned on for 30 minutes, irrigation corresponds to the drip irrigation device being turned on for 10 minutes, pest and disease early warning corresponds to pushing abnormal images, main risk categories, and risk causes to the management terminal, and suspended operation means not triggering equipment control. The execution duration of a single basic action unit is determined based on the rated parameters of the equipment, the spatial scale of the facility, and historical management records within the target crop planting area, and can be recorded in the action configuration table of the edge computing node.

[0040] Subsequently, the candidate agricultural action set was combined. Establish action-risk coverage vector Each candidate agricultural action Each corresponds to a four-dimensional binary vector. ,in, , , , They represent candidate agricultural actions. Whether it can cover the risks of pests and diseases, abnormal growth, abnormal color, and environmental stress. If it can cover them, the corresponding value is 1; otherwise, the value is 0. Index of candidate agricultural actions, .

[0041] Next, following the design principles of preserving the original diagnosis, cross-modal attention calibration, and minimizing residual risk, an explicit calculation formula for the minimum necessary agricultural actions is constructed. Preserving the original diagnosis ensures that the risk outcome of the final action selection does not deviate from the comprehensive diagnostic risk vector. This ensures that agricultural action selection is always based on the completed comprehensive diagnostic risk vector. Cross-modal attention calibration incorporates evidence from pest and disease images, growth images, color images, stage-specific environmental deviations, overall image anomalies, and forward-looking weather deviations. Attention weights are calculated separately for pest and disease risk, growth anomaly risk, color anomaly risk, and environmental stress risk, allowing the impact of different evidence sources on different risk types to be adaptively adjusted. Risk residual minimization combines the risk coverage relationships of candidate agricultural actions to select the action with the smallest total uncovered risk after execution, thereby achieving necessary intervention with minimal cost. The explicit calculation formula is as follows: , in, Indicates the time of diagnosis From the set of candidate agricultural actions The single minimum necessary agricultural action selected refers to the candidate action with the smallest total uncovered risk after cross-modal attention calibration, provided that multiple actions are not triggered simultaneously at the current diagnosis time. The first cross-modal evidence matrix is ​​used to calculate the cross-modal attention weight matrix, and the second cross-modal evidence matrix is ​​used to calculate the attention enhancement amount for each risk type by multiplying it element-wise with the cross-modal attention weight matrix. This means that a Softmax operation is performed on each row of the cross-modal evidence matrix. By performing the Softmax operation on each row, the relative contributions of the four types of evidence sources can be compared within the same risk type, so that the modality with higher evidence value receives higher attention weight. As a result, the risks of pests and diseases, abnormal growth, abnormal color, and environmental stress have different evidence weight allocation results. Indicates element-wise multiplication; It is a four-dimensional all-1 row vector used in the comprehensive diagnostic risk vector. Based on this, a cross-modal attention enhancement amount is introduced; It is a three-dimensional column vector of all 1s, used to sum the weighted results of the three pieces of evidence in each row of the cross-modal evidence matrix; Indicates transpose; This represents the sum of four uncovered risk components, used to derive candidate agricultural actions. Total amount of uncovered risk after execution; To preset a fixed action cost, a value of 0.1 is preferred; This indicates that the operation will be temporarily suspended. This is an indicator function; if the candidate agricultural action is not a postponed operation, it is set to 1; if the candidate agricultural action is a postponed operation, it is set to 0. This indicates how much space is left for each type of risk to reach the maximum risk value of 1; The result of cross-modal attention fusion means that the attention weight of each row is multiplied element-wise by the cross-modal evidence corresponding to the row, and the sum is obtained over the three evidence sources to obtain the attention enhancement amount corresponding to the four types of risks. This indicates that cross-modal evidence can only supplement... The portion that is not yet fully occupied; This indicates the final decision risk after joint calibration of images, environment, and future weather conditions. Used to indicate candidate agricultural actions Uncovered risk types; This is the cost item for action execution, used to avoid frequent equipment startups due to minor risk fluctuations.

[0042] In practical implementation, if the total uncovered risk corresponding to ventilation and dehumidification is the smallest, the edge computing node outputs a control command to turn on the fan for 20 minutes; if the total uncovered risk corresponding to supplemental lighting is the smallest, it outputs a control command to turn on the supplemental lighting for 30 minutes; if the total uncovered risk corresponding to irrigation is the smallest, it outputs a control command to turn on the drip irrigation device for 10 minutes; if the total uncovered risk corresponding to pest and disease early warning is the smallest, it pushes abnormal images, main risk categories, and risk sources to the management terminal; if the total uncovered risk corresponding to suspending operations is the smallest, it does not trigger equipment control and continues to collect data at the next diagnostic moment.

[0043] After the action is executed, subsequent time-lapse image data of the target crop, environmental sensor data, and crop status changes are collected, and the comprehensive diagnostic risk vector is recalculated according to steps S1 and S2. The comprehensive diagnostic risk vector before action execution, the cross-modal attention fusion result, the selected agricultural action, and the comprehensive diagnostic risk vector after action execution are used as execution feedback data to adjust the action-risk coverage vector. The configuration records enable subsequent agricultural decisions to gradually adapt to the actual planting environment, seasonal changes, and management habits of the target crop.

[0044] In summary, a method for intelligent diagnosis and decision-making of crop growth status based on multimodal data fusion has been developed.

[0045] The order of the embodiments is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0046] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0047] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for intelligent diagnosis and decision-making of crop growth status based on multimodal data fusion, characterized by: S1. Simultaneously acquire image data of the target crop, environmental sensor data, future weather forecast data, and current growth stage, and form a multimodal synchronous data set; preprocess, quality judge, effective organ extraction, and three-channel recognition of the image data of the target crop to obtain image evidence of pests and diseases, image evidence of growth, and color image evidence. S2. Based on the current growth stage of the target crop, retrieve the suitable environmental parameters for that stage to obtain the stage-suitable environment center vector and the stage-tolerance scale vector; preprocess the environmental sensor data, and calculate the stage-environment deviation evidence based on the preprocessed environmental sensor data, the stage-suitable environment center vector, and the stage-tolerance scale vector; calculate the overall image anomaly evidence based on pest and disease image evidence, growth image evidence, and color image evidence, and then calculate the consistency coefficient between the overall image anomaly evidence and the stage-environment deviation evidence; generate a comprehensive diagnostic risk vector based on the pest and disease image evidence, growth image evidence, color image evidence, stage-environment deviation evidence, and the consistency coefficient. S3. Calculate the forward deviation value of future weather based on future weather forecast data and the current growth stage, and determine the set of candidate agricultural actions based on the comprehensive diagnostic risk vector; establish an action-risk coverage vector based on the set of candidate agricultural actions, and generate the minimum necessary agricultural actions by combining evidence of pests and diseases in images, evidence of growth in images, evidence of color in images, evidence of deviation of stage environment, evidence of overall image anomalies, and forward deviation value of future weather. After the action is executed, target crop image data, environmental sensor data and future weather forecast data continue to be collected, and the comprehensive diagnostic risk vector is recalculated according to S1 and S2. Agricultural decisions are adjusted by updating the action-risk coverage vector.

2. The intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion according to claim 1, characterized in that, Environmental sensor data includes air temperature, air humidity, soil moisture content, and light intensity; the current growth stage of the target crop is determined based on the number of days between the sowing date or planting date and the current date; during data collection, the clock of the edge computing node is used as a unified time reference, and the image data of the target crop, environmental sensor data, future weather forecast data, and current growth stage within the same time window are timestamped to form a multimodal synchronous data set at any given moment.

3. The intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion according to claim 2, characterized in that, Distortion correction is performed on the image data of the target crop. After distortion correction, color space conversion and brightness equalization are performed on the image data, and noise reduction is carried out to obtain a preprocessed image. The preprocessed images are quality-assessed, and the results are categorized as overexposed, blurred, or images without crop targets or severely occluded. Images that meet any of these quality assessment results are marked as low-quality images, while the remaining images are considered qualified preprocessed images. The qualified preprocessed images are then input into the effective organ detection model to obtain effective organ image blocks. These effective organ image blocks are then input into the joint recognition model, which outputs evidence of pests and diseases, growth, and color through three parallel classification output layers.

4. The intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion according to claim 1, characterized in that, The formula for calculating the evidence of deviation from the stage environment is: in, Evidence of deviation from the phased environment; The temperature of the air after pretreatment; The humidity of the air after pretreatment; The soil moisture content after pretreatment; The pre-processed light intensity; the suitable environmental center vector for the stage. ; These represent the center values ​​of the suitable ranges for pretreated air temperature, air humidity, soil moisture content, and light intensity at the current growth stage; the stage tolerance scale vector. ; These represent half the width of the suitable range for pretreated air temperature, air humidity, soil moisture content, and light intensity at the current growth stage. Used to limit environmental deviation results to a range of 0 to 1.

5. The intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion according to claim 3, characterized in that, The overall evidence of image anomalies is the average of the evidence of pests and diseases, growth, and color.

6. The intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion according to claim 5, characterized in that, First, calculate the absolute value of the difference between the overall evidence of image anomalies and the evidence of deviation from the stage environment. Then, take the minimum value with 1. Finally, subtract the minimum value from 1 to obtain the consistency coefficient between the overall evidence of image anomalies and the evidence of deviation from the stage environment.

7. The intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion according to claim 6, characterized in that, The comprehensive diagnostic risk vector includes pest and disease risk, abnormal growth risk, abnormal color risk, and environmental stress risk; evidence of environmental deviation at different stages is considered as environmental stress risk.

8. The intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion according to claim 1, characterized in that, Deviation judgment is made on the future meteorological forecast data composed of air temperature forecast, air humidity forecast and rainfall forecast, and the deviation of temperature deviation, humidity deviation and rainfall deviation are calculated hourly. The deviation of temperature deviation, humidity deviation and rainfall deviation within the same hour are averaged to obtain the meteorological deviation for the corresponding hour. Based on the meteorological deviation of each hour, the forward deviation value of future meteorology is obtained.

9. The intelligent diagnosis and decision-making method for crop growth status based on multimodal data fusion according to claim 8, characterized in that, The formula for calculating the minimum necessary agricultural actions is: in, Indicates the time of diagnosis From the set of candidate agricultural actions The single least necessary agricultural action selected from the list; This is a comprehensive diagnostic risk vector; A cross-modal evidence matrix; Image evidence of pests and diseases; Evidence of growth through images; Evidence of color images; This is overall evidence of image anomalies; Evidence of deviation from the phased environment; This represents the forward-looking deviation value for future weather conditions; Indicates element-wise multiplication; It is a four-dimensional row vector of all 1s; It is a three-dimensional column vector of all 1s; Indicates transpose; Action-risk coverage vector; This represents the sum of four uncovered risk components, used to derive candidate agricultural actions. Total amount of uncovered risk after execution; Preset fixed action costs; This indicates that the operation will be temporarily suspended. This is an indicator function.

Images