Litchi orchard remote sensing image decision analysis system based on multi-model fusion

Abstract

The application relates to the technical field of artificial intelligence and precision agriculture, and particularly discloses a litchi orchard remote sensing image decision analysis system based on multi-model fusion. The same frame of litchi orchard unmanned aerial vehicle remote sensing image is input into multiple special target detection models in parallel to obtain structured detection results of each task; the heterogeneous detection data is integrated into structured text in a unified format; then the original unmanned aerial vehicle remote sensing image and the structured text are jointly input into a low-rank-adapted multi-modal large language model, four-step thinking chain reasoning is carried out, scene-mismatched noise output is automatically filtered, and a structured decision report is generated in combination with litchi phenology field knowledge. The application realizes end-to-end full-automatic analysis, solves the posterior selection dilemma of multiple special models, and improves the automation degree and decision reliability of orchard remote sensing analysis.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and precision agriculture, and more specifically to a decision analysis system for remote sensing images of litchi orchards based on multi-model fusion. Background Technology

[0002] Currently, UAV remote sensing technology is widely used in orchard management. It can acquire high-resolution images of litchi canopy in a non-contact manner, achieving millimeter-level ground resolution at a flight altitude of 5-15 meters, sufficient to distinguish fine structures such as flowers, fruits, and branches. However, orchard agronomic analysis focuses on fine target areas (ROIs) in the images. The scale of litchi targets such as flower buds, fruits, and shoots varies greatly, and the core indicators to be monitored differ at different phenological stages, placing high demands on the fine-grained recognition capabilities of the detection model.

[0003] YOLO object detection models, with their balanced advantages in speed and accuracy, have been applied to scenarios such as litchi flowering rate estimation and fruit detection. However, existing research trains independent models for single agronomic indicators, making them incompatible with each other. Litchi orchard management requires coverage of multiple indicators, including flowering rate, fruit maturity, shoot growth, and autumn / winter shoot identification. Furthermore, users cannot predict the scene type before acquiring images, making it impossible to pre-select a matching detection model, leading to a dilemma of posterior selection among multiple models. If images are input into all models, models unsuitable for the scene will produce systematic false detections due to visual feature similarity, with false detection confidence levels mostly in the 0.5-0.7 range. These false detections cannot be effectively filtered out using simple confidence thresholds, and manual screening and summarization are not only inefficient but also result in information loss.

[0004] Existing agricultural diagnostic solutions that combine target detection with large language models are all based on the integration of a single detection model and a large model. They only realize the transformation of detection results into decisions and cannot solve the problems of scene matching and noise filtering for multiple specialized target detection models. They are also difficult to adapt to the monitoring needs of lychee orchards with multiple phenological stages and multiple indicators.

[0005] Therefore, how to provide a remote sensing image decision analysis system that can adapt to the multi-dimensional monitoring needs of different phenological stages in lychee orchards and can make independent inferences is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a decision analysis system for remote sensing images of litchi orchards based on multi-model fusion that overcomes or at least partially solves the above problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] This invention relates to a decision analysis system for remote sensing images of litchi orchards based on multi-model fusion, comprising: Multi-target parallel detection module: The same frame of original UAV remote sensing image of lychee orchard is input into multiple pre-trained dedicated target detection models in parallel. Each dedicated target detection model performs inference operations independently and outputs structured detection results in heterogeneous formats.

[0009] Structured text fusion module: Receives heterogeneous structured detection results from all dedicated target detection models, converts the heterogeneous structured detection results into structured text of a unified format, generates structured text of a unified format that corresponds one-to-one with the original UAV remote sensing image, and establishes a correlation mapping with the original UAV remote sensing image.

[0010] Multimodal Large Language Model Decision Engine Module: The original UAV remote sensing images and structured text are input into the multimodal large language model after low-rank adaptation processing. The multimodal large language model performs multi-stage thought chain reasoning and, combined with the litchi phenological period domain knowledge base, outputs a structured decision report for the litchi orchard.

[0011] Preferably, the multiple pre-trained dedicated target detection models in the multi-target parallel detection module are four dedicated target detection models based on the YOLO architecture, namely, flowering rate detection model, fruit maturity detection model, shoot length measurement model, and autumn and winter shoot identification model.

[0012] The four models were independently trained and deployed for four detection tasks: litchi flower organ classification and counting, fruit maturity grading, shoot length measurement, and branch type classification.

[0013] Preferably, each dedicated target detection model outputs a structured detection result corresponding to its detection task. The structured detection result includes the total number of detected targets, confidence data, and status markers, specifically including: Flowering rate detection model: used to detect the number of flower buds, the number of open flowers, the calculated flowering rate, the confidence data of each detected target, and the spatial location information in the original UAV remote sensing image.

[0014] Fruit maturity detection model: used to detect the number of green fruits, ripening fruits and red fruits in the original UAV remote sensing images, the calculated maturity rate, the confidence data of each detected target and the spatial location information.

[0015] Shoot length measurement model: used to detect the number of shoots, the pixel length value of a single shoot, length statistics, confidence data and spatial location information of each detected target in raw UAV remote sensing images.

[0016] Autumn and winter shoot identification model: used to detect the number of autumn shoots, the number of winter shoots, the proportion of autumn and winter shoots, the confidence data and spatial location information of each detected target in the original UAV remote sensing image.

[0017] The dedicated target detection model that fails to detect the corresponding target outputs a zero detection flag and invalid response information.

[0018] Preferably, the structured text fusion module also includes: Construct a summary table of detection results, convert the records into structured text in a unified format, and mark the dedicated target detection models with a status of pending verification for those with a total number of detected targets lower than a preset minimum threshold and confidence data lower than a preset confidence threshold.

[0019] In the summary table of detection results, an independent information block is generated for each dedicated target detection model. The information block records the structured detection results of the corresponding dedicated target detection model.

[0020] The confidence data of all detected targets are retained. For dedicated target detection models that detect zero targets, the corresponding information block in the detection result summary table is marked with a status flag indicating that no valid targets were detected.

[0021] Preferably, the multi-stage reasoning in the multimodal large language model decision engine module includes four reasoning stages: scene recognition, confidence assessment, cross-validation, and decision generation.

[0022] Preferably, the scene recognition stage includes: Using raw UAV remote sensing images as visual input, without relying on structured detection results, the multimodal large language model outputs scene judgment results, which include the corresponding litchi phenological scene type, target density, and image quality.

[0023] Preferably, the confidence assessment phase includes: Based on the scene judgment results, the results are compared one by one with the structured detection results, and a corresponding confidence label is output for each model.

[0024] Among them, the detection target of the dedicated target detection model matches the scene type, and the structured detection results are consistent with visual observation. The confidence label of the structured detection results of the corresponding dedicated target detection model is marked as high confidence.

[0025] The target detected by the dedicated target detection model is clearly inconsistent with the scene type, so the confidence label of the structured detection result of the corresponding dedicated target detection model is marked as low confidence.

[0026] If the dedicated target detection model detects zero targets and there are no targets corresponding to the model in the scene judgment results, the confidence label of the structured detection results of the corresponding dedicated target detection model will be marked as "the scene confirms that there are no such targets".

[0027] If the number of detected targets and the number of detected targets by the dedicated target detection model are lower than the preset minimum threshold and the confidence data is lower than the preset confidence threshold, then the confidence label of the structured detection result of the corresponding dedicated target detection model will be marked as noise output.

[0028] Model outputs marked as pending verification need to undergo scene matching verification. Statistical information and state labels marked as high confidence and indicating that there is no such target in the scene will proceed to the next inference stage.

[0029] Preferably, the cross-validation phase includes: The high-confidence detection results and the results confirming that there is no such target in the scenario are used as inputs. The mutual verification is carried out by using the constraint relationship between the detection indicators in the pre-set litchi phenological period domain knowledge base. The constraint relationship includes that the flowering period and mature fruit do not coexist on the same branch, the phenological sequence of flowering rate and fruit maturity is misaligned, and the negative correlation between the proportion of autumn shoots and fruit yield.

[0030] When contradictions are detected between the structured detection results of different dedicated target detection models, the scene judgment results are re-compared with the detection confidence distribution of the corresponding dedicated target detection model. If it is confirmed that a certain dedicated target detection model has made a false detection, its confidence label is adjusted to low confidence. If there are visual differences in branches on the same plant or local phenological differences, it is determined to be a special scene, the structured detection results are retained and anomalies are marked.

[0031] If the constraints between the detection indicators conform to the corresponding dedicated target detection model, the detection confidence is improved, and reliable detection information after cross-validation is output.

[0032] Preferably, the decision generation stage includes: Using reliable detection information obtained through cross-validation as input, and combining it with a knowledge base on litchi phenology, a structured decision report is generated.

[0033] The knowledge base for lychee phenology includes standard indicator ranges, anomaly judgment rules, agricultural operation specifications and precautions for each phenological stage.

[0034] The structured decision report includes an assessment of the current state of the orchard, detailed evaluations of various agronomic indicators, abnormal warning information, and graded agricultural operation recommendations. Each agricultural operation recommendation is accompanied by the corresponding source reasoning.

[0035] As can be seen from the above technical solutions, compared with the prior art, the beneficial effects of the present invention include at least the following: ① End-to-end elimination of information loss in manual processes, achieving full-process automation. This invention, through parallel operation of multiple models and posterior scene arbitration of multimodal large models, can automatically select reliable detection results without manual prediction of image scenes. This eliminates the information loss and efficiency bottlenecks in the manual selection of models and manual summarization and interpretation of results, realizing fully automated end-to-end inference from raw UAV remote sensing images to decision recommendations.

[0036] ② Multi-indicator collaborative coverage and cross-validation improve decision reliability. Four dedicated YOLO models can simultaneously output four types of agronomic indicators: flowering, fruit, shoots, and branches. Cross-validation is performed by combining the inherent constraints of litchi phenology. By verifying information from multiple sources, the impact of false positives from a single model is reduced, and the bias of a single data source is prevented from being transmitted to the final decision.

[0037] ③ Efficient parameter fine-tuning reduces adaptation costs and supports rapid iteration. Low-rank adaptation technology is used to fine-tune the multimodal large model for domain-specific optimization. Only a very small number of parameters need to be trained to improve the model's adaptability to structured detection text and thought chain reasoning formats, resulting in low training costs. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 This is a diagram illustrating the overall system architecture and end-to-end data flow provided in this embodiment of the invention. Figure 2 This is a schematic diagram of multiple YOLO parallel detection layers provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structured text fusion layer output template provided in an embodiment of the present invention; Figure 4 This is a flowchart of the thought chain reasoning process of the multimodal large language model provided in this embodiment of the invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] like Figure 1As shown, this invention discloses a decision analysis system for remote sensing images of litchi orchards based on multi-model fusion, comprising: Multi-target parallel detection module: The same frame of original UAV remote sensing image of lychee orchard is input into multiple pre-trained dedicated target detection models in parallel. Each dedicated target detection model performs inference operations independently and outputs structured detection results in heterogeneous formats.

[0042] The multi-target parallel detection module contains four pre-trained dedicated target detection models based on the YOLO architecture: a flowering rate detection model, a fruit maturity detection model, a shoot length measurement model, and an autumn / winter shoot identification model.

[0043] like Figure 2 As shown, each dedicated target detection model outputs structured detection results corresponding to its detection task. These results include the total number of detected targets, confidence scores, and status markers, specifically: Flowering rate detection model: used to detect the number of flower buds, the number of open flowers, the calculated flowering rate, the confidence data of each detected target, and the spatial location information in the original UAV remote sensing image.

[0044] Fruit maturity detection model: used to detect the number of green fruits, ripening fruits and red fruits in the original UAV remote sensing images, the calculated maturity rate, the confidence data of each detected target and the spatial location information.

[0045] Shoot length measurement model: used to detect the number of shoots, the pixel length value of a single shoot, length statistics, confidence data and spatial location information of each detected target in raw UAV remote sensing images.

[0046] Autumn and winter shoot identification model: used to detect the number of autumn shoots, the number of winter shoots, the proportion of autumn and winter shoots, the confidence data and spatial location information of each detected target in the original UAV remote sensing image.

[0047] The dedicated target detection model that fails to detect the corresponding target outputs a zero detection flag and invalid response information.

[0048] The four models were independently trained and deployed for four detection tasks: litchi flower organ classification and counting, fruit maturity grading, shoot length measurement, and branch type classification.

[0049] Structured text fusion module: Receives heterogeneous structured detection results from all dedicated target detection models, converts the heterogeneous structured detection results into structured text of a unified format, generates structured text of a unified format that corresponds one-to-one with the original UAV remote sensing image, and establishes a correlation mapping with the original UAV remote sensing image.

[0050] The structured text fusion module also includes: Construct a summary table of detection results, convert the records into structured text in a unified format, and mark the dedicated target detection models with a status of pending verification for those with a total number of detected targets lower than a preset minimum threshold and confidence data lower than a preset confidence threshold.

[0051] In the summary table of detection results, an independent information block is generated for each dedicated target detection model. The information block records the structured detection results of the corresponding dedicated target detection model.

[0052] The confidence data of all detected targets are retained. For dedicated target detection models that detect zero targets, the corresponding information block in the detection result summary table is marked with a status flag indicating that no valid targets were detected.

[0053] Multimodal Large Language Model Decision Engine Module: The original UAV remote sensing images and structured text are input into the multimodal large language model after low-rank adaptation processing. The multimodal large language model performs multi-stage thought chain reasoning and, combined with the litchi phenological period domain knowledge base, outputs a structured decision report for the litchi orchard.

[0054] The multi-stage reasoning in the decision engine module of the multimodal large language model includes four reasoning stages: scene recognition, confidence assessment, cross-validation, and decision generation.

[0055] The scene recognition stage includes: Using raw UAV remote sensing images as visual input, without relying on structured detection results, the multimodal large language model outputs scene judgment results, which include the corresponding litchi phenological scene type, target density, and image quality.

[0056] The confidence assessment phase includes: Based on the scene judgment results, the results are compared one by one with the structured detection results, and a corresponding confidence label is output for each model.

[0057] Among them, the detection target of the dedicated target detection model matches the scene type, and the structured detection results are consistent with visual observation. The confidence label of the structured detection results of the corresponding dedicated target detection model is marked as high confidence.

[0058] The target detected by the dedicated target detection model is clearly inconsistent with the scene type, so the confidence label of the structured detection result of the corresponding dedicated target detection model is marked as low confidence.

[0059] If the dedicated target detection model detects zero targets and there are no targets corresponding to the model in the scene judgment results, the confidence label of the structured detection results of the corresponding dedicated target detection model will be marked as "the scene confirms that there are no such targets".

[0060] If the number of detected targets and the number of detected targets by the dedicated target detection model are lower than the preset minimum threshold and the confidence data is lower than the preset confidence threshold, then the confidence label of the structured detection result of the corresponding dedicated target detection model will be marked as noise output.

[0061] Model outputs marked as pending verification need to undergo scene matching verification. Statistical information and state labels marked as high confidence and indicating that there is no such target in the scene will proceed to the next inference stage.

[0062] The cross-validation phase includes: The high-confidence detection results and the results confirming that there is no such target in the scenario are used as inputs. The mutual verification is carried out by using the constraint relationship between the detection indicators in the pre-set litchi phenological period domain knowledge base. The constraint relationship includes that the flowering period and mature fruit do not coexist on the same branch, the phenological sequence of flowering rate and fruit maturity is misaligned, and the negative correlation between the proportion of autumn shoots and fruit yield.

[0063] When contradictions are detected between the structured detection results of different dedicated target detection models, the scene judgment results are re-compared with the detection confidence distribution of the corresponding dedicated target detection model. If it is confirmed that a certain dedicated target detection model has made a false detection, its confidence label is adjusted to low confidence. If there are visual differences in branches on the same plant or local phenological differences, it is determined to be a special scene, the structured detection results are retained and anomalies are marked.

[0064] If the constraints between the detection indicators conform to the corresponding dedicated target detection model, the detection confidence is improved, and reliable detection information after cross-validation is output.

[0065] The decision generation phase includes: Using reliable detection information obtained through cross-validation as input, and combining it with a knowledge base on litchi phenology, a structured decision report is generated.

[0066] The knowledge base for lychee phenology includes standard indicator ranges, anomaly judgment rules, agricultural operation specifications and precautions for each phenological stage.

[0067] The structured decision report includes an assessment of the current state of the orchard, detailed evaluations of various agronomic indicators, abnormal warning information, and graded agricultural operation recommendations. Each agricultural operation recommendation is accompanied by the corresponding source reasoning.

[0068] The technical solution of the present invention will be described in detail through specific embodiments.

[0069] Example 1 This invention proposes a decision analysis system for remote sensing images of litchi orchards based on multi-model fusion, which consists of three modules: a multi-target parallel detection module, a structured text fusion module, and a multimodal large language model decision engine module.

[0070] The multi-target parallel detection module is responsible for performing multi-task fine-grained target detection on the input UAV remote sensing images. This module includes four dedicated YOLO models: a flowering rate detection model, a fruit maturity detection model, a shoot length measurement model, and an autumn / winter shoot identification model. The four models perform parallel inference on the same input image, are independent of each other, and each outputs a structured detection result containing the detected targets, their quantity, confidence level, and spatial distribution.

[0071] The structured text fusion module is responsible for unifying the heterogeneous outputs of the four YOLO models into a single structured text representation. This module does not perform any filtering or discarding operations; all YOLO outputs are retained. However, confidence statistics (mean, standard deviation, total number of detected targets) are added to the output of each model. Model results with low detection rates or low confidence are marked with a validation tag for evaluation by the Multimodal Large Language Model (MLLM).

[0072] The multimodal large language model decision engine module receives two inputs simultaneously: raw UAV remote sensing images (visual modality) and the output of the structured text fusion module (text modality). The engine reasones according to a pre-defined four-step thought process, sequentially completing scene recognition and visual verification, confidence assessment of each YOLO model output, cross-validation of high-confidence detection results, and finally, generation of a structured decision report. The engine internally embeds a domain knowledge base on lychee phenological stages (typical image features of each phenological stage, normal ranges of key indicators, and phenological correlation constraints between indicators), providing domain knowledge support for reasoning.

[0073] The data flow between the three modules is as follows: UAV remote sensing images are simultaneously input into the four models of the multi-target parallel detection module, each dedicated target detection model outputs structured detection results, the structured text fusion module summarizes them into a unified template, and the multimodal large language model decision engine module receives the original UAV remote sensing images and structured text, performs a four-step thinking chain reasoning, and outputs a structured decision report.

[0074] The multi-object parallel detection module comprises four dedicated YOLO detection models that perform parallel and independent inference on the same input image. All four models are trained on a labeled dataset independently constructed in this invention and are deployed in parallel within the system, simultaneously receiving input images and outputting detection results.

[0075] The detection targets, output definitions, and data construction methods for each specialized target detection model are as follows: Flowering rate detection model: Trained on a dataset of litchi inflorescence UAV images collected using this invention, the model is labeled into two categories: flower buds and open flowers. Flower buds are defined as yellowish-green, closed flower organs without exposed stamens; open flowers are defined as flower organs with visible white stamens. The model output includes: the number of detected flower buds, the number of open flowers, the spatial distribution of both (center coordinates and size of each detection box), and the confidence score for each detection box.

[0076] The flowering rate is automatically calculated based on the output statistics: number of open flowers / (number of buds + number of open flowers). If no flower type is detected, the output flower type detection count = 0, and the confidence score = N / A.

[0077] Fruit maturity detection model: Trained based on a dataset of litchi fruit images collected by UAVs using this invention, the model is labeled with three maturity levels: green fruit, ripening fruit, and red fruit. Green fruit has a greenish-blue peel and is immature; ripening fruit has a yellow-red peel and is in the ripening stage; red fruit has a bright red, glossy peel and is ripe. The model output includes: the number, proportion, and spatial distribution of fruits at each level, as well as the confidence score for each detection box. The maturity rate is automatically calculated as: number of red fruits / total number of fruits. If no fruit is detected, the output is: fruit detection count = 0, confidence score = N / A.

[0078] The shoot length measurement model is trained based on a dataset of litchi shoot images collected by UAVs using this invention. The annotation type is shoot detection boxes and their pixel length regression values ​​(Euclidean distance from the top to the base). The YOLO model outputs the length value of each detected target directly through an additional regression branch while outputting the detection boxes. The model output includes: the number of detected shoots, the pixel length of each shoot, the confidence score for each detection, and length statistics (mean, maximum, and minimum). The growth stage of the shoots is determined based on the length distribution, indicating whether it is an active growth period or a dormant period.

[0079] Autumn and Winter Shoot Recognition Model: Trained based on a dataset of UAV images of litchi branches collected using this invention, the model is categorized into two classes: autumn shoots and winter shoots. Autumn shoots are characterized by thicker branches, darker green color, and higher lignification, and are considered fruiting branches for the following year. Winter shoots are characterized by thinner branches, lighter color, and lower lignification, and should be controlled. The model output includes: the number of autumn shoots, the number of winter shoots, their ratio, spatial distribution, and the confidence score for each detection box. If no shoot category is detected, the output is: Shoot Category Detection Count = 0, Confidence Score = N / A.

[0080] Each of the four models independently outputs the aforementioned structured detection data, which is then passed to the downstream structured text fusion module via a standardized JSON format.

[0081] The structured text fusion module receives the outputs of the four models in the multi-object parallel detection module and unifies them into a standardized structured text template that can be directly consumed by the multimodal large language model. The fusion layer does not perform any filtering, screening, or thresholding; all original outputs are retained and presented to the downstream multimodal large language model, which then determines the confidence level of the outputs of each dedicated object detection model based on image visual information.

[0082] like Figure 3 As shown, the design of structured text templates follows these principles: The output of each YOLO model occupies an independent information block, identified by the model name.

[0083] Each information block contains: the total number of detected targets (including zero detection cases), the count and percentage of each target category, confidence statistics (mean, standard deviation), and a description of the approximate spatial distribution characteristics of the targets (concentrated / scattered / marginal, etc.).

[0084] A model information block with a total detection count of zero indicates that the model did not detect any valid targets in the current image.

[0085] The original confidence list is retained, which facilitates fine-grained evaluation of multimodal large language models at the individual target granularity.

[0086] An additional summary section is provided, listing a brief overview of the detection results for the four models (model name, total number of detections, confidence score data, and whether the model is labeled pending verification), which helps the multimodal large language model quickly grasp the overall picture.

[0087] The structured text, along with the original UAV remote sensing image, is used as input to the multimodal large language model decision engine module.

[0088] The multimodal large language model decision engine module is the core reasoning module of the system. It receives two types of information: raw UAV remote sensing images (visual input) and the output of the structured text fusion module (text input), and performs step-by-step reasoning according to a preset four-step thinking chain.

[0089] Because the input of the multimodal large language model in this invention includes highly structured multi-source detection text and a strictly regulated four-step thought chain reasoning format, this differs significantly from the distribution of natural image-natural language dialogue data in the pre-training stage of the general visual language model. If the general visual language model is used directly, it is difficult to guarantee the model's semantic understanding of each field in the structured template (such as the meaning of confidence statistics and the pragmatic function of the tag to be verified) and its adherence to the dependencies between steps.

[0090] To address this, this invention employs Low-Rank Adaptation (LoRA) to efficiently fine-tune the parameters of the selected visual language model, adapting it to the input / output format and inference task of this system. Low-Rank Adaptation fine-tuning only introduces a low-rank decomposition matrix next to the original model's weight matrix for incremental updates, while the original pre-trained weights remain frozen. The number of training parameters is less than one-thousandth of that of full fine-tuning, and can be completed on a single consumer-grade GPU.

[0091] Low-rank adaptation fine-tuning data construction: The fine-tuning data consists of triplets of (image, structured detection text, target output). The images are UAV remote sensing images of lychee orchards; the structured detection text is automatically generated by the multi-target parallel detection module and the structured text fusion module; the target output is a manually annotated complete reasoning process and decision report conforming to the four-step thinking chain format. Annotators wrote standard answers step by step according to the requirements of steps one to four to ensure that the reasoning logic of each step is correct, the format is standardized, and the scene judgment is accurate. The dataset covers four typical scenarios: flowering period, fruiting period, shoot growth period, and mixed period. Each scenario contains samples with different combinations of YOLO model confidence (all matches, some noise, all zero detections, etc.) to ensure that the fine-tuned model is robust to various posterior selection scenarios.

[0092] Visual input: Raw RGB remote sensing images from the UAV, retaining the original resolution and color information without any preprocessing; Text input: The complete structured detection report output by the structured text fusion module, which includes the structured detection results and confidence information of all four YOLO models.

[0093] like Figure 4 As shown, the four-step reasoning process is as follows: Step 1: Scene Recognition (Visual Verification). The multimodal large language model directly observes the original remote sensing image and independently judges the overall scene type of the image. Judgment dimensions include: which phenological stage of litchi (flowering / fruiting / shooting / mixed stage) the main visual content of the image belongs to, the density of the target object, lighting conditions, image quality, etc.

[0094] This judgment does not rely on any YOLO output; it is entirely based on the visual understanding capabilities of the multimodal large language model itself, providing an independent visual benchmark for subsequent YOLO output confidence evaluation. For example, if the multimodal large language model judges from an image that the main subject is a lychee inflorescence in full bloom, with dense inflorescences and no obvious fruit, then this judgment will be used as a reference anchor point for evaluating the confidence of each YOLO model output in subsequent steps.

[0095] Step Two: Confidence Assessment. Compare the visual scene judgments formed in Step One with the outputs of the four YOLO models in the structured text fusion module one by one. The comparison rules are as follows: If the target detected by a YOLO model matches the scene type determined by the visual judgment of a multimodal large language model, and the output of the model is on the same order of magnitude as the visual observation, then the detection result output of the model is labeled as high confidence.

[0096] If the detection target of a YOLO model is significantly different from the scene type judged by the multimodal large language model (for example, the multimodal large language model judges it as flowering season but the model outputs a large number of fruit detections), then the detection result output of the model is labeled as low confidence or scene mismatch.

[0097] If a YOLO model detects zero targets and the multimodal large language model does not observe such targets in its visual judgment, then the model's detection output is labeled as "scene confirmed to be free of such targets" (zero detection is a correct result, not a model failure).

[0098] If a YOLO model has extremely low detection rate, low mean confidence level but large standard deviation, then the model's detection output is labeled as noise output.

[0099] The output of the confidence assessment step is the confidence labels and rationale for the four YOLO models. Only model results labeled as high confidence or where the scenario confirms no such target are allowed to proceed to the next step of cross-validation.

[0100] Step 3: Cross-validation. Based on the high-confidence YOLO outputs selected in Step 2, cross-correlation inference is performed between indicators. Cross-validation utilizes the inherent constraints in litchi phenology: for example, flowering and fruiting periods do not overlap in time (a large number of blooming flowers and mature fruits will not exist on the same branch at the same time), fruits should be in the young fruit stage rather than the mature stage during periods of high flowering rate, and the color and texture of leaves during the active growth period of shoots and buds have their own characteristics.

[0101] Typical patterns of phenological constraints for cross-validation checks include: A high flowering rate combined with a high confidence level in fruit maturity detection presents a contradiction, requiring a re-examination of the confidence levels for both. This could indicate a situation where different branches are from the same plant, or a model misjudgment.

[0102] A high proportion of autumn shoots and a low detection rate of ripe fruit indicate a period of vigorous vegetative growth, which may be a response to pruning or a sign of a young tree.

[0103] All indicators are consistent with the expected phenological period, indicating high data consistency and improved decision-making confidence.

[0104] Step 4: Decision Generation. Based on the reliable multi-source information obtained from the cross-validation in Step 3, LoRA parameters are efficiently fine-tuned. Combined with the litchi phenological stage domain knowledge base and agricultural management rules embedded in the system, a structured decision report is generated.

[0105] Structured decision reports include the following fixed sections: Orchard current status assessment results: A one-sentence summary of the current phenological stage and overall condition of the orchard; Detailed evaluation of each indicator: List the reliable indicators that have been cross-validated and their evaluations (flowering rate, fruit ripening progress, shoot growth status, etc.).

[0106] Anomaly alert: List any findings that deviate from the normal range or expected phenological period, and indicate their priority.

[0107] Agricultural operation suggestions: Based on the current status and abnormal findings, specific agricultural operation suggestions (such as flower thinning, topdressing, irrigation, pest and disease control, etc.) are given in priority order, and each suggestion is accompanied by reasoning.

[0108] Four dedicated YOLO models infer in parallel on the same image, and a single analysis can simultaneously cover four agronomic indicators: flowering rate, fruit maturity, shoot growth status, and autumn and winter shoot distribution.

[0109] More importantly, the cross-validation step in the thinking chain reasoning utilizes the inherent constraints between indicators in the knowledge of litchi phenology to mutually validate the outputs of different YOLO models. The mutual corroboration of high-confidence multi-source information enhances the reliability of the decision, while contradictory signals trigger a re-examination and retrospective analysis, avoiding the direct adoption of misjudgments from a single model.

[0110] The training data in the pre-training stage of general visual language models mainly consists of natural images and natural language dialogues, and lacks targeted adaptation to highly structured multi-source detection text and strictly standardized four-step thinking chain reasoning format.

[0111] This invention employs low-rank adaptation technology for efficient parameter fine-tuning of a visual language model. While freezing the original pre-training weights, only the low-rank decomposition matrices injected into the attention modules of each layer are trained. The training data consists of structured detection text automatically generated by the system paired with manually annotated four-step thought chain standard reasoning processes, covering four scenarios: flowering, fruiting, shoot growth, and mixed growth stages, as well as different confidence combinations of YOLO model outputs. The fine-tuned multimodal large language model shows improvements in three dimensions: accuracy of structured field semantic parsing, thought chain step adherence rate, and scenario-confidence consistency. Furthermore, the fine-tuning introduces only a very small number of trainable parameters (0.1%-1% of the original parameters), resulting in low training costs and facilitating incremental updates when adding new litchi varieties or phenological data.

[0112] Example 2 All four dedicated YOLO models are trained on a labeled dataset independently constructed in this invention. The YOLOv8 architecture is used as the base detector, and each model is independently trained and deployed for its respective detection target. The shoot length measurement model adds a length regression branch to the standard detector head, directly outputting pixel length values.

[0113] Training data construction: The flowering rate detection dataset uses images of litchi inflorescences collected by drones under different lighting conditions and shooting angles, labeled as buds and fully opened flowers. The fruit maturity dataset uses images of litchi fruits, labeled as green fruit, ripening fruit, and red fruit. The shoot length measurement dataset labels the shoot detection boxes and their pixel length regression values. The autumn and winter shoot recognition dataset labels autumn shoots and winter shoots. To ensure the robustness of the model in actual deployment, data augmentation strategies are used during training, including random scaling, rotation, color jitter, and brightness and contrast adjustment.

[0114] Model Deployment: After being exported in a unified format and optimized for inference, the four models are deployed on the same edge computing device or server. All four model instances are loaded upon system startup. When a UAV remote sensing image is received as input, the four models initiate inference in parallel threads, and their respective inference results are aggregated into the structured text fusion module via a message queue. The latency of parallel inference is determined by the model with the longest inference time among the four, and the total latency is related to the parallel computing capabilities of the deployed hardware platform.

[0115] Output format: The raw output of each model (predicted bounding box coordinates, class labels, confidence data) is converted into a unified JSON format by a post-processing script.

[0116] Post-processing includes: non-maximum suppression (NMS) to remove duplicate detection boxes, calculating statistics (number, proportion, confidence mean and standard deviation) according to predefined rules, and converting the detection box coordinates to normalized coordinates relative to the image size for downstream processing.

[0117] The structured text fusion module is implemented as a Python function that takes a list of four dictionaries (each dictionary corresponds to the output of a YOLO model) and returns a formatted text string.

[0118] Template construction logic: First, a summary table of detection results is built. The outputs of the four models are traversed, and the name, total number of detections, and confidence score data of each model are extracted. For models with zero detections, no confidence score data is calculated, and N / A is entered. Then, information blocks for each dedicated target detection model are constructed: For models with detections, detailed statistics (number, proportion, mean and standard deviation of confidence scores for each subcategory) and spatial distribution descriptions are written; for models with zero detections, a preset prompt text indicating that no valid targets were detected is written. The method for generating the spatial distribution description is as follows: the center coordinates of the detection box are divided into grids according to the image region (such as 3x3 or 5x5), the target density in each grid is calculated, and it is converted into natural language descriptions such as concentrated on the image, evenly distributed in the center, and sparse at the edges.

[0119] Confidence Preservation Strategy: For each block of model information with detected targets, the confidence range (minimum to maximum value) is listed. Simultaneously, the confidence scores are statistically analyzed by interval (e.g., 0.5-0.7, 0.7-0.8, 0.8-0.9, 0.9-1.0), and the distribution is presented in text form. This enables the multimodal large language model to recognize situations where a specific object detection model detects a large number of targets but has a large number of low confidence scores, thus classifying it as noise output.

[0120] Validation-Pending Marking Rules: Models with a detection rate below a preset minimum threshold (e.g., 5 targets) and a confidence score below 0.6 will be marked as "Validation Pending" in the status column of the detection results summary table to draw the attention of the multimodal large language model. This marking does not indicate that the system has made a judgment, but rather prompts the multimodal large language model to pay special attention to this model in the confidence assessment step.

[0121] Multimodal Large Language Model Decision Engine Module Model Selection: For the multimodal large language model, Qwen2-VL-7B or InternVL2-8B are selected. Unlike pure text large language models, these models have a visual encoder-language model bridging architecture, enabling them to accept multimodal inputs of both images and text.

[0122] Efficient Fine-Tuning of Parameters Using Low-Rank Adaptation: Directly using a general visual language model to process the structured text detection and four-step thought chain reasoning tasks of this invention suffers from insufficient adaptation, as the general model has never encountered similar input-output formats during the pre-training phase. To address this issue, low-rank adaptation is employed for efficient parameter fine-tuning of the visual language model.

[0123] The low-rank adaptation configuration is as follows: In the self-attention module of each Transformer layer in the language model part of the visual language model, low-rank decomposition matrices A and B are injected into the query (Q) and value (V) projection matrices, with the rank r set to 16 or 32 and the scaling factor α set to 16. During training, all weights of the original model are frozen, and only the low-rank matrix parameters introduced by the low-rank adaptation are updated. The number of trainable parameters is approximately 0.1%-0.5% of the total parameters.

[0124] Training is performed using the AdamW optimizer with a learning rate of 1e-4 for 3-5 epochs. In one implementation, low-rank adaptation fine-tuning can be done on a single consumer-grade RTX 4090 (24GB VRAM) GPU.

[0125] Fine-tuning the dataset construction: The dataset consists of 200-500 triplets (images, structured detection text, and standard inference output). The images cover four typical scenarios of litchi flowering, fruiting, shoot growth, and mixed growth stages. The structured detection text is automatically generated from the images by the system's multi-target parallel detection module and structured text fusion module. The standard inference output is manually written by agricultural experts according to a four-step thought process format, ensuring that each step is logically correct, uses standardized terminology, and that the decision recommendations conform to agronomic best practices.

[0126] To cover various scenarios in actual deployments, the dataset was designed with four sample distributions: (a) regular samples where the YOLO output perfectly matches the scenario (approximately 50%); (b) partially applicable samples where 1-2 YOLO models produce zero detection or low confidence (approximately 30%); (c) extreme samples where most YOLO models are not applicable and only 1 model is effective (approximately 10%); and (d) adversarial samples where the YOLO output is contradictory and cross-validation is required for judgment (approximately 10%).

[0127] The four-step thinking chain design: After fine-tuning, the system prompts for the multimodal large language model are divided into static and dynamic parts. The static part contains domain knowledge of lychee phenology and format instructions for four-step reasoning, which are loaded once when the system starts. The dynamic part is filled in by the output of the structured text fusion module.

[0128] Post-output processing: The text returned by the multimodal large language model is segmented into four parts according to step N - title using regular expressions, and then the decision report section from step four is extracted for use by downstream systems or users. If the format of the decision report in step four does not meet expectations (missing necessary fields), a second call for format correction is triggered.

[0129] The system adopts an edge-cloud hybrid deployment architecture. The multi-target parallel detection module and the structured text fusion module are deployed on edge computing devices, directly connected to UAV ground stations or image acquisition devices, to achieve low-latency local inference. The four YOLO models are optimized with TensorRT or equivalent inference frameworks and run at FP16 or INT8 precision, with the total latency of parallel inference controlled within a few seconds.

[0130] The multimodal large language model decision engine module is deployed on a cloud server or a local GPU server within the campus. It receives raw UAV remote sensing images and structured text uploaded by edge devices via an API interface. The deployed multimodal large language model is a model fine-tuned with low-rank adaptation. The low-rank adaptation weights are stored separately from the model weights in independent adapter files and are dynamically merged during loading. This deployment method balances the real-time requirements of YOLO inference (low latency at the edge) and the computational resource requirements of multimodal large language model inference (high-performance GPUs on the server). The edge devices communicate with the multimodal large language model server via 4G / 5G or WiFi networks. When new litchi variety data or phenological period samples are added, the low-rank adaptation weights can be retrained only for the new data, without the need for a full retraining, achieving lightweight incremental updates to the model.

[0131] The system provides a RESTful API interface to receive drone remote sensing images and optional user query text (such as checking orchard flowering status or comprehensive assessment), and returns a structured JSON-formatted decision report, which is easy to integrate into existing smart agriculture management platforms.

[0132] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0133] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A decision analysis system for remote sensing images of litchi orchards based on multi-model fusion, characterized in that, include: Multi-target parallel detection module: The same frame of original UAV remote sensing image of lychee orchard is input into multiple pre-trained dedicated target detection models in parallel. Each dedicated target detection model performs inference operations independently and outputs structured detection results in heterogeneous formats. Structured text fusion module: Receives heterogeneous structured detection results output by all dedicated target detection models, converts the heterogeneous structured detection results into structured text in a unified format, generates structured text in a unified format that corresponds one-to-one with the original UAV remote sensing image, and establishes an association mapping with the original UAV remote sensing image; Multimodal Large Language Model Decision Engine Module: The original UAV remote sensing images and structured text are input into the multimodal large language model after low-rank adaptation processing. The multimodal large language model performs multi-stage thought chain reasoning and, combined with the litchi phenological period domain knowledge base, outputs a structured decision report for the litchi orchard.

2. The decision analysis system for remote sensing images of litchi orchards based on multi-model fusion according to claim 1, characterized in that, The multi-target parallel detection module contains four pre-trained dedicated target detection models based on the YOLO architecture: a flowering rate detection model, a fruit maturity detection model, a shoot length measurement model, and an autumn / winter shoot identification model. The four models were independently trained and deployed for four detection tasks: litchi flower organ classification and counting, fruit maturity grading, shoot length measurement, and branch type classification.

3. The decision analysis system for remote sensing images of litchi orchards based on multi-model fusion according to claim 2, characterized in that, Each dedicated target detection model outputs structured detection results corresponding to its detection task. These results include the total number of detected targets, confidence scores, and status markers, specifically: Flowering rate detection model: used to detect the number of flower buds, the number of open flowers, the calculated flowering rate, the confidence data and spatial location information of each detected target in the original UAV remote sensing image; Fruit maturity detection model: used to detect the number of green fruits, ripening fruits and red fruits in the original UAV remote sensing images, the calculated maturity rate, the confidence data of each detected target and the spatial location information; Shoot length measurement model: used to detect the number of shoots, the pixel length value of a single shoot, length statistics, confidence data and spatial location information of each detected target in raw UAV remote sensing images; Autumn and winter shoot identification model: used to detect the number of autumn shoots, the number of winter shoots, the proportion of autumn and winter shoots, the confidence data and spatial location information of each detected target in the original UAV remote sensing image; The dedicated target detection model that fails to detect the corresponding target outputs a zero detection flag and invalid response information.

4. The decision analysis system for remote sensing images of litchi orchards based on multi-model fusion according to claim 3, characterized in that, The structured text fusion module also includes: Construct a summary table of detection results, convert the records into structured text in a unified format, and mark the special target detection model with a status to be verified if the total number of detected targets is lower than the preset minimum threshold and the confidence data is lower than the preset confidence threshold. In the summary table of detection results, an independent information block is generated for each dedicated target detection model. The information block records the structured detection results of the corresponding dedicated target detection model. The confidence data of all detected targets are retained. For dedicated target detection models that detect zero targets, the corresponding information block in the detection result summary table is marked with a status flag indicating that no valid targets were detected.

5. The decision analysis system for remote sensing images of litchi orchards based on multi-model fusion according to claim 1, characterized in that, The multi-stage reasoning in the decision engine module of the multimodal large language model includes four reasoning stages: scene recognition, confidence assessment, cross-validation, and decision generation.

6. The decision analysis system for remote sensing images of litchi orchards based on multi-model fusion according to claim 5, characterized in that, The scene recognition stage includes: Using raw UAV remote sensing images as visual input, without relying on structured detection results, the multimodal large language model outputs scene judgment results, which include the corresponding litchi phenological scene type, target density, and image quality.

7. The decision analysis system for remote sensing images of litchi orchards based on multi-model fusion according to claim 6, characterized in that, The confidence assessment phase includes: Based on the scene judgment results, the results are compared one by one with the structured detection results, and a corresponding confidence label is output for each model. Among them, the detection target of the dedicated target detection model matches the scene type, and the structured detection results are consistent with visual observation. The confidence label of the structured detection results of the corresponding dedicated target detection model is marked as high confidence. The target detected by the dedicated target detection model is clearly inconsistent with the scene type, so the confidence label of the structured detection result of the corresponding dedicated target detection model is marked as low confidence. If the dedicated target detection model detects zero targets and there are no targets corresponding to the model in the scene judgment results, the confidence label of the structured detection results of the corresponding dedicated target detection model will be marked as "the scene confirms that there are no such targets". If the number of detected targets and the number of detected targets of the dedicated target detection model are lower than the preset minimum threshold and the confidence data is lower than the preset confidence threshold, then the confidence label of the structured detection result of the corresponding dedicated target detection model will be marked as noise output. Model outputs marked as pending verification need to undergo scene matching verification. Statistical information and state labels marked as high confidence and indicating that there is no such target in the scene will proceed to the next inference stage.

8. The decision analysis system for remote sensing images of litchi orchards based on multi-model fusion according to claim 7, characterized in that, The cross-validation phase includes: The high-confidence detection results and the results confirming that there is no such target in the scenario are used as input. The mutual verification is carried out by using the constraint relationship between the detection indicators in the pre-set litchi phenological period domain knowledge base. The constraint relationship includes that flowering period and mature fruit do not coexist on the same branch, the phenological sequence of flowering rate and fruit maturity is misaligned, and the negative correlation between autumn shoot ratio and fruit yield. When contradictions are detected between the structured detection results of different dedicated target detection models, the scene judgment results are re-compared with the detection confidence distribution of the corresponding dedicated target detection model. If it is confirmed that a certain dedicated target detection model has made a false detection, its confidence label is adjusted to low confidence. If there are visual differences such as different branches on the same plant or local phenological differences, it is determined to be a special scene, the structured detection results are retained and anomalies are marked. If the constraints between the detection indicators conform to the corresponding dedicated target detection model, the detection confidence is improved, and reliable detection information after cross-validation is output.

9. The decision analysis system for remote sensing images of litchi orchards based on multi-model fusion according to claim 8, characterized in that, The decision generation phase includes: Using reliable detection information obtained through cross-validation as input, and combining it with a knowledge base on litchi phenology, a structured decision report is generated. The knowledge base for litchi phenology includes standard indicator ranges, anomaly judgment rules, agricultural operation specifications and precautions for each phenological stage. The structured decision report includes an assessment of the current state of the orchard, detailed evaluations of various agronomic indicators, abnormal warning information, and graded agricultural operation recommendations. Each agricultural operation recommendation is accompanied by the corresponding source reasoning.

Images