A visual token sparsification method based on growth stage perception and a gardening planning device based on growth stage perception

Abstract

The application provides a growth stage-aware visual token sparsification method and a growth stage-aware gardening planning device, including the following steps: step S1: obtaining current growth stage information of crops; step S2: according to the current growth stage information; step S3: a visual encoder performs visual token aggregation and redundant token pruning on an input image according to the visual attention area and a token retention quantity threshold: generating a compact visual representation; step S4: inputting the compact visual representation into a visual-language-action large model to generate a gardening task planning result; step S5: according to task execution feedback data, online fine-tuning the mapping relationship in the growth stage-visual attention area mapping table. The technical scheme is suitable for long-period automatic management of corm plants in an open environment.

Description

Technical Field

[0001] This invention relates to the fields of agricultural robots and artificial intelligence technology, and in particular to a visual token sparsification method based on growth stage perception and a horticultural planning device based on growth stage perception. Background Technology

[0002] With the deepening application of artificial intelligence and robotics in agriculture, agricultural robot planning systems based on the Vision-Language-Motion (VLA) model have attracted widespread attention. The VLA model can directly generate robotic arm motion sequences based on natural language commands and visual observations, demonstrating powerful capabilities in general robot operation tasks.

[0003] However, VLA models face significant computational efficiency bottlenecks in practical agricultural deployments. VLA models typically require processing a large number of visual tokens—hundreds of visual tokens are generated after each frame of image is processed by the visual encoder, and these tokens, along with language tokens, are input into a large language model for cross-modal inference. The standard Transformer's self-attention mechanism has a computational complexity of O(n log n). 2 ), where n is the total number of tokens. When the number of visual tokens is large, inference latency and computational resource consumption increase dramatically, making it difficult to run in real time on edge computing devices for agricultural robots.

[0004] To reduce the computational overhead of VLA models, researchers have proposed various visual token sparsity methods. For example, ATP-LLaVA reduces the number of visual tokens by adaptively determining the token pruning ratio for each LLM layer; AdaptInfer adaptively prunes visual tokens during VLM inference using a text-guided dynamic pruning mechanism; and CogVLA achieves three-stage progressive sparsity through instruction-driven routing and sparsification. However, the sparsity strategies of these methods are all based on the principle of instruction content or instance adaptation, without considering the biological prior information specific to agricultural scenarios.

[0005] In agricultural scenarios, crop growth stages represent a crucial set of structured prior information. Different growth stages—germination, seedling, vegetative growth, flowering, fruiting, maturity, and harvest—have fundamentally different requirements for the area of ​​focus and detail needed for visual perception. For example, germination requires attention to seedling emergence on the soil surface, vegetative growth demands high-resolution leaf details for pest and disease detection, and maturity requires global plant morphology to assess bulb size. Existing visual token sparsification methods fail to utilize this biological prior to guide their sparsification strategies, resulting in the processing of numerous redundant visual tokens during growth stages where high resolution is not required, or the loss of critical information due to over-pruning during growth stages where detail is essential.

[0006] Existing patents, such as "A Method, System, Device, and Medium for Identifying the Growth Stage of Rice in the Field," focus on identifying which growth stage the crop is in; "An Image-Based Plant Growth Cycle Monitoring System and Method" focuses on fertilization management based on growth stages. None of these patents address using growth stage information to guide visual token sparsity in embodied intelligent robot (VLA) models.

[0007] Therefore, there is an urgent need for a method and apparatus that can incorporate biological prior information about the growth stage into the visual token sparsification process, so as to significantly reduce the computational overhead and inference latency of the VLA model while maintaining task accuracy. Summary of the Invention

[0008] In view of this, the purpose of the present invention is to provide a visual token sparsification method based on growth stage perception and a horticultural planning device based on growth stage perception, which is suitable for long-term automated management of bulbous plants in open-air environments.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: a visual token sparsification method based on growth stage perception, comprising the following steps: Step S1: Obtain the current growth stage information of the crop, which includes the crop type identifier and the category identifier of the current growth stage; Step S2: Based on the current growth stage information, determine the visual attention region corresponding to the current growth stage from the preset growth stage-visual attention region mapping table, and determine the corresponding token retention threshold. Step S3: The visual encoder performs visual token aggregation and redundant token pruning on the input image based on the visual attention region and the token retention threshold: retaining visual tokens within the visual attention region and pruning visual tokens outside the visual attention region according to their importance scores, so that the number of retained tokens does not exceed the token retention threshold, thereby generating a compact visual representation. Step S4: Input the compact visual representation into the visual-language-action large model for cross-modal reasoning to generate the gardening task planning result; Step S5: Based on the task execution feedback data, fine-tune the mapping relationship in the growth stage-visual attention area mapping table online.

[0010] In a preferred embodiment, the category identifier of the growth stage includes at least one of the following: germination period, seedling period, growth period, flowering period, fruiting period, maturity period, and harvesting period; the visual attention area corresponding to different growth stages includes at least one of the following differences: different spatial location of the attention area, different size of the attention area, and different number of attention areas.

[0011] In a preferred embodiment, the token retention threshold in step S3 is dynamically determined based on the growth stage: a first token retention threshold is set in the growth stage that requires high-resolution details, and a second token retention threshold is set in the growth stage that requires global morphological observation, wherein the first token retention threshold is greater than the second token retention threshold.

[0012] In a preferred embodiment, the importance score in step S3 is calculated based on at least one of the following factors: the attention weight of the visual token, the relevance score of the visual token to the current growth stage, and the overlap between the spatial location of the visual token and the visual attention area.

[0013] In a preferred embodiment, the visual attention area includes at least one of the following types: root area, stem area, leaf area, flower organ area, and fruit / bulb area; the combination of visual attention area types varies for different growth stages.

[0014] In a preferred embodiment, the online fine-tuning in step S5 includes: when the execution feedback data indicates that the task has failed or the accuracy is below a threshold, adjusting the visual attention area range or token retention threshold of the corresponding growth stage in the growth stage-visual attention area mapping table.

[0015] The present invention also provides a horticultural planning device based on growth stage perception, which implements the aforementioned visual token sparsification method based on growth stage perception, including: The growth stage acquisition module is used to acquire information about the current growth stage of the crop. A mapping storage module is used to store a growth stage-visual attention area mapping table, which defines the correspondence between different growth stages, visual attention areas, and token retention thresholds. The visual token sparsification module is used to determine the corresponding visual attention region and token retention threshold from the mapping table based on the current growth stage information, and to perform visual token aggregation and redundant token pruning on the input image to generate a compact visual representation. The Visual-Language-Motion Large Model (VLA) inference module is used to perform cross-modal inference on the compact visual representation input to generate gardening task planning results; The feedback fine-tuning module is used to fine-tune the mapping relationship in the mapping table online based on task execution feedback data.

[0016] In a preferred embodiment, it further includes: The cloud-based planning subsystem, deployed on a cloud server, includes the growth stage acquisition module, mapping storage module, visual token sparsification module, VLA inference module, and first 5G communication module. The vehicle-mounted execution subsystem, deployed on a wheeled robotic arm gardening vehicle, includes a second 5G communication module, a task parsing module, a deep reinforcement learning control module, a chassis drive module, a robotic arm drive module, a multi-sensor perception module, and a feedback acquisition module. The cloud-based planning subsystem and the vehicle-side execution subsystem communicate bidirectionally via a 5G network, and the execution feedback data collected by the feedback acquisition module is transmitted to the feedback fine-tuning module via the 5G network.

[0017] In a preferred embodiment, the growth stage acquisition module acquires the current growth stage information through at least one of the following methods: automatic recognition based on an image classification model, time inference based on a growth calendar, or manual input by the user.

[0018] Compared with the prior art, the present invention has the following beneficial effects: First, biological priors are incorporated into visual token sparsification. Unlike instruction-driven sparsification methods such as CogVLA, this invention utilizes structured prior information unique to the agricultural scenario of growth stages to drive sparsification strategies. This makes the retention and pruning of visual tokens more in line with the actual visual needs of crops, achieving biologically rational sparsification that "focuses on areas that should be viewed and reduces the viewing of areas that should not be viewed."

[0019] Second, an adaptive differentiated sparsity strategy for the growth stage. Different visual attention areas and token retention thresholds are used for different growth stages. Sufficient tokens are retained to ensure accuracy in stages where details are needed, while significant pruning is performed in stages where details are not needed to reduce computational overhead, achieving an on-demand balance between accuracy and efficiency.

[0020] Third, a closed-loop optimization mechanism based on execution feedback. By feeding back the task execution results to the online fine-tuning of the growth stage-visual attention region mapping table, the sparsity strategy can continuously learn and optimize from actual tasks, which is different from the static sparsity strategy of existing methods.

[0021] Fourth, a fundamental difference from existing growth stage identification technologies. Existing patents, such as the "Method for Identifying the Growth Stage of Rice in the Field," only focus on "identifying which stage the crop is in," and the "Plant Growth Cycle Monitoring System" only focuses on "fertilization management based on the stage." This invention uses growth stage information as a control signal for visual token sparsity, solving a completely different problem: "how to make the VLA model process visual information more efficiently during inference." Attached Figure Description

[0022] Figure 1 This is a flowchart of the overall process of the method of the present invention.

[0023] Figure 2 The intention is to represent the visual attention area mapping for the growth stage.

[0024] Figure 3 This is a schematic diagram of the visual token sparsification process.

[0025] Figure 4 A diagram of the architecture for a horticultural planning device. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0029] refer to Figure 1-4 Example 1: A method for sparsifying visual tokens perceived during growth stages This embodiment takes the open-air horticultural management scenario of daffodil bulbs as an example to illustrate the specific implementation of the visual token sparsification method based on growth stage perception.

[0030] (I) Definition of growth stages and construction of mapping table The following is a pre-constructed visual attention area mapping table for the growth stages of daffodil bulbs:

[0031] The token retention threshold and the scope of interest in the mapping table are initial values ​​set based on prior knowledge of horticulture. Different crop varieties can be customized according to their morphological characteristics and growth patterns.

[0032] (ii) Visual token sparsification process Suppose that the input image, after being processed by the visual encoder, generates m visual tokens (e.g., m=576, corresponding to a 24×24 grid). This embodiment uses m=576 as an example for illustration, but the invention is not limited thereto—the actual number of visual tokens depends on the specific architecture of the visual encoder and the resolution of the input image.

[0033] (1) Growth stage acquisition: The system automatically identifies the current daffodil in the "growth period" through the image classification model and obtains the growth stage category identifier.

[0034] (2) Determining the sparsity strategy: Query the mapping table to determine the visual attention area corresponding to the growth period as the "leaf area" and the token retention threshold is 256.

[0035] (3) Visual token aggregation and pruning: 1) Identify leaf regions in images using a semantic segmentation model and generate a visual interest region mask; 2) Preserve the visual tokens within the leaf area; 3) For tokens outside the leaf region, sort them based on their relevance score to the semantics of "leaf", and retain the tokens with the highest scores, so that the total number of retained tokens does not exceed 256. After the above processing, the number of visual tokens is reduced from the original m to no more than 256.

[0036] (4) Compact visual representation input to VLA model: The compressed visual token and the language instruction token are input into the VLA model for reasoning to generate horticultural tasks (such as "inspecting the daffodils in the A3 planting basket for pests and diseases").

[0037] (III) Expected Results Compared with existing technologies, the adaptive sparsification method for the growth stage of the present invention is expected to be able to: 1. Compared to the original VLA inference without sparsification, it significantly reduces inference latency and computational resource consumption while maintaining task accuracy; 2. Compared to a uniform sparsity strategy, it maintains higher detection accuracy at growth stages that require detail (such as pest and disease detection during the growth period), avoiding the loss of key information due to excessive pruning; 3. Compared to instruction-driven sparsification methods (such as CogVLA), it provides orthogonal optimization at the biological prior level, and the two can be used together to obtain better overall efficiency.

[0038] Example 2: Comparison of Adaptive Sparsity During Growth Stage and CogVLA This embodiment illustrates the essential difference between the present invention and CogVLA.

[0039] CogVLA's sparsity strategy is "instruction-driven"—dynamically routing and sparsifying visual tokens based on the content of the natural language instruction. For example, when the instruction is "pick up the red cup," the model will enhance attention to tokens related to the red area and the shape of the cup.

[0040] The sparsification strategy of this invention is "growth stage driven"—the visual attention area and sparsification degree are pre-determined based on the current biological growth stage of the crop. For example, when the system identifies that the daffodil is in the "growth period", the visual encoder will prioritize retaining tokens in the leaf area, regardless of whether the current instruction is "watering" or "inspection".

[0041] The two strategies can be used in combination: growth phase-driven approaches determine "which regions should be preferentially preserved" (biological priors), while instruction-driven approaches further determine "which are most relevant to the current task" among the preserved tokens (task dynamic selection). This invention can add a growth phase-aware token pre-screening layer to the visual encoder of existing frameworks such as CogVLA, and use it as a plug-in front-end module.

[0042] Example 3: Online Fine-tuning Based on Execution Feedback This embodiment illustrates an online fine-tuning mechanism for mapping tables based on execution feedback.

[0043] Taking the pest and disease inspection task during the daffodil growing season as an example: Suppose that in multiple consecutive inspections, the feedback data indicates that the missed detection rate of pests and diseases is higher than a preset threshold. If the system analyzes the failure as being due to excessive pruning by the visual token causing loss of leaf details, then online fine-tuning is triggered. 1) Increase the threshold for the number of tokens to be retained during the growth period from the current value (e.g., from 256 to a higher value); 2) At the same time, expand the area of ​​visual attention (such as adding the "back of the leaf area" to the "leaf area").

[0044] After the adjusted mapping table takes effect, subsequent pest and disease inspection tasks are expected to improve. The specific adjustment step size of the token retention threshold and the expansion range of the visual attention area can be flexibly set according to the actual crop variety, image resolution, and task accuracy requirements; this invention does not impose specific numerical limitations on these.

[0045] Example 4: Hardware Configuration and System Integration This embodiment provides a specific hardware configuration for implementing the present invention: (1) Planting platform: PVC hydroponic pipe planting system (Φ160mm~Φ200mm), with anti-algae coating; planting basket 100mm, with QR code label; support height 750mm; double row facing layout, area 50㎡.

[0046] (2) Wheeled robotic arm gardening vehicle: four-wheel differential chassis (900×700×750mm), aluminum body, maximum load 70kg, battery life ≥4 hours; equipped with a six-degree-of-freedom robotic arm (load 2kg, repeatability ±0.1mm, working radius 625mm), dexterous hand (five fingers); the perception system includes a 16-line lidar (30m detection distance) and a binocular depth camera (1280×720@90fps); the edge computing platform has a computing power ≥21TOPS and supports 5G communication.

[0047] (3) Cloud server: dual EPYC processors, 8 NVIDIA H20 GPUs (each with 96GB HBM3e video memory), deploying VLA large model and visual token sparsification module for growth stage perception.

[0048] (4) Communication: 5G network, end-to-end latency ≤50ms.

[0049] The above hardware configuration has been deployed in a 50㎡ open-air test field, and the parameters of each device are consistent with the configuration described above.

Claims

1. A visual token sparsity method based on growth stage perception, characterized in that, Includes the following steps: Step S1: Obtain the current growth stage information of the crop, which includes the crop type identifier and the category identifier of the current growth stage; Step S2: Based on the current growth stage information, determine the visual attention region corresponding to the current growth stage from the preset growth stage-visual attention region mapping table, and determine the corresponding token retention threshold. Step S3: The visual encoder performs visual token aggregation and redundant token pruning on the input image based on the visual attention region and the token retention threshold: retaining visual tokens within the visual attention region and pruning visual tokens outside the visual attention region according to their importance scores, so that the number of retained tokens does not exceed the token retention threshold, thereby generating a compact visual representation. Step S4: Input the compact visual representation into the visual-language-action large model for cross-modal reasoning to generate the gardening task planning result; Step S5: Based on the task execution feedback data, fine-tune the mapping relationship in the growth stage-visual attention area mapping table online.

2. The visual token sparsity method based on growth stage perception according to claim 1, characterized in that, The category identifiers for the growth stages include at least one of the following: germination stage, seedling stage, growth stage, flowering stage, fruiting stage, maturity stage, and harvesting stage; the visual attention areas corresponding to different growth stages include at least one of the following differences: different spatial location of the attention area, different size of the attention area, and different number of attention areas.

3. The visual token sparsity method based on growth stage perception according to claim 1, characterized in that, The token retention threshold in step S3 is dynamically determined based on the growth stage: a first token retention threshold is set in the growth stage that requires high-resolution details, and a second token retention threshold is set in the growth stage that requires global morphological observation. The first token retention threshold is greater than the second token retention threshold.

4. The visual token sparsity method based on growth stage perception according to claim 1, characterized in that, The importance score in step S3 is calculated based on at least one of the following factors: the attention weight of the visual token, the correlation score of the visual token with the current growth stage, and the overlap between the spatial location of the visual token and the visual attention area.

5. The visual token sparsity method based on growth stage perception according to claim 1, characterized in that, The visual attention area includes at least one of the following types: root area, stem area, leaf area, flower organ area, and fruit / bulb area; the combination of visual attention area types varies at different growth stages.

6. The visual token sparsity method based on growth stage perception according to claim 1, characterized in that, The online fine-tuning in step S5 includes: when the execution feedback data indicates that the task has failed or the accuracy is lower than the threshold, adjusting the visual attention area range or token retention threshold of the corresponding growth stage in the growth stage-visual attention area mapping table.

7. A horticultural planning device based on growth stage perception, characterized in that, Running a visual token sparsification method based on growth stage perception as described in any one of claims 1-6 includes: The growth stage acquisition module is used to acquire information about the current growth stage of the crop. A mapping storage module is used to store a growth stage-visual attention area mapping table, which defines the correspondence between different growth stages, visual attention areas, and token retention thresholds. The visual token sparsification module is used to determine the corresponding visual attention region and token retention threshold from the mapping table based on the current growth stage information, and to perform visual token aggregation and redundant token pruning on the input image to generate a compact visual representation. The Visual-Language-Motion Large Model (VLA) inference module is used to perform cross-modal inference on the compact visual representation input to generate gardening task planning results; The feedback fine-tuning module is used to fine-tune the mapping relationship in the mapping table online based on task execution feedback data.

8. A horticultural planning device based on growth stage perception according to claim 7, characterized in that, Also includes: The cloud-based planning subsystem, deployed on a cloud server, includes the growth stage acquisition module, mapping storage module, visual token sparsification module, VLA inference module, and first 5G communication module. The vehicle-mounted execution subsystem, deployed on a wheeled robotic arm gardening vehicle, includes a second 5G communication module, a task parsing module, a deep reinforcement learning control module, a chassis drive module, a robotic arm drive module, a multi-sensor perception module, and a feedback acquisition module. The cloud-based planning subsystem and the vehicle-side execution subsystem communicate bidirectionally via a 5G network, and the execution feedback data collected by the feedback acquisition module is transmitted to the feedback fine-tuning module via the 5G network.

9. A horticultural planning device based on growth stage perception according to claim 7, characterized in that, The growth stage acquisition module obtains the current growth stage information through at least one of the following methods: automatic recognition based on an image classification model, time inference based on a growth calendar, or manual input by the user.

Images