Ploughing depth intelligent recognition method and system based on cross-modal knowledge distillation
By integrating RGB and event cameras through cross-modal knowledge distillation technology, an intelligent tillage system was constructed, which solved the accuracy and adaptability problems of traditional tillage depth control and achieved high-precision, low-cost tillage depth adjustment and quality control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHIHEZI UNIVERSITY
- Filing Date
- 2026-04-23
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391735A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of agricultural machinery technology, and in particular to a method and system for intelligent identification of plowing depth based on cross-modal knowledge distillation, as well as a storage medium and electronic device. Background Technology
[0002] The development of precision agriculture has placed higher demands on the quality and efficiency of tillage operations. Among these, tillage depth, as one of the core agronomic parameters, is crucial for ensuring uniform crop growth and high yields through precise and stable control. Traditional tillage depth control relies entirely on the driver's experience and visual estimation, which is highly subjective and prone to delays. Drivers cannot accurately perceive the real-time depth of the plowshare in complex operating environments characterized by severe bumps and dust, leading to uneven tillage depth. This directly affects the effectiveness of subsequent operations such as sowing and irrigation, becoming a primary bottleneck restricting the improvement of tillage quality and efficiency.
[0003] Current sensor-based mechanical adjustment schemes typically estimate or fix tillage depth indirectly by installing angle sensors, pressure sensors, or mechanical depth-limiting wheels on the plow. However, this method is difficult to adapt to dynamic changes caused by different soil textures, moisture, and surface undulations. It has limited adjustment accuracy and poor universality, and cannot make intelligent real-time adjustments based on actual tillage results.
[0004] In recent years, machine vision technology has shown great potential in agriculture due to its rich information content and non-contact operation. Existing research has attempted to mount traditional RGB cameras on tillage machinery, using image processing or deep learning models to identify furrows and detect tillage depth. This approach elevates the perception dimension from simple mechanical signals to a richer level of visual information. However, the drastic changes in lighting conditions in farmland, coupled with the unavoidable vibrations and dust generated by tractors, lead to severe motion blur, overexposure, or underexposure problems in images captured by traditional RGB cameras. Furthermore, complex recognition models require high computing power from the vehicle, resulting in high costs and energy consumption. A high-precision, real-time visual perception solution for adjusting tillage depth that can operate stably in real, dynamic, and harsh field environments is lacking.
[0005] Event cameras address the shortcomings of traditional RGB cameras. These biomimetic sensors record only pixel brightness changes, offering advantages such as high dynamic range, microsecond-level latency, and low power consumption. They are insensitive to strong light, shadows, and motion blur in dynamic farmland scenes. However, event cameras output asynchronous sparse event streams, which are drastically different from traditional images. The lack of large-scale, high-quality, well-labeled farmland event datasets makes it difficult to directly train high-performance, reliable dedicated recognition models. Currently, how to transform the hardware advantages of event cameras into intelligent perception capabilities usable in the field remains a core, unresolved challenge.
[0006] Therefore, there is an urgent need in this field for an innovative technical solution that can integrate the advantages of different visual modalities to provide agricultural machinery with an intelligent farming system that can perceive in real time and make intelligent decisions, thereby realizing a new type of intelligent farming technology and system that is high-precision, low-cost, and easy to install. Summary of the Invention
[0007] To address the problems existing in the prior art, this invention proposes a method and system for intelligent recognition of plowing depth based on cross-modal knowledge distillation, as well as a storage medium and electronic device. By integrating traditional RGB cameras and event cameras and installing a visual perception module on the plow, an intelligent farming system capable of real-time perception and intelligent decision-making is constructed, thereby improving farming quality and achieving high-precision, low-cost intelligent farming in agricultural operations.
[0008] According to one aspect of the present invention, a method for intelligent recognition of plowing depth based on cross-modal knowledge distillation is provided, comprising the following steps: S1, synchronously acquiring RGB image sequences and event stream data of a target tillage scene through a visual perception module detachably mounted on the plow, wherein the visual perception module includes an RGB camera and an event camera, and the RGB camera and the event camera are aligned spatiotemporally and pixel-level; S2, uploading the RGB image sequences and event stream data to a cloud server, wherein the cloud server is loaded with a general visual teacher model pre-trained on the RGB dataset; S3, using cross-modal knowledge distillation technology, utilizing the RGB image sequences as a supervision medium, the general visual... The visual features and depth estimation knowledge in the visual teacher model are transferred to a lightweight student model that processes event streams. The lightweight student model that processes event streams adopts a spiking neural network or sparse convolutional neural network architecture. S4, the student model obtained by distillation is edge-optimized, including model quantization, pruning and / or compilation operations, and deployed to the vehicle edge controller. S5, in the application stage, the RGB camera can be selectively removed or turned off, and only the event camera is used to collect real-time event streams. The student model identifies the tillage depth and soil breaking rate in real time. S6, the tillage depth identification result is compared with the preset optimal depth range, and a control signal is generated to drive the tillage depth adjustment actuator to adjust the tillage depth of the plow in real time.
[0009] Optionally, in step S1, the RGB camera and the event camera ensure time synchronization through hardware synchronization trigger signals or software timestamps, and ensure spatial overlap of the field of view through coplanar mounting and joint calibration; in step S2, the general visual teacher model is a visual Transformer model pre-trained based on a self-supervised learning method; in step S3, the cross-modal knowledge distillation mechanism includes: extracting high-level feature maps of the teacher model and encoding them into meta-knowledge vectors; dynamically generating adaptation parameters based on the meta-knowledge vectors; and adjusting the feature processes within the student model using the adaptation parameters; in step S5, the student model output also includes a tillage quality confidence score, and the system dynamically switches between intelligent control, assisted prompting, and manual control modes based on the confidence score.
[0010] Optionally, the self-supervised learning method is unlabeled knowledge distillation or a variant thereof; the loss function used in the distillation includes feature alignment loss and task distillation loss.
[0011] Optionally, it also includes: when a change in the tillage environment or plow type is detected, generating a prompt message for the user to restart or install the RGB camera, and executing steps S1 to S4 after the RGB camera is started, generating a dedicated student model suitable for the new tillage environment or new plow through a new round of data acquisition and distillation process, and maintaining a model library that associates and stores different condition configurations and dedicated models, which is used to intelligently call the corresponding student model according to the tillage environment and plow type or user selection.
[0012] According to another aspect of the present invention, a plowing depth intelligent recognition system based on cross-modal knowledge distillation is provided, comprising: a detachable visual perception module for data acquisition, including a detachable and spatiotemporally aligned RGB camera and an event camera, and detachably mounted on various types of plows via a standardized mechanical interface; a cloud server for receiving and processing paired data sent by the detachable visual perception module, loading a pre-trained general visual teacher model, performing cross-modal knowledge distillation, and generating a lightweight student model for processing event streams; an on-board edge controller for deploying the student model and, during the application phase, outputting plowing depth and soil breaking rate evaluation values based on the real-time event streams collected by the event camera, comparing them with a preset optimal depth range, and generating control signals; a plowing depth adjustment actuator for receiving instructions from the on-board edge controller to adjust the plowing depth in real time; and a driver's cab interactive terminal for displaying real-time plowing parameters, soil breaking rate, and a plowing quality heatmap, and providing a model training and switching interface.
[0013] Optionally, when the system detects a change in the tillage environment or plow type, it generates a prompt message for the user to restart or install the RGB camera, and after the RGB camera is started, it generates a dedicated student model suitable for the new tillage environment or new plow through a new round of data acquisition and distillation process.
[0014] Optionally, the cloud server is also used to maintain a model library, which stores different condition configurations and corresponding dedicated models, and is used to intelligently call the corresponding student model according to the farming environment and plow type or user selection.
[0015] Optionally, the detachable visual perception module is mounted on the plow via a replaceable base with a snap-on, bolt-on, or magnetic interface, and the replaceable base is pre-installed with mounting adapters for various plows; the vehicle-mounted edge controller is an embedded AI computing platform; the standardized mechanical interface is a waterproof and dustproof structure including electrical connectors to ensure reliable connection of the visual perception module in harsh field environments.
[0016] According to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent identification method for plowshare tillage depth based on cross-modal knowledge distillation as described above.
[0017] According to another aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the intelligent recognition method for plow depth based on cross-modal knowledge distillation as described above.
[0018] The beneficial effects of this invention are: 1. This invention solves the problem of difficult dynamic visual perception in farmland environments. By employing cross-modal knowledge distillation technology, the general visual knowledge of a large model (teacher model) pre-trained on large-scale RGB data is successfully transferred to a lightweight student model specifically designed to handle event streams. This avoids the problem of scarce labeled data in the event camera domain, enabling the final system to achieve high-precision and high-stability farming with only a low-power event camera, thus solving the problem of traditional RGB vision failing in environments with strong light, vibration, and dust.
[0019] 2. This invention significantly reduces system cost and power consumption. It proposes a "dual-mode acquisition during training, single-mode operation during application" working mode. Users only need one-time dual-mode data acquisition and cloud distillation in a specific environment to obtain a dedicated lightweight event stream recognition model. In subsequent long-term daily operations, only a single event camera and edge computing unit need to be retained and run, achieving a low-cost, low-power intelligent farming method.
[0020] 3. This invention is convenient and highly practical. When the soil, field environment, or type of plow changes, the system will guide the user to reactivate the RGB camera for new training, generate and store a dedicated model for the new environment. This system can flexibly adapt to different farming conditions, significantly improving the product's practical value and user-friendliness.
[0021] 4. This invention constructs a complete "perception-decision-control" intelligent closed loop, improving operational quality and consistency. Through real-time perception of tillage depth and soil breakage rate by an event camera, the edge intelligent unit makes decisions and controls the tillage depth adjustment actuator in real time. This closed loop automatically maintains tillage parameters within a preset optimal range, completely changing the operation mode that relies on driver experience. It ensures the uniformity of tillage depth and quality, laying a solid foundation for subsequent operations such as sowing and irrigation, and contributing to increased crop yields.
[0022] 5. This invention ensures the real-time performance and reliability of the system. The lightweight student model (such as a spiking neural network, SNN) used in this invention is specifically designed for processing asynchronous event streams. Combined with edge optimization techniques such as model quantization and pruning, it can achieve ultra-fast inference of less than 100 milliseconds on resource-constrained onboard computing units. Simultaneously, the inherent anti-interference characteristics of the event camera and the miniaturization of the model jointly ensure the long-term reliable operation of the system in harsh field environments. Attached Figure Description
[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the installation of the visual perception module of the present invention; Figure 3 This is a flowchart of the training phase of the present invention; Figure 4 This is a schematic diagram of the data flow and cross-modal knowledge distillation during the training phase of this invention; Figure 5 This is a flowchart illustrating the application phase of the present invention. Figure 6 This is a flowchart illustrating the workflow of the system of the present invention, which adapts to different soil environments and different types of plows. Figure 7 This is a schematic diagram of the overall structure of the system of the present invention. Detailed Implementation
[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present application can be combined with each other.
[0025] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.
[0026] The terms “comprising” and “having”, and any variations thereof, in the specification and claims of this application are intended to cover non-exclusive inclusion, for example, a process, method, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.
[0027] Example 1: This example provides an intelligent plow depth recognition method based on cross-modal knowledge distillation. By fusing the rich semantics of RGB images and the high dynamic characteristics of event streams during the training phase, knowledge distillation is used to transfer the capabilities of the pre-trained large model (teacher model) to a lightweight dedicated model (student model). Ultimately, it can achieve high-precision real-time plow depth (cultivation depth) and soil breakage rate recognition by relying on only a single, low-power event camera.
[0028] like Figure 1 As shown, the method mainly includes the following steps: S1, through a visual perception module that can be detachably installed on the plow, synchronously collects RGB image sequences and event stream data of the target farming scene; like Figure 2 As shown, the detachable visual sensing module is mounted on the plow beam via a standardized quick-release interface and includes an RGB camera and an event camera, both of which are fixed in the same protective housing in a coplanar manner.
[0029] Among them, an event-based camera is a new type of visual sensor (also known as a dynamic visual sensor) that can capture fast-moving objects in a high dynamic range. The biggest difference between it and a traditional camera is that a traditional camera outputs a sequence of images at a fixed frame rate, while an event camera outputs an asynchronous event stream. It outputs asynchronous signals by measuring the brightness changes of each pixel, and these changes are called "events".
[0030] The RGB camera and the event camera are hardware synchronized and jointly calibrated to ensure strict alignment of data in time and space. Time synchronization is achieved through hardware synchronization trigger signals or software timestamps, and the field of view is ensured to overlap through coplanar installation and joint calibration.
[0031] Preferably, the RGB camera uses a global shutter CMOS sensor, and the event camera uses an iniVation DAVIS346.
[0032] Preferably, in training mode (i.e., the training phase before generating an applicable student model), the tractor travels at a speed of 5 km / h, and the system simultaneously acquires a 1280x720 resolution, 30fps RGB video stream, as well as the corresponding event stream data packets.
[0033] S2, the RGB image sequence and event stream data are uploaded to a cloud server, wherein the cloud server is loaded with a general visual teacher model pre-trained on a large-scale RGB dataset; Specifically, paired data consisting of RGB image sequences and event stream data is uploaded to the cloud via a wireless network to drive the subsequent cross-modal knowledge distillation process. The cloud server is deployed with a deep learning framework based on PyTorch or TensorFlow, and loaded with a VisionTransformer teacher model pre-trained on a large-scale RGB dataset using self-supervised learning methods (such as DINO (Distillation with No Labels) or its variants). This model already possesses powerful general visual feature extraction and scene understanding capabilities.
[0034] S3, using cross-modal knowledge distillation technology, the RGB image sequence is used as a supervision medium to transfer the visual features and depth estimation knowledge in the general visual teacher model to a lightweight student model that processes event streams. The cross-modal knowledge distillation mechanism is as follows: extracting high-level feature maps from the teacher model and encoding them into meta-knowledge vectors; dynamically generating adaptation parameters based on the meta-knowledge vectors; and using the adaptation parameters to adjust the feature processes within the student model, thereby distilling a student model that can mimic the feature representation and prediction capabilities of the teacher model.
[0035] In deep learning networks, high-level feature maps refer to the following: shallow networks extract shallow, concrete, low-level features such as edge colors, while deep networks output deep, abstract, and semantically rich high-level features that contain other information, such as object categories, overall outlines, and scene layouts. In this embodiment, the high-level feature map output by the teacher model after processing each frame of RGB image will clearly display annotation information. For example, plowed furrows will be shown as dark areas, while uncultivated soil will be shown as light areas. In addition, details such as the boundaries of the plowshare and the size distribution of soil fragments will also be displayed.
[0036] Meta-knowledge vectors typically refer to compressed encoded vectors used to summarize the core feature attributes of the current input data, such as lighting conditions and movement speed. In this embodiment, targeting a farming scenario, the meta-knowledge vector encodes the current farming conditions, such as the current soil type being wet clay, the surface being uneven, the lighting being strong backlight, and the plow being driven into the soil at a steep angle.
[0037] The adaptation parameters refer to the neural network weights or adjustment coefficients dynamically generated based on the "meta-knowledge vector," used to modify the processing logic of the student model in real time. In this embodiment, based on meta-knowledge (such as the current environment being wet clay and backlit), the adaptation parameters dynamically adjust the convolutional kernel weights in the student model. For example, they enhance the filtering ability of dense noise generated by the collision between soil and plowshares and increase the sensitivity to the edges of large soil particles, while temporarily reducing attention to the distant background. This is equivalent to customizing the model's processing rules for the current soil type and lighting conditions.
[0038] The feature process refers to the computational steps such as transformation, activation, pooling, or normalization that occur when data flows between different network layers of the student model. In this embodiment, the feature processes that event stream data undergoes after entering the student model include, but are not limited to: Pulse conversion: converting continuous event signals into sparse spatiotemporal point clouds; Spatial aggregation: For example, aggregating point clouds within 10ms into "event frames"; Spatiotemporal feature extraction: Extract "the flow velocity of soil particles on the time axis" and "the event burst pattern at the moment the plowshare enters the soil" through 3D convolution; Mapping Regression: Ultimately, the features are mapped to specific numerical values, such as a tillage depth of 25cm and a soil breakage rate of 85%.
[0039] like Figure 3 and Figure 4 The diagram illustrates the training phase workflow and the data flow and cross-modal knowledge distillation process.
[0040] Simultaneously, RGB data and event stream data are collected. The teacher model and student model process the RGB data and event stream data in parallel. After repeated training, a lightweight student model that can be deployed and used is obtained for subsequent cultivation.
[0041] Specifically, a lightweight student network is constructed using a spiking neural network (SNN) or sparse convolutional neural network architecture. The backbone network preferably adopts a sparse convolutional neural network, such as Sparse-ResNet18 under the MinkowskiEngine framework. This architecture is designed to process highly sparsity event stream data and can efficiently extract spatiotemporal features.
[0042] The feature maps output from the intermediate layers of the teacher model are treated as "knowledge" and projected onto dimensions that match the features of the student model through a lightweight adapter network. The cosine similarity or mean squared error between teacher and student features is calculated as the feature alignment loss, forcing the student model to learn semantic representations equivalent to RGB images from the event stream. Simultaneously, the RGB images from the paired data are input into the teacher model to obtain its predicted tillage depth pseudo-labels. The corresponding event streams are then input into the student model to obtain its predicted tillage depth.
[0043] The distillation loss function includes feature alignment loss and task distillation loss. Feature alignment loss measures the degree of matching between the student model and the teacher model on the intermediate layer feature maps, aiming to make the semantic features extracted by the student model from the event stream as similar as possible to those extracted by the teacher model from the RGB image. Task distillation loss measures the difference between the final prediction output of the student model and the final prediction output of the teacher model, aiming to make the student model's prediction result closer to the teacher model's prediction result. Furthermore, KL divergence or smoothed L1 loss is used to calculate the task distillation loss.
[0044] Through training and iterative optimization, the student model gradually learns to infer the knowledge of tillage depth and soil fragmentation rate that originally required RGB images from the event stream alone.
[0045] S4. The student model obtained from distillation is edge-optimized and deployed to the vehicle edge controller; Marginalization optimization includes model quantization, pruning, and / or compilation operations.
[0046] Pruning: Remove components that have little impact on the result, such as reducing the number of heads and removing layers with little impact, sharing parameters, etc. Model quantization: reducing the precision of model values, for example, reducing float32 (32-bit floating-point number) to float8 (8-bit floating-point number). Compilation operation: Transforms the model from a framework description into highly optimized machine code that can be executed on specific hardware.
[0047] The edge-optimized student model is deployed on an in-vehicle edge controller (computing unit). This in-vehicle edge controller is an embedded AI computing platform with sufficient computing power to complete the inference of a frame of event data by the student model within 100 milliseconds, so as to meet the stringent requirements of field embedded devices for computing efficiency and power consumption and enter the application stage.
[0048] In the application phase, the S5 can selectively remove or turn off the RGB camera, and only collect real-time event streams through the event camera, and the student model can identify the tillage depth and soil breaking rate in real time. Specifically, such as Figure 5 As shown, in subsequent daily cultivation, users can physically remove or turn off the RGB camera. The event camera continuously outputs an asynchronous event stream. After preprocessing such as format conversion and timing alignment, the vehicle-mounted edge controller accumulates the events into event frames at fixed time windows (e.g., 10ms) and inputs them into the deployed lightweight student model for real-time monitoring and perception. The (vehicle-mounted) student model outputs the current cultivation depth estimate and soil breakage rate in real time.
[0049] S6 compares the tillage depth identification result with the preset optimal depth range, generates a control signal, and drives the tillage depth adjustment actuator to adjust the tillage depth of the plow in real time. The vehicle-mounted edge controller compares the current tillage depth (i.e., the tillage depth recognition result of the student model) with the preset optimal depth range. If it is within the optimal tillage range, the current tillage state is maintained. If it is not within the optimal tillage range, the PID controller calculates the control signal of the hydraulic valve and sends the control signal to the tillage depth adjustment actuator to drive the hydraulic cylinder / electric adjuster to adjust the plow height, thereby controlling the tillage depth of the plow and forming a closed-loop control of "perception-decision-execution".
[0050] Simultaneously, the current tillage depth and soil breakage rate are displayed in real-time on the graphical interface of the cab's interactive terminal and stored in the data. Furthermore, a tillage depth curve can be plotted and displayed based on historical tillage depths.
[0051] S7 guides users to start a new round of training and distillation when the farming environment or plow type changes, generating and managing dedicated student models suitable for the new farming environment or plow.
[0052] Specifically, such as Figure 6As shown, when the system detects a change in the tillage environment or plow type through onboard sensors or user input, such as a change in soil environment from sandy soil to clay soil, or a change in plow from a moldboard plow to a furrowing plow, the driver's cab interactive terminal will display a prompt to guide the user to reinstall or start the RGB camera. After the RGB camera is reinstalled or started, a new round of data acquisition and distillation process will be automatically started, that is, repeating steps S1 to S4 to generate a dedicated student model for the new conditions. Then, the new model (i.e., the dedicated student model for the new conditions) will be deployed to the onboard edge controller for subsequent identification of tillage depth and soil breaking rate, as well as tillage depth control.
[0053] Furthermore, the output of the student model also includes a confidence score for tillage quality (soil breakage rate). Based on this confidence score, the system dynamically switches between intelligent control, auxiliary prompts, and manual control modes. For example, when the confidence score is within a preset first range, the confidence is low and the result is unreliable (can be ignored), so the system switches to manual control mode, where the operator controls the tillage depth. When the confidence score is within a preset second range (the preset second range is higher than the preset first range), the system switches to auxiliary prompts (the second range is higher than the preset first range), and the recognition result serves as an auxiliary prompt for the operator without executing tillage control. When the confidence score is within a preset third range (or higher than a certain preset threshold), the recognition result is accepted, and intelligent tillage control of the tillage can be executed, driving the tillage depth adjustment actuator to perform intelligent real-time adjustments.
[0054] Optionally, before loading the teacher model for cross-modal knowledge distillation, the cloud server will preprocess the spatiotemporally synchronized paired data composed of the received RGB image sequence and event stream data (such as data cleaning, transformation, standardization, etc.), and provide collection quality feedback. The server will evaluate the quality of the collected data (such as clarity, completeness, motion artifacts, image noise, etc.) and provide feedback to the collection end (visual perception module) based on the evaluation results to re-collect or adjust the collection strategy.
[0055] In addition, the cloud server maintains a model library to associate and store different condition configurations (including farming environment and plow type) with dedicated models. In the model library, various combinations of environments and plows correspond to corresponding dedicated models. The vehicle edge controller can intelligently download and load the most matching model from the model library based on GPS positioning information, the detected current surrounding environment and plow type, or manual selection by the user, so as to achieve continuous self-adaptation of the system.
[0056] In summary, this invention discloses an intelligent plow depth recognition method based on cross-modal knowledge distillation. It utilizes spatiotemporally aligned RGB cameras and event cameras to collect paired data. Through cross-modal knowledge distillation, the rich general visual and depth estimation knowledge from a pre-trained large RGB model (teacher model) is transferred to a lightweight student model specifically designed for event streams. The student model not only inherits high-precision visual understanding capabilities but also possesses the high dynamic range and anti-blurring characteristics unique to event cameras, solving the problem of scarce labeled data for event cameras. Simultaneously, the lightweight model significantly reduces the workload of onboard computers, substantially lowering costs and driving the intelligent development of agriculture.
[0057] Furthermore, the integration of traditional RGB cameras and event cameras creates a dual-modal fusion working mode during training and a single-modal lightweight operation mode during application. Based on real-time working conditions, it can intelligently adjust the tillage depth to the set standard in real time without human intervention, and simultaneously evaluate the tillage quality (soil breakage rate). This significantly reduces hardware costs and power consumption, improves tillage precision and quality, and realizes agricultural intelligence.
[0058] When the farming environment, such as soil type or plow type, changes, this invention can proactively sense and guide users to generate new dedicated models by re-collecting data. At the same time, it archives optimized models under different configurations into a unified model library for management. This is not only applicable to soil farming in different regions or under different crop conditions, but also can be started with one click and intelligently call the model library, greatly reducing costs and the barrier to entry.
[0059] Example 2: Based on the overall concept of Example 1, this example provides an intelligent plow depth recognition system based on cross-modal knowledge distillation, such as... Figure 7 The diagram shown is a system architecture schematic of a preferred embodiment of the present invention. The entire system mainly consists of four parts: Detachable visual perception module: As a data acquisition end, it is detachably installed on the plow through a standardized mechanical interface. It includes a detachable and spatiotemporally aligned RGB camera and event camera, which are used to synchronously acquire paired data through the RGB camera and event camera during the training phase, and to retain only the event camera to acquire real-time event streams during the application phase. Cloud server / main controller: As the algorithm center, it is responsible for performing cross-modal knowledge distillation training. By loading a pre-trained general visual teacher model, it receives and processes paired data, performs cross-modal knowledge distillation, and generates a lightweight student model that processes event streams. Vehicle-mounted edge controller: As the control center, it deploys the student model obtained after distillation and outputs the evaluation values of tillage depth and soil breaking rate based on the real-time event stream collected by the event camera during the application phase. Tillage depth adjustment actuator: As the execution end, it is used to receive instructions from the vehicle-mounted edge controller to adjust the tillage depth of the plow in real time. It is usually an electro-hydraulic proportional valve or an electric push rod, which directly drives the plow to lift and lower.
[0060] Furthermore, the system may also include a cab-mounted interactive terminal for displaying real-time tillage parameters, soil breakage rate, and tillage quality heatmaps (intuitively showing differences in tillage quality through color depth) and providing a model training and switching interface.
[0061] The system supports generating a dedicated student model suitable for new farming environments or new plows by reinstalling the RGB camera and performing steps S1 to S4 in Example 1.
[0062] The detachable visual sensing module is mounted on the plow via a replaceable base with a snap-on, bolt-on, or magnetic interface. The interface base is pre-installed with mounting adapters for various plows.
[0063] The vehicle-mounted edge controller is an embedded AI computing platform, whose computing power is sufficient to complete the student model's inference of a frame of event data within 100 milliseconds.
[0064] The standardized mechanical interface is a waterproof and dustproof structure that includes electrical connectors, ensuring reliable connection of the visual perception module in harsh field environments.
[0065] When changing the type of plow, the detachable module is removed from the original plow and installed on the new plow, triggering an adaptation process to the new plow and the farming environment: first, a complex teacher model is trained, and then the capabilities of the teacher model are transferred to a lightweight student model as a dedicated recognition model through cross-modal knowledge distillation technology.
[0066] In the system, solid arrows represent mechanical or electrical connections, while dashed arrows represent the flow of data and control signals, clearly demonstrating the complete link from data acquisition, cloud training, model distribution to real-time edge control.
[0067] In summary, this invention discloses an intelligent plowing depth recognition system based on cross-modal knowledge distillation. By installing a visual perception module on the plow, a closed-loop system is formed with the vehicle-mounted control unit (vehicle-mounted edge controller) and the tillage depth adjustment actuator. The visual perception module collects data to train and distill a lightweight model, which is then deployed on the vehicle-mounted control unit to monitor the soil tillage depth in real time. The actuator intelligently adjusts the tillage depth in real time during the tillage process, generating a tillage quality report. This achieves cost reduction and efficiency improvement, effectively enhancing tillage precision and quality, and realizing intelligent agriculture.
[0068] This invention presents an intelligent tillage system based on cross-modal knowledge distillation, forming a complete closed loop of "multimodal perception - edge intelligent decision-making - real-time precise control." It can reliably and in real-time identify tillage depth and soil fragmentation rate in complex farmland environments such as strong light, dust, and severe vibration. The system intelligently compares the identification results with a preset optimal range, generating control signals to drive the actuators and achieve intelligent adjustment of tillage depth. Furthermore, it is applicable long-term after a single training session. When the tillage environment changes or the type of plow is altered, only a single click of retraining is required, greatly improving its practicality and versatility, and realizing intelligent operation of agricultural machinery.
[0069] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.
[0070] This invention also provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when it is run.
[0071] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0072] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0073] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.
[0074] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0075] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0076] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0078] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0079] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0080] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A method for intelligent recognition of plow depth based on cross-modal knowledge distillation, characterized in that, Includes the following steps: S1, synchronously acquire RGB image sequences and event stream data of the target farming scene through a visual perception module that can be detachably installed on the plow. The visual perception module includes an RGB camera and an event camera, and the RGB camera and the event camera are aligned in time, space and at the pixel level. S2, the RGB image sequence and event stream data are uploaded to a cloud server, wherein the cloud server is loaded with a general visual teacher model pre-trained on the RGB dataset; S3, using cross-modal knowledge distillation technology, the RGB image sequence is used as a supervision medium to transfer the visual features and depth estimation knowledge in the general visual teacher model to a lightweight student model that processes event streams. The lightweight student model that processes event streams adopts a spiking neural network or sparse convolutional neural network architecture. S4, the student model obtained by distillation is edge-optimized, including model quantization, pruning and / or compilation operations, and then deployed to the vehicle edge controller; In the application phase, the S5 can selectively remove or turn off the RGB camera, and only collect real-time event streams through the event camera, and the student model can identify the tillage depth and soil breaking rate in real time. S6 compares the tillage depth identification result with the preset optimal depth range, generates a control signal, and drives the tillage depth adjustment actuator to adjust the tillage depth of the plow in real time.
2. The intelligent plowing depth recognition method based on cross-modal knowledge distillation according to claim 1, characterized in that, In step S1, the RGB camera and the event camera ensure time synchronization through hardware synchronization trigger signals or software timestamps, and ensure that the field of view space coincides through coplanar installation and joint calibration. In step S2, the general visual teacher model is a visual Transformer model pre-trained based on a self-supervised learning method; In step S3, the cross-modal knowledge distillation mechanism includes: extracting high-level feature maps from the teacher model and encoding them into meta-knowledge vectors; dynamically generating adaptation parameters based on the meta-knowledge vectors; and adjusting the feature processes within the student model using the adaptation parameters. In step S5, the student model output also includes a tillage quality confidence level, and the system dynamically switches between intelligent control, assisted prompts and manual control modes based on the confidence level.
3. The intelligent plowing depth recognition method based on cross-modal knowledge distillation according to claim 2, characterized in that, The self-supervised learning method is unlabeled knowledge distillation or a variant thereof; the loss function used in distillation includes feature alignment loss and task distillation loss.
4. The intelligent plowing depth recognition method based on cross-modal knowledge distillation according to claim 1, characterized in that, Also includes: When a change in the tillage environment or plow type is detected, a prompt message is generated for the user to restart or install the RGB camera. After the RGB camera is started, steps S1 to S4 are executed to generate a dedicated student model suitable for the new tillage environment or plow through a new round of data acquisition and distillation process. A model library is maintained that stores different condition configurations and dedicated models, which is used to intelligently call the corresponding student model according to the tillage environment, plow type or user selection.
5. A plow depth intelligent recognition system based on cross-modal knowledge distillation, characterized in that, include: A detachable vision sensing module for data acquisition includes a detachable and spatiotemporally aligned RGB camera and event camera, which can be detachably mounted on various types of plows via a standardized mechanical interface; A cloud server is used to receive and process paired data sent by the detachable visual perception module, load a pre-trained general visual teacher model, perform cross-modal knowledge distillation, and generate a lightweight student model that processes event streams. The vehicle-mounted edge controller is used to deploy the student model and, during the application phase, outputs the evaluation values of tillage depth and soil breaking rate based on the real-time event stream collected by the event camera, compares them with the preset optimal depth range, and generates control signals. The tillage depth adjustment actuator is used to receive instructions from the vehicle-mounted edge controller to adjust the tillage depth of the plow in real time. The cab-interactive terminal displays real-time tillage parameters, soil breakage rate, and tillage quality heatmaps, and provides an interface for model training and switching.
6. The intelligent plow depth recognition system based on cross-modal knowledge distillation according to claim 5, characterized in that, When the system detects a change in the tillage environment or plow type, it generates a prompt message for the user to restart or install the RGB camera. After the RGB camera is started, it generates a dedicated student model suitable for the new tillage environment or new plow through a new round of data acquisition and distillation process.
7. The intelligent plow depth recognition system based on cross-modal knowledge distillation according to claim 5, characterized in that, The cloud server is also used to maintain a model library, which stores different condition configurations and corresponding dedicated models, and is used to intelligently call the corresponding student model according to the farming environment and plow type or user selection.
8. The intelligent plow depth recognition system based on cross-modal knowledge distillation according to claim 5, characterized in that, The detachable visual perception module is mounted on the plow via a replaceable base with a snap-on, bolt-on, or magnetic interface. The replaceable base is pre-installed with mounting adapters for various plows. The vehicle-mounted edge controller is an embedded AI computing platform. The standardized mechanical interface is a waterproof and dustproof structure that includes electrical connectors, ensuring reliable connection of the visual perception module in harsh field environments.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent recognition method for plowing depth based on cross-modal knowledge distillation as described in any one of claims 1 to 4.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the intelligent recognition method for plowing depth based on cross-modal knowledge distillation as described in any one of claims 1 to 4.