A field dynamic obstacle detection, segmentation, and precise ranging device and method

Abstract

This invention provides a device and method for dynamic obstacle detection, segmentation, and precise ranging in fields. The invention aims to solve the problems of large ranging errors at the center point of the detection frame, limited computing power of edge devices, and unreliable control switching in unstructured field environments. The method involves adaptive preprocessing of the video stream; utilizing spatiotemporal sparse inference to filter active blocks, reducing computational load; introducing an entropy-guided cueing strategy to calculate pixel Shannon entropy within the detection frame, generating composite cueing to drive a lightweight segmentation decoder to output an obstacle mask; and constructing a point cloud based on mask-constrained back projection and outputting precise distances. The device uses an embedded AI module as its core, integrating a dual-power supply switching circuit and a relay-based CAN bus control switching circuit to achieve hardware-level isolation and smooth switching between local obstacle avoidance and global navigation modes. This invention achieves high-precision, low-latency obstacle perception and reliable control at the edge.

Description

Technical Field

[0001] This invention belongs to the field of cross-integration technology of intelligent equipment, machine vision algorithms and embedded automatic control technology, and in particular relates to a device and method for dynamic obstacle detection, segmentation and accurate distance measurement in the field. Background Technology

[0002] Unlike the structured road environment faced by urban autonomous driving, the field environment is a typical unstructured or semi-structured complex scenario with extremely high uncontrollability, uncertainty, and strong dynamic time-varying characteristics. This poses unprecedented challenges to existing machine vision perception systems. Specifically, the lighting conditions in the field are complex, with interference such as high light spillover, low illumination, and fast-moving shadows; the obstacles in the field are extremely irregular in shape, including workers, farm tools, water conservancy facilities, and fallen tree trunks in various postures, often forming a "camouflage effect" with the background; in addition, dynamic interference such as agricultural machinery bumps, dust, and dense fog further exacerbate the perception difficulty.

[0003] Existing technologies have the following significant limitations in addressing the above challenges:

[0004] First, methods based on traditional handcrafted features (such as HOG, SIFT, LBP combined with SVM) have poor generalization ability, are extremely sensitive to changes in lighting and object deformation, and have a high false negative rate; traditional binocular stereo matching algorithms are prone to parallax errors or holes in areas with weak or repetitive textures.

[0005] Second, while deep learning-based object detection methods (such as the YOLO series) offer high detection accuracy, they suffer from a fatal geometric contradiction between the rectangular bounding box and accurate spatial ranging. Existing methods simply select the geometric center point of the detection box and map it to the depth map for ranging. However, most obstacles in the field are irregularly shaped (such as bent farmers or leaning utility poles), and their detection box center points often fall into the background area, leading to a serious distance estimation bias of "measuring far but hitting close," directly threatening operational safety.

[0006] Third, while instance segmentation techniques (such as Mask R-CNN) can generate pixel-level masks for accurate ranging, their computational cost is enormous, making them unsuitable for real-time operation on edge devices commonly used in agricultural machinery (such as the Jetson series) (frame rate less than 5 FPS, far below the required 15-30 FPS). Furthermore, pixel-level annotation is extremely expensive (more than 10 times that of bounding box annotation), making rapid deployment for specific farmland scenarios difficult. The recently developed Segment Anything Model also suffers from excessively large parameter counts, hindering real-time deployment at the edge.

[0007] Fourth, existing solutions lack integrated hardware and software reliability design at the system engineering and hardware architecture levels. During agricultural machinery operation in the field, the drastic voltage fluctuations at engine start-up can easily cause the main control unit to restart or crash; existing solutions lack integrated power management and uninterruptible power supply protection circuits. More importantly, when switching between the "global path tracking" and "local emergency obstacle avoidance" control modes, existing solutions mostly rely on software-level communication protocols (such as ROS topic switching), which inherently carries risks of latency, deadlock, or race conditions. Once the sensing main control's operating system or communication program crashes, the agricultural machinery will be in an "out-of-control" state, leading to a safety accident.

[0008] In summary, existing technologies face multiple contradictions: the contradiction between the need for high-precision ranging and the limited computing power of edge devices; the contradiction between the potential risks of rectangular frame center ranging and the complex shapes of unstructured obstacles; the contradiction between high annotation costs and rapid iterative deployment; and the contradiction between the sophistication of algorithms and the fragility of hardware and electrical architecture. Therefore, a system-level solution is urgently needed. Summary of the Invention

[0009] To address the aforementioned technical problems, this invention provides a field dynamic obstacle detection, segmentation, and precise distance measurement device and method. This solves the technical problems in existing technologies, such as large distance deviations due to reliance on the center point of the detection frame, difficulty in real-time operation of high-precision segmentation algorithms on edge devices, high pixel-level annotation costs, and the lack of reliable power management and hardware-level control arbitration in agricultural machinery control systems. This enables low-cost, high-real-time, and highly robust obstacle perception and safe obstacle avoidance control in complex farmland environments.

[0010] This invention focuses on solving the real-time environmental perception challenge of agricultural mobile robots and autonomous agricultural machinery on edge devices with limited computing power. Its technical scope covers lightweight object detection based on deep learning, spatiotemporal sparse convolutional inference acceleration, Shannon entropy-driven cue-based segmentation, and 3D spatial reconstruction technology based on binocular stereo vision. Specifically, this invention addresses the unstructured features of irregular crop obstacle shapes in farmland scenarios by proposing a robust depth estimation technique using pixel-level mask constraints. This aims to overcome the background perspective error and positioning drift problems inherent in traditional two-dimensional object detection box center point distance measurement. Furthermore, this invention relates to the integrated design of the underlying electronic and electrical architecture of agricultural machinery, specifically including power management and communication control technology for an integrated embedded main control terminal. This encompasses a multi-source power supply (vehicle battery / built-in battery) switching circuit, a CAN bus control arbitration mechanism based on physical relay logic, and a hardware-level safety switching strategy for local obstacle avoidance and global navigation modes. This invention is applicable to the construction of active safety defense and assisted driving systems for agricultural automation equipment such as unmanned tractors, orchard plant protection robots, and intelligent harvesters.

[0011] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0012] A field dynamic obstacle detection, segmentation, and precise ranging device includes a sensing module, a computing processing core, and a power supply and control management module;

[0013] The sensing module is used to collect RGB images and depth information of the field environment, and the sensing module is connected to the computing and processing core.

[0014] The computational processing core employs an embedded AI computing module as the main control unit. Within this core are deployed a spatiotemporal sparse inference module, a feature alignment and detection module, an entropy-guided cue segmentation module, and a 3D point cloud ranging module. The spatiotemporal sparse inference module performs inter-frame differencing and saliency filtering on the input video stream, performs convolution calculations only on active regions, and outputs the generated sparse feature map to the feature alignment and detection module. The feature alignment and detection module receives the sparse feature map, extracts multi-scale features, and uses deformable convolution for feature alignment to generate dynamic obstacles. The initial bounding box and category probability map of the object are generated, and then the initial bounding box is passed to the entropy-guided cue segmentation module. The entropy-guided cue segmentation module is used to calculate pixel-level Shannon entropy within the region of interest defined by the received initial bounding box, select high-entropy boundary points and low-entropy center points as cue points based on the entropy value, generate a pixel-level segmentation mask, and send it to the 3D point cloud ranging module. The 3D point cloud ranging module is used to perform mask filtering on the depth map according to the received segmentation mask, back-project the effective pixels within the mask to 3D space to construct a point cloud, remove outliers, and calculate the obstacle distance.

[0015] The power supply and control management module is connected between the computing core, the external power supply, and the underlying actuators of the agricultural machinery. The module includes a dual-power supply switching circuit and a control switching circuit. The input of the dual-power supply switching circuit is connected to both the external vehicle power interface and the built-in battery interface, and is connected to a voltage regulator circuit via a switching unit. The output of the voltage regulator circuit powers the computing core. The control switching circuit includes a relay unit and signal switching logic. The relay unit is connected in series in the CAN bus communication link and is used to switch the sending entity of control commands between a first control state and a second control state. The first control state is a local obstacle avoidance mode, where the relay is in the off state to connect the communication link between the computing core and the underlying actuators, and the computing core sends control commands. The second control state is a global path tracking mode, where the relay is in the energized state to connect the external interface used to connect the external navigation terminal to the underlying actuators, shielding the control commands from the computing core.

[0016] In the above scheme, the perception module uses a ZED 2i binocular depth camera; the computing processing core uses an NVIDIA Jetson Orin NX module.

[0017] The external navigation terminal is an Android tablet or industrial control computer running a global path planning algorithm;

[0018] The dual-power supply switching circuit is configured such that when the external power circuit is closed, the vehicle battery provides power; when the external power circuit is disconnected and the internal circuit is closed, the built-in lithium battery pack provides power.

[0019] In the above scheme, the control switching circuit adopts a common-ground single-wire switching topology:

[0020] The CAN-Low signal lines of the agricultural machinery's underlying controller, external navigation terminal, and computing core are permanently shorted.

[0021] The common terminal of the relay is connected to the CAN-High pin of the agricultural machinery's underlying controller, the normally open terminal of the relay is connected to the CAN-High pin of the external navigation terminal, and the normally closed terminal of the relay is connected to the CAN-High pin of the computing processing core.

[0022] When the computing core outputs a low level, the relay remains in the normally closed state (i.e., open state), and control belongs to the computing core; when the computing core outputs a high level, the relay is energized, and control is switched to the external navigation terminal.

[0023] A method for detecting, segmenting, and accurately measuring dynamic obstacles in a field using the aforementioned field dynamic obstacle detection, segmentation, and precise ranging device includes the following steps:

[0024] Step S1: Obtain real-time video stream image sequences and synchronized depth information of the field environment;

[0025] Step S2: Perform resolution adaptive adjustment and multi-dimensional denoising preprocessing on each frame of image to construct a multi-scale image pyramid;

[0026] Step S3: Use the sparse inference controller to perform spatiotemporal saliency region screening on the multi-scale image pyramid to generate a set of active state blocks; determine whether the current frame is a key frame based on the inter-frame temporal correlation, reuse the features of the previous key frame for non-key frames and perform convolution calculation only on newly added active state blocks;

[0027] Step S4: Input the sparse features into the shared backbone network containing the feature alignment module, use deformable convolution to perform feature alignment, and output the preliminary bounding box and class probability map of the dynamic obstacle.

[0028] Step S5: For each preliminary bounding box, calculate the pixel-level information entropy distribution within its corresponding region of interest. Based on the entropy value, automatically select high-entropy boundary points and low-entropy center points as point cues, and combine them with the bounding box to form a composite cues. Input the composite cues into the lightweight segmentation decoder and output the pixel-level segmentation mask of the dynamic obstacles.

[0029] Step S6: Use the segmentation mask to perform mask filtering on the depth map, back-project the effective pixels in the mask to the three-dimensional space to construct a point cloud, remove outliers, and then use robust statistics to calculate the obstacle distance.

[0030] Step S7: Based on the calculated obstacle distance, the calculation processing core determines whether the obstacle avoidance logic is triggered; if triggered, the control switching circuit ensures that it is in the first control state and sends an obstacle avoidance command.

[0031] In the above scheme, the sparse inference controller in step S3 specifically includes:

[0032] The input frame is divided into N×N grid blocks;

[0033] Calculate the sum of gradient magnitudes G for each grid block and the temporal difference magnitude T between the current grid block and the previous frame;

[0034] Constructing an activation scoring formula ,in, It is the maximum value of the temporal difference amplitude of all blocks in the current frame. This represents the maximum sum of gradient magnitudes across all blocks in the current frame. These are the weighting coefficients; if Greater than the set threshold If a block is marked as active, it is marked as dormant.

[0035] Introducing a safety fallback mechanism: Force a full frame refresh every preset number of frames, or trigger a full inference rollback when the detection confidence is lower than the threshold.

[0036] In the above scheme, the sparse inference controller in step S3 is also equipped with a spatiotemporal memory decay mechanism:

[0037] Maintain a state hold frame counter for each grid block. When a block is determined to be active, the counter is set to the preset hold frame number. If activation is not triggered again in each subsequent frame, the counter is decremented until it reaches zero before the block is put into a dormant state.

[0038] In the above scheme, the entropy-guided prompt generation strategy in step S5 is as follows:

[0039] Calculate the pixel Shannon entropy within the region of interest. Where C is the total number of categories, and c is the category index. For pixels ( The predicted probability of belonging to category c;

[0040] Center point prompt: Select The smallest K pixels represent the high-confidence foreground;

[0041] Boundary point hint: Select The top M largest pixels represent the edge of uncertainty;

[0042] The coordinates of the points mentioned above are concatenated with the coordinates of the bounding box and used as geometric constraints to input into the segmentation network.

[0043] In the above scheme, the logic for precise ranging in step S6 is as follows:

[0044] Using camera intrinsic matrix pixel coordinates within the mask With depth value Back projection yields three-dimensional coordinates ,in This represents the inverse of the camera intrinsic parameter matrix, and the superscript T indicates the matrix transpose;

[0045] Perform statistical filtering on the generated point cloud set to remove outlier noise points whose distance from the mean exceeds a multiple of the standard deviation.

[0046] Sort the depth values ​​of the remaining valid point cloud and take the distance values ​​of the top 5% to 15% as the final obstacle distance.

[0047] In the above scheme, the preprocessing in step S2 includes:

[0048] The image is scaled to the preset network input resolution using the Letterbox method, which maintains the aspect ratio.

[0049] Convert the image from RGB color space to HSV or LAB color space;

[0050] Bilateral filtering is used to smooth the image, removing field environmental noise while preserving obstacle edge gradient information.

[0051] In the above scheme, the lightweight segmentation decoder is based on the Tiny-ViT architecture. It learns weights from the SegmentAnything Model through knowledge distillation and uses a cross-attention mechanism with composite prompts as queries and image features as keys and values ​​to output a binary segmentation mask.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] Firstly, regarding ranging accuracy, this invention abandons the traditional method of ranging using the center point of the bounding box, and utilizes entropy derivative...

[0054] A 3D point cloud is constructed using pixel-level mask constraints generated by the prompt segmentation module. Statistical filtering and percentile distance output (e.g., the first 5%~15%) completely solve the problem of "overestimation of distance" caused by the center point's perspective into the background area. Experiments show that its average absolute error in distance measurement of irregular obstacles (such as bent-over farmers and utility poles) has been reduced from over 100 centimeters in traditional methods to less than 10 centimeters, significantly improving the active safety of agricultural machinery operations.

[0055] Secondly, in terms of computational efficiency, the spatiotemporal sparse inference module (based on inter-frame difference and gradient saliency activation block selection, key frame feature reuse, and spatiotemporal memory decay mechanism) fully utilizes the video stream characteristics of field scenes, significantly reducing convolution calculations in invalid background areas. Combined with a lightweight segmentation decoder (based on Tiny-ViT architecture and knowledge distillation), it enables complex perception algorithms to achieve high frame rate real-time operation on low-power edge devices (measured average frame rate of 34.8 FPS, a 54.6% improvement over full inference), meeting the real-time requirements of high-speed agricultural machinery operation (>1.5 m / s).

[0056] Furthermore, in terms of practicality and cost, the entropy-guided prompting strategy of this invention (calculating pixel-level Shannon entropy within the detection bounding box and automatically selecting high-entropy boundary points and low-entropy center points to form composite prompts) directly utilizes the weakly supervised information in the detection stage to drive the segmentation decoder, without requiring any manual pixel-level data annotation. This greatly reduces the threshold for model training and scene adaptation, making rapid deployment for different farmland environments possible.

[0057] Finally, regarding system reliability, the dual-power supply switching circuit (supporting flexible switching between the vehicle battery and the built-in battery, in conjunction with a wide-voltage regulator module) effectively suppresses voltage fluctuations caused by the start-up of the agricultural machinery engine, preventing the main control unit from restarting or data loss; the CAN bus control arbitration circuit based on the relay unit (adopting a common-ground single-wire topology, with the normally closed terminal of the relay connected to the computing core and the normally open terminal connected to the external navigation terminal interface) realizes hardware-level logical isolation and "fail-safe" switching between local obstacle avoidance mode and global path tracking mode, completely avoiding the risks of delay, deadlock, or system crash that may be caused by pure software switching, providing a solid engineering foundation for the industrial application of unmanned agricultural machinery. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the overall process architecture of a field dynamic obstacle detection, segmentation, and precise ranging method according to an embodiment of the present invention.

[0059] Figure 2 This is a hardware electrical architecture block diagram of a field dynamic obstacle detection, segmentation, and precise ranging device according to an embodiment of the present invention.

[0060] Figure 3 This is a schematic diagram illustrating the working principle of the spatiotemporal sparse inference controller in this invention.

[0061] Figure 4 This is a schematic diagram of the internal network structure of the feature alignment module in this invention.

[0062] Figure 5 This is a flowchart illustrating the generation process of the entropy-guided prompting strategy in this invention.

[0063] Figure 6 This is a visualization of the Shannon entropy calculation in this invention.

[0064] Figure 7 This is a visualization of the box / dot prompt results in this invention.

[0065] Figure 8 This is a network architecture diagram of the lightweight prompt guidance segmentation decoder in this invention.

[0066] Figure 9 This is a comparison chart of the effectiveness of the method of this invention and traditional target detection methods. Detailed Implementation

[0067] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention. The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment takes an intelligent obstacle avoidance system mounted on an unmanned combine harvester as an example, but the present invention is also applicable to agricultural automation equipment such as orchard plant protection robots and rice transplanters.

[0068] I. Hardware Installation and Electrical Architecture Implementation Details

[0069] like Figure 2 As shown, the present invention provides a field dynamic obstacle detection, segmentation and precise ranging device, which adopts a modular design integrating hardware and software, and consists of a sensing module, a computing processing core, a power supply and control management module and an underlying execution mechanism.

[0070] 1. Sensing Module and Computing Processing Core

[0071] The core computing power utilizes an NVIDIA Jetson Orin NX 16GB embedded AI computing module. Based on the Ampere architecture GPU, this module provides up to 100 TOPS of AI computing power, meeting the edge inference requirements of deep learning models. The core computing power is extended with a WiFi / Bluetooth module via an M.2 Key E interface for debugging and data backhaul.

[0072] Perception Module: Employs a Stereolabs ZED 2i binocular depth camera. This camera has an IP67 protection rating and integrates an IMU (Inertial Measurement Unit), magnetometer, and barometer. The camera connects to the computing core via a USB 3.0 Type-A interface and is secured with screws to prevent it from falling off due to vibration. In this embodiment, the camera is configured in HD720 resolution mode with a frame rate of 10 FPS to acquire low-latency RGB images and depth information.

[0073] 2. Dual power supply switching circuit in the power supply and control management module

[0074] To address the harsh power environment and ease of maintenance requirements for agricultural machinery operating in the field, this device's power supply and control management module implements a dual-power supply switching circuit. This circuit abandons complex electronic automatic switching schemes and adopts a mechanical multi-source power supply architecture based on physical isolation. Specifically, the device has an external 3-position 6-pin double-pole double-throw (DPDT) heavy-duty toggle switch as the main power arbiter, i.e., the "switching unit." This switch has a three-stage "ON-OFF-ON" logic: when the lever is in the upper position (built-in power supply mode), the internal contacts are closed, and the device is powered by the built-in 4S lithium battery pack (14.8V). This mode is specifically used for data export, algorithm debugging, or portable mobile testing when the agricultural machinery is off; when the lever is in the middle position, the physical circuit is completely disconnected, achieving safe power-off of the entire machine; when the lever is in the lower position (external power supply mode), the device draws power directly from the agricultural machinery's onboard battery (12V-36V) via an aviation connector, suitable for long-term continuous field operations.

[0075] Regulated Output: Following the power switching circuit, this device connects an industrial-grade high-power DC-DC step-down regulator module (model: BM-DPS3512-5A) in series as the core power conditioning unit, i.e., the "voltage regulator circuit". This module has a wide DC input voltage capability from 5V to 32V, effectively withstanding input voltage surges caused by changes in the engine speed of agricultural machinery; its output provides a high-precision 12V DC voltage (deviation controlled within ±0.2V), with a maximum rated power of 60W, and can provide a continuous current output of up to 5A. This high-current redundancy design not only meets the peak power consumption requirements of the Jetson Orin NX computing core when performing high-load deep learning inference, preventing system restarts due to insufficient instantaneous power supply, but also its high conversion efficiency of up to 95% and built-in overcurrent, overtemperature, and short-circuit protection mechanisms, combined with an all-aluminum alloy heat dissipation shell, effectively ensure the long-term operational stability of the device in high-temperature and dusty field environments. The above structure together constitutes the dual-power supply switching circuit.

[0076] 3. Control switching circuit in the power supply and control management module

[0077] At the execution control level, to ensure absolutely reliable logical isolation and smooth switching between the "global path tracking" and "local emergency obstacle avoidance" modes for agricultural machinery, this device is designed with a control control switching circuit based on physical link interruption. This embodiment specifically adopts a common-ground single-wire switching topology to simplify wiring and reduce the risk of signal interference. The control control switching circuit uses a 3.3V driven single-channel electromagnetic relay module, i.e., a "relay unit." This module integrates an optocoupler isolation circuit and can be directly driven by the 3.3V logic level of the Jetson Orin NX.

[0078] For the physical connection of the CAN bus, a connection strategy of "CAN-Low common line, CAN-High switching" is adopted. The CAN-Low signal lines of the agricultural machinery's underlying controller (VCU), the external navigation terminal (such as an Android tablet running a global navigation algorithm), and this device (JetsonOrin NX) are permanently shorted together to form a common reference potential. The critical CAN-High signal line is physically selected through a relay: the CAN-High pin of the agricultural machinery's underlying controller is connected to the common terminal (COM) of the relay; the CAN-High pin of the external navigation terminal is connected to the normally open terminal (NO) of the relay; and the CAN-High pin of this device, JetsonOrin NX, is connected to the normally closed terminal (NC) of the relay.

[0079] This wiring logic forms the "default local priority" safety strategy, which is the hardware basis for the first control state (local obstacle avoidance mode): When the relay coil is not energized or the control signal is low, the common terminal COM is connected to the normally closed terminal NC by default. At this time, the chassis control is physically connected to the Jetson master controller, ensuring that the vehicle can immediately respond to local safety braking commands in the initial stage of system power-on or in the event of an abnormal reset. When the system enters the normal global navigation operation process, the Jetson master controller outputs a high level to drive the relay to engage, the COM terminal switches to the NO terminal, and the chassis control is transferred to the external navigation terminal, thus entering the second control state (global path tracking mode). Once the vision algorithm detects an obstacle in the field that is less than the preset safety threshold (e.g., 6 meters), the Jetson master controller will immediately flip the GPIO level, release the relay, and cause the COM terminal to instantly bounce back to the NC terminal, physically cutting off the signal path of the external navigation terminal, thereby forcibly taking over the chassis for obstacle avoidance and restoring the first control state.

[0080] To achieve the aforementioned microsecond-level relay control, this device utilizes pin 31 (physical board pin 31) of the Jetson Orin NX's 40-pin expansion interface as a dedicated control signal output. This pin corresponds to GPIO number PQ.06 (GPIO3_PQ.06) within the Tegra SoC, with a register base address of 0x02430000 (PADCTL_A0) and an offset of 0x70. In terms of software driver implementation, this embodiment abandons the traditional sysfs file system control method, instead employing a more efficient approach of directly manipulating registers or calling the libgpiod library to achieve low-latency control in JetPack 6.x and later kernel versions. The specific control logic is as follows: First, during the initialization phase, the system configures the PQ.06 pin multiplexing function to GPIO output mode by writing to the register (or using the instruction `sudo busybox devmem 0x2430070 w 0x004`). When switching to global navigation mode is required, the algorithm calls the `gpiod_line_set_value` function to write a logic "1" (high level) to the pin, and the relay is activated. When emergency obstacle avoidance needs to be triggered, the algorithm writes a logic "0" (low level), and the relay is deactivated. This direct drive method based on the underlying hardware address avoids the scheduling delay of the upper-level operating system, ensuring that the response time for switching control is controlled within milliseconds at the moment an obstacle is detected, thereby effectively ensuring the active safety of agricultural machinery under high-speed operation.

[0081] II. Implementation Steps and Mathematical Principles of the Algorithm

[0082] The method flow of the present invention is as follows: Figure 1 As shown, the specific steps include:

[0083] Step S1: Data Acquisition and Refined Preprocessing

[0084] Image pyramid construction:

[0085] To accommodate obstacles of varying scales (such as distant pedestrians and nearby farm implements), the input resolution is [not specified]. The original image Gaussian pyramid downsampling is performed. In this embodiment, scaling is selected to... As network input resolution Scaling uses the Letterbox method (maintaining aspect ratio, with the shorter side filled with gray pixels), and the transformation formula is:

[0086]

[0087] scaling factor .

[0088] Bilateral filtering for noise reduction:

[0089] To address field dust and light noise, bilateral filtering is performed on the A and B channels of the LAB color space. For each pixel... its filtered value The calculation is as follows:

[0090]

[0091] in: It is a spatial domain Gaussian kernel used to reduce spatial noise. The value is 15; It is a pixel-level Gaussian kernel used to preserve edge gradients. The value is 0.1 (after normalization). It is the normalization coefficient.

[0092] This step effectively smooths the field texture (such as grass texture) while preserving the sharp edges of obstacles, providing clear boundaries for subsequent segmentation.

[0093] Step S2: Spatiotemporal Sparse Inference and Keyframe Selection

[0094] To achieve high frame rates (>30 FPS) on the Jetson Orin NX computing core, a spatiotemporal sparse inference module was designed. Figure 3 ).

[0095] Mesh generation and feature calculation:

[0096] Divide the image into Grid blocks, each block is sized as follows: Pixel. For the first The first frame Block Calculate two indicators:

[0097] Temporal Difference:

[0098]

[0099] Gradient saliency:

[0100]

[0101] in, The horizontal and vertical gradients are calculated for the Sobel operator.

[0102] Activation status determination:

[0103] Determine all blocks in the current frame Maximum value and Maximum value Construct the activation scoring formula:

[0104] ,in .

[0105] Set dynamic threshold .like The block is marked as active; otherwise, it is marked as inactive. A safety fallback mechanism is introduced: a full frame refresh is forced every preset number of frames, or a full inference rollback is triggered when the detection confidence falls below a threshold.

[0106] Sparse convolution and memory decay:

[0107] For active blocks, perform full convolutional feature extraction; for dormant blocks, directly copy the feature vector from the corresponding position of the previous keyframe. .

[0108] Spatiotemporal memory decay mechanism: A state retention counter is maintained. If a block is determined to be active, the counter is set to a preset retention frame number (e.g., 5 frames). If activation is not triggered again in subsequent frames, the counter is decremented until it reaches zero, at which point the block is put into a dormant state. Specifically, to prevent the short-term stagnation of moving targets from being missed, a state matrix is ​​maintained. If the block is activated, set (Keep 5 frames); if it becomes still, each frame It only truly enters hibernation when its value reaches 0.

[0109] Step S3: Feature Alignment and Detection

[0110] An improved YOLOv8 backbone network is adopted, and a feature alignment and detection module (FAM) is introduced. Figure 4 Solve the problem of misalignment in multi-scale fusion.

[0111] The principle of Deformable Convolution:

[0112] Standard convolution on a regular grid Upsampling, while deformable convolution introduces an offset. For the position on the output feature map Its value is calculated as follows:

[0113]

[0114] in, From the input features by a bypass convolutional layer I learned this in school. Because... The coordinates are usually non-integer coordinates, and bilinear interpolation is used to calculate the pixel values.

[0115]

[0116] in It is a two-dimensional bilinear interpolation kernel.

[0117] Detection output:

[0118] The aligned feature layer outputs the prediction result, including A preliminary boundary box .

[0119] Step S4: Generating Entropy-Driven Hints

[0120] This is the core of the invention for achieving label-free segmentation. Figure 5 ).

[0121] Shannon entropy calculation ( Figure 6 ):

[0122] For the detection box Each pixel within The network outputs a class probability vector. , where C is the total number of categories. Calculate its Shannon entropy:

[0123]

[0124] Low Entropy: Indicates that the model is extremely confident in classifying the point, usually located in the center of the object where the texture is clear.

[0125] High entropy: indicates that the model is uncertain, usually located at the edge of an object or at an occluded area.

[0126] Tip-based filtering strategy ( Figure 7 ):

[0127] Positive Points: Sort all pixels within the ROI by entropy value from smallest to largest, and select the first... These points are highly likely to be obstacles in the foreground.

[0128] Uncertainty Points: Select the point with the highest entropy. These points indicate the ambiguous regions that the segmentation network needs to focus on.

[0129] Composite hint: Combine the coordinates of the above point set With bounding box Concatenate to form a hint vector .

[0130] Step S5: Lightweight Segmentation and Mask Generation

[0131] Input the composite prompt into the lightweight segment decoder ( Figure 8 This decoder is based on the Tiny-ViT architecture and learns weights from a large SAM (Segment Anything Model) model through knowledge distillation. The decoder utilizes a cross-attention mechanism, using the cue vector as the query and image features as the key and value, with an output resolution of [resolution value missing]. binary mask .

[0132] Step S6: Robust Ranging Based on Masks

[0133] like Figure 9 As shown, this method addresses the shortcomings of traditional center-point distance measurement.

[0134] Depth back projection:

[0135] Using camera intrinsic matrix All pixels within the mask area Three-dimensional points back-projected into the camera coordinate system :

[0136]

[0137] Only keep The pixels form the original point cloud set. .

[0138] Statistical outlier removal (SOR):

[0139] Calculate the distance from each point in the point cloud to its corresponding point. The average distance between the nearest neighbors Calculate the global mean. and standard deviation Remove points that satisfy the following condition:

[0140]

[0141] The purified point cloud set .

[0142] Robust distance calculation:

[0143] To obtain the safest obstacle avoidance distance (i.e., the point where the obstacle is closest to the agricultural machinery) and to avoid the influence of noise, the minimum value is not directly taken. Instead, the depth value at the 5th percentile is used as the final output distance. :

[0144]

[0145] In other embodiments, the percentile can be selected as any value between 5% and 15% to balance the robustness and sensitivity of the ranging.

[0146] Step S7: Obstacle avoidance control execution

[0147] Real-time monitoring of computing processing core Obstacle avoidance is performed via a control switching circuit.

[0148] Specifically, the edge controller Jetson Orin NX provides real-time monitoring. :

[0149] like When the GPIO output is low, the relay is disconnected, maintaining the second control state (global path tracing mode).

[0150] like When the GPIO outputs a high level, the relay is activated, the computing core takes over the chassis, and sends a "enter obstacle avoidance mode" command (e.g., speed limit is 0.5m / s), entering the first control state (partial obstacle avoidance mode).

[0151] like When the GPIO outputs a high level, the relay is activated, and the computing core sends an "Emergency Stop (E-Stop)" command.

[0152] III. Experimental Results and Comparative Analysis

[0153] To verify the effectiveness of this invention, a test dataset containing 3,000 images was collected in a real cornfield and orchard environment, including obstacles such as people, tractors, utility poles, and wells. The test platform was a computing processing core (NVIDIA Jetson Orin NX).

[0154] Distance measurement accuracy comparison:

[0155] The method of this invention is compared with the traditional "YOLOv8 detection frame + center point distance measurement" method, as shown in Table 1:

[0156] Table 1 Comparison of Distance Measurement Accuracy

[0157]

[0158] Conclusion: When facing regular objects (upright pedestrians), the accuracy of this invention is improved by about 3-4 times; however, when facing irregular objects (bending farmers, utility poles), traditional methods result in huge errors of several meters due to the center point falling into the background. In contrast, this invention uses the mask generated by the entropy-guided segmentation module for distance measurement, which stably controls the error within 10cm, improving the positioning accuracy by more than 90% and completely eliminating the safety hazards caused by background perspective.

[0159] Comparison of real-time performance and resource consumption:

[0160] Table 2 shows a comparison of the performance of enabling and disabling the "spatial-temporal sparse inference module":

[0161] Table 2 Comparison of Real-Time Performance and Resource Consumption

[0162]

[0163] Conclusion: By introducing a spatiotemporal sparse inference module, this invention successfully achieved real-time detection of over 30 FPS on the computing core (Jetson Orin NX) and significantly reduced power consumption, making it more suitable for battery-powered agricultural machinery applications.

[0164] This invention aims to solve the problems of large distance measurement error of the center point of traditional target detection boxes, strong background interference, limited computing power of edge devices, and unreliable hardware control switching in unstructured field environments.

[0165] In terms of methodology, the real-time video stream is first preprocessed with adaptive resolution adjustment and multi-dimensional denoising. Then, a sparse inference controller is used to select activation blocks based on inter-frame difference and gradient saliency, and a periodic keyframe refresh mechanism is combined to reduce invalid convolution calculations. Next, an entropy-guided cueing strategy is introduced to calculate pixel-level Shannon entropy within the detection bounding box, and high-entropy boundary points and low-entropy center points are selected to form a composite cue, driving a lightweight segmentation decoder to generate dynamic obstacle masks. Finally, the mask is used to constrain depth back projection to construct a 3D point cloud, and the accurate distance is output through robust statistics.

[0166] In terms of the device, a modular design is adopted, using a Jetson Orin NX as the embedded master controller, and connecting to a ZED 2i binocular camera via a USB interface to acquire environmental perception data. The device integrates a dual power supply and control switching circuit: the power supply section includes a power switching switch and a voltage regulator module, supporting flexible switching between the built-in battery and an external 12V vehicle battery; the control communication section is equipped with a relay switching module, used to switch the main body sending control commands in CAN bus communication, realizing hardware-level logical isolation and smooth switching between local obstacle avoidance mode (master controller sending commands) and global path tracking mode (external Android tablet sending commands). Through innovative hardware and software integrated design, this invention, while ensuring real-time performance at the edge, fundamentally solves the problem of inaccurate distance measurement of complex obstacles in the field by utilizing entropy-guided cue segmentation and mask ranging technology, and ensures the underlying control security of the system through the control switching circuit (hardware-level physical isolation based on relay units).

[0167] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0168] The detailed descriptions listed above are merely specific illustrations of feasible embodiments of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A field dynamic obstacle detection, segmentation, and precise ranging device, characterized in that, It includes a sensing module, a computing processing core, and a power and control management module; The sensing module is used to collect RGB images and depth information of the field environment, and the sensing module is connected to the computing and processing core. The computing core uses an embedded AI computing module as the main control unit. The computing core is equipped with a spatiotemporal sparse inference module, a feature alignment and detection module, an entropy-guided prompting segmentation module, and a 3D point cloud ranging module. The spatiotemporal sparse inference module is used to perform inter-frame difference and saliency filtering on the input video stream, perform convolution calculation only on the active region, and output the generated sparse feature map to the feature alignment and detection module. The feature alignment and detection module receives the sparse feature map, extracts multi-scale features, and performs feature alignment using deformable convolution to generate a preliminary bounding box and category probability map of the dynamic obstacle. The preliminary bounding box is then passed to the entropy-guided cue segmentation module. The entropy-guided cue segmentation module calculates pixel-level Shannon entropy within the region of interest defined by the received preliminary bounding box, selects high-entropy boundary points and low-entropy center points as cue points based on the entropy value, generates a pixel-level segmentation mask, and sends it to the 3D point cloud ranging module. The three-dimensional point cloud ranging module is used to perform mask filtering on the depth map based on the received segmentation mask, back-project the effective pixels within the mask to the three-dimensional space to construct a point cloud, and calculate the distance to obstacles after removing outliers. The power supply and control management module is connected between the computing core, the external power supply, and the underlying actuators of the agricultural machinery. The module includes a dual-power supply switching circuit and a control switching circuit. The input of the dual-power supply switching circuit is connected to both the external vehicle power interface and the built-in battery interface, and is connected to a voltage regulator circuit via a switching unit. The output of the voltage regulator circuit powers the computing core. The control switching circuit includes a relay unit and signal switching logic. The relay unit is connected in series in the CAN bus communication link and is used to switch the sending entity of control commands between a first control state and a second control state. The first control state is a local obstacle avoidance mode, where the relay is in the off state to connect the communication link between the computing core and the underlying actuators, and the computing core sends control commands. The second control state is a global path tracking mode, where the relay is in the energized state to connect the external interface used to connect the external navigation terminal to the underlying actuators, shielding the control commands from the computing core.

2. The field dynamic obstacle detection, segmentation, and precise ranging device according to claim 1, characterized in that, The perception module uses a ZED 2i binocular depth camera; the computing processing core uses an NVIDIA Jetson Orin NX module. The external navigation terminal is an Android tablet or industrial control computer running a global path planning algorithm; The dual-power supply switching circuit is configured such that when the external power circuit is closed, the vehicle battery provides power; when the external power circuit is disconnected and the internal circuit is closed, the built-in lithium battery pack provides power.

3. The field dynamic obstacle detection, segmentation, and precise ranging device according to claim 1, characterized in that, The control switching circuit adopts a common-ground single-wire switching topology: The CAN-Low signal lines of the agricultural machinery's underlying controller, external navigation terminal, and computing core are permanently shorted. The common terminal of the relay is connected to the CAN-High pin of the agricultural machinery's underlying controller, the normally open terminal of the relay is connected to the CAN-High pin of the external navigation terminal, and the normally closed terminal of the relay is connected to the CAN-High pin of the computing processing core. When the computing core outputs a low level, the relay remains in the normally closed state (i.e., open state), and control belongs to the computing core; when the computing core outputs a high level, the relay is energized, and control is switched to the external navigation terminal.

4. A method for detecting, segmenting, and accurately measuring dynamic obstacles in a field using the field dynamic obstacle detection, segmentation, and precise ranging device according to any one of claims 1-3, characterized in that, Includes the following steps: Step S1: Obtain real-time video stream image sequences and synchronized depth information of the field environment; Step S2: Perform resolution adaptive adjustment and multi-dimensional denoising preprocessing on each frame of image to construct a multi-scale image pyramid; Step S3: Use the sparse inference controller to perform spatiotemporal saliency region screening on the multi-scale image pyramid to generate a set of active state blocks; determine whether the current frame is a key frame based on the inter-frame temporal correlation, reuse the features of the previous key frame for non-key frames and perform convolution calculation only on newly added active state blocks; Step S4: Input the sparse features into the shared backbone network containing the feature alignment module, use deformable convolution to perform feature alignment, and output the preliminary bounding box and class probability map of the dynamic obstacle. Step S5: For each preliminary bounding box, calculate the pixel-level information entropy distribution within its corresponding region of interest. Based on the entropy value, automatically select high-entropy boundary points and low-entropy center points as point cues, and combine them with the bounding box to form a composite cues. Input the composite cues into the lightweight segmentation decoder and output the pixel-level segmentation mask of the dynamic obstacles. Step S6: Use the segmentation mask to perform mask filtering on the depth map, back-project the effective pixels in the mask to the three-dimensional space to construct a point cloud, remove outliers, and then use robust statistics to calculate the obstacle distance. Step S7: Based on the calculated obstacle distance, the calculation processing core determines whether to trigger obstacle avoidance logic; If triggered, the control switching circuit ensures that the system is in the first control state and sends an obstacle avoidance command.

5. The field dynamic obstacle detection, segmentation, and precise ranging method according to claim 4, characterized in that, The sparse inference controller in step S3 specifically includes: The input frame is divided into N×N grid blocks; Calculate the sum of gradient magnitudes G for each grid block and the temporal difference magnitude T between the current grid block and the previous frame; Constructing an activation scoring formula ,in, It is the maximum value of the temporal difference amplitude of all blocks in the current frame. This represents the maximum sum of gradient magnitudes across all blocks in the current frame. These are the weighting coefficients; if Greater than the set threshold If a block is marked as active, it is marked as dormant. Introducing a safety fallback mechanism: Force a full frame refresh every preset number of frames, or trigger a full inference rollback when the detection confidence is lower than the threshold.

6. The method for detecting, segmenting, and accurately measuring dynamic obstacles in the field according to claim 4, characterized in that, The sparse inference controller in step S3 is also equipped with a time-space memory decay mechanism: Maintain a state hold frame counter for each grid block. When a block is determined to be active, the counter is set to the preset hold frame number. If activation is not triggered again in each subsequent frame, the counter is decremented until it reaches zero before the block is put into a dormant state.

7. The method for detecting, segmenting, and accurately measuring dynamic obstacles in the field according to claim 4, characterized in that, The entropy-guided prompt generation strategy in step S5 is as follows: Calculate the pixel Shannon entropy within the region of interest. Where C is the total number of categories, and c is the category index. For pixels ( The predicted probability of belonging to category c; Center point prompt: Select The smallest K pixels represent the high-confidence foreground; Boundary point hint: Select The top M largest pixels represent the edge of uncertainty; The coordinates of the points mentioned above are concatenated with the coordinates of the bounding box and used as geometric constraints to input into the segmentation network.

8. The method for detecting, segmenting, and accurately measuring dynamic obstacles in the field according to claim 4, characterized in that, The logic for precise ranging in step S6 is as follows: Using camera intrinsic matrix pixel coordinates within the mask With depth value Back projection yields three-dimensional coordinates ,in This represents the inverse of the camera intrinsic parameter matrix, and the superscript T indicates the matrix transpose; Perform statistical filtering on the generated point cloud set to remove outlier noise points whose distance from the mean exceeds a multiple of the standard deviation. Sort the depth values ​​of the remaining valid point cloud and take the distance values ​​of the top 5% to 15% as the final obstacle distance.

9. The method for detecting, segmenting, and accurately measuring dynamic obstacles in the field according to claim 4, characterized in that, The preprocessing in step S2 includes: The image is scaled to the preset network input resolution using the Letterbox method, which maintains the aspect ratio. Convert the image from RGB color space to HSV or LAB color space; Bilateral filtering is used to smooth the image, removing field environmental noise while preserving obstacle edge gradient information.

10. The method for detecting, segmenting, and accurately measuring dynamic obstacles in the field according to claim 4, characterized in that, The lightweight segmentation decoder is based on the Tiny-ViT architecture. It learns weights from the Segment Anything Model through knowledge distillation and uses a cross-attention mechanism with composite prompts as queries and image features as keys and values ​​to output a binary segmentation mask.

Images