Agricultural picking visual monitoring method

Abstract

A kind of agricultural picking visual monitoring method, based on YOLOv11, BN is replaced by DyTb to construct fruit detection model;Picking robot sets up acquisition equipment to collect field original image and standardization processing, obtains standardization field image;According to the real-time motion data of picking robot, the local feature gain compensation is carried out to the standardization field image, generates the field image after compensation and inputs fruit detection model, carries out feature extraction, multi-scale fusion and target preliminary prediction, obtains original detection frame;The spatial topological verification position rationality is carried out to original detection frame, and the candidate target frame set is formed by combining equipment state and target priority screening, obtains the final visual detection output containing fruit category label, spatial positioning coordinates by coordinate decoding, non-maximum suppression.The DyTb of the application is used to replace BN, can be seamlessly embedded in a variety of mainstream detection architecture, provides a set of high practicality solution for intelligent agricultural picking robot.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of artificial intelligence, computer vision, and intelligent agricultural equipment, and in particular to a target visual monitoring method based on an improved dynamic activation function applied to agricultural harvesting robots. Background Technology

[0002] With the rapid development of smart agriculture, harvesting robots have become core equipment for alleviating labor shortages and improving operational efficiency. Visual inspection, as the core of robot perception, directly determines positioning accuracy, recognition robustness, and operational reliability.

[0003] Currently, YOLO (You Only Look Once) series deep learning models are widely used, with Batch Normalization (BN) to improve training stability. However, this approach has significant drawbacks in complex agricultural scenarios. First, because BN relies on batch statistics, statistical bias is severe in small batches, leading to large fluctuations in inference accuracy and making it difficult to adapt to embedded platforms such as Raspberry Pi and Jetson. Second, strong light, backlight, shadows, occlusion, and robotic arm motion blur can easily cause feature distortion, parameter oscillation, and false positives and false negatives. Third, when multiple robots operate in parallel, there is no parameter sharing mechanism, resulting in random repetitive calculations, slow environmental adaptation, and low collaborative efficiency. Fourth, visual models are prone to outputting high false positives in non-fruit areas such as the sky and ground, lacking a spatial topology blacklisting mechanism and physical rule constraints. Fifth, under low power and high load conditions, computing power cannot be allocated as needed, compromising the recognition of core targets.

[0004] The existing Dynamic Tanh-base (DyTb) improvement scheme only replaces the normalization layer and does not solve the problems of distributed collaboration, motion fuzzing, physical rule filtering and resource scheduling, making it difficult to meet the real-world harvesting requirements of multi-machine collaboration, high-speed dynamics and low power consumption at the edge. Summary of the Invention

[0005] (I) Purpose of the invention: In order to solve the problems existing in the prior art, the purpose of this invention is to provide a visual monitoring method for agricultural harvesting, based on YOLOv11-DyTb, to achieve multi-machine collaborative rapid adaptation, stable motion recognition, suppression of false detection, and low power consumption to protect the core.

[0006] (II) Technical Solution: To solve the above-mentioned technical problems, this technical solution provides a visual monitoring method for agricultural harvesting, including the following steps: Based on YOLOv11, the batch normalization layer was replaced with the DyTb normalization layer to build a fruit detection model; The harvesting robot is equipped with a data acquisition device to collect original field images, which are then processed to obtain standardized field images. Based on the real-time motion data of the harvesting robot, local feature gain compensation is applied to the standardized field images to generate compensated field images. The compensated field image is input into the fruit detection model for feature extraction, multi-scale fusion and preliminary target prediction to obtain the original detection box containing all potential fruits. The original detection boxes are spatially topologically verified for location rationality, and combined with device status and target priority for screening, ultimately forming a set of candidate target boxes; The candidate target box set is processed by coordinate decoding and non-maximum suppression to obtain the final visual detection output containing fruit category labels and spatial positioning coordinates.

[0007] Preferably, the DyTb normalization layer is embedded in the backbone network, neck network and detection head of YOLOv11 to form a structure of convolutional layer conv, dynamic hyperbolic tangent base normalization layer DyTb and SiLU activation function; The dynamic hyperbolic tangent base normalization layer , where x is the feature map of the previous convolutional layer; tanh(αx) is the hyperbolic tangent function; α is the curvature parameter with an initial value of 1.0; γ is the scaling parameter with an initial value of 1.0; β is the translation parameter with an initial value of 0.0.

[0008] Preferably, the harvesting robot is equipped with a data acquisition device to collect raw field images, which are then standardized to obtain standardized field images. Based on the real-time motion data of the harvesting robot, local feature gain compensation is performed on the standardized field images to generate compensated field images, including: The harvesting robot is equipped with a data acquisition device. Through the image acquisition device, it can collect images in the field under different lighting conditions, different postures, and different occlusion conditions, and obtain original field images containing fruits, branches, leaves, and backgrounds. The collected raw field images were preprocessed to obtain standardized field images; Based on the real-time motion data of the harvesting robot, local feature gain compensation is performed on the standardized field image to generate a compensated field image.

[0009] Preferably, the width-to-height ratio of the original field image is calculated; if the aspect ratio is not equal to 1, the original field image is scaled, the long side of the original field image is scaled to the target size fixed input by the fruit detection model, the short side is scaled by the same ratio, the scaled image is placed in the center of the canvas, the left and right sides are uniformly filled with background color, and the pixel normalization value is mapped to [0, 1] or [-1, 1] according to the pixel normalization value = original pixel value / 255 to obtain a standardized field image of channel number × width × height.

[0010] Preferably, the harvesting robot has a six-axis IMU sensor installed at the end of the robotic arm or on the camera gimbal to acquire posture motion data and / or robotic arm motion command data in real time, and calculate the instantaneous angular velocity amplitude and linear velocity amplitude at the moment of image acquisition. When the angular velocity and / or linear velocity are greater than or equal to the preset angular velocity and / or linear velocity motion blur threshold, the original field image of the current frame has a risk of motion blur. Region of Interest (ROI) for each frame of the original field image i Applying a Gaussian gain inside it yields the gain mask. , among which, (u x u y () represents the center point of the region of interest (ROI); ; A represents the gain amplitude. The original standardized field image is multiplied pixel by pixel with the gain mask to obtain the compensated field image.

[0011] Preferably, the compensated field image first enters the backbone network of the fruit detection model. After passing through each layer, the size of the feature map is gradually reduced while the number of channels gradually increases. Finally, three feature maps of different scales are output and enter the neck network of the fruit detection model. The neck network adopts the PANet path aggregation network structure, which includes two fusion paths: top-down and bottom-up. After bidirectional fusion, the three feature maps are output and sent to three detection heads respectively, resulting in three feature tensors with a specific number of channels. The number of channels of the feature tensor = (5 + number of categories) × number of anchor boxes per grid point. The feature tensors of the three scales are merged to obtain the original detection boxes containing all potential fruits.

[0012] Preferably, the global curvature adjustment coefficient is calculated based on the current motion intensity. When motion blur exists, for each DyTb normalization layer in the fruit detection model, When there is no motion blur, λ takes the value of 1.

[0013] Preferably, when multiple harvesting robots are working collaboratively, each harvesting robot is equipped with an environment context manager to collect, cache, broadcast, and apply DyTb parameters. After the preliminary picking robot completes the stable detection of a frame image, the environment context manager extracts the dynamic parameters and corresponding metadata of all DyTb normalized layers from the fruit detection model to obtain the parameter broadcast packet. When the illumination intensity parameter of the following harvesting robot exceeds a preset first intensity threshold, the environment context manager checks if the current GPS coordinates match the parameter effective region of any cached entry, and if the timestamp of the cached entry is within the validity period. The environment context manager directly reads the DyTb parameter corresponding to the spatial region from the cache table and assigns it to the corresponding DyTb layer in the fruit detection model. When the illumination intensity parameter is less than the preset first intensity threshold but exceeds a preset second intensity threshold, the environment context manager checks if the current GPS coordinates match the parameter effective region of any cached entry, and if the timestamp of the cached entry is within the validity period. The environment context manager performs an exponentially weighted moving average of the default DyTb parameter of the current fruit detection model and the DyTb parameter corresponding to the spatial region read from the cache table. When the illumination intensity parameter is less than the preset second intensity threshold, the default DyTb parameter of the fruit detection model is still used. After the parameters are updated, the subsequent harvesting robot uses the updated fruit detection model to process the current frame.

[0014] Preferably, the cache table is indexed by coordinate region, and each cache entry in the cache table includes a region identifier, a DyTb parameter set, an update timestamp, and the source harvesting robot ID; For each cache entry in the cache table, the effective region of the parameters is divided according to the spatial coordinates and the preset radius.

[0015] Preferably, the original detection boxes undergo spatial topology verification to determine their location rationality, and are then filtered based on device status and target priority to ultimately form a set of candidate target boxes, including: A preset verification whitelist is used to verify the rationality of spatial locations. By combining device status and target priority, a set of candidate target boxes is obtained.

[0016] Preferably, the verification whitelist is a binary mask image with the same size as the input image, where pixels with a value of 1 represent allowed detection areas and pixels with a value of 0 represent prohibited detection areas; The coordinates of the center point or four corner points of the original detection box are reverse-mapped to the actual pixel coordinates of the original field image. The value of the corresponding position in the whitelist is queried. If the value is 0, the original detection box is discarded; if the value is 1, the original detection box is retained.

[0017] Preferably, the candidate target box set is subjected to coordinate decoding and non-maximum suppression to obtain the final visual detection output containing fruit category labels and spatial positioning coordinates, including: Coordinate decoding is performed on each entry in the candidate target box set to obtain the position and size of each candidate target box in the compensated field image; For each candidate bounding box after coordinate decoding, calculate the probability of target existence based on its position and size in the compensated field image; Overlapping detection boxes are removed by nonmaximum suppression to obtain the final set of detection boxes; The final set of detection boxes is inversely mapped to obtain the final visual detection output.

[0018] Preferably, the center point offset is added to the row and column indices of the grid, and then multiplied by the downsampling step size of the feature map relative to the compensated field image to obtain the pixel coordinates of the bounding box center point on the compensated field image. The upper left corner coordinate of the bounding box is calculated by subtracting half of the actual width from the center point x-coordinate and half of the actual height from the center point y-coordinate. The lower right corner coordinate is calculated by adding half of the actual width to the center point x-coordinate and half of the actual height to the center point y-coordinate.

[0019] Preferably, the fruit detection model outputs an independent original value P for the presence of the target for each detection box. obj and the original scores P for each category cls ; Calculate the probability of the final detected target's existence = P obj ×P cls .

[0020] Preferably, all detection boxes are sorted from high to low according to the probability of the final detected target. Starting from the detection box with the highest probability of the final detected target, each remaining detection box is traversed, and the intersection-union ratio (IoU) between it and the last detection box in the current retention list is calculated. If the IoU is greater than the intersection-union threshold, the detection box is discarded; if the IoU is less than or equal to the intersection-union threshold, the detection box is retained.

[0021] (III) Beneficial Effects: This invention provides a visual monitoring method for agricultural harvesting. First, it uses a DyTb module to replace the traditional batch normalization layer, which can be seamlessly embedded into various mainstream detection architectures such as YOLOv11, YOLOv8, ResNet, and Transformer. This facilitates migration and deployment between different edge platforms and solves the performance degradation problem caused by small batch sizes on edge devices, providing a highly practical technical solution for intelligent agricultural harvesting robots. Second, DyTb contains only three learnable parameters and dynamically adjusts the feature amplitude to predict the position of the detection box in the previous frame in the current frame and generate a Gaussian local gain mask. This mask is used to specifically enhance areas that may contain fruit, effectively offsetting feature attenuation caused by motion blur, enhancing robustness in complex environments, reducing computational and memory overhead, and meeting real-time harvesting requirements. Furthermore, by using a preset whitelist, the center point of the detection box is compared with the allowed detection area, enhancing decision security. By adjusting the task based on device status and target priority, the effective completion of the task is ensured. Finally, the dynamic parameters of the DyTb normalization layer are designed to enable complete multi-machine collaborative perception and parameter sharing, so that the follower robot does not need to go through dozens of frames of online adaptation process, which greatly improves the collaborative operation efficiency and environmental adaptation speed of multi-robot clusters. Attached Figure Description

[0022] Figure 1 This is a flowchart of the steps of a visual monitoring method for agricultural harvesting according to the present invention; Figure 2 This invention relates to a convolutional layer structure where DyTb replaces BN.

[0023] Figure 3 These are the different curves of tanh(αx) under different α values ​​according to the present invention. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to preferred embodiments. Further details are set forth in the following description to provide a full understanding of the invention. However, the present invention can obviously be practiced in many other ways different from those described herein. Those skilled in the art can make similar extensions and derivations based on actual applications without departing from the spirit of the invention. Therefore, the scope of protection of the present invention should not be limited by the content of this specific embodiment.

[0025] The accompanying drawing is a schematic diagram of an embodiment of the present invention. It should be noted that this drawing is for illustrative purposes only and is not drawn to scale, and should not be construed as limiting the actual scope of protection of the present invention.

[0026] A visual monitoring method for agricultural harvesting, such as Figure 1 As shown, the specific steps include: Step 1: Based on YOLOv11, replace the batch normalization layer with the DyTb normalization layer to build a fruit detection model.

[0027] Step 2: The harvesting robot is equipped with a data acquisition device to collect original field images, which are then processed to obtain standardized field images. Based on the real-time motion data of the harvesting robot, local feature gain compensation is performed on the standardized field images to generate compensated field images.

[0028] Step 3: Input the compensated field image into the fruit detection model to perform feature extraction, multi-scale fusion and preliminary target prediction to obtain the original detection box containing all potential fruits.

[0029] Step 4: Perform spatial topology verification on the original detection boxes to check the rationality of their positions, and combine the device status and target priority to filter and finally form a set of candidate target boxes.

[0030] Step 5: The candidate target box set is subjected to coordinate decoding and non-maximum suppression to obtain the final visual detection output containing fruit category labels and spatial positioning coordinates.

[0031] Step 1 specifically includes: Based on the YOLOv11 detection model, the Batch Normalization (BN) layer after each convolutional layer in YOLOv11 is replaced with Dynamic Tanh-base (DyTb) to obtain the fruit detection model.

[0032] The YOLOv11 detection model, as the latest generation of single-stage detection framework, has efficient end-to-end detection capabilities, including a backbone network, a neck network, and a head network.

[0033] The backbone network is used to extract multi-scale semantic features from the input image, including the CBS module, the C3k2 (Cross Stage Partial with kernel size 2) module, and the terminal SPPF (Spatial Pyramid Pooling Fast) and C2PSA (Convolutional block with Parallel Spatial Attention) modules. The CBS module consists of sequentially connected convolutional layers (conv), batch normalization layers (BN), and the SiLU activation function. The C3k2 module controls its internal structure through the parameter c3k. On server-side applications requiring high accuracy, c3k=True can be set to enhance feature representation. The C3k2 module can use convolutional kernels of different sizes for feature extraction. On resource-constrained edge devices, c3k=False can be set to speed up the process, degenerating into the standard Bottleneck module (C2F). The C3k2 module divides the feature map into two parts: one part undergoes minimal convolutional processing, and the other part is directly connected to the next stage, effectively reducing repetitive gradient information and improving training efficiency. The SPPF module captures multi-scale information in images by cascading multiple pooling operations of different scales, enhancing the model's ability to perceive targets of different sizes. The C2PSA module divides the input feature map into two branches, processes it through a multi-head attention mechanism, and combines channel and spatial information to weighted output the feature responses of important regions, improving the extraction of important regions in small targets or partially occluded objects and enhancing the feature fusion effect in complex scenes.

[0034] The Neck network enhances the multi-scale features output by the Backbone network through upsampling, concatenation, and multi-layer convolution and feature fusion such as C3k2, resulting in three different scale predicted feature maps used for the detection of small, medium, and large targets, respectively. When the Feature Pyramid Network (FPN) is used, it follows a top-down path, passing high-level semantic information to lower levels, thus enhancing the detection capability for small targets. When the Path Aggregation Network (PANet) is used, it follows a bottom-up path, passing low-level positional information to higher levels, thus enhancing the target localization accuracy.

[0035] The detection head performs multi-level target recognition on prediction feature maps at three different scales, outputting target bounding box regression and category prediction results on the prediction feature maps. It includes a CBS module, a DSC module, and a Conv2d module, employing a decoupled design, divided into three independent branches: a classification branch, a detection bounding box branch, and a circular mask branch. The classification branch predicts the target category probability and outputs the number of channels. The detection bounding box branch is used to locate and regress the bounding box coordinates (x, y, w, h). The circular mask branch is only used for instance segmentation when performing a segmentation task.

[0036] This invention addresses the issue of unstable performance of batch normalization (BN) layers in complex agricultural scenarios. It iterates through all convolutional layers in the YOLOv11 network, replacing all BN layers in the C2f and C3k2 modules of the backbone network, the feature fusion convolutional layers in the neck network, and the prediction convolutional layers in the detection head with dynamic hyperbolic tangent base normalization layers (DyTb). Figure 2 As shown, all other structures remain unchanged, resulting in a fruit detection model with an adaptive feature normalization structure that does not depend on batch statistics, thereby improving the generalization ability and environmental adaptability of the fruit detection model.

[0037] The dynamic hyperbolic tangent basis normalization layer DyTb is implemented in an object-oriented manner, containing three trainable parameters, each with a dimension equal to the number of channels in the input feature map. During forward computation, a channel-by-channel nonlinear mapping is directly performed on the input feature map. As shown in the following equation, for each channel of the input feature map, the dynamic hyperbolic tangent basis normalization layer DyTb first multiplies it by a learnable curvature parameter, then performs nonlinear compression using the hyperbolic tangent function, and finally multiplies it by a learnable scaling parameter and adds a learnable translation parameter.

[0038] , Where x is the feature map of the previous convolutional layer; tanh(αx) is the hyperbolic tangent function, which achieves smooth nonlinear compression and suppresses extreme activation values, such as... Figure 3 As shown, tanh(αx) exhibits different compression curves under different α values; α is the curvature parameter, which enables DyTb to adapt to the input amplitude range. The initial value is set to 1.0, which is used to control the compression sensitivity of the tanh function; γ is the scaling parameter, with an initial value set to 1.0, which is used to adjust the output amplitude; β is the translation parameter, with an initial value set to 0.0, which is used to adjust the output center. α, β, and γ are all independently learnable for each channel.

[0039] The final structure consists of a convolutional layer (conv), a dynamic hyperbolic tangent base normalization layer (DyTb), and a SiLU activation function. Specifically, the original two-dimensional convolutional layer (conv) is kept unchanged, and its output channel number is denoted as C. The dynamic hyperbolic tangent base normalization layer (DyTb) is directly connected after the convolutional layer (conv), and the number of features of the dynamic hyperbolic tangent base normalization layer (DyTb) is set to C. The SiLU activation function is then connected after the dynamic hyperbolic tangent base normalization layer (DyTb).

[0040] Step 2 specifically includes: Step 201: Set up the harvesting robot with the image acquisition equipment to collect images under different lighting, postures, and occlusion conditions in the field, and obtain original field images containing fruits, branches, leaves, and background.

[0041] Image acquisition equipment is deployed at the target crop planting site. This equipment is mounted on the robotic arm base of the harvesting robot or on a separate vision support. The distance between the lens and the target crop is adjusted according to the actual plant height, typically 0.5 to 1.5 meters. The image acquisition equipment includes at least one RGB industrial camera or RGB-D depth camera, and can be configured with an optional multi-view camera array or pan-tilt unit to acquire images from different angles.

[0042] The image acquisition device continuously acquires images at a fixed frame rate, with a resolution that can be set to 1920×1080, 1280×720, or other formats compatible with the input size of the fruit detection model. To cover different lighting conditions and occlusion scenarios, image acquisition should be conducted at multiple time periods, including: strong sunlight at noon, cloudy days, dawn and dusk, and artificial lighting scenarios. Simultaneously, it should include diverse samples such as fruits obscured by leaves, overlapping fruits, fruits at different stages of maturity, and fruits with surface reflections, blemishes, or diseases, to obtain original field images containing fruits, branches, leaves, and background.

[0043] Each captured image frame is saved in a lossless or low-compression format to avoid detail loss caused by JPEG compression. Simultaneously, for RGB-D cameras, depth and color maps must also be saved, along with camera parameters. If the quality of the captured raw field image is too low, the frame is automatically discarded, and the camera parameters are adjusted before re-capturing.

[0044] During the image acquisition process, the image acquisition device also needs to record metadata such as timestamps, GPS coordinates, light intensity, fruit variety, and prior information required for labeling. All original field images and metadata are stored on a local solid-state drive or uploaded to a server via a wireless network to form an original image library.

[0045] The timestamp refers to the current moment, accurate to milliseconds, obtained from the robot's internal clock. The time can be saved in a year-month-day hour:minute:second format, or as the cumulative number of milliseconds starting from a fixed point. When multiple robots work collaboratively, all robots' clocks are synchronized with the same time server via a network time protocol, ensuring consistent timestamps on all original field images captured by the robots.

[0046] When the robot collects images in outdoor orchards, it can be equipped with a high-precision GPS module or a regular GPS device. It continuously sends its current location information, including longitude, latitude, altitude, positioning accuracy, the number of satellites involved in the positioning, and the GPS time, to the main controller via a data cable. The internationally recognized WGS84 coordinate system is preferred for representing latitude and longitude. If images are collected inside greenhouses or in orchards without GPS signals, local coordinates output by an ultra-wideband positioning system or a visual positioning algorithm can be used instead of GPS.

[0047] The light intensity can be measured directly using a digital illuminance meter sensor or estimated using the camera's automatic exposure control parameters. Stronger light requires a shorter exposure time and lower gain; weaker light requires a longer exposure time and higher gain. An approximate illuminance value is then calculated. Where K and C are the coefficients to be calibrated, T is the exposure time, and G is the gain factor. For example, if the exposure time is 5000μs = 0.005s, the gain is 6dB = 2, and the calibration result is K = 120, C = 15, then... The recorded information includes specific illuminance values ​​and whether a special lighting mode is currently in effect.

[0048] Step 202: Perform image standardization preprocessing on the collected raw field images to obtain standardized field images.

[0049] Since the fruit detection model has a fixed input size of square, while the collected original field images are mostly rectangular, the width-to-height ratio of the original field image is calculated first. If the aspect ratio is not equal to 1, the original field image is scaled. Specifically, the longer side of the original field image is scaled to the target size fixed in the fruit detection model, and the shorter side is scaled proportionally to preserve the entire image. For example, if the original field image is 1920×1080 with a longer side of 1920 and a target size of 640×640, the scaling factor is 640 / 1920≈0.3333, and the scaled height is 1080×0.3333≈360 pixels, and the width is 640 pixels.

[0050] Create a 640×640 solid color canvas, place the scaled image in the center of the canvas, and fill the left and right sides evenly with the background color. Record the width offset (dw) and height offset (dh) of the fill for inverse transformation when mapping the detection box coordinates back to the original field image later.

[0051] By normalizing the pixel value to the original pixel value by 255, the pixel values ​​of the resized and padded image are normalized from [0, 255] to [0, 1] or [-1, 1], resulting in pixel values ​​for each channel (R, G, B) ranging from 0 to 1. Ensure the image channel order is RGB.

[0052] After the above processing, a standardized field image I with channel number × width × height is obtained. norm (x, y). Simultaneously, preprocessing parameters are saved, including scaling factor, width offset dw, height offset dh, and original image size w×h. These preprocessing parameters will be used later when mapping the detection boxes back to the original field image coordinates.

[0053] Step 203: Based on the real-time motion data of the harvesting robot, perform local feature gain compensation on the standardized field image to generate a compensated field image.

[0054] The harvesting robot has a six-axis IMU sensor mounted on its end effector or camera gimbal to acquire real-time attitude motion data and / or robot motion command data. Based on the attitude motion data and / or robot motion command data, it determines whether the harvesting robot's movement is causing blurring of the original field image. If motion blur exists in the original field image, a gain mask G is obtained to compensate for the blurring caused by the robot's movement, ultimately resulting in a clear, compensated field image.

[0055] The attitude motion data includes triaxial angular velocities (ω). x w y w z ), triaxial acceleration (a x a y a z ).

[0056] A low-pass filter, such as a first-order IIR filter with a smoothing coefficient of 0.2, is applied to the attitude motion data and / or robotic arm motion command data to eliminate high-frequency noise. The filtered angular velocity amplitude is then... Linear velocity amplitude v inst Instead of using IMU integration or visual optical flow for estimation, the linear velocity modulus is directly taken from the last moment of the robotic arm motion command data feedback.

[0057] For each frame of the original field image, if the angular velocity and / or linear velocity are greater than or equal to a preset motion blur threshold, the original field image of the current frame is determined to have a risk of motion blur. At this point, the camera's motion vector (Δu, Δv) in the current frame is estimated using the historical detection results and motion data of the most recent N frames. N takes values ​​from 3 to 5.

[0058] Specifically, the set of all retained fruit detection boxes in the previous frame (t-1) of the original field image is obtained. Using the IMU integration method, the rotation angle Δθ is obtained by integrating the angular velocity over a time interval Δt = 33ms between adjacent frames, and the displacement ΔT is obtained by quadratic integration of the acceleration. The pixel offset is then calculated by combining the camera focal length f and the target depth Z. , , .

[0059] Obtain the set B of fruit detection boxes that are finally retained from the original field image of the previous frame. t-1 Each box is represented as b i =(x min ymin x max y max The center point of each fruit detection frame. =( , Based on the center point of each fruit detection frame. Predict the center point of the fruit detection box in the current frame. = +M. Expand the predicted fruit detection box to obtain the Region of Interest (ROI) of the original field image in the current frame. i .

[0060] Create a single-channel mask G(x, y) of the same size as the standardized field image, with width and height set to w respectively. i h i The initial values ​​are all set to 1.0. For each frame of the original field image, the Region of Interest (ROI) is... i Apply a Gaussian gain inside it. , among which, (u x u y () represents the center point of the region of interest (ROI); ; A represents the gain amplitude, ranging from 0.2 to 0.5, with a larger value for faster movement.

[0061] Each pixel channel of the standardized field image is independently multiplied with the gain mask pixel by pixel, and the result is truncated to the pixel value range [0, 1] or [-1, 1] of the standardized field image to obtain the compensated field image. .

[0062] The motion blur thresholds for angular velocity and linear velocity , It is not a fixed value, but a configurable hyperparameter that can be adjusted according to the actual camera parameters used, the motion characteristics of the robotic arm, and the expected speed of the fruit. The motion blur threshold for angular velocity in this invention... The value range is 15~60° / s, preferably 30° / s; the motion fuzziness threshold for linear velocity. The value range is 0.3~0.5m / s, with 0.4m / s being preferred.

[0063] Step 3 specifically includes: The compensated field image first enters the backbone network of the fruit detection model. The backbone network consists of multiple stacked convolutional blocks and C3k2 modules. Each convolutional block employs a sequential structure of convolutional layers, DyTb normalization layers, and SiLU activation functions. The global curvature adjustment coefficient is calculated based on the current motion intensity. When motion blur exists, for each DyTb normalization layer in the fruit detection model, Otherwise, when there is no motion blur, λ is set to 1, and each DyTb normalization layer in the fruit detection model uses the original α. learned .

[0064] After compensation, the field image passes through each layer sequentially, with the feature map size decreasing progressively while the number of channels increases progressively. For example, the input image size is 3×640×640. After the first-level processing, low-level features such as edges and textures are extracted, resulting in an output feature map size of approximately 64×320×320. After the second-level processing, mid-level features such as local shapes and color patches are extracted, resulting in an output feature map size of approximately 128×160×160. After the third-level processing, the output feature map size is approximately 256×80×80, with the number of channels increasing to 256. At this point, the feature map possesses good semantic information while still retaining high resolution, making it suitable for small target detection. After the fourth-level processing, the output feature map size is approximately 512×40×40. At this point, the resolution is moderate, the receptive field is increased, and it can detect medium-sized fruits. After the fifth-level processing, the output feature map size is approximately 1024×20×20. At this point, the resolution is the lowest, the semantics are the strongest, and it is suitable for large targets or distant fruits.

[0065] Following the final stage, an SPPF module and C2PSA are connected to aggregate multi-scale contextual information and enhance attention to important regions. The backbone network ultimately outputs three feature maps at different scales: 256×80×80, 512×40×40, and 1024×20×20. These three feature maps are then fused into the neck network.

[0066] The neck network of the fruit detection model adopts the PANet path aggregation network structure, which includes two fusion paths: top-down and bottom-up.

[0067] The top-down path enlarges the 1024×20×20 deep feature map to 256×40×40 through convolution and upsampling. This is then concatenated with the 512×40×40 feature map output from the backbone network along the channel direction, forming a 768×40×40 feature map. The concatenated feature map is then processed through several convolutional layers and DyTb normalization to obtain an enhanced 256×40×40 feature map. Similarly, this enhanced 256×40×40 feature map is further upsampled to 256×80×80, concatenated with the 256×80×80 feature map output from the backbone network to obtain a 512×80×80 feature map, and then convolved to obtain an enhanced 128×80×80 feature map.

[0068] The bottom-up path downsamples the enhanced 256×80×80 feature map to 256×40×40 size through a convolution with a stride of 2. This is then concatenated with the enhanced 256×40×40 feature map obtained from the top-down path to obtain a 512×40×40 feature map. This 512×40×40 feature map is then convolved to obtain a further fused 256×40×40 feature map. Similarly, this fused 256×40×40 feature map is downsampled again to 512×20×20, concatenated with the 1024×20×20 feature map output from the backbone network, and convolved to obtain a final fused 512×20×20 feature map.

[0069] After bidirectional fusion of top-down semantic information transmission and bottom-up positional information transmission, the neck network outputs three feature maps: 128×80×80, 256×40×40, and 512×20×20. These three feature maps ensure that the details of small targets are preserved in the large feature map, while the semantics of large targets are preserved in the small feature map, thus enabling fruits of different sizes to achieve good detection responses at their respective scales.

[0070] The three feature maps output by the neck network are fed into three separate detection heads, with each feature map entering one detection head. Each detection head has the same structure, consisting of several convolutional layers, and finally outputs a feature tensor with a specific number of channels.

[0071] The number of channels in the feature tensor is calculated as (5 + number of categories) × number of anchor frames per grid point. For example, if the number of categories is set to 4 (including ripe fruit, unripe fruit, shaded fruit, and reflective fruit), and each grid point has 3 anchor frames of different sizes, then the number of channels in the output feature tensor of each detection head is (5 + 4) × 3 = 27.

[0072] Taking a 128×80×80 feature map as an example, after convolution processing, the number of channels is transformed to a final output of 27 channels, resulting in a 27×80×80 feature map. Here, 80×80 represents the number of grid points on the feature map, with each grid point corresponding to a region in the original image. For each anchor box at each grid point, the detector predicts five basic values: the lateral offset of the center point relative to the top-left corner of the grid, the vertical offset, the scaling factor of the anchor box width, the scaling factor of the anchor box height, and the probability of an object existing within that anchor box. In addition, a probability score for each category is also predicted.

[0073] Similarly, a 256×40×40 feature map also outputs a corresponding scale prediction tensor of 27×40×40, and a 512×20×20 feature map also outputs a corresponding scale prediction tensor of 27×20×20.

[0074] By merging the feature tensors at the three scales, we obtain the original detection boxes containing all potential fruits during the forward propagation of the fruit detection model. These original detection boxes can be understood as a list containing 80×80×3 + 40×40×3 + 20×20×3 = 19200 + 4800 + 1200 = 25200 entries. Each entry corresponds to the original predicted value of an anchor box, including center offset, width and height scaling, the original target value, and the original score for each category.

[0075] When multiple harvesting robots work together, in order to quantify the image blurring caused by changes in ambient light and motion, the leading harvesting robot introduces a light and shadow motion mask to dynamically adjust the DyTb parameter when it detects changes in ambient light. The dynamic amplitude adjustment factor generated by the dynamic adjustment of DyTb is then synchronized to the following harvesting robots via a wireless network, enabling rapid transfer of environmental features and real-time sharing of fruit detection model parameters. This avoids the following harvesting robots from repeatedly performing environmental adaptation calculations when entering areas already explored by the leading harvesting robots.

[0076] Specifically, each harvesting robot is equipped with a lightweight environment context manager, which runs in the edge computing module of the harvesting robot and is responsible for collecting, caching, broadcasting and applying DyTb parameters.

[0077] After the preliminary harvesting robot completes stable detection of a frame of image under specific environmental conditions, the environment context manager automatically extracts three dynamic parameters from all DyTb normalized layers of the fruit detection model, including curvature parameter α, scaling parameter γ, and translation parameter β. Each parameter is a one-dimensional tensor of the same length as the number of feature channels. The parameters of all DyTb layers are serialized into a compact data packet DyTb={(α...} k γ k ,β k |k=1,2,…,N layer}, where N layer This represents the total number of DyTb layers in the fruit detection model. When extracting the parameters of the DyTb normalized layers, the corresponding metadata is also extracted and packaged to obtain a parameter broadcast packet.

[0078] The packaged parameter broadcast packet is broadcast to all subsequent harvesting robots within the same collaborative network via UDP multicast or MQTT using the wireless communication module of the harvesting robot. The broadcast frequency is configurable, with a typical value of broadcasting once for each successfully detected frame or once every 0.5 seconds.

[0079] The subsequent harvesting robot's environment context manager continuously listens to the wireless network, receiving parameter broadcast packets from the preceding harvesting robot. For each received parameter broadcast packet, the subsequent harvesting robot checks if the timestamp has expired and if the GPS coordinates are within its preset collaborative area. Parameter broadcast packets that pass the verification are cached; otherwise, they are discarded.

[0080] The environment context manager stores a cache table for caching validated parameter broadcast packets. The cache table is indexed by coordinate region, and each cache entry includes a region identifier, a DyTb parameter set, an update timestamp, and the source harvesting robot ID. For each cache entry in the cache table, the parameter effective region is divided according to spatial coordinates and a preset radius. When the harvesting robot's own GPS coordinates enter the parameter effective region, the cache entry is marked as available for the current parameter effective region. In this invention, the cache table size is limited to 100 entries, and LRU (Least Recently Used) is used to filter and evict older entries.

[0081] When the illumination intensity parameter of the following harvesting robot exceeds a preset first intensity threshold, it indicates that the robot is entering a new spatial area with a significant environmental change, such as moving from a sunny area to a shady area. The environment context manager then checks if the current GPS coordinates match the effective region of any cached entry's parameters. If a match is found, and the cached entry's timestamp is within its validity period, the environment context manager does not use the default initial parameters of the fruit detection model or begin online adaptation from scratch. Instead, it directly reads the DyTb parameter corresponding to that spatial area from the cache table and assigns it to the corresponding DyTb layer in the fruit detection model, completely replacing the original default value or old value. This is then used to initialize or smoothly update the corresponding layer of its own fruit detection model. Conversely, when the illumination intensity parameter is less than the preset first intensity threshold but exceeds a preset second intensity threshold, it indicates that the following harvesting robot is entering a new spatial area with a slow environmental change, such as at sunset. In this case, the environment context manager checks if the current GPS coordinates match the effective region of any cached entry's parameters. If a match is successful and the timestamp of the cached entry is within its validity period, the environment context manager performs an exponentially weighted moving average of the default DyTb parameter of the current fruit detection model and the DyTb parameter corresponding to the spatial region read from the cache table. When the light intensity parameter is less than the preset second intensity threshold, it indicates that the environment in which the harvesting robot is located has not changed significantly, for example, it is always in a lit area, and in this case, the default DyTb parameter of the current fruit detection model is still used.

[0082] More specifically, when the motion speed exceeds the motion blur threshold, a motion weighting factor is calculated based on the linear velocity and angular velocity, which is then used to generate a light and shadow motion mask to suppress the interference of motion blur on the detection of illumination changes.

[0083] Normalized Motion Intensity Motion weighting factor It can be seen that the motion weighting factor λ decreases linearly, reducing the sensitivity to feature differences under rapid motion.

[0084] The light and shadow motion mask M LM This is used to quantify the feature gradient changes caused by abrupt changes in illumination between consecutive frames, while also considering the effects of motion velocity modulation and the DyTb amplitude adjustment factor. Where Δf is the absolute value of the pixel-by-pixel difference between the feature maps at time t and time t-1; λ is the motion weighting factor; γ is the scaling parameter of the current DyTb layer; K norm This is a normalization constant to keep the values ​​input to the tanh function within a reasonable range. When the ambient light suddenly dims, Δf increases significantly, thus M... LM The value of M approaches 1; in the stable region or when motion blur is severe, M LM Approaching 0.

[0085] For the light and shadow motion mask M LM For pixels whose values ​​exceed a preset threshold, the depth map of the current frame image and the camera intrinsic parameter K are used to back-project the pixel to a 3D point (X) in the camera coordinate system. c Y c Z c Then, by combining the robot's current pose P(x, y, z) with GPS coordinates, a set of spatial coordinate points is formed.

[0086] The pre-harvesting robot packages the data into an environmental feature data package. The environmental feature data package includes a set of spatial coordinate points, parameters of all DyTb layers in the fruit detection model at the current moment, timestamps corresponding to parameter generation, and corresponding environmental labels.

[0087] After receiving the environmental feature data packet, the environment context manager of the following harvesting robot first compares the received spatial coordinates of the preceding harvesting robot with the current GPS coordinates of the following harvesting robot. When the distance between the following harvesting robot and any point in the spatial coordinates of the preceding harvesting robot is less than a preset threshold, it is determined that it is about to enter a region of sudden change in illumination. The parameters of all DyTb layers in the fruit detection model of the following harvesting robot are directly replaced with snapshot values ​​from the environmental feature data packet, without the need for statistical iterative adaptation. To avoid a sudden drop in detection performance due to parameter mutations, if the GPS coordinates successfully match the effective region of any cached entry's parameters, the environment context manager performs an exponentially weighted moving average of the DyTb parameters in the cache table with the DyTb parameters of the current fruit detection model. , where θ is one of α, γ or β; η is the fusion coefficient, usually taken as 0.6~0.8.

[0088] After the parameters are updated, the subsequent harvesting robot immediately uses the updated fruit detection model to process the current frame. Simultaneously, the environment context manager logs this application and can optionally broadcast the local detection results as new preceding harvesting robots, enabling iterative propagation and continuous optimization of parameters.

[0089] Step 4 specifically includes: Step 401: Preset a verification whitelist and verify the rationality of the spatial location based on the verification whitelist.

[0090] On the image plane, the pixel regions corresponding to the canopy layer and fruit growth zone are pre-marked, generating a binary mask image of the same size as the input image. Pixels with a value of 1 represent allowed detection areas, and pixels with a value of 0 represent prohibited detection areas. This binary mask image serves as a whitelist for image space verification.

[0091] The verification whitelist is a list of spatial conditions or image regions that allow fruit to appear. The verification whitelist can be generated through manual annotation, semantic segmentation models, or fixed geometry, and can be configured and switched according to different orchards and different crops.

[0092] Using the preprocessing parameters of the compensated field image, the coordinates of the center point or four corner points of the original detection box are reverse-mapped to the actual pixel coordinates of the original field image. Based on the mapped center point coordinates of the original detection box or all pixels within the box, the value at the corresponding position is searched in the verification whitelist.

[0093] Specifically, when using the center point coordinates of the original detection box for judgment, the whitelist value of the pixel position is directly read. If the center point coordinates of the original detection box fall within the whitelist-allowed area (i.e., the mask value is 1), indicating that detection is allowed, the original detection box is retained and proceeds to the next filtering step. If the center point of the original detection box falls within the whitelist-prohibited area (i.e., the mask value is 0), indicating that detection is prohibited, the original detection box is discarded directly without any further processing.

[0094] When using all pixels within the original detection box for judgment, the proportion of pixels within the original detection box belonging to the whitelist region is calculated. If the proportion exceeds a set threshold, the original detection box is retained; otherwise, it is discarded. The preferred proportion threshold is 50%.

[0095] Step 402: Combine device status and target priority to obtain a set of candidate target boxes.

[0096] Even after verifying the spatial location validity using the whitelist, the number of remaining detection boxes may still be significant. However, harvesting robots typically have limited computing resources and may be operating under low battery or high load conditions. Therefore, it is crucial to monitor the device status in real time, read the current battery level or computing power of the harvesting robot, determine the operating mode, and then filter the remaining detection boxes after verifying the spatial location validity using the threshold corresponding to the operating mode.

[0097] The device status includes, but is not limited to, battery level, CPU / GPU utilization, available memory / video memory, and robotic arm movement status. The operating modes include normal mode, power-saving mode, and emergency mode.

[0098] When the battery power or computing power is lower than the preset power threshold or computing power threshold, switch to power saving mode or emergency mode, and retain only the top K detection boxes or only the detection boxes located in the active area of ​​the space.

[0099] Specifically, the normal mode refers to a battery level >30%, indicating sufficient computing power, and retains all detection boxes that have passed the whitelist verification. The energy-saving mode refers to a battery level between 15% and 30%, or a computing load >80%, retaining only the top K detection boxes and hiding detection boxes in low-priority areas. The emergency mode refers to a battery level <15%, or computing power nearing saturation, retaining detection boxes only in the spatially active areas determined by motion planning and historical detection results, discarding all detection boxes in other areas. The spatially active area is a dynamic region of interest that combines the robot arm's prior motion knowledge with the temporal continuity of visual detection. This allows the harvesting robot to focus on the most likely harvest target when resources are limited.

[0100] In scenarios involving multiple fruits and multiple categories, the detection boxes are further sorted and filtered according to preset task priorities. The task priority filtering does not directly discard low-priority boxes, but sorts them to obtain a set of candidate target boxes, and then processes the high-priority detection boxes first.

[0101] The task priorities include maturity priority, occlusion priority, distance priority, and time priority. Maturity priority means that the detection boxes for mature fruits have a higher priority than those for unripe fruits, and mature fruits are picked first. Occlusion priority means that the detection boxes for fruits with no occlusion or slight occlusion have a higher priority than those for severely occluded fruits, and fruits with no occlusion and easier picking are picked first. Distance priority means that the detection boxes for fruits closer to the robotic arm have a higher priority than those farther away from the robotic arm, reducing movement time. Time priority means that if the same fruit is detected in multiple consecutive frames, its priority is increased.

[0102] When fruits are extremely dense or lighting is complex, the number of original prediction boxes generated by multi-scale feature fusion surges, leading to an increase in the inference latency τ of the edge controller. inf The load increased significantly, potentially exceeding the obstacle avoidance safety threshold of the harvesting robot's robotic arm. This was addressed through a comprehensive algorithmic load assessment. To quantify the processing load of the current frame, the task load of the harvesting robot is determined by considering target density, environmental complexity, and penalties for actual inference latency, and the original prediction box of the harvesting robot is comprehensively evaluated. Wherein, D obj C represents the target detection density, i.e., the number of entries in the original predicted bounding boxes of the current frame; env The environmental complexity factor is calculated as the light and shadow motion mask M. LM The proportion of pixels with a value greater than 0.7 in the total number of pixels in the image reflects the complexity of regions with abrupt changes in illumination; W obj W env The preset weighting coefficients are empirically set to 0.6 and 0.4 respectively; σ(τ) inf ) is the time delay penalty function. The delay penalty function is 1 when the delay does not exceed the safety threshold, and increases exponentially after it exceeds the threshold, forcing the system to unload the load as soon as possible.

[0103] Therefore, in this invention, based on the obstacle avoidance planning of the robotic arm of the harvesting robot and historical detection results, the working area mask M is used. mask The currently prioritized active spatial regions are marked, thereby filtering and sorting the original detection boxes, retaining important working area sub-images, and unloading unimportant unloadable feature sub-images. Specifically, when L>L max When the workload exceeds the load, task slice unloading is triggered. First, based on the job area mask M... mask The feature map f of the current framet The data is divided into a work area feature sub-map and an unloadable feature sub-map. The work area feature sub-map... This includes the area where the robotic arm of the harvesting robot will be working. The unloadable feature sub-map. This is the remaining feature map.

[0104] The work area mask M mask It is a binary matrix with the same size as the feature map. Pixels with a value of 1 correspond to the active spatial region (determined jointly by the prior of the robotic arm's motion and the historical detection results), while regions with a value of 0 represent non-urgent or background regions.

[0105] However, when officially unloading the unloadable feature subgraph, it is also necessary to calculate the total distributed processing latency. ,in, τ represents the network latency for transmitting the offloadable feature subgraph from the edge to the generational computing node. remote_inf The time delay for the computing node to complete the inference of the offloadable feature subgraph. When τ tota l<τ local_inf When τ, the unloadable feature subgraph is unloaded; when τ tota l≥τ local_inf At that time, the unloadable feature subgraph is retained.

[0106] The edge end reverse maps the returned detection box coordinates back to the original field image coordinate system and merges them with the core operation area detection results processed locally to form a complete set of candidate target boxes.

[0107] Visual algorithms in agricultural environments are susceptible to interference from strong reflections such as glare from mulch film, water surfaces, and fruit surfaces, resulting in high-confidence false features. These false features consume significant computational resources and can potentially cause erroneous actions by the robotic arm. To further suppress visual bias at its source, this invention introduces a topological semantic mask and a self-healing reverse suppression mechanism in the front-end feature processing stage. This mechanism utilizes the physical growth logic of the fruit to correct the feature response pixel-by-pixel, preventing invalid calculations and erroneous operations.

[0108] Based on prior agricultural knowledge, different fruits must grow within a specific height range and within the canopy projection area. First, define a spatial discriminant function. , where (X, Y, Z) are the local spatial coordinates with the center of the fruit tree trunk as the origin, XY are the horizontal plane, and Z is the vertical plane.

[0109] However, in practical applications, the work area mask needs to be superimposed on the lighting and motion mask. Only areas where the unloadable feature sub-images are removed and where lighting and motion cause image blurring are unloaded. Therefore, the harvesting robot obtains its relative position to each fruit tree based on GPS coordinates, maps each 3D point in the current frame to a local coordinate system, and maps the lighting and motion mask to 3D physical space, forming a topological semantic mask M. TSM .

[0110] Specifically, using the camera intrinsic parameter K and the depth map of the current frame, the spatial discriminant function value of the three-dimensional physical space is reprojected onto the image plane to obtain a topological semantic mask M of the same size as the input image. TSM (i, j), where (i, j) are pixel coordinates. In the topological semantic mask, a pixel with a value of 1 indicates that the location physically meets the fruit growth position constraints, i.e., fruit may grow there; a pixel with a value of 0 indicates that fruit cannot physically appear, such as in open areas other than the ground, sky, or tree trunk.

[0111] Light and shadow motion mask M LM With topological semantic mask M TSM Perform element-wise Hadamard product to obtain the consistency score. The consistency score combines visual change confidence and physical plausibility. A high score is given when a pixel simultaneously satisfies significant changes in ambient lighting and is located within a legitimate fruit growth space; a high score is also given when a pixel has a high-light-shadow motion mask M. LM However, if it is located in a physically impossible region, the consistency score approaches 0.

[0112] For the M-type light and shadow motion mask with extremely low consistency score LM The extremely high scores indicate a visual illusion, suggesting that the fruit detection model generated strong false feature responses in areas where fruits should not physically exist. At this point, the consistency score for the light and shadow motion mask M is extremely low. LM The dynamic amplitude adjustment factor γ of the DyTb layer corresponding to the extremely high region is suppressed online, and a new adjustment factor is generated. .

[0113] Due to consistency score and light and shadow motion mask M LM The difference is larger in the visual illusion region, where the exponential term exp(-k·Δ) will be significantly less than 1, making γ′ in this region much smaller than γ. γ, as the scaling parameter of the DyTb layer, directly controls the amplitude of the features after the tanh nonlinear mapping. A decrease in γ′ means that the feature response in this region is exponentially suppressed, making it difficult for subsequent detection heads to generate candidate boxes from this region.

[0114] Before the feature map is input into the detection head, the output of the DyTb layer is modulated using γ′. The feature map after inverse suppression enters the detection head, preventing interference regions from generating high-confidence candidate boxes. This achieves false alarm filtering at the operator level, avoiding invalid calculations and erroneous actions by the robotic arm.

[0115] Step 5 specifically includes: Each detection box in the candidate target box set still retains the original predicted value directly output by the fruit detection model. The original predicted value cannot be directly used for robotic arm positioning or visualization; it must undergo a series of decoding, transformation, and filtering operations to obtain fruit category labels and spatial positioning coordinates with actual physical meaning.

[0116] Step 501: Decode the coordinates of each entry in the candidate target box set to obtain the position and size of each candidate target box in the compensated field image.

[0117] For each entry in the candidate bounding box set, based on its grid position, anchor box preset size, and original prediction value, the precise location (x, y, x) of each candidate bounding box on the compensated field image is calculated. min y min x max y max (and width and height).

[0118] Specifically, the center point offset (t) of the original predicted value output by the fruit detection model x , t y The offset is relative to the top-left corner of the corresponding grid cell, ranging from 0 to 1. Adding the grid's row and column indices to this center point offset, and then multiplying by the downsampling step size of the feature map relative to the compensated field image, yields the pixel coordinates of the bounding box center point on the compensated field image. For example, on an 80×80 feature map, each grid corresponds to an 8×8 pixel area in the original image. If the grid index is (5, 5) and the offset is (0.3, 0.4), then the center point coordinates are ((5+0.3)×8=42.4 pixels, (5+0.4)×8=43.2 pixels).

[0119] The scaling factor (d) of the original predicted value output by the fruit detection model w d h The scaling factor is a logarithmic scale relative to the preset anchor frame width and height. First, take the exponent of the scaling factor (...). , Then multiply by the preset width and height of the corresponding anchor frame to obtain the actual width and height of the bounding box. For example, if the preset width of the anchor frame is 20 pixels, d w =0.5, then the actual width = 20 × ≈20×1.6487=32.97 pixels.

[0120] Calculate the top-left corner coordinates of the bounding box by subtracting half the actual width from the center point's x-coordinate and half the actual height from the center point's y-coordinate; calculate the bottom-right corner coordinates by adding half the actual width to the center point's x-coordinate and half the actual height to the center point's y-coordinate, and clip any excess portion to the boundary.

[0121] Step 502: Calculate the probability of target presence for each candidate target box after coordinate decoding, based on its position and size in the compensated field image.

[0122] The fruit detection model outputs an independent original value of target presence and an original score for each category for each detection box.

[0123] The raw probability of target existence represents the logarithmic probability of whether the anchor frame contains any target, reflecting the reliability of the presence of an object within the frame. Applying the sigmoid function to the raw probability of target existence converts it into a probability value between 0 and 1, denoted as P. obj .

[0124] The fruit detection model outputs raw scores for each preset category. The Softmax function is applied to each raw score to convert it into a probability distribution between 0 and 1, with a sum of 1. The probability corresponding to each category is denoted as . The maximum value is taken as the class probability P of the bounding box. cls =max i The corresponding category is the predicted category of the box.

[0125] Calculate the probability of the final detected target's existence = P obj ×P cls The probability of the final detected target being present is between 0 and 1. This probability is associated with each detection box for subsequent filtering.

[0126] Step 503: Remove overlapping detection boxes by nonmaximum suppression to obtain the final set of detection boxes.

[0127] Since the fruit detection model may generate multiple overlapping detection boxes for the same fruit at adjacent grid points or at different scales, non-maximum suppression is used to merge the overlapping detection boxes and retain the one with the highest probability of the final detected target.

[0128] Specifically, all detection boxes are sorted from highest to lowest probability of the final detected target, and an empty list is created to store all the detection boxes to be retained. Starting with the detection box with the highest probability of the final detected target, it is added to the retention list. For each remaining detection box, the Intersection over Union (IoU) is calculated between it and the last detection box in the current retention list. If the IoU is greater than the IoU threshold, the two detection boxes are considered to be detecting the same target, and since the currently retained detection box has a higher probability of the final detected target, it is discarded. If the IoU is less than or equal to the IoU threshold, they are considered to be detecting different targets, and the box is also added to the retention list. This process is repeated until all detection boxes have been processed, resulting in the final set of detection boxes.

[0129] The Intersection over Union (IoU) is the area of ​​the intersection of two detection boxes divided by the area of ​​their union, and its value ranges from 0 to 1. A larger IoU indicates a higher degree of overlap. In this invention, the intersection over union threshold is set to 0.45 to 0.55.

[0130] Step 504: The final detection box set is inversely mapped to obtain the final visual detection output.

[0131] Using the scaling factor, fill width, and fill height saved in step 2, as well as the size of the original field image, the four corner points of each detection box are inversely transformed to obtain the final visual detection output containing fruit category labels, spatial location coordinates, and detection.

[0132] Specifically, the coordinates on the compensated field image are subtracted from the padding offset. Then, this is divided by the scaling factor to obtain the coordinates on the original scaled image. Finally, these coordinates are constrained within the width and height range of the original field image. This yields the pixel position of the fruit bounding box in the original field image.

[0133] The fruit category label corresponds to the category index output by the fruit detection model, such as ripe apple, unripe tomato, etc. The spatial positioning coordinates refer to the coordinates of the top-left and bottom-right corners of the bounding box in the original field image, usually in pixels. The detection value is a floating-point number between 0 and 1, representing the reliability of the detection.

[0134] The pixel location of the fruit bounding box in the original field image is sent to the robotic arm of the harvesting robot via an internal communication interface to guide the robot's grasping and harvesting of the fruit. Simultaneously, the final visual detection output can also be used for visualization and monitoring by the operator.

[0135] Example 1: An apple-picking robot equipped with an RGB-D camera, a Jetson Orin NX edge computing module, and a six-axis robotic arm operates in a Fuji apple orchard.

[0136] The apple-picking robot features a four-wheel independently driven tracked chassis and is equipped with a six-degree-of-freedom collaborative robotic arm. The robotic arm has a two-finger flexible gripper at its end, with a maximum end-effector linear velocity of 0.8 m / s and a maximum angular velocity of 90° / s. The robot is also equipped with an RGB-D camera with a resolution of 1920×1080 at 30fps and a depth resolution of 640×480, and integrates a six-axis IMU sensor.

[0137] The Red Fuji apple orchard has a row spacing of 4m, a plant spacing of 2m, and a tree height of approximately 3m.

[0138] YOLOv11 was used as the baseline network. All convolutional modules in the network were traversed, and all batch normalization layers were replaced with DyTb modules to construct the fruit detection model.

[0139] In the orchard, an RGB-D camera captured raw field images at a resolution of 1920×1080, while simultaneously recording timestamps, GPS coordinates, light intensity, fruit variety, and prior information. The long side of the raw field image was scaled to 640 pixels. In this example, the original width was 1920 and the height was 1080, with a scaling factor of 0.3333, resulting in a scaled size of 640×360. A 640×640 gray canvas was created, and the scaled image was placed vertically centered, with 140 pixels of padding above and below. The pixel values ​​were normalized to [0, 1] by dividing by 255. The scaling factor, padding size, and original image size were recorded to obtain a standardized field image.

[0140] When the robotic arm performs rapid movements, for each frame, the robotic arm controller reads the current motion command linear velocity of the end effector as 0.6 m / s, and the IMU measures ω. inst =15° / s. Motion blur threshold for angular velocity. =30° / s, motion fuzziness threshold for linear velocity =0.4m / s. Therefore, r = 0.6 / 0.4 = 1.5. Calculate... λ = 1 - (1.5 - 1) / 2 = 0.75. Multiply the standardized field image pixel by pixel with the gain mask to obtain the compensated field image, and truncate it to the range [0, 1].

[0141] The compensated field image is input into the fruit detection model. The backbone network extracts features step by step, outputting F3 (256×80×80), F4 (512×40×40), and F5 (1024×20×20). The neck network generates P3 (128×80×80) and P4 (256×40×40) from top to bottom; and N4 (256×40×40) and N5 (512×20×20) from bottom to top. The three detection heads output raw prediction tensors of 27×80×80, 27×40×40, and 27×20×20 respectively, containing a total of 25,200 candidate entries. Each entry includes center offset, width and height scaling, original target value, and original class score, which are output as raw detection boxes.

[0142] By broadcasting the DyTb normalized layer parameters to the first harvesting robot, and then directly loading the cached parameters when the subsequent harvesting robot enters the same coordinate area, the rapid transfer of environmental features and instantaneous adaptation of the model state are achieved.

[0143] A whitelist mask is pre-built for the image space, where the tree canopy region has a value of 1, and the ground and sky regions have values ​​of 0. For each original detection box, its center point is mapped back to the original image coordinates, and the value at the corresponding position in the whitelist is queried. If the value is 0, it is discarded; if the value is 1, it is retained. After whitelist filtering, approximately 10,000 boxes remain. The robot's current battery level is read as 22%, and GPU usage is 85%, indicating that it is entering power-saving mode, retaining only the first 100 boxes. These are then sorted by maturity priority to obtain 100 candidate target boxes, while still maintaining the original prediction values.

[0144] Perform coordinate decoding on each candidate bounding box. Obtain P by passing the original target values ​​through a Sigmoid function. obj The original class scores are then processed by Softmax to obtain the maximum class probability P. cls The probability of the final detected target being present is P. obj ×P cls The IoU threshold is 0.45, and non-maximum suppression is applied to all decoded detection boxes in descending order. The coordinates of the retained detection boxes are inversely mapped back to the original image coordinates (1920×1080) using a scaling factor and padding values. Finally, the category label and bounding box coordinates of each fruit are output.

[0145] This example outputs 32 valid detection results, with a total time of 48ms and a frame rate of 20FPS.

[0146] This invention provides a visual monitoring method for agricultural harvesting. First, it replaces the traditional batch normalization layer with a DyTb module, which can be seamlessly embedded into various mainstream detection architectures such as YOLOv11, YOLOv8, ResNet, and Transformer. This facilitates deployment across different edge platforms and solves the performance degradation problem caused by small batch sizes on edge devices, providing a highly practical technical solution for intelligent agricultural harvesting robots. Second, DyTb contains only three learnable parameters and dynamically adjusts the feature amplitude to predict the position of the detection box from the previous frame in the current frame and generates a Gaussian local gain mask. This mask specifically enhances areas that may contain fruit, effectively offsetting feature attenuation caused by motion blur, enhancing robustness in complex environments, reducing computational and memory overhead, and meeting real-time harvesting requirements. Third, a preset whitelist compares the center point of the detection box with the allowed detection area, enhancing decision-making security. Finally, the task is adjusted based on device status and target priority to ensure effective task completion. Finally, the dynamic parameters of the DyTb normalization layer are designed to enable complete multi-machine collaborative perception and parameter sharing, so that the follower robot does not need to go through dozens of frames of online adaptation process, which greatly improves the collaborative operation efficiency and environmental adaptation speed of multi-robot clusters.

[0147] The above description illustrates preferred embodiments of the present invention and helps those skilled in the art to more fully understand the technical solution of the present invention. However, these embodiments are merely illustrative and should not be construed as limiting the specific implementation of the present invention to these embodiments. For those skilled in the art, several simple deductions and modifications can be made without departing from the inventive concept, and all such modifications should be considered within the protection scope of the present invention.

Claims

1. A visual monitoring method for agricultural harvesting, characterized in that, Including the following steps: Based on YOLOv11, the batch normalization layer was replaced with the DyTb normalization layer to build a fruit detection model; The harvesting robot is equipped with a data acquisition device to collect original field images, which are then processed to obtain standardized field images. Based on the real-time motion data of the harvesting robot, local feature gain compensation is applied to the standardized field images to generate compensated field images. The compensated field image is input into the fruit detection model for feature extraction, multi-scale fusion and preliminary target prediction to obtain the original detection box containing all potential fruits. The original detection boxes are spatially topologically verified for location rationality, and combined with device status and target priority for screening, ultimately forming a set of candidate target boxes; The candidate target box set is processed by coordinate decoding and non-maximum suppression to obtain the final visual detection output containing fruit category labels and spatial positioning coordinates.

2. The method for visual monitoring of agricultural harvesting according to claim 1, characterized in that, The DyTb normalization layer is embedded in the backbone network, neck network and detection head of YOLOv11, forming a structure of convolutional layer conv, dynamic hyperbolic tangent base normalization layer DyTb and SiLU activation function; The dynamic hyperbolic tangent base normalization layer , where x is the feature map of the previous convolutional layer; tanh(αx) is the hyperbolic tangent function; α is the curvature parameter with an initial value of 1.0; γ is the scaling parameter with an initial value of 1.0; β is the translation parameter with an initial value of 0.

0.

3. The method for visual monitoring of agricultural harvesting according to claim 1, characterized in that, A harvesting robot is equipped with data acquisition devices to collect raw field images, which are then standardized to obtain standardized field images. Based on the real-time motion data of the harvesting robot, local feature gain compensation is applied to the standardized field images to generate compensated field images, including: The harvesting robot is equipped with a data acquisition device. Through the image acquisition device, it can collect images in the field under different lighting conditions, different postures, and different occlusion conditions, and obtain original field images containing fruits, branches, leaves, and backgrounds. The collected raw field images were preprocessed to obtain standardized field images; Based on the real-time motion data of the harvesting robot, local feature gain compensation is performed on the standardized field image to generate a compensated field image.

4. The agricultural harvesting visual monitoring method according to claim 3, characterized in that, Calculate the width-to-height ratio of the original field image; if the aspect ratio is not equal to 1, scale the original field image by scaling the long side of the original field image to the target size fixed input by the fruit detection model, and scaling the short side by the same ratio. Place the scaled image in the center of the canvas, fill the left and right sides with the background color evenly, and map the pixel normalization value to [0, 1] or [-1, 1] according to the pixel normalization value = original pixel value / 255 to obtain the standardized field image of channel number × width × height.

5. The agricultural harvesting visual monitoring method according to claim 3, characterized in that, The harvesting robot has an IMU sensor installed at the end of its robotic arm or on the camera gimbal to acquire posture motion data and / or robotic arm motion command data in real time, and calculate the instantaneous angular velocity amplitude and linear velocity amplitude at the moment of image acquisition. When the angular velocity and / or linear velocity are greater than or equal to the preset angular velocity and / or linear velocity motion blur threshold, the original field image of the current frame is at risk of motion blur. Region of Interest (ROI) for each frame of the original field image i Applying a Gaussian gain inside it yields the gain mask. , among which, (u x u y () represents the center point of the region of interest (ROI); ; A represents the gain amplitude. The original standardized field image is multiplied pixel by pixel with the gain mask to obtain the compensated field image.

6. The method for visual monitoring of agricultural harvesting according to claim 1, characterized in that, After compensation, the field image first enters the backbone network of the fruit detection model. After passing through each layer, the size of the feature map is gradually reduced while the number of channels gradually increases. Finally, three feature maps of different scales are output and enter the neck network of the fruit detection model. The neck network adopts the PANet path aggregation network structure, which includes two fusion paths: top-down and bottom-up. After bidirectional fusion, the three feature maps are output and sent to three detection heads respectively, resulting in three feature tensors with a specific number of channels. The number of channels of the feature tensor = (5 + number of classes) × number of anchor boxes per grid point. The feature tensors of the three scales are merged to obtain the original detection boxes containing all potential fruits.

7. The agricultural harvesting visual monitoring method according to claim 6, characterized in that, Calculate the global curvature adjustment coefficient based on the current motion intensity. When motion blur exists, for each DyTb normalization layer in the fruit detection model, When there is no motion blur, λ is 1.

8. The method for visual monitoring of agricultural harvesting according to claim 1, characterized in that, When multiple harvesting robots are working collaboratively, each robot is equipped with an environment context manager to collect, cache, broadcast, and apply DyTb parameters. After the preliminary picking robot completes the stable detection of a frame image, the environment context manager extracts the dynamic parameters and corresponding metadata of all DyTb normalized layers from the fruit detection model to obtain the parameter broadcast packet. When the illumination intensity parameter of the following harvesting robot exceeds a preset first intensity threshold, the environment context manager checks if the current GPS coordinates match the parameter effective region of any cached entry, and if the timestamp of the cached entry is within the validity period. The environment context manager directly reads the DyTb parameter corresponding to the spatial region from the cache table and assigns it to the corresponding DyTb layer in the fruit detection model. When the illumination intensity parameter is less than the preset first intensity threshold but exceeds the preset second intensity threshold, the environment context manager checks if the current GPS coordinates match the parameter effective region of any cached entry, and if the timestamp of the cached entry is within the validity period. The environment context manager performs an exponentially weighted moving average of the default DyTb parameter of the current fruit detection model and the DyTb parameter corresponding to the spatial region read from the cache table. When the light intensity parameter is less than the preset second intensity threshold, the default DyTb parameter of the fruit detection model will still be used. After the parameters are updated, the subsequent harvesting robot uses the updated fruit detection model to process the current frame.

9. The agricultural harvesting visual monitoring method according to claim 8, characterized in that, The cache table is indexed by coordinate region, and each cache entry in the cache table includes a region identifier, a DyTb parameter set, an update timestamp, and the source harvesting robot ID; For each cache entry in the cache table, the effective region of the parameters is divided according to the spatial coordinates and the preset radius.

10. The method for visual monitoring of agricultural harvesting according to claim 1, characterized in that, The original detection boxes undergo spatial topology verification to ensure their location is reasonable. Combined with device status and target priority filtering, a candidate target box set is ultimately formed, including: A preset verification whitelist is used to verify the rationality of spatial locations. By combining device status and target priority, a set of candidate target boxes is obtained.

11. The method for visual monitoring of agricultural harvesting according to claim 10, characterized in that, The verification whitelist is a binary mask image with the same size as the input image, where pixels with a value of 1 represent allowed detection areas and pixels with a value of 0 represent prohibited detection areas. The coordinates of the center point or four corner points of the original detection box are reverse-mapped to the actual pixel coordinates of the original field image. The value of the corresponding position in the whitelist is queried. If the value is 0, the original detection box is discarded; if the value is 1, the original detection box is retained.

12. The agricultural harvesting visual monitoring method according to claim 1, characterized in that, The candidate bounding box set is processed through coordinate decoding and non-maximum suppression to obtain the final visual detection output, which includes fruit category labels and spatial location coordinates. Coordinate decoding is performed on each entry in the candidate target box set to obtain the position and size of each candidate target box in the compensated field image; For each candidate bounding box after coordinate decoding, calculate the probability of target existence based on its position and size in the compensated field image; Overlapping detection boxes are removed by nonmaximum suppression to obtain the final set of detection boxes; The final set of detection boxes is inversely mapped to obtain the final visual detection output.

13. The agricultural harvesting visual monitoring method according to claim 12, characterized in that, Add the center point offset to the grid row and column indices, and then multiply by the downsampling step size of the feature map relative to the compensated field image to obtain the pixel coordinates of the bounding box center point on the compensated field image. Calculate the top-left corner coordinates of the bounding box by subtracting half the actual width from the center point's x-coordinate and half the actual height from the center point's y-coordinate; calculate the bottom-right corner coordinates by adding half the actual width to the center point's x-coordinate and half the actual height to the center point's y-coordinate.

14. The method for visual monitoring of agricultural harvesting according to claim 12, characterized in that, The fruit detection model outputs an independent original value P for the presence of the target for each detection box. obj and the original scores P for each category cls ; Calculate the probability of the final detected target's existence = P obj ×P cls .

15. The agricultural harvesting visual monitoring method according to claim 14, characterized in that, Sort all detection boxes in descending order of the probability of the final detected target. Starting with the detection box with the highest probability of the final detected target, iterate through each remaining detection box and calculate its Intersection over Union (IoU) with the last detection box in the current retention list. If the IoU is greater than the IoU threshold, discard the detection box; if the IoU is less than or equal to the IoU threshold, retain the detection box.

Images