An integrated identification and positioning method and system for farmland weed plant protection operation

Abstract

The application relates to the technical field of computer vision, and discloses an integrated identification and positioning method and system for farmland weed plant protection operation, which comprises the following steps: synchronously triggering a multi-modal sensor to form a time-aligned multi-modal data group; identifying image data to output crop pixel coordinates and weed pixel coordinates; searching a carrier attitude from pose data according to a time stamp carried by the image; predicting the pose of the carrier after a delay according to a full-link processing delay to obtain predictive crop operation coordinates and predictive weed operation coordinates; transmitting an instruction carrying the predictive weed operation coordinates to an execution mechanism through a real-time bus; and completing a target action in an execution window triggered by a unified clock source timing signal according to the predictive weed operation coordinates. The application can eliminate the influence of random delay jitter on positioning accuracy, provides a crop avoidance reference for target operation, and realizes the dual goals of accurate weed positioning and effective crop protection.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an integrated identification and positioning method and system for weed control operations in farmland. Background Technology

[0002] In farmland weed control operations, accurate weed identification and spatial positioning are key prerequisites for efficient weed control. Existing weed identification and positioning methods mostly rely on vehicle-mounted cameras to collect farmland images, use deep learning networks to identify weeds and output their pixel coordinates, and then convert the pixel coordinates into physical coordinates in the operational coordinate system through camera calibration parameters, so that the subsequent execution mechanism can perform target operations.

[0003] However, existing methods suffer from significant spatiotemporal consistency issues in practical dynamic operations. On one hand, image sensors and pose sensors operate independently, lacking hardware-level time synchronization, resulting in temporal misalignment between image frames and the measurement data of the instantaneous pose of the describing carrier. Even with high-precision algorithms, spatial coordinate calculations based on temporally misaligned image and pose data will still contain inherent errors in the resulting operational coordinates.

[0004] On the other hand, from image acquisition, target recognition, coordinate transformation to the final output of job instructions, there is a significant processing delay throughout the entire process. Under general computing architectures, this delay is affected by factors such as operating system task scheduling and data contention, exhibiting random fluctuations that make compensation schemes based on fixed delay values ​​ineffective, severely limiting the accuracy of dynamic positioning.

[0005] Furthermore, existing methods only focus on the identification and location of weed targets, without simultaneously outputting crop location information. When weeds grow adjacent to or interspersed with crops, the lack of a crop spatial reference makes it difficult to achieve effective crop avoidance in subsequent operations, posing a risk of accidental damage.

[0006] Therefore, this invention proposes an integrated identification and positioning method and system for weed control operations in farmland. Summary of the Invention

[0007] The purpose of this invention is to solve the problems of asynchronous sensor time, random processing delay, and lack of crop avoidance benchmark in the prior art. To this end, an integrated identification and positioning method and system for weed protection operations in farmland is proposed.

[0008] In a first aspect, the present invention provides an integrated identification and positioning method for weed control operations in farmland, comprising the following steps: Step S1: Synchronously trigger the multimodal sensor through a unified clock source, and bind the timestamp generated by the unified clock source frame by frame to the image data and pose measurement data to form a time-aligned multimodal data group; Step S2: Identify the image data in the time-aligned multimodal data set and output the pixel coordinates of crops and weeds; Step S3: Based on the timestamp carried by the image in the time-aligned multimodal data group, retrieve the carrier pose that is time-synchronized with the image frame from the pose data bound to the same clock source timestamp. Step S4: Based on the end-to-end processing delay under the unified clock source constraint, predict the pose of the carrier after the delay, and transform the crop pixel coordinates and weed pixel coordinates to the operation coordinate system under the predicted pose to obtain the predictive crop operation coordinates and the predictive weed operation coordinates. Step S5: The instruction carrying the predictive weeding operation coordinates is transmitted to the actuator via a real-time bus that is synchronized with a unified clock source. In step S6, the actuator completes the target-aiming action within the execution window triggered by a unified clock source timing signal based on the predictive weed operation coordinates, and the operation area of ​​the target-aiming action avoids the crop position corresponding to the predictive crop operation coordinates.

[0009] Secondly, the present invention provides an integrated identification and positioning system for farmland weed control operations, comprising: The clock synchronization module is used to generate synchronization trigger pulses and clock count values; A multimodal sensor module is used to simultaneously acquire image data and pose measurement data on the same clock edge of the synchronous trigger pulse; The timestamp binding module is used to embed the clock count value at the trigger time as a timestamp into the image data and pose measurement data respectively, forming a time-aligned multimodal data set; The identification and localization module is used to identify crop pixel coordinates and weed pixel coordinates from time-aligned multimodal data sets; The spatiotemporal calibration module is used to retrieve the carrier pose synchronized with the image frame time from the pose data bound to the same clock source timestamp based on the timestamp carried by the image in the time-aligned multimodal data group, and generate predictive job coordinates based on the end-to-end processing delay under the constraint of a unified clock source. The constraint marking module is used to mark crop operation coordinates in the predictive operation coordinates as target restricted areas and weed operation coordinates as target operation points, and to establish the spatial constraint relationship between the two. The real-time bus, which keeps in time synchronization with the clock synchronization module, is used to transmit work instructions carrying predictive work coordinates and their spatial constraints to the actuator. The actuator, synchronized with the clock synchronization module, is used to complete the target-aligning action within the execution window triggered by the timing signal of the clock synchronization module according to the work instructions, and the working area of ​​the target-aligning action avoids the crop location.

[0010] The beneficial effects of the technical solution provided by this invention include at least the following: This invention synchronously triggers multimodal sensors by using a unified clock source and binds timestamps to image data and pose measurement data frame by frame, thereby forming time-aligned multimodal data sets and eliminating spatial calculation errors caused by asynchronous operation between sensors.

[0011] This invention identifies time-aligned multimodal data sets and outputs crop and weed pixel coordinates simultaneously, providing target locations for weeds and reference points for crop avoidance in subsequent operations.

[0012] This invention achieves a precise correspondence between image frames and carrier poses by retrieving carrier poses synchronized with the time of image frames from pose data constrained by the same clock source based on timestamps, thus avoiding coordinate transformation errors introduced by asynchronous pose data.

[0013] This invention predicts the future pose of the carrier based on the end-to-end processing delay under a unified clock source constraint, and transforms the pixel coordinates to the working coordinate system under the predicted pose. This can transform random and uncontrollable processing delays into a precisely compensable deterministic quantity, eliminating positioning deviations caused by delay jitter.

[0014] This invention transmits instructions carrying predictive weeding coordinates via a real-time bus synchronized with a unified clock source, ensuring a fixed instruction transmission delay and eliminating random jitter caused by bus contention.

[0015] This invention completes the target-targeting action within an execution window triggered by a unified clock source timing signal, and the working area avoids the crop position corresponding to the predictive crop operation coordinates, thus accurately hitting weeds while protecting crops from accidental damage. Attached Figure Description

[0016] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the method flow provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the system architecture provided for an embodiment of the present invention. Detailed Implementation

[0018] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of an integrated identification and positioning method and system for farmland weed protection operations proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0020] The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0021] The following description, in conjunction with the accompanying drawings, details the specific scheme of the integrated identification and positioning method and system for farmland weed protection operations provided by this invention.

[0022] Please see Figure 1 The diagram illustrates a flowchart of an integrated identification and positioning method for weed control operations in farmland, provided by an embodiment of the present invention, comprising the following steps: Step S1: Synchronously trigger the multimodal sensor through a unified clock source, and bind the timestamp generated by the unified clock source frame by frame to the image data and pose measurement data to form a time-aligned multimodal data group; Step S2: Identify the image data in the time-aligned multimodal data set and output the pixel coordinates of crops and weeds; Step S3: Based on the timestamp carried by the image in the time-aligned multimodal data group, retrieve the carrier pose that is time-synchronized with the image frame from the pose data bound to the same clock source timestamp. Step S4: Based on the end-to-end processing delay under the unified clock source constraint, predict the pose of the carrier after the delay, and transform the crop pixel coordinates and weed pixel coordinates to the operation coordinate system under the predicted pose to obtain the predictive crop operation coordinates and the predictive weed operation coordinates. Step S5: The instruction carrying the predictive weeding operation coordinates is transmitted to the actuator via a real-time bus that is synchronized with a unified clock source. In step S6, the actuator completes the target-aiming action within the execution window triggered by a unified clock source timing signal based on the predictive weed operation coordinates, and the operation area of ​​the target-aiming action avoids the crop position corresponding to the predictive crop operation coordinates.

[0023] It should be noted that a unified clock source refers to a high-precision clock generator capable of generating synchronous trigger pulses and clock count values. Its clock count value increases monotonically over time, providing a unified time reference for all links in the entire chain.

[0024] Multimodal sensors refer to a combination of sensors installed in the field of crops that can simultaneously acquire information about different modes of farmland. They include at least an RGB camera for acquiring visible light texture information, a multispectral imager for acquiring spectral differences between crops and weeds, and an inertial measurement unit for acquiring instantaneous pose information of the work vehicle.

[0025] A timestamp refers to writing the clock count value from a unified clock source at the moment of triggering sampling into the frame header metadata of each frame of image data and the data packet of each set of pose measurement data obtained in that sampling, so that all data corresponding to the same trigger pulse carries the same time reference mark.

[0026] A time-aligned multimodal data set refers to a data set consisting of a frame of RGB image, a frame of multispectral image, and a set of inertial measurement data generated by the same synchronization trigger pulse and carrying the same timestamp. The modal data in this data set are precisely aligned in the time dimension.

[0027] Crop pixel coordinates refer to the pixel-level coordinates of the crop plant's center position, output by the recognition network in the image coordinate system.

[0028] Weed pixel coordinates refer to the pixel-level coordinates of the center position of a weed plant, output by a recognition network in an image coordinate system.

[0029] The carrier pose for image frame time synchronization refers to the spatial pose information of the working carrier that is precisely corresponding to the current image frame at the sampling time, retrieved from the pose data stream through timestamp matching or timestamp interpolation.

[0030] End-to-end processing latency refers to the total time consumed by the entire processing chain from the moment image acquisition is triggered until the moment the actuator receives the instruction and completes the action. This time is a fixed value under the constraint of a unified clock source and includes the sum of image acquisition time, recognition time, coordinate transformation time, instruction transmission time, and actuator action time.

[0031] Predictive crop operation coordinates refer to the spatial coordinates representing the crop position obtained in the operation coordinate system after transforming the crop pixel coordinates through camera intrinsic and extrinsic parameters under the predicted future pose of the carrier.

[0032] Predictive weed control coordinates refer to the spatial coordinates representing the position of weeds obtained in the control coordinate system after transforming the pixel coordinates of weeds through camera intrinsic and extrinsic parameters under the predicted future pose of the carrier.

[0033] A real-time bus is an industrial fieldbus that maintains timing synchronization with a unified clock source. Its communication cycle is determined by the unified clock source, and each node completes data transmission and reception within a predetermined time offset, ensuring that the instruction transmission delay is a fixed value.

[0034] The execution window refers to a preset duration defined by a timed trigger signal from a unified clock source, during which the actuator completes the initiation, execution, and reset of the target action.

[0035] Targeting action refers to the application of herbicides or physical weeding operations performed by the actuator on the location of weeds based on the predicted weeding operation coordinates.

[0036] Avoiding crop locations in the work area means that during the target-oriented operation, the actuator uses the crop location corresponding to the predicted crop operation coordinates as a no-go zone, so that the spraying area or weeding range does not cover the no-go zone.

[0037] In one specific implementation, the operating vehicle is a self-propelled high-clearance crop protection machine, which travels at a constant speed of 5 km / h along the direction of the corn crop rows. The unified clock source is a temperature-controlled crystal oscillator clock generator installed in the control cabinet of the crop protection machine, with a clock frequency of 10 MHz, generating a synchronous trigger pulse with a period of 1 ms and a 64-bit unsigned integer clock count value. The multimodal sensors include an RGB camera facing the crop rows, a multispectral imager, and an inertial measurement unit. The RGB camera is a global shutter industrial camera with a resolution of 2048×1536. The multispectral imager covers a wavelength range of 600nm to 1000nm and includes 50 spectral bands with a spatial resolution of 1024×768. The RGB camera and the multispectral imager are mounted side by side on the front crossbeam of the crop protection machine, 1.5m above the ground, with the lens pointing vertically downwards to cover three rows of corn crops. The inertial measurement unit is installed at the geometric center of the crop protection machine chassis, with a sampling frequency of 1000Hz. The actuator is a high-speed solenoid valve nozzle array mounted on the spray boom of the agricultural sprayer. The nozzle spacing is 15cm, with a total of 24 nozzles. The response time of a single nozzle is less than 5ms. The real-time bus is an EtherCAT bus with a communication cycle of 1ms. The preset safety threshold is set to 5cm. The end-to-end processing latency was obtained by averaging the time from image acquisition triggering to the completion of the solenoid valve action; the calibration value is 45ms.

[0038] The specific implementation of step S1 is as follows: A unified clock source generates a synchronous trigger pulse with a period of 1ms. The rising edge of the trigger pulse is simultaneously connected to the external trigger interface of the RGB camera, the external trigger interface of the multispectral imager, and the synchronous latch interface of the inertial measurement unit (IMU) via shielded twisted-pair cables. Each sensor samples synchronously at the moment the rising edge of the trigger pulse arrives. The exposure time of the RGB camera and the multispectral imager is set to 0.5ms. The IMU outputs triaxial acceleration and triaxial angular velocity values. After the sampled data is output, within the first clock cycle, the current 64-bit clock count value at the trigger moment is used as a timestamp and written into the first 8 bytes of the frame header of the RGB image frame, the first 8 bytes of the frame header of the multispectral image frame, and the first 8 bytes of the message header of the inertial measurement data packet, respectively. This ensures that the RGB image, multispectral image, and inertial measurement data corresponding to the same trigger pulse carry the same timestamp, forming a time-aligned multimodal data group. After the timestamp binding is completed, the data is framed according to the trigger sequence number and sent to the calculation unit to execute subsequent steps.

[0039] The specific implementation of step S2: Receive the time-aligned multimodal data set from step S1. First, perform spectral analysis on the multispectral image using a spectral angle filling algorithm. Calculate the spectral angle of each pixel's 50-dimensional spectral vector with pre-labeled corn spectral reference vectors and typical weed spectral reference vectors. Pixels with spectral angles less than a preset threshold are assigned the corresponding crop or weed label, while the remaining pixels are marked as background, generating a category mask that distinguishes between crop and weed pixels. Simultaneously, perform texture and morphological analysis on the RGB image using a YOLOv8 object detection network to detect bounding boxes of all corn and weed plants in the image. Each bounding box is defined by its center point coordinates, width, and height. Then, register the category mask with the bounding boxes. Registration is achieved by mapping the coordinates of the four corner points of the bounding box to the corresponding pixel positions of the category mask according to the spatial resolution ratio of the multispectral image and the RGB image. Assign a category label to each bounding box, determined by majority voting within its coverage area. Extract the center point of the bounding box labeled as crop as the crop pixel coordinates, and extract the center point of the bounding box labeled as weed as the weed pixel coordinates. Next, for each weed pixel coordinate within the same frame, its Euclidean distance to all crop pixel coordinates is calculated. A KD-tree is used to accelerate the nearest neighbor search, finding the crop pixel coordinate with the smallest distance. A correspondence is established between this weed and its spatially nearest crop, recorded in JSON format and appended to the output coordinate data. Finally, the timestamps of the time-aligned multimodal data set are assigned to all output crop and weed pixel coordinates, and written to the Unix timestamp field of the coordinate data.

[0040] The specific implementation of step S3 is as follows: First, the pose measurement data carrying timestamps in each time-aligned multimodal data group are arranged in chronological order of acquisition time to form a pose data stream. The pose data stream is stored in the form of a circular buffer with a capacity of 2000 frames, covering a 2-second time window. The timestamps carried by the image frames are extracted from the current time-aligned multimodal data group, and the corresponding 64-bit unified clock source count value is parsed as the acquisition time of the image frame. The acquisition time is used as the query index to search in the pose data stream, and the matching judgment uses a precise comparison of 64-bit integers. If pose data with a timestamp exactly equal to the acquisition time exists in the pose data stream, then this pose data is directly extracted as the carrier pose synchronized with the image frame time. If no precisely matching pose data exists, the closest pose data frame with a timestamp less than the acquisition time and the closest pose data frame with a timestamp greater than the acquisition time are selected. Linear interpolation is performed with the acquisition time between the two timestamps as the weight. The position values ​​are interpolated separately according to the three-dimensional components in the northeast-northeast coordinate system. The attitude angles are converted to Euler angles after quaternion spherical linear interpolation to obtain the carrier attitude synchronized with the image frame time. Then, the real-time position and real-time heading angle are extracted from the carrier attitude synchronized with the image frame time. Using the crop row centerline obtained by fitting the crop pixel coordinates identified in step S2 using the least squares method as the reference, the vertical distance from the carrier centerline to the crop row centerline is calculated as the lateral position deviation. The distance from the carrier along the crop row direction to the nearest identified crop plant is calculated as the longitudinal position deviation. The angle between the carrier heading angle and the crop row direction is calculated as the heading deflection angle. Lateral position deviation, longitudinal position deviation, and heading deflection angle are encapsulated in the form of a structure as plant protection operation pose parameters. The units of lateral position deviation and longitudinal position deviation are meters (m), and the unit of heading deflection angle is degrees (degrees).

[0041] The specific implementation of step S4 is as follows: First, obtain the plant protection operation pose parameters output in step S3 and the carrier pose synchronized with the image frame time. Starting from the carrier pose synchronized with the image frame time, the carrier's travel direction is corrected according to the heading deflection angle to make the corrected travel direction parallel to the crop row direction. Based on the corrected travel direction, within a full-link processing delay of 45ms under a unified clock source constraint, the carrier pose is predicted using the Kalman filter algorithm. The Kalman filter adopts a constant velocity motion model. The state vector contains six state variables, including the carrier's three-dimensional position and three-dimensional velocity. The observation vector consists of six observations, including the three-dimensional velocity and three-dimensional attitude angle obtained by integrating the three-axis acceleration and three-axis angular velocity output by the inertial measurement unit. The system noise covariance matrix and the observation noise covariance matrix are statistically obtained from the inertial measurement unit data in a stationary state. Then, the predicted pose is compensated for laterally and longitudinally based on the lateral and longitudinal position deviations to obtain the plant protection operation pose. Using camera intrinsic parameters pre-calibrated using a checkerboard calibration method and camera extrinsic parameters pre-calibrated using a hand-eye calibration method, the crop and weed pixel coordinates are transformed from the image coordinate system to the operational coordinate system, respectively, to obtain predictive crop and predictive weed operational coordinates. The operational coordinate system has its origin at the starting position of the plant protection machine, with the X-axis perpendicular to the corn crop row direction and pointing to the right, and the Y-axis pointing forward along the corn crop row direction. Predictive crop operational coordinates are marked as target restricted areas with a protection radius set to 5 cm, while predictive weed operational coordinates are marked as target operational points. For each target operational point, the Euclidean distance between it and the center points of all target restricted areas is calculated. The target restricted area with the smallest distance is selected as the pairing object, and the Euclidean distance between the center points of this pair is calculated as the minimum safe distance. The minimum safe distance of each target operational point and the coordinates of the nearest paired target restricted area are used as spatial constraints and appended to the coordinate data of that target operational point. The minimum safe distance is used to determine whether to suppress the target-aiming action at the target-aiming operation point, and the coordinates of the nearest target-aiming restricted area are used to determine the avoidance direction of the target-aiming action.

[0042] The specific implementation of step S5 is as follows: Read the coordinates of each predictive weeding operation output from step S4, and obtain the minimum safe distance and the coordinates of the nearest prohibited area for that operation point. Compare the minimum safe distance with a preset safety threshold of 5cm. If the minimum safe distance is less than 5cm, suppress the generation of the instruction for that operation point, and the weeds corresponding to that operation point are abandoned. If the minimum safe distance is greater than or equal to 5cm, calculate the avoidance direction based on the coordinates of the nearest prohibited area. The avoidance direction is a unit vector pointing in the opposite direction from the target operation point to the nearest prohibited area. Generate an operation instruction carrying the coordinates of the operation point and the avoidance direction. The operation instruction is encapsulated as an EtherCAT data frame. The frame structure includes a frame header, node address field, operation point X coordinate field, operation point Y coordinate field, avoidance direction X component field, avoidance direction Y component field, and cyclic redundancy check field. The length of a single frame is 64 bytes. Within a 1ms communication period determined by a unified clock source, the operation instruction is sent to the corresponding actuator node via the EtherCAT bus. The actuator node receives and parses commands within a predetermined time offset allocated during system initialization. The first node's receiving offset is 100μs, and each subsequent node increments by 40μs. After obtaining the work point coordinates and avoidance direction, it stores them in its local buffer and returns an acknowledgment frame to the EtherCAT master station. If no work command is received within three consecutive communication cycles, the actuator node automatically enters an idle state, and the nozzle remains closed.

[0043] The specific implementation of step S6: The actuator node calculates the action offset according to the avoidance direction in the operation instruction. The deflection angle is the angle between the avoidance direction and the vertical downward direction. The offset distance is calculated using a ratio function of the preset offset upper limit of 8cm, the minimum safe distance, and the preset safe threshold of 5cm. The smaller the minimum safe distance, the larger the offset distance. When the minimum safe distance is greater than 15cm, the offset distance is zero. Then, the plant protection operation pose parameters obtained from the latest sampled data of the inertial measurement unit at the current moment are obtained. The lateral and longitudinal position deviations at the current moment are subtracted from the deviation values ​​stored during the prediction in step S4. The difference is added to the deflection angle and offset distance with a proportional coefficient of 0.8 for real-time compensation. After that, the next timed trigger signal from the unified clock source is waited for. The timed trigger signal period is 1ms. When the timed trigger signal arrives, within the preset 20ms time window, the target nozzle is determined based on the closest distance matching between the X coordinate of the operation point and the X coordinate of each nozzle, and the corresponding solenoid valve is opened. The nozzle is adjusted to spray the direction according to the compensated deflection angle and offset distance to complete the target spraying action. At the end of the 20ms time window, the solenoid valve closes, the nozzle rebounds to the vertically downward position and automatically resets, and the system returns to the ready state waiting for the next instruction.

[0044] In the above specific implementation, the self-propelled high-clearance plant protection machine travels at a speed of 5 km / h in the corn crop row. A unified clock source drives the multi-modal sensors to sample synchronously with a 10MHz frequency and a 1ms synchronous trigger pulse. Within the first clock cycle of the sampled data output, a 64-bit clock count value is embedded as a timestamp into each modal data to form a time-aligned multi-modal data group. The machine sequentially performs multispectral and RGB collaborative recognition and outputs crop and weed pixel coordinates with timestamps and spatial correspondences; synchronous attitude retrieval and plant protection operation pose parameter extraction; Kalman filter prediction and coordinate transformation and spatial constraint relationship establishment; safety threshold judgment and avoidance direction calculation; and EtherCAT command transmission. The actuator calculates the action offset based on the avoidance direction and compensates using real-time pose parameters, completing target spraying and automatic reset within a 20ms time window controlled by the unified clock source's timed trigger signal. The total delay from image acquisition to the completion of the actuator action is determined to be 45ms. Each link forms a deterministic processing pipeline under the constraint of a unified clock source. The random jitter component in the target positioning error is eliminated, and the residual error is determined only by the quantifiable system delay compensation.

[0045] Step S1 further includes the following sub-steps: S1-1 generates a synchronous trigger pulse through a unified clock source and outputs it to the hardware trigger interface of the multimodal sensor, which includes an RGB camera for crop rows in farmland, a multispectral imager, and an inertial measurement unit. S1-2, through the multimodal sensor, synchronously samples on the same clock edge of each trigger pulse to obtain a frame of RGB image containing crops and weeds, a frame of multispectral image reflecting the spectral differences between crops and weeds, and a set of inertial measurement data for calculating the instantaneous pose of the working vehicle. S1-3, the unified clock source count value at the trigger moment is used as a timestamp and embedded into the RGB image data, multispectral image data and inertial measurement data obtained in this sampling, so that the RGB image, multispectral image and inertial measurement data corresponding to the same trigger pulse carry the same timestamp, forming a set of time-aligned multimodal data for weed identification and crop positioning.

[0046] It should be noted that the synchronous trigger pulse refers to an electrical signal with a fixed period generated by a unified clock source. The rising or falling edge of this signal arrives at the hardware trigger interface of each multimodal sensor at the same time, so that each sensor starts exposure or sampling at the same moment.

[0047] The hardware trigger interface refers to the physical pin or signal port on a multimodal sensor used to receive external trigger signals. The sensor's exposure or sampling actions are directly controlled by the edge of the trigger signal received by this interface, rather than being driven by the freely running clock inside the sensor.

[0048] The same clock edge refers to the same rising edge or the same falling edge of the synchronous trigger pulse. Each sensor performs sampling operation synchronously at the moment the clock edge arrives, ensuring that the sampling time of each modal data is consistent at the physical level.

[0049] An RGB camera facing the rows of crops in farmland refers to a visible light industrial camera mounted on the front beam of the work carrier, with its lens pointing vertically or tilted downwards at the area of ​​the rows of crops in farmland, and its optical axis direction forming a fixed geometric relationship with the direction of the crop rows.

[0050] A multispectral imager is an imaging device that covers multiple spectral bands, including at least the visible light band and the near-infrared band. It is used to distinguish between crops and weeds by the difference in reflectance under different bands.

[0051] An inertial measurement unit (IMU) is a microelectromechanical system sensor that integrates a triaxial accelerometer and a triaxial gyroscope. It is installed at the geometric center of the chassis of the work vehicle to measure the triaxial acceleration and triaxial angular velocity of the vehicle in real time, and to obtain the instantaneous position and attitude of the vehicle through integration calculation.

[0052] An RGB image containing crops and weeds refers to a two-dimensional digital image obtained by imaging in the visible light band, which fully records the texture, color, and morphological information of crop plants and weed plants within the crop row area of ​​farmland.

[0053] Multispectral images reflecting the spectral differences between crops and weeds refer to multichannel digital images obtained by imaging multiple spectral bands separately, recording the differences in reflectance between crops and weeds in each band. There are significant differences in reflectance between crops and weeds in certain specific bands.

[0054] The inertial measurement data for calculating the instantaneous attitude of the work vehicle refers to the three-axis acceleration and three-axis angular velocity values ​​output by the inertial measurement unit at the moment of sampling. These values ​​are used to calculate the spatial position and attitude angle of the vehicle at that sampling moment using the strapdown inertial navigation algorithm.

[0055] The unified clock source count value at the trigger moment refers to the current value of the internal counter of the unified clock source at the instant the synchronous trigger pulse is generated. This value increases monotonically with the oscillation of the unified clock source.

[0056] Embedding refers to writing timestamp data into the frame header metadata area of ​​the image data frame and the packet header of the inertial measurement data, making the timestamp an integral part of the data frame.

[0057] Carrying the same timestamp means that the timestamp values ​​recorded in the RGB image data frame, multispectral image data frame and inertial measurement data packet corresponding to the same trigger pulse are completely consistent, indicating that these three sets of data come from the same synchronous trigger sampling.

[0058] Step S1 addresses the asynchronous sampling timing issue caused by the independent operating clocks of the multimodal sensors. When the multimodal sensors operate independently with their internal clocks, the time deviation at the moment of sampling by each sensor leads to temporal misalignment between image frames and pose data in subsequent steps. This asynchronous error is particularly significant in high-speed travel and bumpy operating environments and cannot be completely eliminated by software compensation. This step generates periodic synchronization trigger pulses through a unified clock source and simultaneously connects these pulses to the hardware trigger interfaces of all multimodal sensors. This forces all sensors to synchronously perform sampling actions on the same electrical signal edge, physically eliminating the sampling timing differences between sensors. Simultaneously, the clock count value at the moment of triggering is directly embedded as a timestamp into each frame of image data and each set of pose measurement data, forming a time-aligned multimodal data set. This mechanism establishes a unified time reference across the entire link, providing the data prerequisite for retrieving the carrier pose precisely synchronized with the image frame by timestamp in step S3, and also providing quantifiable time parameters for the predictive coordinate transformation based on deterministic delay in step S4.

[0059] Step S2 further includes the following sub-steps: S2-1, Perform spectral analysis on multispectral images in time-aligned multimodal datasets to generate category masks that distinguish between crop pixels and weed pixels; S2-2, perform texture and morphological analysis on RGB images in time-aligned multimodal data sets to detect bounding boxes of all plants in the images; S2-3, Register the category mask with the bounding box, and assign the category label of the corresponding region in the category mask to each bounding box; S2-4, output the center point of the bounding box labeled as crop as the crop pixel coordinates, and output the center point of the bounding box labeled as weed as the weed pixel coordinates; S2-5, Establish the correspondence between the coordinates of each weed pixel in the same frame and the coordinates of the crop pixel with the nearest spatial distance, and append this correspondence to the output coordinate data; S2-6 assigns the timestamps of the time-aligned multimodal data sets to the output crop and weed pixel coordinates.

[0060] It should be noted that spectral analysis refers to classifying the spectral feature vector of each pixel in a multispectral image by utilizing the differences in reflectance between crops and weeds in different bands, thus distinguishing pixels belonging to crops from pixels belonging to weeds.

[0061] A category mask is a binary or labeled image with the same spatial dimensions as a multispectral image, where the value at each pixel location represents the category to which the pixel belongs; that is, crop pixels are labeled as the first category value, weed pixels as the second category value, and background pixels as the third category value.

[0062] Texture and morphology analysis refers to using information such as texture patterns, edge contours, and shape features of plant leaves and stems in RGB images to identify each individual plant in the image and determine its bounding rectangle through an object detection network.

[0063] A bounding box is a rectangular bounding box output by an object detection network that completely selects a single plant in an RGB image. Each bounding box is defined by four parameters: center point coordinates, width, and height.

[0064] Registration refers to aligning the category mask generated on the multispectral image with the bounding box generated on the RGB image in spatial position. Since the multispectral image and the RGB image originate from the same synchronous sampling and have the same field of view, registration is achieved through pixel-level direct mapping.

[0065] Category labels are semantic identifiers assigned to each bounding box, with values ​​of crop or weed. The label is determined by the majority of pixel categories in the category mask within the area covered by the bounding box.

[0066] The closest spatial distance refers to calculating the Euclidean distance between a weed pixel coordinate and all crop pixel coordinates within the same frame of an image, starting from a certain weed pixel coordinate, and taking the crop pixel coordinate corresponding to the minimum distance as the nearest crop for that weed.

[0067] The correspondence relationship refers to the one-to-one or one-to-many mapping relationship between the pixel coordinates of each weed in the same frame of an image and the pixel coordinates of the nearest crop in terms of spatial distance. This relationship is recorded in the form of a data table, which contains the pixel coordinate values ​​of weeds and the corresponding nearest crop pixel coordinate values.

[0068] Adding the correspondence to the output coordinate data means that when outputting the pixel coordinate data of weeds, the pixel coordinates of the nearest crop corresponding to the weed are written as an additional field into the same data record.

[0069] The timestamp of the time-aligned multimodal data group refers to the unified clock source count value embedded in step S1, which is shared by all modal data within the data group. Assigning the timestamp to the output crop pixel coordinates and weed pixel coordinates means writing the timestamp field into the output data record, so that the output coordinate data maintains time traceability with the original image data.

[0070] Step S2 addresses the problem of existing weed identification methods only outputting weed locations while ignoring crop locations and lacking a crop spatial reference. Existing weed identification methods typically only output weed locations, ignoring crop location information. In scenarios where weeds and crops grow adjacently or interspersed, the lack of a crop spatial reference makes it impossible to determine safe distances in subsequent operations, posing a risk of accidental damage. This step utilizes multispectral and RGB images from a time-aligned multimodal data set for collaborative identification. Multispectral images provide the reflectance differences between crops and weeds at different wavelengths, generating category masks that distinguish crop pixels from weed pixels through spectral analysis. RGB images provide rich texture and morphological information, detecting bounding boxes for all plants in the image using an object detection network. After registering the category masks with the bounding boxes, each bounding box obtains a crop or weed category label, thereby extracting crop pixel coordinates and weed pixel coordinates. Based on this, a correspondence is established between each weed pixel coordinate within the same frame and the coordinates of its spatially nearest crop pixel, and the timestamp of the time-aligned multimodal data set is assigned to all coordinates. This collaborative identification mechanism establishes the spatial constraint relationship between weeds and crops explicitly in the identification and positioning stage for the first time. This provides a basis for marking crop coordinates as the target restricted area in step S4, and also provides spatial reference data for the suppression judgment and avoidance direction calculation based on the minimum safe distance in step S5.

[0071] Step S3 further includes the following sub-steps: S3-1, Time-aligned pose measurement data with timestamps in the multimodal data set are arranged in chronological order of acquisition time to form a pose data stream; S3-2: Extract the timestamp carried by the image frame from the current time-aligned multimodal data group, parse the unified clock source count value corresponding to the timestamp as the acquisition time of the image frame, and use the acquisition time as the query index to search in the pose data stream. S3-3, If there is pose data in the pose data stream that matches the acquisition time, then extract the pose data as the carrier pose synchronized with the image frame time. S3-4. If there is no pose data with a matching timestamp in the pose data stream, select the pose data of one frame before and one frame after the acquisition time, and perform interpolation calculation with the acquisition time as the interpolation point to obtain the carrier pose synchronized with the image frame time.

[0072] Furthermore, step S3 also includes extracting the pose parameters for plant protection operations: The real-time position and real-time heading angle of the carrier are obtained from the carrier attitude synchronized with the image frame time. Based on the spatial relationship between the real-time location and the predetermined crop row centerline, the vertical distance between the carrier centerline and the crop row centerline is calculated as the lateral position deviation. The longitudinal position deviation is calculated based on the difference between the real-time position along the direction of travel and the coordinates of the identified plants in the crop row; The angle between the real-time heading angle and the crop row direction is taken as the heading deflection angle; Lateral position deviation, longitudinal position deviation, and heading deflection angle are used as plant protection operation pose parameters to characterize the real-time pose of the carrier relative to the crop row.

[0073] It should be noted that the pose measurement data refers to the spatial position and attitude angle data of the carrier obtained by the strapdown inertial navigation system after the three-axis acceleration and three-axis angular velocity values ​​output by the inertial measurement unit in each synchronous trigger sampling are calculated.

[0074] The acquisition time refers to the point in time corresponding to the unified clock source count value recorded by the timestamp.

[0075] Pose data stream refers to an ordered data sequence formed by arranging pose measurement data generated from each sampling in ascending order according to the acquisition time represented by the timestamps it carries, supporting fast retrieval by timestamp.

[0076] A query index is a search keyword used to locate target data in a pose data stream; here, the acquisition time of an image frame is used as the query index.

[0077] Matching refers to the fact that the timestamp carried by a frame of pose data in the pose data stream is numerically equal to the acquisition time of the image frame.

[0078] Interpolation calculation refers to the process of selecting the previous and next frame of pose data with the timestamps closest to the acquisition time when there is no pose data with an exact timestamp in the pose data stream. Using the ratio between the acquisition time and these two timestamps as weights, linear interpolation is performed on the position and attitude angle values ​​of the two frames of pose data to obtain the pose estimate corresponding to the acquisition time.

[0079] Real-time position refers to the three-dimensional spatial coordinates of the carrier in the geodetic coordinate system, which are contained in the carrier's posture synchronized with the image frame time.

[0080] The real-time heading angle refers to the angle of deflection of the longitudinal axis of the carrier relative to true north, which is included in the carrier attitude synchronized with the image frame time.

[0081] The crop row centerline is a spatial straight line representing the direction of the crop row, obtained by fitting the crop pixel coordinates output in step S2 using the least squares method.

[0082] Lateral positional deviation refers to the vertical distance from the centerline of the carrier to the centerline of the crop row, which is measured in a direction perpendicular to the direction of the crop row.

[0083] Longitudinal position deviation refers to the distance from the current position of the carrier along the crop row direction to the nearest identified crop plant.

[0084] The heading deflection angle refers to the angle between the longitudinal axis of the carrier and the direction of the crop row.

[0085] Plant protection operation pose parameters refer to the pose description vector composed of three parameters: lateral position deviation, longitudinal position deviation, and heading deflection angle. It is used to describe the spatial offset relationship between the operation carrier and the crop row.

[0086] Step S3 addresses the asynchronous error problem caused by the lack of time synchronization between carrier attitude data and image frames in existing methods. Existing methods typically directly read the latest output value of the inertial measurement unit when acquiring carrier attitude, but this output value does not have a precise correspondence with the sampling time of the current image frame. When the carrier is in motion, this asynchronous error directly translates into spatial deviation in subsequent coordinate transformations. This step first arranges the pose measurement data in each time-aligned multimodal data group according to the timestamp sequence to form a pose data stream. Then, using the timestamp carried by the current image frame as the query index, it retrieves pose data matching the acquisition time in the pose data stream. When a precise match exists, it is directly extracted. When the pose sampling frequency is higher than the image sampling frequency, resulting in no precise match, the carrier attitude synchronized with the image frame time is obtained by interpolating the pose data of the two frames before and after. This time-synchronized retrieval mechanism utilizes the time-aligned data group established in step S1 to achieve a precise correspondence between the image frame and the carrier attitude in the time dimension. Based on this, three plant protection operation pose parameters—lateral position deviation, longitudinal position deviation, and heading deflection angle—are extracted from the synchronous attitude. This transforms the general carrier pose information into a spatial deviation quantification index for plant protection operations, providing directly usable parameters for pose prediction in step S4 and real-time compensation in step S6.

[0087] Step S4 further includes the following sub-steps: S4-1: Obtain the plant protection operation pose parameters, and take the carrier attitude synchronized with the image frame time as the starting point, and make angle correction on the carrier's travel direction according to the heading deflection angle; S4-2, based on the corrected direction of travel, predicts the pose of the carrier using the Kalman filter algorithm within the duration of the end-to-end processing delay under a unified clock source constraint. S4-3, based on the lateral and longitudinal position deviations, perform lateral and longitudinal position compensation on the predicted pose to obtain the plant protection operation pose; S4-4: Using the camera intrinsic parameters and the camera extrinsic parameters under the plant protection operation pose, the crop pixel coordinates are transformed to the operation coordinate system to obtain the predictive crop operation coordinates. S4-5: Using the camera's intrinsic parameters and extrinsic parameters under the plant protection operation pose, the weed pixel coordinates are transformed to the operation coordinate system to obtain the predictive weed operation coordinates. S4-6 marks the predictive crop operation coordinates as the target restricted area and the predictive weed operation coordinates as the target operation points, and adds the spatial constraint relationship between the target restricted area and the target operation points to the predictive weed operation coordinates.

[0088] Furthermore, in sub-steps S4-6, spatial constraints are established in the following manner: Each target engagement point is paired with its nearest spatially distant target restricted area; Calculate the Euclidean distance between each target engagement point and the center point of its paired target restricted area, and use this distance as the minimum safe distance for that target engagement point. The minimum safe distance of each target engagement point and the coordinates of the nearest target restricted area are used as the spatial constraint relationship of that target engagement point and are added to the coordinate data of that target engagement point. Among them, the minimum safe distance is used to determine whether to suppress the target-aiming action at the target-aiming operation point, and the coordinates of the nearest target-aiming restricted area are used to determine the avoidance direction of the target-aiming action.

[0089] It should be noted that the plant protection operation pose parameters refer to the pose description vector composed of the lateral position deviation, longitudinal position deviation, and heading deflection angle output in step S3.

[0090] The heading deflection angle refers to the angle between the longitudinal axis of the carrier and the direction of the crop row.

[0091] Angle correction refers to subtracting the heading deflection angle from the heading angle in the carrier's attitude, which is synchronized with the image frame time, so that the corrected direction of travel is consistent with the crop row direction, eliminating the directional error caused by the carrier deviating from the crop row direction due to terrain or control reasons.

[0092] The corrected direction of travel refers to the direction of carrier movement that is parallel to the crop's direction of movement, obtained after compensation for heading deflection angle.

[0093] The Kalman filter algorithm is a filtering algorithm that uses the modified motion model of the carrier in the direction of travel as the state equation and the real-time acceleration and angular velocity observations of the inertial measurement unit as the observation equation to recursively estimate the carrier's pose after a delay.

[0094] Predicted pose refers to the estimated pose of the carrier at a future time, output by the Kalman filter algorithm.

[0095] Lateral positional deviation refers to the vertical distance from the centerline of the carrier to the centerline of the crop row.

[0096] Longitudinal positional deviation refers to the distance from the carrier along the crop row direction to the nearest identified crop plant.

[0097] Position compensation refers to subtracting the lateral position deviation from the lateral coordinates and the longitudinal position deviation from the longitudinal coordinates in the predicted pose, so that the origin of the compensated system is aligned with the center line of the crop row and the longitudinal axis is aligned with the position of the identified plants in the crop row.

[0098] Plant protection operation posture refers to the spatial posture of the carrier based on crop behavior, obtained after lateral and longitudinal position compensation.

[0099] Camera intrinsic parameters refer to the internal geometric parameters of an RGB camera, such as focal length, principal point coordinates, and distortion coefficients.

[0100] Camera extrinsic parameters refer to the rotation matrix and translation vector between the camera coordinate system and the carrier coordinate system.

[0101] The operation coordinate system refers to a plane rectangular coordinate system with the intersection of the crop row centerline and the operation starting line as the origin, the crop row direction as the vertical axis, and the direction perpendicular to the crop row as the horizontal axis.

[0102] The target-targeting restricted area refers to a circular area centered on the predictive crop operation coordinates and with a preset protection radius, within which target-targeting actions are prohibited.

[0103] The target operation point refers to the spatial location point represented by the predictive weeding operation coordinates.

[0104] Pairing refers to calculating the Euclidean distance between each target operation point and the center point of all target restricted areas, and selecting the target restricted area with the smallest distance as its pairing object.

[0105] The minimum safe distance refers to the Euclidean distance between the target operation point and the center point of its paired target restricted area.

[0106] The coordinates of the nearest target-restricted area refer to the coordinates of the center point of the target-restricted area paired with the target-targeting operation point in the operation coordinate system.

[0107] Spatial constraint relationships refer to data pairs consisting of the minimum safe distance and the coordinates of the nearest prohibited target area, which are used in subsequent steps to determine whether to perform a target-aiming action and to determine the direction of the action.

[0108] Suppression refers to canceling the target-aiming action command for the target-aiming operation point when the minimum safe distance is less than the preset safe threshold.

[0109] The avoidance direction refers to the unit vector pointing in the opposite direction from the target operation point to the nearest target restricted area. The actuator adjusts the spraying or weeding angle along this direction to move away from the crop.

[0110] Step S4 addresses the issues of coordinate transformation lag errors caused by carrier movement during processing delays and the lack of crop safety constraint information in the operational coordinates in existing methods. Existing methods typically assume that the carrier's pose at the moment of image acquisition is consistent with its pose at the moment of execution when transforming image coordinates to operational coordinates. However, during high-speed travel, there is a non-negligible processing delay between acquisition and execution, during which the carrier undergoes significant displacement and attitude changes. This step first corrects the carrier's travel direction based on the heading deflection angle output in step S3, eliminating directional errors caused by the carrier deviating from the crop's path due to terrain or manipulation. Based on the corrected travel direction, within the duration of the end-to-end processing delay under a unified clock source constraint, the Kalman filter algorithm predicts the carrier's future pose, transforming the processing delay from a random and uncontrollable factor into a deterministic prediction duration parameter. Then, position compensation is performed on the predicted pose based on lateral and longitudinal position deviations to obtain the crop protection operational pose with crop behavior as the reference. Finally, the crop pixel coordinates and weed pixel coordinates are transformed to the operational coordinate system under this pose using camera intrinsic and extrinsic parameters, resulting in predictive operational coordinates. Finally, crop operation coordinates are marked as target restricted areas, and weed operation coordinates are marked as target operation points. The minimum safe distance is calculated by pairing each operation point with the nearest restricted area to establish spatial constraint relationships. This predictive spatiotemporal calibration mechanism embeds crop safety constraint information into operation coordinates for the first time, allowing subsequent steps to make safety decisions directly based on these constraints without recalculation.

[0111] Step S5 further includes the following sub-steps: S5-1, for each predicted weeding operation coordinate, read the minimum safe distance and the coordinates of the nearest target-restricted area for that operation point; S5-2, compare the minimum safe distance with the preset safe threshold. If it is less than the preset safe threshold, suppress the generation of instructions for that work point. S5-3, if it is greater than or equal to the preset safety threshold, calculate the avoidance direction based on the coordinates of the nearest target restricted area, and generate an operation instruction carrying the coordinates of the operation point and the avoidance direction; S5-4, within the communication cycle determined by a unified clock source, sends the work instructions to the corresponding actuator nodes via the real-time bus; S5-5, the actuator node receives and parses instructions within a defined time offset constrained by a unified clock source, and obtains the corresponding work point coordinates and spatial constraint relationships.

[0112] It should be noted that the minimum safe distance refers to the Euclidean distance between the target operation point and the center point of its paired target restricted area, calculated in step S4.

[0113] The coordinates of the nearest target restricted area refer to the coordinates of the center point of the target restricted area in the working coordinate system that is paired with the target working point in step S4.

[0114] The preset safety threshold refers to the minimum allowable operating distance set in advance. When the distance between the target operation point and the nearest crop is less than the threshold, the target operation point shall not be subjected to the target operation to avoid accidentally damaging the crop.

[0115] Suppression refers to not generating a target operation instruction for that operation point, so that the weeds corresponding to that operation point are not sprayed or hoeed.

[0116] The avoidance direction refers to the unit vector pointing in the opposite direction from the target operation point to the nearest prohibited area, indicating the safe direction that the actuator should face when spraying or weeding.

[0117] A work instruction is a data packet consisting of the coordinates of the work point and the avoidance direction, which is sent to the execution node. The execution node then performs the target-aiming action based on the data packet.

[0118] The communication cycle determined by the unified clock source refers to the fixed time slices that the real-time bus divides based on the clock signal of the unified clock source, and one data transmission and reception is completed within each time slice.

[0119] An actuator node refers to a microcontroller node mounted on a real-time bus that independently controls a single nozzle or weeding blade. Each node corresponds to a physical actuator.

[0120] The time offset of the unified clock source constraint refers to the fixed offset of the data reception time allocated to each actuator node within the communication cycle relative to the start of the communication cycle. This offset is determined during system initialization and remains unchanged during operation.

[0121] Spatial constraint relationships refer to data pairs consisting of the minimum safe distance and the coordinates of the nearest target restricted area.

[0122] Step S5 addresses the problem in existing methods where safety assessment lags behind instruction generation, and the execution mechanism needs to independently determine crop avoidance, leading to high complexity at the execution layer. Existing methods typically send the coordinates of all identified weed targets directly to the execution mechanism when generating operation instructions, without performing safety assessments at the instruction stage. This forces the execution mechanism to independently determine whether crop damage is possible during operation, or to completely delegate safety assessments to subsequent execution stages, increasing the complexity of the execution layer. This step first reads each predictive weed operation coordinate output from step S4, along with its associated minimum safety distance and the coordinates of the nearest prohibited area. The minimum safety distance is compared with a preset safety threshold. For operation points too close to crops, instruction generation is directly suppressed to avoid accidental damage at the source. For operation points within the safety range that require avoidance, the avoidance direction is calculated based on the coordinates of the nearest prohibited area, generating an operation instruction carrying the operation point coordinates and the avoidance direction. The instruction is sent via a real-time bus within a communication cycle determined by a unified clock source, and the execution mechanism node receives and parses it within a defined time offset. The instruction transmission delay is a fixed value, not a random one. This proactive safety decision-making mechanism moves crop avoidance judgments from the execution level to the instruction generation stage, allowing the implementing agency to simply follow instructions without having to make its own judgments, thus separating decision-making from execution.

[0123] Step S6 further includes the following sub-steps: S6-1, Calculate the action offset of the actuator according to the avoidance direction in the work instruction. The action offset includes the deflection angle and offset distance of the nozzle or weeding blade relative to the default work direction. S6-2, obtain the plant protection operation posture parameters at the current moment, and compensate for the action offset in real time based on the lateral and longitudinal position deviations; S6-3 waits for the next timing trigger signal from the unified clock source. When the timing trigger signal arrives, within the preset time window, it locates the target position according to the work point coordinates in the work instruction, and adjusts the spraying angle or weeding angle of the actuator according to the compensated action offset. S6-4: When the time window ends, the actuator automatically resets and waits for the next instruction.

[0124] It should be noted that the avoidance direction refers to the unit vector carried in the operation instruction generated in step S5, which points from the target operation point in the opposite direction to the nearest target restricted area.

[0125] Action offset refers to the amount of adjustment of the actuator relative to the default working direction when performing a target-aiming action, including two components: deflection angle and offset distance.

[0126] The default operating direction refers to the spraying or weeding direction adopted by the actuator when no avoidance direction is received. It is usually directly below or vertically downward.

[0127] The deflection angle refers to the angle at which the nozzle or weeding blade rotates from the default working direction to the avoidance direction.

[0128] Offset distance refers to the amount of displacement of the nozzle or weeding blade in the avoidance direction. This distance is inversely proportional to the minimum safe distance. The smaller the minimum safe distance, the larger the offset distance, so as to ensure that the spraying or weeding range is far away from the crop.

[0129] The plant protection operation pose parameters at the current moment refer to the lateral position deviation and longitudinal position deviation calculated from the latest sampled data of the inertial measurement unit when step S6 is executed.

[0130] Real-time compensation refers to comparing the current lateral position deviation with the lateral position deviation used in step S4 prediction, calculating the difference, and then adding it to the deflection angle for correction. Similarly, it compares the current longitudinal position deviation with the longitudinal position deviation used in step S4 prediction, calculates the difference, and then adds it to the offset distance for correction, in order to eliminate the error caused by the change in carrier pose during the period from the issuance of the command in step S5 to the actual execution in step S6.

[0131] A timed trigger signal refers to a periodic timed pulse generated by a unified clock source, which is used to trigger the actuator to start its action at a preset time.

[0132] The preset time window refers to a fixed period of time from the arrival of the timed trigger signal to the start of timing, during which the actuator completes all mechanical movements of the target action.

[0133] The duration of the time window is longer than the shortest time required for the actuator to complete a single spraying or weeding action.

[0134] Positioning the target according to the work point coordinates refers to moving the end of the actuator nozzle or weeding blade to the spatial position indicated by the work point coordinates in the work instruction.

[0135] Adjusting the spraying angle or weeding angle according to the compensated action offset means deflecting the working direction of the actuator nozzle or weeding blade from the default direction by the deflection angle after real-time compensation, and shifting the working range along the avoidance direction by the compensated offset distance.

[0136] Automatic reset refers to the actuator automatically restoring the nozzle or weeding blade from its deflected working position to the default direction position at the end of the time window, closing the nozzle valve or retracting the weeding blade, and returning to the initial standby state.

[0137] Step S6 addresses the problem in existing methods where the actuator operates in a fixed direction and cannot dynamically avoid obstacles based on crop position. In existing methods, the actuator typically performs spraying or weeding actions in a fixed direction, unable to dynamically adjust according to crop position. When the actuator operates in the predetermined direction, if the spraying or weeding area covers adjacent crops, it will cause accidental damage. This step converts the avoidance direction in the operation command into an executable mechanical action offset for the actuator, including the deflection angle and offset distance of the nozzle or weeding blade relative to the default direction. Before execution, the latest plant protection operation pose parameters are obtained and compared with the parameters used in step S4 for prediction. The difference is calculated, and the action offset is compensated in real time to eliminate additional pose changes caused by carrier movement during the period from command issuance to actual execution. Within the execution window triggered by a unified clock source, the target position is located according to the operation point coordinates, and the execution angle is adjusted according to the compensated offset. The window automatically resets at the end. This execution mechanism transforms the avoidance direction generated in step S5 into precise mechanical actions, and through secondary compensation before execution and timed actions within a defined time window, ensures that the target operation accurately hits the weeds while effectively protecting the crops.

[0138] Please see Figure 2 The diagram illustrates a system architecture of an integrated identification and positioning system for farmland weed control operations, according to an embodiment of the present invention, comprising: The clock synchronization module is used to generate synchronization trigger pulses and clock count values; A multimodal sensor module is used to simultaneously acquire image data and pose measurement data on the same clock edge of the synchronous trigger pulse; The timestamp binding module is used to embed the clock count value at the trigger time as a timestamp into the image data and pose measurement data respectively, forming a time-aligned multimodal data set; The identification and localization module is used to identify crop pixel coordinates and weed pixel coordinates from time-aligned multimodal data sets; The spatiotemporal calibration module is used to retrieve the carrier pose synchronized with the image frame time from the pose data bound to the same clock source timestamp based on the timestamp carried by the image in the time-aligned multimodal data group, and generate predictive job coordinates based on the end-to-end processing delay under the constraint of a unified clock source. The constraint marking module is used to mark crop operation coordinates in the predictive operation coordinates as target restricted areas and weed operation coordinates as target operation points, and to establish the spatial constraint relationship between the two. The real-time bus, which keeps in time synchronization with the clock synchronization module, is used to transmit work instructions carrying predictive work coordinates and their spatial constraints to the actuator. The actuator, synchronized with the clock synchronization module, is used to complete the target-aligning action within the execution window triggered by the timing signal of the clock synchronization module according to the work instructions, and the working area of ​​the target-aligning action avoids the crop location.

[0139] It should be noted that this system solves the problems of inconsistent time bases, random processing delays, and lack of crop avoidance constraints among modules in existing identification and positioning systems by assembling eight functional modules: clock synchronization, multimodal sensing, timestamp binding, identification and positioning, spatiotemporal calibration, constraint marking, real-time bus transmission, and avoidance execution, into a full-link clock-constrained processing pipeline. Specifically, the clock synchronization module generates a unified synchronization trigger pulse and clock count value, serving as the trigger source for synchronous sampling by the multimodal sensor module and the timing reference for the timestamp binding module. The multimodal sensor module synchronously acquires image data and pose measurement data on the same clock edge of the trigger pulse. The timestamp binding module writes the clock count value into each data frame at the first moment of data output, forming a time-aligned multimodal data set. The identification and positioning module collaboratively utilizes multispectral and RGB modal information from this data set, simultaneously outputting the pixel coordinates of crops and weeds and establishing the spatial correspondence between them. The spatiotemporal calibration module retrieves the carrier pose precisely synchronized with the image frame from the pose data stream using timestamps as indexes. It then uses a Kalman filter to predict the future pose based on the deterministic end-to-end processing delay, transforming pixel coordinates into predictive operational coordinates. The constraint marking module performs semantic marking and spatial constraint calculations on the predictive operational coordinates, marking crop coordinates as restricted areas and weed coordinates as operational points, and binding minimum safe distance and nearest restricted area coordinates to each operational point. The real-time bus, based on the timing reference of the clock synchronization module, transmits operational instructions carrying spatial constraints to the actuator with a deterministic delay. Within the execution window triggered by the timing signal from the clock synchronization module, the actuator adjusts its operational direction according to the spatial constraints in the operational instructions to complete the target-aligning action and automatically avoids crops. All modules operate collaboratively under a unified clock source constraint, with all end-to-end delays being deterministic values. Random jitter components in the target positioning error are eliminated by the deterministic timing coupling between modules.

[0140] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An integrated identification and positioning method for weed control operations in farmland, characterized in that, Includes the following steps: Step S1: Synchronously trigger the multimodal sensor through a unified clock source, and bind the timestamp generated by the unified clock source frame by frame to the image data and pose measurement data to form a time-aligned multimodal data group; Step S2: Identify the image data in the time-aligned multimodal data set and output the pixel coordinates of crops and weeds; Step S3: Based on the timestamp carried by the image in the time-aligned multimodal data group, retrieve the carrier pose that is time-synchronized with the image frame from the pose data bound to the same clock source timestamp. Step S4: Based on the end-to-end processing delay under the unified clock source constraint, predict the pose of the carrier after the delay, and transform the crop pixel coordinates and weed pixel coordinates to the operation coordinate system under the predicted pose to obtain the predictive crop operation coordinates and the predictive weed operation coordinates. Step S5: The instruction carrying the predictive weeding operation coordinates is transmitted to the actuator via a real-time bus that is synchronized with a unified clock source. In step S6, the actuator completes the target-aiming action within the execution window triggered by a unified clock source timing signal based on the predictive weed operation coordinates, and the operation area of ​​the target-aiming action avoids the crop position corresponding to the predictive crop operation coordinates.

2. The integrated identification and positioning method for farmland weed protection operations according to claim 1, characterized in that: Step S1 further includes the following sub-steps: S1-1 generates a synchronous trigger pulse through a unified clock source and outputs it to the hardware trigger interface of the multimodal sensor, which includes an RGB camera, a multispectral imager, and an inertial measurement unit for crop rows in farmland. S1-2, through the multimodal sensor, synchronously samples on the same clock edge of each trigger pulse to obtain a frame of RGB image containing crops and weeds, a frame of multispectral image reflecting the spectral differences between crops and weeds, and a set of inertial measurement data for calculating the instantaneous pose of the working vehicle. S1-3, the unified clock source count value at the trigger moment is used as a timestamp and embedded into the RGB image data, multispectral image data and inertial measurement data obtained in this sampling, so that the RGB image, multispectral image and inertial measurement data corresponding to the same trigger pulse carry the same timestamp, forming a set of time-aligned multimodal data for weed identification and crop positioning.

3. The integrated identification and positioning method for farmland weed protection operations according to claim 1, characterized in that: Step S2 further includes the following sub-steps: S2-1, Perform spectral analysis on multispectral images in time-aligned multimodal datasets to generate category masks that distinguish between crop pixels and weed pixels; S2-2, perform texture and morphological analysis on RGB images in time-aligned multimodal data sets to detect bounding boxes of all plants in the images; S2-3, Register the category mask with the bounding box, and assign the category label of the corresponding region in the category mask to each bounding box; S2-4, output the center point of the bounding box labeled as crop as the crop pixel coordinates, and output the center point of the bounding box labeled as weed as the weed pixel coordinates; S2-5, Establish the correspondence between the coordinates of each weed pixel in the same frame and the coordinates of the crop pixel with the nearest spatial distance, and append this correspondence to the output coordinate data; S2-6 assigns the timestamps of the time-aligned multimodal data sets to the output crop and weed pixel coordinates.

4. The integrated identification and positioning method for farmland weed protection operations according to claim 1, characterized in that: Step S3 further includes the following sub-steps: S3-1, Time-aligned pose measurement data with timestamps in the multimodal data set are arranged in chronological order of acquisition time to form a pose data stream; S3-2: Extract the timestamp carried by the image frame from the current time-aligned multimodal data group, parse the unified clock source count value corresponding to the timestamp as the acquisition time of the image frame, and use the acquisition time as the query index to search in the pose data stream; S3-3, If there is pose data in the pose data stream that matches the acquisition time, then extract the pose data as the carrier pose synchronized with the image frame time. S3-4. If there is no pose data with a matching timestamp in the pose data stream, select the pose data of one frame before and one frame after the acquisition time, and perform interpolation calculation with the acquisition time as the interpolation point to obtain the carrier pose synchronized with the image frame time.

5. The integrated identification and positioning method for farmland weed protection operations according to claim 4, characterized in that, Step S3 also includes extracting plant protection operation pose parameters: The real-time position and real-time heading angle of the carrier are obtained from the carrier attitude synchronized with the image frame time. Based on the spatial relationship between the real-time location and the predetermined crop row centerline, the vertical distance between the carrier centerline and the crop row centerline is calculated as the lateral position deviation. The longitudinal position deviation is calculated based on the difference between the real-time position along the direction of travel and the coordinates of the identified plants in the crop row; The angle between the real-time heading angle and the crop row direction is taken as the heading deflection angle; Lateral position deviation, longitudinal position deviation, and heading deflection angle are used as plant protection operation pose parameters to characterize the real-time pose of the carrier relative to the crop row.

6. The integrated identification and positioning method for farmland weed protection operations according to claim 1, characterized in that: Step S4 further includes the following sub-steps: S4-1: Obtain the plant protection operation pose parameters, and take the carrier attitude synchronized with the image frame time as the starting point, and make angle correction on the carrier's travel direction according to the heading deflection angle; S4-2, based on the corrected direction of travel, predicts the pose of the carrier using the Kalman filter algorithm within the duration of the end-to-end processing delay under a unified clock source constraint. S4-3, based on the lateral and longitudinal position deviations, perform lateral and longitudinal position compensation on the predicted pose to obtain the plant protection operation pose; S4-4: Using the camera intrinsic parameters and the camera extrinsic parameters under the plant protection operation pose, the crop pixel coordinates are transformed to the operation coordinate system to obtain the predictive crop operation coordinates. S4-5: Using the camera's intrinsic parameters and extrinsic parameters under the plant protection operation pose, the weed pixel coordinates are transformed to the operation coordinate system to obtain the predictive weed operation coordinates. S4-6 marks the predictive crop operation coordinates as the target restricted area and the predictive weed operation coordinates as the target operation points, and adds the spatial constraint relationship between the target restricted area and the target operation points to the predictive weed operation coordinates.

7. The integrated identification and positioning method for farmland weed protection operations according to claim 6, characterized in that, In sub-steps S4-6, the spatial constraint relationship is established in the following way: Each target engagement point is paired with its nearest spatially distant target restricted area; Calculate the Euclidean distance between each target engagement point and the center point of its paired target restricted area, and use this distance as the minimum safe distance for that target engagement point. The minimum safe distance of each target engagement point and the coordinates of the nearest target restricted area are used as the spatial constraint relationship of that target engagement point and are added to the coordinate data of that target engagement point. The minimum safe distance is used to determine whether to suppress the target-aiming action at the target-aiming operation point, and the coordinates of the nearest target-aiming restricted area are used to determine the avoidance direction of the target-aiming action.

8. The integrated identification and positioning method for farmland weed protection operations according to claim 1, characterized in that: Step S5 further includes the following sub-steps: S5-1, for each predicted weeding operation coordinate, read the minimum safe distance and the coordinates of the nearest target-restricted area for that operation point; S5-2, compare the minimum safe distance with the preset safe threshold. If it is less than the preset safe threshold, suppress the generation of instructions for that work point. S5-3, if it is greater than or equal to the preset safety threshold, calculate the avoidance direction based on the coordinates of the nearest target restricted area, and generate an operation instruction carrying the coordinates of the operation point and the avoidance direction; S5-4, within the communication cycle determined by a unified clock source, sends the work instructions to the corresponding actuator nodes via the real-time bus; S5-5, the actuator node receives and parses instructions within a defined time offset constrained by a unified clock source, and obtains the corresponding work point coordinates and spatial constraint relationships.

9. The integrated identification and positioning method for farmland weed protection operations according to claim 1, characterized in that: Step S6 further includes the following sub-steps: S6-1, Calculate the action offset of the actuator according to the avoidance direction in the work instruction. The action offset includes the deflection angle and offset distance of the nozzle or weeding blade relative to the default work direction. S6-2, obtain the plant protection operation posture parameters at the current moment, and compensate for the action offset in real time based on the lateral and longitudinal position deviations; S6-3 waits for the next timing trigger signal from the unified clock source. When the timing trigger signal arrives, within the preset time window, it locates the target position according to the work point coordinates in the work instruction, and adjusts the spraying angle or weeding angle of the actuator according to the compensated action offset. S6-4: When the time window ends, the actuator automatically resets and waits for the next instruction.

10. An integrated identification and positioning system for farmland weed control operations, characterized in that, include: The clock synchronization module is used to generate synchronization trigger pulses and clock count values; A multimodal sensor module is used to simultaneously acquire image data and pose measurement data on the same clock edge of the synchronous trigger pulse; The timestamp binding module is used to embed the clock count value at the trigger time as a timestamp into the image data and pose measurement data respectively, forming a time-aligned multimodal data set; The identification and localization module is used to identify crop pixel coordinates and weed pixel coordinates from time-aligned multimodal data sets; The spatiotemporal calibration module is used to retrieve the carrier pose synchronized with the image frame time from the pose data bound to the same clock source timestamp based on the timestamp carried by the image in the time-aligned multimodal data group, and generate predictive job coordinates based on the end-to-end processing delay under the constraint of a unified clock source. The constraint marking module is used to mark crop operation coordinates in the predictive operation coordinates as target restricted areas and weed operation coordinates as target operation points, and to establish the spatial constraint relationship between the two. The real-time bus, which keeps in time synchronization with the clock synchronization module, is used to transmit work instructions carrying predictive work coordinates and their spatial constraints to the actuator. The actuator, synchronized with the clock synchronization module, is used to complete the target-aligning action within the execution window triggered by the timing signal of the clock synchronization module according to the work instructions, and the working area of ​​the target-aligning action avoids the crop location.

Images