An image stitching method for visual navigation of an orchard working robot

Abstract

The application discloses a kind of orchard operation robot visual navigation image splicing methods, it is related to image splicing technical field, comprising: S1: the image sequence of the central axis area of the trunk of fruit tree in the coverage fruit tree of the front and both sides of operation robot with ground path area is collected, and the trunk center axis in each frame image is extracted using fruit tree structure identification module;S2: the trunk center axis of each frame in image sequence is connected, and the structure skeleton curve is generated;S3: with the trunk center axis of fruit tree as the central axis, the structure fusion band region is formed by expanding preset pixel distance in image, and the extraction and matching of feature points are limited.The application extracts and connects the trunk center axis of fruit tree in continuous image frame, constructs structure skeleton curve, as the space leading path of image splicing, realizes the change of image splicing from "image content drive" to "structure line drive", significantly improves the coherence and geometric consistency of the trunk structure in spliced image.

Description

Technical Field

[0001] This invention relates to the field of image stitching technology, specifically to a method for stitching images for visual navigation of orchard operation robots. Background Technology

[0002] With the development of smart agriculture technology, orchard operation robots have been widely used in tasks such as fruit tree inspection, information collection, and yield estimation. Visual perception, as a key subsystem of the operation robot, continuously acquires fruit tree image sequences through cameras and performs image stitching, target recognition, and path guidance, making it one of the core components for achieving efficient operation.

[0003] In orchard environments, the objects being processed exhibit strong structural regularity. If the tree trunk typically displays relatively vertical and continuous geometric features in the image, then incorporating these structural characteristics into image processing becomes a beneficial approach. Simultaneously, during orchard operations, the robot's trajectory and posture changes also significantly impact the image's field of view and the trunk's position.

[0004] A search revealed a Chinese patent (publication number: CN114519671A) that discloses a method for dynamic and rapid stitching of UAV remote sensing images. This patent includes: locating a target area using a multi-sensor integrated navigation system mounted on a UAV, acquiring POS data and image information; obtaining the overlap relationship between the images based on the POS data and image information, and determining an overlap threshold based on the overlap relationship; constructing an image transformation model based on the overlap threshold, and acquiring transformation parameters through the image transformation model; and performing image stitching operations through a backtracking mechanism based on the image transformation model and transformation parameters.

[0005] Compared with existing technologies, one of the research directions is to effectively introduce the structural continuity features of the main trunk of the fruit tree during the image stitching process, clarify the structural control range of the feature point extraction area, and thus improve the geometric consistency of image stitching registration. At the same time, one of the research directions is to reasonably predict the future image content based on the robot's pose changes and optimize the image frame selection logic accordingly, thereby reducing redundant data processing and improving stitching robustness. Therefore, this invention proposes a visual navigation image stitching method for orchard operation robots. Summary of the Invention

[0006] The purpose of this invention is to provide a visual navigation image stitching method for orchard operation robots to solve the problems mentioned in the background art.

[0007] This invention can be achieved through the following technical solution: a method for stitching together visual navigation images for an orchard operation robot, the method comprising the following steps:

[0008] S1. Install visible light cameras on the front and sides of the robot, and the camera's field of view covers the central axis area of ​​the fruit tree and the ground path area.

[0009] During the operation of the robot, an image sequence of visible light images of its front and sides is acquired by a visible light camera, and the image sequence is analyzed by a fruit tree structure recognition module to extract the central axis of the fruit tree trunk.

[0010] S2. Connect the central axes of the continuous fruit tree trunks in the image sequence to construct the structural skeleton curve of the fruit tree trunk in the working path. The structural skeleton curve provides the dominant spatial path for subsequent image stitching.

[0011] This makes image stitching no longer a "matching of images" but a target-driven stitching "around the structural skeleton", and clearly controls the stitching path, solving the problem of stitching frame skipping or breakage caused by visual repetition, occlusion, misalignment, etc. in orchards.

[0012] By establishing geometric constraints for image registration based on the continuity of the target structure, the stitching results are ensured to have coherence and physical interpretability.

[0013] Furthermore, the structural skeleton curve, as a "spatial anchor" in the splicing process, can effectively prevent image misalignment, drift, or even splicing skew caused by motion.

[0014] S3. Extract the central axis of the fruit tree trunk from each frame of the image sequence as the central axis, and extend it to the left and right sides and the top and bottom sides of the image by a preset pixel distance to form a structural fusion zone area. The structural fusion zone area is used as the core area for image stitching.

[0015] For each frame of image, feature point extraction is performed. A corner point extraction algorithm is used to extract all corner point features. All feature points are then filtered, and only feature points located within the structural fusion zone are retained.

[0016] The retained feature points are used for feature matching between adjacent image frames. The matching operation is limited to the corresponding regions of the structural fusion band to construct the initial registration relationship between image frames.

[0017] Furthermore, it only uses stable feature points within the main control area, improving registration quality and positioning accuracy. It shrinks image registration from "full-image feature point matching" to "feature alignment within the structural control area", effectively filtering out unstable feature points, thereby improving the robustness of stitching.

[0018] S4. Equip the robot with an inertial measurement unit and a wheel encoder to collect the current position and attitude data of the work path in real time.

[0019] Based on current position and pose data and historical position and pose data, predict the range of viewpoint changes within a few time steps in the future, and estimate the location of the main trunk structure of the fruit tree in future image frames.

[0020] Image frames that meet the predicted viewing angle range are loaded into the image cache pool, and the candidate image frames are scored and ranked based on sharpness score, backbone region integrity score and occlusion rate score.

[0021] Image frames with a score value above a preset threshold are selected for the current stitching process, while image frames with a score value below the score threshold are discarded. This improves stitching accuracy and removes image frames that are not suitable for stitching due to problems such as occlusion, blurring, or overexposure, thereby reducing the post-processing burden.

[0022] S5. For each pair of adjacent image frames, calculate the initial image registration transformation relationship based on the registered feature points in the structural fusion band region;

[0023] During the transformation calculation, a continuity constraint model of the main trunk structure of the fruit tree is constructed to detect whether the position of the main trunk structure of the fruit tree in the continuous frame images after the registration transformation is offset, bent or misaligned.

[0024] If the continuity is not satisfied, the image registration relationship is readjusted to ensure the linear consistency and coherence of the main trunk structure of the fruit tree in the final stitched image;

[0025] If the main trunk structure of the fruit tree is found to be discontinuous or offset beyond the preset range, the stitching parameters will be automatically adjusted and the image registration relationship will be recalculated to ensure that the main trunk structure line of the fruit tree is continuous and the linear features are consistent in the stitched image.

[0026] Furthermore, during the image fusion process, a weighted fusion algorithm is adopted, in which the structural fusion zone is set as the high-weight fusion region and other regions are set as the low-weight fusion region, ultimately generating a stitched image with a continuous main trunk structure of the fruit tree, clear image content, and uniform brightness distribution.

[0027] A further technical improvement of the present invention is that: in step S1, the analysis of the image sequence by the fruit tree structure recognition module, including image gradient direction analysis, symmetrical shape recognition, and vertical region continuity judgment, specifically includes the following steps:

[0028] A. Analyze the vertical edge concentration zone of the main trunk area of ​​the fruit tree by analyzing the gradient direction of the image;

[0029] a1. Perform grayscale processing on each frame of the visible light image in the image sequence to obtain the corresponding grayscale image, and perform vertical edge extraction processing on each frame of the grayscale image.

[0030] a2. Calculate the average vertical gradient intensity of each column in the grayscale image to form the column gradient intensity curve;

[0031] a3. Traverse the gradient intensity curves to detect whether there are peaks in each grayscale image in the image sequence within a preset region;

[0032] If it exists, its position is recorded, and then the presence of a stable high gradient band in the region of each grayscale image is detected.

[0033] a4. Compare the traversal results with the preset requirements to obtain candidate regions;

[0034] B. Verify whether the candidate region conforms to the characteristics of the fruit tree trunk by identifying symmetrical morphology;

[0035] b1. Divide the grayscale image into consecutive column numbers along the horizontal direction, and construct a detection region with symmetrical width centered on the candidate region;

[0036] b2. Divide the detection area into left and right symmetrical parts according to columns, and cut multiple horizontal height bands from the bottom of the corresponding grayscale image upwards at a fixed length, and compare the grayscale value difference of the left and right symmetrical columns in each height band in turn.

[0037] b3. For each height band, calculate the average absolute value of the gray value difference for each symmetrical grayscale image;

[0038] b4. If the absolute average value in 80% of the height band is lower than the preset grayscale threshold, then the detection area is considered to have strong left-right symmetry in the vertical direction.

[0039] C. By judging the continuity of the vertical region, confirm whether the features of the main trunk of the fruit tree are continuously present in the vertical direction of the grayscale image;

[0040] c1. Divide the grayscale image into multiple judgment regions of equal height from bottom to top along the Y-axis, and perform the following judgment in each judgment region;

[0041] c11. Count the column numbers of the stable high gradient bands in the judgment region;

[0042] c12. Compare whether the column number of the stable high gradient band in this judgment region overlaps or intersects with the column number of the stable high gradient band in the previous segment, that is, the length of the column number intersection exceeds 60% of the width of the column number band in the previous segment.

[0043] c13. If two adjacent judgment regions both satisfy c12, then mark them as "structurally continuous";

[0044] c2. If a judgment region that occupies more than half of the height of the corresponding grayscale image is marked as "structural continuity", it is considered that there is a vertical continuous structure of the main trunk of the fruit tree in the grayscale image.

[0045] c3. The fruit tree structure recognition module outputs a "fruit tree trunk structure exists" signal and provides the column coordinate values ​​of the center points of the trunk in all height segments, which are used as the central axis of the fruit tree trunk for subsequent fitting.

[0046] A further technical improvement of the present invention is that, in step c2, if there is a break between the judgment areas that are continuously marked as "structural continuity", the fruit tree structure recognition module uses the average column number of the front and rear segments to perform linear interpolation to reconstruct the center point coordinates of the middle segment, so as to ensure continuous output of the structure line.

[0047] A further technical improvement of the present invention is that: in step S3, an infrared camera is configured for the working robot, and the field of view of the infrared camera is arranged parallel to that of the visible light camera to acquire infrared images from the same field of view.

[0048] By setting up an ambient light monitoring module, when the brightness of the structural fusion zone region in the image is not within the preset brightness threshold, the visible light image and the infrared image are registered accordingly. The uniformity of brightness distribution in the structural fusion zone is used as the dominant parameter to perform pixel-level image fusion. The fusion method can adopt any mature method in the existing technology based on the requirements, and the enhanced fused image is output.

[0049] Furthermore, the feature point extraction and structural fusion band matching steps are re-executed on the fused image to update the initial registration relationship between image frames.

[0050] A further technical improvement of the present invention is that, in S2, the step of constructing the structural skeleton curve of the fruit tree trunk includes:

[0051] Z1. Collect the coordinates (Xt, Yt) of the bottom center point of the central axis of the main trunk of the fruit tree in each frame of the visible light image in the image sequence, and form the center point sequence P = {P1, P2, ... Pn} of the time series arrangement;

[0052] Z2. Calculate the column number difference ΔX between the bottom center points of the central axis of the main trunks of adjacent fruit trees;

[0053] Z3. Construct a direction vector for the center point sequence P;

[0054] And calculate the change in the angle between adjacent direction vectors;

[0055] If the angle between adjacent direction vectors is continuously less than a preset angle threshold, it is recorded as a "continuous segment".

[0056] The proportion of the center point sequence P marked as "continuous segment" is calculated. If it exceeds the preset proportion, it indicates that the skeleton path structure has strong continuity.

[0057] Conversely, the current jump point frame is removed, and the path is reconstructed backward from the end point of the previous stable segment as the new starting point to avoid structural distortion caused by local misidentification.

[0058] Z4. Calculate the vector angle in step Z3 and compare it with the preset turning angle threshold and redundant angle threshold.

[0059] If the angle between a vector is greater than a preset angle threshold, it will be retained as a path feature point.

[0060] If the included angle of e consecutive vectors is less than the redundancy angle threshold, then the included angles of the remaining vectors excluding the first and last ones are considered redundant, and the corresponding line segments are considered to be approximately straight line segments. They can be expressed by merging the two endpoints to reduce the number of unnecessary points.

[0061] Finally, the compressed path point sequence is output as the structural skeleton curve in the image stitching module.

[0062] The final compressed and optimized set of bottom center point coordinates is: {(X1,Y1), (X2,Y3), ...,(Xm,Ym)}, and m≤n.

[0063] A further technical improvement of the present invention is that, in step Z2, a weighted smoothing process is performed on the center point sequence (Pt-1, Pt, Pt+1) of three frames as a group, which is carried out in the following manner:

[0064] The center point position is calculated as (0.25*Pt-1)+(0.5*Pt)+(0.25*Pt+1), and the updated center point sequence is output.

[0065] A further technical improvement of the present invention is that, in S4, the method for predicting the position of the main trunk structure of a fruit tree includes the following steps:

[0066] Q1. Input the current position coordinates of the robot (X... z t,Y z t), orientation angle θt, linear velocity vt, angular velocity ωt, and historical pose sequence (X z ,Y z ,θ), where the historical pose sequence (X z ,Y z θ) is used to determine whether the robot is in an acceleration phase, a turning phase, or a constant speed straight-line phase.

[0067] Q2. Set the prediction time range ΔT, and perform step-by-step prediction with a fixed prediction step size Δt to obtain the prediction step number K, K=ΔT÷Δt;

[0068] Q3. Calculate the angle difference Δθ and velocity change Δv using a continuous frame pose sequence to determine the robot's operating state, including:

[0069] For straight-line conditions (Δθ < 5°, Δv < 0.05 m / s), a linear straight-line model is used for prediction, including:

[0070] For the prediction at step k (k∈[1,K]):

[0071] X z (t+k)=X z t+vt×cos(θt)×k×Δt;

[0072] Y z (t+k)=Y z t+vt×sin(θt)×k×Δt;

[0073] Predicted orientation angle: θ(t+k)=θt;

[0074] For turning (Δθ≥5°), a small-angle line segment iterative model is used for prediction, and a continuous deflection at a fixed angular velocity is set, including:

[0075] X z (t+k)=X z (t+(k-1))+vt×cosθ(t+k)×Δt;

[0076] Y z (t+k)=Y z (t+(k-1))+vt×cosθ(t+k)×Δt;

[0077] Predicted orientation angle: θ(t+k)=θt+ωt×k×Δt;

[0078] Last record {X z (t+k),Y z (t+k),θ(t+k)} represents the predicted location points, including the predicted path points {X}. z (t+k),Y z (t+k)} and the corresponding viewpoint direction θ(t+k);

[0079] Q4. At each predicted location point {X z (t+k),Y z On (t+k),θ(t+k)}, construct a field of view model, with θ as the center direction, the subtended angle is the field of view angle of the corresponding visible light camera, and the detection radius is the farthest effective imaging distance of the visible light camera;

[0080] Within the field of view model, it is overlaid with the orchard layout map or the known fruit tree row and column model, and the image area where the main structure is expected to appear in the future image frame is marked with the orientation angle θ as the reference, which serves as the basis for the subsequent structure fusion region localization.

[0081] Furthermore, the predicted path and the region where the backbone image appears at each step are spatially matched with the existing frames in the image cache pool to determine whether they "hit" and are used for image filtering.

[0082] Q5. Output the predicted path point sequence {X} z (t+1),Y z (t+1),...,X z (t+K),Y z (t+K)}, and the column interval of the structure fusion zone corresponding to each predicted path point.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] This invention extracts and connects the central axis of the main trunk of a fruit tree in consecutive image frames to construct a structural skeleton curve, which serves as the dominant spatial path for image stitching. This enables the transformation of image stitching from "image content-driven" to "structural line-driven," significantly improving the coherence and geometric consistency of the main trunk structure of the fruit tree in the stitched image.

[0085] Furthermore, by setting a structural fusion zone, corner features are extracted only in the extended area of ​​the main trunk's central axis, and the corner points are filtered at multiple levels to eliminate invalid points and false features, thereby improving the accuracy of feature point matching and effectively reducing the interference of non-structural information such as leaf texture and background areas on registration.

[0086] Meanwhile, this invention introduces a robot pose trajectory prediction mechanism to calculate the camera observation direction and the expected area of ​​the image backbone structure within a future time step. By matching the shooting pose with the image frames in the image cache pool, and combining image clarity, occlusion rate and backbone integrity for scoring and sorting, only image frames that meet the prediction range and quality requirements are retained for stitching, thereby improving stitching efficiency and accuracy. Attached Figure Description

[0087] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0088] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0089] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0090] Please see Figure 1As shown, this invention provides a method for stitching together visual navigation images for an orchard operation robot. The method includes the following steps:

[0091] S1. Install visible light cameras on the front and sides of the robot, and the camera's field of view covers the central axis area of ​​the fruit tree and the ground path area.

[0092] During the operation of the robot, an image sequence of visible light images of its front and sides is acquired by a visible light camera, and the image sequence is analyzed by a fruit tree structure recognition module to extract the central axis of the fruit tree trunk.

[0093] It is used to transform "image content" into "structural representation", that is, to change from passively stitching image content to actively extracting target structure, providing spatial constraints for stitching and improving stability;

[0094] In step S1, the fruit tree structure recognition module analyzes the image sequence, including image gradient direction analysis, symmetrical shape recognition, and vertical region continuity determination, specifically including the following steps:

[0095] A. Analyze the vertical edge concentration zone of the main trunk area of ​​the fruit tree by analyzing the gradient direction of the image;

[0096] a1. Perform grayscale processing on each frame of the visible light image in the image sequence to obtain the corresponding grayscale image;

[0097] Vertical edge extraction is performed on each frame of grayscale image. In this embodiment, a 3*3 fixed-size Sobel operator is used to obtain the gradient intensity of each column of pixels in the vertical direction.

[0098] a2. Calculate the average vertical gradient intensity of each column in the grayscale image to form the column gradient intensity curve;

[0099] a3. Traverse the gradient intensity curves to detect whether there are peaks in the 1 / 3 region of each grayscale image in the image sequence;

[0100] If it exists, the location is recorded, and then it is detected whether there is a stable high gradient band in the 1 / 3 region of each grayscale image (i.e., the gradient values ​​of multiple adjacent columns are more than 1.5 times higher than the average vertical gradient intensity).

[0101] a4. Compare the traversal results with the preset requirements to obtain candidate regions;

[0102] In this embodiment, if a stable high gradient band is detected in the middle region of each grayscale image and the width does not exceed 1 / 5 of the width of the grayscale image, i.e., the width of the visible light image is 640 pixels, and the middle stable high gradient band does not exceed 128 pixels, then the region is initially determined to be a region that may contain the trunk structure of the fruit tree, and the region is determined to be a candidate region.

[0103] B. Verify whether the candidate region conforms to the characteristics of the fruit tree trunk by identifying symmetrical morphology;

[0104] b1. Divide the grayscale image into consecutive column numbers along the horizontal direction, and construct a symmetrical detection region with the candidate region as the center. For example, expand the candidate region to the left and right by 40 pixels to form a symmetrical detection region with a width of 80 pixels.

[0105] b2. Divide the detection area into left and right symmetrical parts according to columns, and cut multiple horizontal height bands from the bottom of the corresponding grayscale image upwards at a fixed length, and compare the grayscale value difference of the left and right symmetrical columns in each height band in turn.

[0106] b3. For each height band, calculate the average absolute value of the gray value difference for each symmetrical grayscale image;

[0107] b4. If the absolute average value in 80% of the height band is lower than the preset grayscale threshold, then the detection area is considered to have strong left-right symmetry in the vertical direction.

[0108] C. By judging the continuity of the vertical region, confirm whether the features of the main trunk of the fruit tree are continuously present in the vertical direction of the grayscale image;

[0109] c1. Divide the grayscale image into multiple judgment regions of equal height from bottom to top along the Y-axis, and perform the following judgment in each judgment region;

[0110] c11. Count the column numbers of the stable high gradient bands in the judgment region;

[0111] c12. Compare whether the column number of the stable high gradient band in this judgment region overlaps or intersects with the column number of the stable high gradient band in the previous segment, that is, the length of the column number intersection exceeds 60% of the width of the column number band in the previous segment.

[0112] c13. If two adjacent judgment regions both satisfy c12, then mark them as "structurally continuous";

[0113] c2. If a judgment region that occupies more than half of the height of the corresponding grayscale image is marked as "structural continuity", it is considered that there is a vertical continuous structure of the main trunk of the fruit tree in the grayscale image.

[0114] In step c2, if there is a break between the judgment areas that are continuously marked as "structural continuity", the fruit tree structure recognition module uses the average column number of the front and rear segments to perform linear interpolation to reconstruct the center point coordinates of the middle segment, so as to ensure the continuous output of the structure line.

[0115] c3. The fruit tree structure recognition module outputs the signal "the main trunk structure of the fruit tree exists" and provides the column coordinate values ​​of the center points of the main trunk in all height segments, which are used as the central axis of the main trunk of the fruit tree for subsequent fitting.

[0116] S2. Connect the central axes of the continuous fruit tree trunks in the image sequence to construct the structural skeleton curve of the fruit tree trunk in the working path. The structural skeleton curve provides the dominant spatial path for subsequent image stitching.

[0117] This makes image stitching no longer a "matching of images" but a target-driven stitching "around the structural skeleton", and clearly controls the stitching path, solving the problem of stitching frame skipping or breakage caused by visual repetition, occlusion, misalignment, etc. in orchards.

[0118] In S2, the steps for constructing the structural skeleton curve of the fruit tree trunk include:

[0119] Z1. Collect the coordinates (Xt, Yt) of the bottom center point of the central axis of the main trunk of the fruit tree in each frame of the visible light image in the image sequence, and form the center point sequence P = {P1, P2, ... Pn} of the time series arrangement;

[0120] Z2. Calculate the column number difference ΔX between the bottom center points of the central axis of the main trunks of adjacent fruit trees;

[0121] In step Z2, the center point sequence (Pt-1, Pt, Pt+1) of three frames is subjected to weighted smoothing in the following manner:

[0122] The center point position is calculated as (0.25*Pt-1)+(0.5*Pt)+(0.25*Pt+1), and the updated center point sequence is output.

[0123] Z3. Construct a direction vector for the center point sequence P:

[0124] And calculate the change in the angle between adjacent direction vectors;

[0125] If the angle between adjacent direction vectors is continuously less than a preset angle threshold, it is recorded as a "continuous segment".

[0126] The proportion of the center point sequence P marked as "continuous segment" is calculated. If it exceeds the preset proportion, it indicates that the skeleton path structure has strong continuity.

[0127] Conversely, the current jump point frame is removed, and the path is reconstructed backward from the end point of the previous stable segment as the new starting point to avoid structural distortion caused by local misidentification.

[0128] Z4. Calculate the vector angle in step Z3 and compare it with the preset turning angle threshold and redundant angle threshold.

[0129] If the angle between a vector is greater than a preset angle threshold, it will be retained as a path feature point.

[0130] If the included angle of e consecutive vectors is less than the redundancy angle threshold, then the included angles of the remaining vectors excluding the first and last ones are considered redundant, and the corresponding line segments are considered to be approximately straight line segments. They can be expressed by merging the two endpoints to reduce the number of unnecessary points.

[0131] Finally, the compressed path point sequence is output as the structural skeleton curve in the image stitching module.

[0132] The final compressed and optimized set of bottom center point coordinates is: {(X1,Y1), (X2,Y3), ...,(Xm,Ym)}, and m≤n;

[0133] Furthermore, the compression optimization is completed within each frame output cycle, with a single calculation delay of no more than 10ms, ensuring that the real-time constraints of the robot embedded system are met.

[0134] By establishing geometric constraints for image registration based on the continuity of the target structure, the stitching results are ensured to have coherence and physical interpretability.

[0135] Furthermore, the structural skeleton curve, as a "spatial anchor" in the splicing process, can effectively prevent image misalignment, drift, or even splicing skew caused by motion.

[0136] S3. Extract the central axis of the fruit tree trunk from each frame of the image sequence as the central axis, and extend it to the left and right sides and the top and bottom sides of the image by a preset pixel distance to form a structural fusion zone area. The structural fusion zone area is used as the core area for image stitching.

[0137] For each frame of image, feature point extraction is performed. A corner point extraction algorithm is used to extract all corner point features. All feature points are then filtered, and only feature points located within the structural fusion zone are retained.

[0138] The retained feature points are used for feature matching between adjacent image frames. The matching operation is limited to the corresponding regions of the structural fusion band to construct the initial registration relationship between image frames.

[0139] Used to clearly define the stitching area and avoid feature redundancy and mismatch caused by unstructured areas such as image edges, sky, and branches;

[0140] Furthermore, it only uses stable feature points within the main control area, improving registration quality and positioning accuracy. It shrinks image registration from "full-image feature point matching" to "feature alignment within the structural control area", effectively filtering out unstable feature points, thereby improving the robustness of stitching.

[0141] In this embodiment, the central axis of the fruit tree trunk in each frame of the image is extended outward by 40 pixels in the left and right directions and by 20 pixels in the up and down directions to form a continuous vertical rectangular area.

[0142] Ultimately, the pixel coordinate range of the structure fusion band is:

[0143] Horizontal (column coordinate) range: ±40 pixels of the column containing the main trunk centerline;

[0144] Vertical (row coordinate) range: The starting point of the main line extends upward to the top + 20 pixels, and the ending point extends downward to the bottom + 20 pixels;

[0145] Specifically, it uses the ORB algorithm for feature point extraction, including:

[0146] During the extraction process, FAST corner detection is performed using a fixed window size (e.g., 31×31), and feature descriptors are generated using the BRIEF method.

[0147] In each image frame, extraction operations are limited to the structure fusion zone region only, and corner point extraction is prohibited on image edges, sky background, and unstructured ground areas to avoid redundancy and mismatch.

[0148] After extraction, the corner points are filtered as follows: calculate the gray difference between the gray level of the pixel where the corner point is located and the average gray level of the neighboring region. If the gray level contrast is lower than the set gray level contrast threshold, it is considered a weak corner point and is removed.

[0149] If a corner point is more than 5 pixels away from the boundary of the structural fusion zone, or more than 50 pixels away from the center line of the main trunk, it is judged as an "unstable point" and is removed.

[0150] Finally, corner points that meet spatial location requirements, have high response values, and large grayscale contrast are retained as feature points to construct a stable registration reference point set;

[0151] In this example, we take an image with a width of 640 pixels and a height of 480 pixels as an example:

[0152] The main axis of the fruit tree trunk is mainly distributed between column coordinates 280–300, so the column range of the structural fusion zone is 240–340 (i.e., 280±40 pixels).

[0153] If a corner point has a response value of 20, a grayscale contrast of 5, and a position of column 380 and row 200, then this corner point is removed because it is "outside the blending band + weak response".

[0154] Ultimately, 60–80 effective feature points are retained per frame, mainly concentrated in the trunk structure area of ​​the fruit tree;

[0155] S4. Equip the robot with an inertial measurement unit and a wheel encoder to collect the current position and attitude data of the work path in real time.

[0156] Based on current position and pose data and historical position and pose data, predict the range of viewpoint changes within a few time steps in the future, and estimate the location of the main trunk structure of the fruit tree in future image frames.

[0157] The method for predicting the location of the main trunk structure of fruit trees includes the following steps:

[0158] Q1. Input the current position coordinates of the robot (X... z t,Y z t), orientation angle θt, linear velocity vt, angular velocity ωt, and historical pose sequence (X z ,Y z ,θ), where the historical pose sequence (X z ,Y z θ) is used to determine whether the robot is in an acceleration phase, a turning phase, or a constant speed straight-line phase.

[0159] Q2. Set the prediction time range ΔT, and perform step-by-step prediction with a fixed prediction step size Δt to obtain the prediction step number K, K=ΔT÷Δt;

[0160] Q3. Calculate the angle difference Δθ and velocity change Δv using a continuous frame pose sequence to determine the robot's operating state, including:

[0161] For straight-line conditions (Δθ < 5°, Δv < 0.05 m / s), a linear straight-line model is used for prediction, including:

[0162] For the prediction at step k (k∈[1,K]):

[0163] X z (t+k)=X z t+vt×cos(θt)×k×Δt;

[0164] Y z (t+k)=Y z t+vt×sin(θt)×k×Δt;

[0165] Predicted orientation angle: θ(t+k)=θt;

[0166] For turning (Δθ≥5°), a small-angle line segment iterative model is used for prediction, and a continuous deflection at a fixed angular velocity is set, including:

[0167] X z (t+k)=X z (t+(k-1))+vt×cosθ(t+k)×Δt;

[0168] Y z (t+k)=Y z (t+(k-1))+vt×cosθ(t+k)×Δt;

[0169] Predicted orientation angle: θ(t+k)=θt+ωt×k×Δt;

[0170] Last record {X z (t+k),Y z (t+k),θ(t+k)} represents the predicted location points, including the predicted path points {X}. z (t+k),Y z (t+k)} and the corresponding viewpoint direction θ(t+k);

[0171] Q4. At each predicted location point {X z (t+k),Y z On (t+k),θ(t+k)}, construct a field of view model, with θ as the center direction, the subtended angle is the field of view angle of the corresponding visible light camera, and the detection radius is the farthest effective imaging distance of the visible light camera;

[0172] Within the field of view model, it is overlaid with the orchard layout map or the known fruit tree row and column model, and the image area where the main structure is expected to appear in the future image frame is marked with the orientation angle θ as the reference, which serves as the basis for the subsequent structure fusion region localization.

[0173] Furthermore, the predicted path and the region where the backbone image appears at each step are spatially matched with the existing frames in the image cache pool to determine whether they "hit" and are used for image filtering.

[0174] Q5. Output the predicted path point sequence {X} z (t+1),Y z (t+1),...,X z (t+K),Y z (t+K)}, and the column interval where the structure fusion zone corresponding to each predicted path point is located;

[0175] For example, the predicted path point of the robot is {X} z (t+3),Y z (t+3)}, facing angle θ=0°, the resolution of the visible light camera is 640 pixels;

[0176] Centered on image column number 320, extend 80 pixels to the left and right;

[0177] The predicted region of the backbone structure in the third frame image is the column number range: 240–400;

[0178] If the structural fusion band in an image frame falls within this range, the image is valid and enters the image buffer pool.

[0179] Image frames that meet the predicted viewing angle range are loaded into the image cache pool, and the candidate image frames are scored and ranked based on sharpness score, backbone region integrity score and occlusion rate score.

[0180] In this embodiment, the sharpness score is calculated by performing edge detection on the entire image and calculating the average intensity of the edge gradient to measure the image sharpness. If the image gradient value is lower than a set gradient value threshold, it is judged as a blurry image.

[0181] The trunk integrity score is determined by detecting whether a complete and continuous fruit tree trunk structure exists in the image. If there are breaks, occlusions, or recognition failures in the structural fusion zone, the score is reduced.

[0182] The occlusion rate score is calculated by statistically analyzing the proportion of occluded or invalid information within the main structural area of ​​an image, such as overexposure or foliage obstruction. The larger the proportion of occluded area, the lower the score.

[0183] Image frames with a score value above a preset threshold are selected for the current stitching process, while image frames with a score value below the score threshold are discarded. This improves stitching accuracy and removes image frames that are not suitable for stitching due to problems such as occlusion, blurring, or overexposure, thereby reducing the post-processing burden.

[0184] Furthermore, the image cache pool is updated using a sliding time window method: when a new image frame arrives, the earliest image frame is removed from the cache pool to ensure that the cache capacity does not exceed the set upper limit;

[0185] For example, if the current speed of the robot is 0.4 m / s, then 5 time points are predicted in 0.1-second increments, and there are currently 10 images in the image cache pool; the system finds 3 matching images in the prediction path.

[0186] The rating result is:

[0187] Image A: High clarity, intact main structure, low occlusion, score 95;

[0188] Image B: Average clarity, main part obscured, score 78;

[0189] Image C: The main trunk is broken and has low clarity, score 62 (rejected);

[0190] Finally, image A and image B are used as the input for stitching, while image C is excluded;

[0191] S5. For each pair of adjacent image frames, calculate the initial image registration transformation relationship based on the registered feature points in the structural fusion band region;

[0192] In this embodiment, the affine transformation matrix estimated by the least squares method is used to calculate the image registration transformation relationship, including three degrees of freedom: rotation, translation and scaling. The subsequent frame image is aligned to the coordinate system of the previous frame image, and a preliminary stitching and docking is formed.

[0193] During the transformation calculation, a continuity constraint model of the main trunk structure of the fruit tree is constructed to detect whether the position of the main trunk structure of the fruit tree in the continuous frame images after the registration transformation is offset, bent or misaligned.

[0194] In this embodiment, the continuity constraint model extracts key points of the central axis of the tree trunk in adjacent image frames after stitching and judges the positional change trend of these points in the image coordinates. If it is found that the offset of the central point of the trunk between adjacent frames is too large, or the line formed by multiple consecutive central points has obvious bends or breaks, it is determined that the structure is discontinuous. In addition, the continuity constraint model sets column coordinate offset threshold and structural angle change threshold to judge whether the structure after stitching is stable and linear.

[0195] If the continuity is not satisfied, the image registration relationship is readjusted to ensure the linear consistency and coherence of the main trunk structure of the fruit tree in the final stitched image;

[0196] If the main trunk structure of the fruit tree is found to be discontinuous or offset beyond the preset range, the stitching parameters will be automatically adjusted and the image registration relationship will be recalculated to ensure that the main trunk structure line of the fruit tree is continuous and the linear features are consistent in the stitched image.

[0197] Specifically, the column coordinate difference between the positions of the central axis of the trunk in the two images is defined as the trunk offset value. If the trunk offset value exceeds the set tolerance threshold, it is judged as structural misalignment or trunk discontinuity. The current transformation matrix is ​​re-estimated: for example, the weights are increased to feature points near the trunk center point, or unreliable edge points are removed and the registration relationship is recalculated until the structural lines are continuous and smooth.

[0198] If three consecutive frames have the same type of misalignment, the system records the abnormal segment markers for post-processing cleaning.

[0199] Furthermore, during the image fusion process, a weighted fusion algorithm is used, in which the structural fusion zone is set as the high-weight fusion region and other regions are set as the low-weight fusion region, ultimately generating a stitched image with a continuous main trunk structure of the fruit tree, clear image content, and uniform brightness distribution.

[0200] By introducing a structural continuity detection mechanism into the splicing results, if abnormalities such as misalignment or bending of the main structure occur, the splicing parameters are re-optimized. Furthermore, by focusing on preserving the image information of the structural area during the fusion stage, the clarity of the main structure and the naturalness of the image connection are improved.

[0201] By using the continuity of the target structure as the basis for judging the effectiveness of stitching, mis-stitching, frame skipping and structural distortion can be fundamentally suppressed, ensuring that the stitched image can be used for navigation path recognition and control.

[0202] For example, before two frames of images are stitched together, the column numbers of the backbone center are 285 and 270 respectively, and the backbone offset value is 15, which is lower than the threshold of 20 → stitching is done directly.

[0203] The merging region is set between columns 250 and 310, with weights w1 = 0.8 and w2 = 0.2.

[0204] The fused image shows a coherent main structure and a natural transition in brightness between the edges of the two frames, with no obvious stitching marks.

[0205] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for stitching images for visual navigation of an orchard operation robot, characterized in that, include: S1: The visible light camera captures image sequences of the front and sides of the robot, covering the central axis area of ​​the fruit tree and the ground path area, and the fruit tree structure recognition module extracts the central axis of the fruit tree trunk in each frame of the image. The fruit tree structure recognition module analyzes image sequences, including: A. Analyze the vertical edge concentration zone of the main trunk area of ​​the fruit tree by analyzing the gradient direction of the image; B. Verify whether the candidate region conforms to the characteristics of the fruit tree trunk by identifying symmetrical morphology; C. By judging the continuity of the vertical region, confirm whether the features of the main trunk of the fruit tree are continuously present in the vertical direction of the grayscale image; The fruit tree structure recognition module outputs a "fruit tree trunk structure exists" signal and provides column coordinate values ​​of the center points of the trunk in all height segments, which are used as the central axis of the fruit tree trunk for subsequent fitting. S2: Connect the central axes of the main trunks of the fruit trees in each frame of the image sequence to generate structural skeleton curves, including: Z1. Collect the coordinates (Xt, Yt) of the bottom center point of the central axis of the main trunk of the fruit tree in each frame of the visible light image in the image sequence, and form the center point sequence P={P1,P2,...Pn} of the time series arrangement; Z2. Calculate the column number difference between the bottom center points of the central axes of adjacent fruit tree trunks. ; Z3. Construct direction vectors for the center point sequence P, and calculate the change in the angle between adjacent direction vectors; Z4. Calculate the vector angle in step Z3 and compare it with the preset turning angle threshold and redundant angle threshold. If the angle between a vector is greater than a preset angle threshold, it will be retained as a path feature point. If the included angle of e consecutive vectors is less than the redundancy angle threshold, then the included angles of the e vectors excluding the first and last vectors are considered redundant, and the corresponding line segments are considered straight line segments. Finally, the compressed path point sequence is output as the structural skeleton curve in the image stitching module. S3: Extract the central axis of the main trunk of the fruit tree in each frame of the image sequence as the central axis, and extend it to the left and right sides and the top and bottom sides of the image by a preset pixel distance to form a structural fusion zone area, which is the core area for image stitching; For each frame of image, feature point extraction is performed. A corner point extraction algorithm is used to extract all corner point features. All feature points are then filtered, and only feature points located within the structural fusion zone are retained. The retained feature points are used for feature matching between adjacent image frames. The matching operation is limited to the corresponding regions of the structural fusion band to construct the initial registration relationship between image frames. S4: Based on the current position and posture data of the robot and historical posture data, predict the range of future viewpoint changes and the expected location of the main trunk structure of the fruit tree in the image, and select the required image frames from the image cache pool to participate in the stitching. S5: Based on the registered feature points in the structural fusion zone region, calculate the registration transformation relationship between adjacent image frames. During the transformation calculation process, construct a continuity constraint model for the main trunk structure of the fruit tree, detect whether the position of the main trunk structure of the fruit tree in the continuous frame images is shifted, bent or misaligned after the registration transformation, and use a weighted fusion algorithm with the structural fusion zone region as the high weight region to output the stitched image.

2. The method for stitching together visual navigation images for an orchard operation robot according to claim 1, characterized in that, In step S3, an infrared camera is configured for the robot, and the field of view of the infrared camera is aligned parallel to that of the visible light camera to capture infrared images under the same visual perspective. When the brightness of the structural fusion zone region in the image is not within the preset brightness threshold, the visible light image and the infrared image are registered accordingly, and the fused image is output. Furthermore, the feature point extraction and structural fusion band matching steps are re-executed on the fused image.

3. The image stitching method for visual navigation of an orchard operation robot according to claim 1, characterized in that, In step Z2, the center point sequence (Pt-1, Pt, Pt+1) of three frames is subjected to weighted smoothing in the following manner: Center point location Output the updated center point sequence. .

4. The method for stitching together visual navigation images for an orchard operation robot according to claim 1, characterized in that, In S4, the method for predicting the location of the main trunk structure of a fruit tree includes the following steps: Q1. Input the current position coordinates of the robot. Orientation Angle linear velocity angular velocity and historical pose sequence ( , , ); Q2. Set the forecast time range and with a fixed prediction step size Perform step-by-step prediction to obtain the prediction step number K, K= ÷ ; Q3. Calculate the angle difference using a continuous frame pose sequence. and speed change To determine the robot's operating status; Q4. At each predicted location point Above, construct a field of view model, and with With the center direction as the center, the angular diameter is the field of view angle of the corresponding visible light camera, and the detection radius is the farthest effective imaging distance of the visible light camera; Within the field of view model, it is overlaid with the orchard layout map or a known tree row and column model, and oriented at an angle. Based on this, mark the image regions where the backbone structure is expected to appear in future image frames; The predicted path and the region where the backbone image appears at each step are spatially matched with the existing frames in the image cache pool to determine whether they are "hit". Q5. Output the predicted path point sequence: { And the column interval where the structure fusion zone corresponding to each predicted path point is located.

5. The image stitching method for visual navigation of an orchard operation robot according to claim 4, characterized in that, In step Q3, the robot's operating status includes: Straight ahead ( <5°, <0.05m / s), predicted using a linear linear model, including: For the prediction at step k (k∈[1,K]): ； ； Predicted orientation angle: ; Turning position The prediction is made using a small-angle line segment iterative model, and the deflection is set to be continuous at a fixed angular velocity, including: ； ； Predicted orientation angle: ; Last record For predicted location points, including predicted waypoints and corresponding perspective direction .

6. The method for stitching together visual navigation images for an orchard operation robot according to claim 1, characterized in that, Methods for determining the continuity of vertical regions include: c1. Divide the grayscale image into multiple judgment regions of equal height from bottom to top along the Y-axis, and perform the following judgment in each judgment region; c11. Count the column numbers of the stable high gradient bands in the judgment region; c12. Compare the column numbers of the stable high gradient band in this judgment region with those of the stable high gradient band in the previous paragraph to see if they overlap or intersect. c13. If two adjacent judgment regions both satisfy c12, then mark them as "structurally continuous"; c2. If a judgment region that occupies more than half of the height of the corresponding grayscale image is marked as "structural continuity", it is considered that there is a vertical continuous structure of the trunk of the fruit tree in the grayscale image. c3. The fruit tree structure recognition module outputs the signal "the main trunk structure of the fruit tree exists" and provides the column coordinate values ​​of the center point of the main trunk for all height segments.

Images