Orchard operation robot visual navigation image splicing method

Abstract

The invention discloses a visual navigation image splicing method for an orchard operation robot, and relates to the technical field of image splicing, and the method comprises the steps: S1, collecting image sequences covering a fruit tree central axis region and a ground path region at the front part and two sides of an operation robot, and extracting a fruit tree trunk central axis in each frame of image through a fruit tree structure recognition module; s2, connecting each frame of fruit tree trunk central axis in the image sequence to generate a structural skeleton curve; and S3, by taking the central axis of the trunk of the fruit tree as a central axis, expanding a preset pixel distance in the image to form a structure fusion zone region, and limiting extraction and matching of feature points. According to the method, the central axes of the fruit tree trunks in the continuous image frames are extracted and connected, the structural skeleton curve is constructed and serves as a space leading path for image splicing, image splicing is converted from image content driving to structural line driving, and the consistency and geometric consistency of the fruit tree trunk structures in the spliced images are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image splicing, and in particular to a method for splicing images of an orchard operation robot through visual navigation. Background Art

[0002] With the development of smart agricultural technology, orchard robots have been widely used for tasks such as fruit tree inspection, information collection, and yield estimation. As a key subsystem of these robots, visual perception continuously captures fruit tree image sequences through cameras and performs image stitching, target recognition, and path guidance, making it a key component in achieving efficient operations.

[0003] In orchards, the objects being worked on have strong structural regularities. Tree trunks typically appear as relatively vertical, continuous geometric features in images. Therefore, incorporating these structural features into image processing is a beneficial approach. Furthermore, during orchard operations, the robot's motion trajectory and posture changes also affect the image's viewing angle and trunk position.

[0004] After searching, a Chinese patent (publication number: CN114519671A) discloses a method for dynamic rapid stitching of UAV remote sensing images. The patent includes: positioning the target area through a multi-sensor combined navigation system equipped on a UAV to obtain POS data and image information; obtaining the overlap relationship between images based on the POS data and image information, and determining the overlap threshold between images based on the overlap relationship; constructing an image transformation model based on the overlap threshold, and obtaining transformation parameters through the image transformation model; and performing image stitching operations through a backtracking mechanism based on the image transformation model and transformation parameters.

[0005] Compared with the existing technology, how to effectively introduce the structural continuity characteristics of the fruit tree trunk in the image stitching process, clarify the structural control range of the feature point extraction area, and thus improve the geometric consistency of the stitching and registration between images; at the same time, how to reasonably predict the future image content according to the robot posture changes, and optimize the image frame selection logic accordingly, so as to reduce redundant data processing and improve the robustness of stitching, is one of the research and development directions. Therefore, the present invention proposes a visual navigation image stitching method for orchard operation robots. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for stitching images of a robot operating in an orchard using visual navigation to solve the problems mentioned in the above background technology.

[0007] The present invention can be implemented by the following technical solution: a method for stitching images of a visual navigation robot in an orchard operation, the method comprising the following steps:

[0008] S1. Install visible light cameras on the front and both sides of the robot, and ensure that the camera captures images covering the central axis of the fruit tree and the ground path area.

[0009] During the operation of the robot, a visible light camera is used to collect a sequence of visible light images of the front and sides of the robot. The fruit tree structure recognition module analyzes the image sequence and extracts the central axis of the fruit tree trunk.

[0010] S2. Connecting the central axes of consecutive fruit tree trunks in the image sequence to construct a structural skeleton curve of the fruit tree trunk in the operation path. The structural skeleton curve provides a spatial dominant path for subsequent image stitching;

[0011] Image stitching is no longer an "image-to-image" match, but a target-driven stitching "around the structural skeleton." The stitching path is clearly controlled, solving the problem of frame skipping or breakage caused by visual duplication, occlusion, and misalignment in orchards.

[0012] Establish geometric constraints for image registration through the continuity of the target structure to ensure the consistency and physical interpretability of the stitching results;

[0013] Furthermore, the structural skeleton curve serves as the "spatial anchor" of the stitching process, effectively preventing image dislocation, drift, and even stitching distortion caused by motion.

[0014] S3, extracting the central axis of the fruit tree trunk of each frame image in the image sequence as the central axis, and extending the preset pixel distance to the left and right sides and the top and bottom sides of the image to form a structural fusion zone area, which is used as the core area for image stitching;

[0015] Perform feature point extraction on each frame of image, use corner point extraction algorithm to extract all corner point features, and filter all feature points to retain only the feature points located in the structural fusion zone;

[0016] The retained feature points are used for feature matching between adjacent image frames, wherein the matching operation is limited to be performed between corresponding areas of the structure fusion zone to establish an initial registration relationship between the image frames;

[0017] In addition, only stable feature points within the backbone control area are used to improve registration quality and positioning accuracy. Image registration is narrowed from "full-image feature point matching" to "feature alignment within the structural control area", effectively filtering out unstable feature points, thereby improving the robustness of stitching.

[0018] S4. Equip the working robot with an inertial measurement unit and a wheel encoder to collect the current position and posture data of the working path in real time;

[0019] Based on the current position and posture data and the historical position and posture data, the viewing angle change range in the next several time steps is predicted, and the appearance position of the main structure of the fruit tree in the future image frame is estimated;

[0020] Load image frames that meet the predicted viewing angle range into the image buffer pool, and rank the candidate image frames based on the clarity score, trunk area integrity score, and occlusion rate score;

[0021] Image frames with scores above the preset threshold are selected for the current stitching process, and image frames with scores below the threshold are discarded to improve stitching accuracy. Image frames that are not suitable for stitching due to occlusion, blur, overexposure, etc. are eliminated, reducing the post-processing burden.

[0022] S5. For each pair of adjacent image frames, calculate the initial image registration transformation relationship based on the registered feature points in the structural fusion zone area;

[0023] During the transformation calculation process, a continuity constraint model of the fruit tree trunk structure is constructed to detect whether the position of the fruit tree trunk structure in the continuous frame images after the registration transformation is offset, bent or misplaced;

[0024] If the continuity is not satisfied, the image registration relationship is readjusted to ensure the linear consistency and coherence of the fruit tree trunk structure in the final spliced ​​image;

[0025] If the main trunk structure of the fruit tree is detected to be discontinuous or the offset exceeds the preset range, the stitching parameters are automatically adjusted and the image registration relationship is recalculated to ensure that the main trunk structure line of the fruit tree in the stitched image is continuous and the linear features are consistent;

[0026] In the image fusion process, a weighted fusion algorithm is used, in which the structural fusion zone area is set as the high-weight fusion area, and other areas are set as the low-weight fusion area, and finally a spliced ​​image with a continuous fruit tree trunk structure, clear image content and uniform brightness distribution is generated.

[0027] A further technical improvement of the present invention is that in step S1, the analysis of the image sequence by the fruit tree structure recognition module includes image gradient direction analysis, symmetric shape recognition and vertical region continuity judgment, specifically including the following steps:

[0028] A. Analyze the vertical edge concentration zone in the trunk area of ​​the fruit tree by image gradient direction;

[0029] a1. Perform grayscale processing on each visible light image in the image sequence to obtain a corresponding grayscale image, and perform vertical edge extraction on each grayscale image;

[0030] a2. Calculate the average vertical gradient intensity of each column in the grayscale image to form a column-wise gradient intensity curve;

[0031] a3. Traverse the gradient intensity curve to detect whether each grayscale image in the image sequence has a peak in a preset area;

[0032] If it exists, the position is recorded, and then the presence of a stable high gradient band in the region of each grayscale image is detected;

[0033] a4. Compare the traversal results with the preset requirements to obtain the candidate area;

[0034] B. Verify whether the candidate area meets the characteristics of the fruit tree trunk through symmetric morphology recognition;

[0035] b1. Divide the grayscale image into consecutive columns in the horizontal direction and construct a detection area with symmetrical width centered on the candidate area.

[0036] b2. Divide the detection area into two parts symmetrically along the columns, and cut multiple horizontal height bands of a fixed length upward from the bottom of the corresponding grayscale image. Compare the grayscale value differences in each height band of the symmetrical columns on the left and right sides in turn;

[0037] b3. For each height band, calculate the average absolute value of the grayscale value difference of each symmetrical grayscale image;

[0038] b4. If the absolute average value in the 80% height band is lower than the preset grayscale threshold, the detection area is considered to have strong bilateral symmetry in the vertical direction;

[0039] C. Verify the vertical continuity of the tree trunk to see whether it is continuous in the vertical direction of the grayscale image.

[0040] c1. Divide the grayscale image into multiple judgment areas of equal height along the Y-axis from bottom to top, and perform the following judgment in each judgment area;

[0041] c11. Count the column numbers of the stable high-gradient bands in the judgment area;

[0042] c12. Compare the column numbers of the stable high-gradient band in the judgment area with the column numbers of the stable high-gradient band in the previous section to see if they overlap or intersect, that is, the length of the column number intersection accounts for more than 60% of the width of the column number band in the previous section;

[0043] c13. If two adjacent judgment areas both meet c12, they are marked as "structurally continuous";

[0044] c2. If the judgment area that occupies more than half of the height of the corresponding grayscale image is marked as "structurally continuous", it is considered that the vertical continuous structure of the fruit tree trunk exists in the grayscale image;

[0045] c3. The fruit tree structure recognition module outputs the "fruit tree trunk structure exists" signal and gives the column coordinate values ​​of the trunk center points of all height segments, which are used as the central axis of the fruit tree trunk for subsequent fitting.

[0046] A further technical improvement of the present invention is that in step c2, if there is a break between the judgment areas that are continuously marked as "structurally continuous", the fruit tree structure recognition module uses the average column number of the front and back segments to perform linear interpolation to reconstruct the center point coordinates of the middle segment to ensure continuous output of the structural line.

[0047] A further technical improvement of the present invention is that: in step S3, an infrared camera is configured for the working robot, and the viewing angle of the infrared camera is arranged parallel to that of the visible light camera to collect infrared images at the same viewing angle;

[0048] By setting up an ambient light monitoring module, when the brightness of the structural fusion zone area in the image is not within the preset brightness threshold, the visible light image and the infrared image are aligned with the corresponding area, and the image pixel-level fusion is performed with the brightness distribution uniformity in the structural fusion zone as the dominant parameter. The fusion method can adopt any mature method in the existing technology based on the needs, and the enhanced fused image is output;

[0049] And the feature point extraction and structure fusion band matching steps are re-executed on the fused image to update the initial registration relationship between the image frames.

[0050] A further technical improvement of the present invention is that in S2, the step of constructing the structural skeleton curve of the fruit tree trunk includes:

[0051] Z1, collect the coordinates (Xt, Yt) of the bottom center point of the central axis of the fruit tree trunk in each frame of the visible light image in the image sequence, and form a time series arrangement of the center point sequence P = {P1, P2, ... Pn};

[0052] Z2. Calculate the column number difference ΔX between the bottom center points of the central axis of the main trunks of adjacent fruit trees;

[0053] Z3, construct the direction vector for the center point sequence P;

[0054] And calculate the change in angle between adjacent direction vectors;

[0055] If the angle between adjacent direction vectors is continuously smaller than the preset angle threshold, it is recorded as a "continuous segment"

[0056] The proportion of the center point sequence P marked as "continuous segments" is calculated. If it exceeds the preset proportion, it means that the skeleton path structure has strong continuity;

[0057] Otherwise, the current jump point frame is removed, and the end point of the previous stable segment is used as the new starting point to redraw the path backward to avoid structural kinks caused by local misidentification.

[0058] Z4, calculating the vector angle in step Z3, and comparing the vector angle with the preset turning angle threshold and redundant angle threshold;

[0059] If the angle of a vector is greater than the preset angle threshold, it will be retained as a path feature point;

[0060] If the angles of e consecutive vectors are less than the redundant angle threshold, the remaining vector angles except the first and last vector angles are considered redundant, and the corresponding line segments are considered to be approximate straight line segments. The two end points can be combined to express them, reducing the number of unnecessary points.

[0061] Finally, the sequence of compressed path points is output as the structural skeleton curve in the image stitching module;

[0062] The final compressed and optimized bottom center point coordinate group is: {(X1, Y1), (X2, Y3), ..., (Xm, Ym)}, and m≤n.

[0063] A further technical improvement of the present invention is that in step Z2, a weighted smoothing process is performed on the center point sequence (Pt-1, Pt, Pt+1) of a group of three frames, which is performed in the following manner:

[0064] Center point position = (0.25*Pt-1)+(0.5*Pt)+(0.25*Pt+1), and output the updated center point sequence

[0065] A further technical improvement of the present invention is that in S4, the method for predicting the position of the trunk structure of the fruit tree comprises the following steps:

[0066] Q1. Input the current position coordinates of the robot (X z t,Y z t), orientation angle θt, linear velocity vt, angular velocity ωt and historical pose sequence (X z ,Y z ,θ), where the historical pose sequence (X z ,Y z ,θ) is used to determine whether the working robot is in the acceleration section, turning section or uniform speed straight section;

[0067] Q2. Set the prediction time range ΔT and perform step-by-step prediction with a fixed prediction step length Δt to obtain the prediction step number K, K = ΔT ÷ Δt;

[0068] Q3. Calculate the angle difference Δθ and velocity change Δv from the continuous frame pose sequence to determine the robot's operating status, including:

[0069] The straight-line state (Δθ<5°, Δv<0.05m / s) is predicted using a linear straight-line model, including:

[0070] For the k-th step (k∈[1,K]) prediction:

[0071] X z (t+k)=X z t+vt×cos(θt)×k×Δt;

[0072] Y z (t+k)=Y z t+vt×sin(θt)×k×Δt;

[0073] Predicted heading angle: θ(t+k)=θt;

[0074] The turning state (Δθ ≥ 5°) is predicted using a small-angle line segment iterative model and set to continuously deflect at a fixed angular velocity, including:

[0075] X z (t+k)=X z (t+(k-1))+vt×cosθ(t+k)×Δt;

[0076] Y z (t+k)=Y z (t+(k-1))+vt×cosθ(t+k)×Δt;

[0077] Predicted heading angle: θ(t+k)=θt+ωt×k×Δt;

[0078] Last record {X z (t+k),Y z (t+k),θ(t+k)} is the predicted location point, including the predicted path point {X z (t+k),Y z (t+k)} and the corresponding viewing direction θ(t+k);

[0079] Q4, at each predicted location {X z (t+k),Y z (t+k),θ(t+k)}, construct the field of view range model, with θ as the center direction, the opening angle as the field of view angle of the corresponding visible light camera, and the detection radius as the farthest effective imaging distance of the visible light camera;

[0080] Within the scope of the field of view model, it is superimposed with the orchard layout map or the known fruit tree row model, and the image area where the trunk structure is expected to appear in the future image frame is marked based on the orientation angle θ, which serves as the basis for the subsequent structural fusion area positioning;

[0081] The predicted path and the area where the main image appears at each step are spatially matched with the existing frames in the image buffer pool to determine whether they are "hit" for image screening;

[0082] Q5. Output predicted path point sequence {X z (t+1),Y z (t+1),...,X z (t+K),Y z (t+K)}, and the column interval where the structural fusion band corresponding to each predicted path point is located.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] The present invention extracts and connects the central axis of the fruit tree trunk in consecutive image frames to construct a structural skeleton curve as the spatial dominant path for image stitching. This realizes the transformation of image stitching from "image content driven" to "structural line driven", significantly improving the coherence and geometric consistency of the fruit tree trunk structure in the stitched image.

[0085] Furthermore, by setting a structural fusion zone, corner features are extracted only in the extended area of ​​the trunk central axis, and multi-level screening is performed on the corner points to eliminate invalid points and pseudo features, thereby improving the matching accuracy of feature points and effectively reducing the interference of non-structural information such as branch and leaf texture and background areas on the registration.

[0086] At the same time, the present invention introduces a robot posture trajectory prediction mechanism to calculate the camera observation direction and the expected area of ​​the image backbone structure in the future time step. By matching the shooting posture of the image frames in the image cache pool, the image clarity, occlusion rate and backbone integrity are combined for scoring and sorting, and only image frames that meet the prediction range and quality requirements are retained for splicing, thereby improving the splicing efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] To facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0088] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0089] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0090] See also Figure 1As shown, the present invention provides a method for stitching visual navigation images of an orchard operation robot, a method for stitching visual navigation images of an orchard operation robot, the method comprising the following steps:

[0091] S1. Install visible light cameras on the front and both sides of the robot, and ensure that the camera captures images covering the central axis of the fruit tree and the ground path area.

[0092] During the operation of the robot, a visible light camera is used to collect a sequence of visible light images of the front and sides of the robot. The fruit tree structure recognition module analyzes the image sequence and extracts the central axis of the fruit tree trunk.

[0093] It is used to convert "image content" into "structural expression", that is, to switch from passive splicing of image content to active extraction of target structure, provide spatial constraints for splicing, and improve stability;

[0094] In step S1, the fruit tree structure recognition module analyzes the image sequence, including image gradient direction analysis, symmetry morphology recognition, and vertical region continuity judgment, specifically including the following steps:

[0095] A. Analyze the vertical edge concentration zone in the trunk area of ​​the fruit tree by image gradient direction;

[0096] a1. Perform grayscale processing on each visible light image in the image sequence to obtain a corresponding grayscale image;

[0097] Perform vertical edge extraction on each frame of grayscale image. In this embodiment, a 3*3 fixed-size Sobel operator is used to obtain the vertical gradient intensity of each column of pixels.

[0098] a2. Calculate the average vertical gradient intensity of each column in the grayscale image to form a column-wise gradient intensity curve;

[0099] a3. Traverse the gradient intensity curve and detect whether there is a peak in the 1 / 3 area of ​​each grayscale image in the image sequence;

[0100] If it exists, the position is recorded, and then the presence of a stable high gradient band (i.e., the gradient values ​​of multiple adjacent columns are more than 1.5 times higher than the average vertical gradient intensity) in the 1 / 3 area of ​​each grayscale image is detected;

[0101] a4. Compare the traversal results with the preset requirements to obtain the candidate area;

[0102] In this embodiment, if a stable high-gradient band is detected in the middle region of each grayscale image and its width does not exceed 1 / 5 of the grayscale image width, that is, if the width of the visible light image is 640 pixels and the middle stable high-gradient band does not exceed 128 pixels, then the region is preliminarily determined to be a region that may contain the trunk structure of a fruit tree, and the region is determined as a candidate region;

[0103] B. Verify whether the candidate area meets the characteristics of the fruit tree trunk through symmetric morphology recognition;

[0104] b1. Divide the grayscale image into consecutive columns along the horizontal direction and construct a detection area with symmetrical width centered on the candidate area. For example, with the candidate area as the center, expand 40 pixels to the left and right to form a symmetrical detection area with a width of 80 pixels.

[0105] b2. Divide the detection area into two parts symmetrically along the columns, and cut multiple horizontal height bands of a fixed length upward from the bottom of the corresponding grayscale image. Compare the grayscale value differences in each height band of the symmetrical columns on the left and right sides in turn;

[0106] b3. For each height band, calculate the average absolute value of the grayscale value difference of each symmetrical grayscale image;

[0107] b4. If the absolute average value in the 80% height band is lower than the preset grayscale threshold, the detection area is considered to have strong bilateral symmetry in the vertical direction;

[0108] C. Verify the vertical continuity of the tree trunk to see whether it is continuous in the vertical direction of the grayscale image.

[0109] c1. Divide the grayscale image into multiple judgment areas of equal height along the Y-axis from bottom to top, and perform the following judgment in each judgment area;

[0110] c11. Count the column numbers of the stable high-gradient bands in the judgment area;

[0111] c12. Compare the column numbers of the stable high-gradient band in the judgment area with the column numbers of the stable high-gradient band in the previous section to see if they overlap or intersect, that is, the length of the column number intersection accounts for more than 60% of the width of the column number band in the previous section;

[0112] c13. If two adjacent judgment areas both meet c12, they are marked as "structurally continuous";

[0113] c2. If the judgment area that occupies more than half of the height of the corresponding grayscale image is marked as "structurally continuous", it is considered that the vertical continuous structure of the fruit tree trunk exists in the grayscale image;

[0114] In step c2, if there are breaks between the consecutive judgment areas marked as "structurally continuous", the fruit tree structure recognition module uses the average column number of the front and back segments to perform linear interpolation to reconstruct the coordinates of the center point of the middle segment to ensure the continuous output of the structural line;

[0115] c3. The fruit tree structure recognition module outputs a "fruit tree trunk structure exists" signal and provides the column coordinate values ​​of the trunk center points of all height segments, which are used as the central axis of the fruit tree trunk for subsequent fitting;

[0116] S2. Connecting the central axes of consecutive fruit tree trunks in the image sequence to construct a structural skeleton curve of the fruit tree trunk in the operation path. The structural skeleton curve provides a spatial dominant path for subsequent image stitching;

[0117] Image stitching is no longer an "image-to-image" match, but a target-driven stitching "around the structural skeleton." The stitching path is clearly controlled, solving the problem of frame skipping or breakage caused by visual duplication, occlusion, and misalignment in orchards.

[0118] In S2, the steps of constructing the structural skeleton curve of the fruit tree trunk include:

[0119] Z1, collect the coordinates (Xt, Yt) of the bottom center point of the central axis of the fruit tree trunk in each frame of the visible light image in the image sequence, and form a time series arrangement of the center point sequence P = {P1, P2, ... Pn};

[0120] Z2. Calculate the column number difference ΔX between the bottom center points of the central axis of the main trunks of adjacent fruit trees;

[0121] In the Z2 step, the weighted smoothing process is performed on the center point sequence (Pt-1, Pt, Pt+1) of three frames in a group, which is adopted as follows:

[0122] Center point position = (0.25*Pt-1)+(0.5*Pt)+(0.25*Pt+1), and output the updated center point sequence

[0123] Z3. Construct direction vector for center point sequence P:

[0124] And calculate the change in angle between adjacent direction vectors;

[0125] If the angle between adjacent direction vectors is continuously smaller than the preset angle threshold, it is recorded as a "continuous segment"

[0126] The proportion of the center point sequence P marked as "continuous segments" is calculated. If it exceeds the preset proportion, it means that the skeleton path structure has strong continuity;

[0127] Otherwise, the current jump point frame is removed, and the end point of the previous stable segment is used as the new starting point to redraw the path backward to avoid structural kinks caused by local misidentification.

[0128] Z4, calculating the vector angle in step Z3, and comparing the vector angle with the preset turning angle threshold and redundant angle threshold;

[0129] If the angle of a vector is greater than the preset angle threshold, it will be retained as a path feature point;

[0130] If the angles of e consecutive vectors are less than the redundant angle threshold, the remaining vector angles except the first and last vector angles are considered redundant, and the corresponding line segments are considered to be approximate straight line segments. The two end points can be combined to express them, reducing the number of unnecessary points.

[0131] Finally, the sequence of compressed path points is output as the structural skeleton curve in the image stitching module;

[0132] The final compressed and optimized bottom center point coordinate set is: {(X1, Y1), (X2, Y3), ..., (Xm, Ym)}, and m ≤ n;

[0133] Furthermore, compression optimization is completed within each frame image output cycle, with a single calculation delay of no more than 10ms, ensuring that the real-time constraints of the robot's embedded system are met.

[0134] Establish geometric constraints for image registration through the continuity of the target structure to ensure the consistency and physical interpretability of the stitching results;

[0135] Furthermore, the structural skeleton curve serves as the "spatial anchor" of the stitching process, effectively preventing image dislocation, drift, and even stitching distortion caused by motion.

[0136] S3, extracting the central axis of the fruit tree trunk of each frame image in the image sequence as the central axis, and extending the preset pixel distance to the left and right sides and the top and bottom sides of the image to form a structural fusion zone area, which is used as the core area for image stitching;

[0137] Perform feature point extraction on each frame of image, use corner point extraction algorithm to extract all corner point features, and filter all feature points to retain only the feature points located in the structural fusion zone;

[0138] The retained feature points are used for feature matching between adjacent image frames, wherein the matching operation is limited to be performed between corresponding areas of the structure fusion zone to establish an initial registration relationship between the image frames;

[0139] It is used to clearly define the stitching area to avoid feature redundancy and mismatching caused by non-structural areas such as image edges, sky, branches and leaves;

[0140] In addition, only stable feature points within the backbone control area are used to improve registration quality and positioning accuracy. Image registration is narrowed from "full-image feature point matching" to "feature alignment within the structural control area", effectively filtering out unstable feature points, thereby improving the robustness of stitching.

[0141] In this embodiment, the central axis of the fruit tree trunk in each frame of the image is expanded 40 pixels in the left and right directions and 20 pixels in the up and down directions to form a continuous longitudinal rectangular area;

[0142] Finally, the pixel coordinate range of the structural fusion zone is:

[0143] Horizontal (column coordinate) range: ±40 pixels from the column where the center line of the backbone is located;

[0144] Vertical (row coordinate) range: The starting point of the main line extends upward to the top + 20 pixels, and the end point extends downward to the bottom + 20 pixels;

[0145] It specifically uses the ORB algorithm to extract feature points, including:

[0146] During the extraction process, a fixed window size (e.g., 31×31) is used to perform FAST corner detection and feature descriptors are generated using the BRIEF method.

[0147] In each image frame, only the extraction operation is performed on the structure fusion zone area, and corner point extraction is prohibited on the image edge, sky background, and ground non-structure area to avoid redundancy and mismatching;

[0148] After extraction, the corner points are screened as follows: the grayscale difference between the grayscale of the pixel where the corner point is located and the mean of the neighborhood area is calculated. If the grayscale contrast is lower than the set grayscale ratio threshold, it is considered a weak corner point and is removed;

[0149] If a corner point is more than 5 pixels away from the boundary of the structural fusion zone, or more than 50 pixels away from the center line of the main trunk, it is determined to be an "unstable position point" and is removed;

[0150] Finally, the corner features that meet the spatial position requirements, have high response values, and large grayscale contrast are retained as feature points to construct a stable registration reference point set;

[0151] In this embodiment, an image with a width of 640 pixels and a height of 480 pixels is used as an example:

[0152] The central axis of the fruit tree trunk is mainly distributed between column coordinates 280–300, so the column range of the structural fusion zone is: 240–340 (i.e., 280 ± 40 pixels);

[0153] Among the extracted ORB corner points, if a corner point has a response value of 20, a grayscale contrast of 5, and is located at column 380 and row 200, then this corner point is removed due to "outside the fusion band + weak response";

[0154] Finally, 60–80 valid feature points are retained in each frame, mainly concentrated in the trunk structure area of ​​the fruit tree;

[0155] S4. Equip the working robot with an inertial measurement unit and a wheel encoder to collect the current position and posture data of the working path in real time;

[0156] Based on the current position and posture data and the historical position and posture data, the viewing angle change range in the next several time steps is predicted, and the appearance position of the main structure of the fruit tree in the future image frame is estimated;

[0157] The method for predicting the position of the trunk structure of a fruit tree comprises the following steps:

[0158] Q1. Input the current position coordinates of the robot (X z t,Y z t), orientation angle θt, linear velocity vt, angular velocity ωt and historical pose sequence (X z ,Y z ,θ), where the historical pose sequence (X z ,Y z ,θ) is used to determine whether the working robot is in the acceleration section, turning section or uniform speed straight section;

[0159] Q2. Set the prediction time range ΔT and perform step-by-step prediction with a fixed prediction step length Δt to obtain the prediction step number K, K = ΔT ÷ Δt;

[0160] Q3. Calculate the angle difference Δθ and velocity change Δv from the continuous frame pose sequence to determine the robot's operating status, including:

[0161] The straight-line state (Δθ<5°, Δv<0.05m / s) is predicted using a linear straight-line model, including:

[0162] For the k-th step (k∈[1,K]) prediction:

[0163] X z (t+k)=X z t+vt×cos(θt)×k×Δt;

[0164] Y z (t+k)=Y z t+vt×sin(θt)×k×Δt;

[0165] Predicted heading angle: θ(t+k)=θt;

[0166] The turning state (Δθ ≥ 5°) is predicted using a small-angle line segment iterative model and set to continuously deflect at a fixed angular velocity, including:

[0167] X z (t+k)=X z (t+(k-1))+vt×cosθ(t+k)×Δt;

[0168] Y z (t+k)=Y z (t+(k-1))+vt×cosθ(t+k)×Δt;

[0169] Predicted heading angle: θ(t+k)=θt+ωt×k×Δt;

[0170] Last record {X z (t+k),Y z (t+k),θ(t+k)} is the predicted location point, including the predicted path point {X z (t+k),Y z (t+k)} and the corresponding viewing direction θ(t+k);

[0171] Q4, at each predicted location {X z (t+k),Y z (t+k),θ(t+k)}, construct the field of view range model, with θ as the center direction, the opening angle as the field of view angle of the corresponding visible light camera, and the detection radius as the farthest effective imaging distance of the visible light camera;

[0172] Within the scope of the field of view model, it is superimposed with the orchard layout map or the known fruit tree row model, and the image area where the trunk structure is expected to appear in the future image frame is marked based on the orientation angle θ, which serves as the basis for the subsequent structural fusion area positioning;

[0173] The predicted path and the area where the main image appears at each step are spatially matched with the existing frames in the image buffer pool to determine whether they are "hit" for image screening;

[0174] Q5. Output predicted path point sequence {X z (t+1),Y z (t+1),...,X z (t+K),Y z (t+K)}, and the column interval of the structural fusion zone corresponding to each predicted path point;

[0175] For example, the predicted path point of the working robot is {X z (t+3),Y z (t+3)}, the heading angle θ = 0°, and the visible light camera resolution is 640 pixels;

[0176] Centered on image column 320, expand 80 pixels to the left and right.

[0177] The predicted area of ​​the backbone structure in the third frame image is the column number interval: 240–400;

[0178] If the structural fusion zone in the image frame is exactly within this range, the image is valid → enters the image cache pool;

[0179] Load image frames that meet the predicted viewing angle range into the image buffer pool, and rank the candidate image frames based on the clarity score, trunk area integrity score, and occlusion rate score;

[0180] In this embodiment, the clarity score is performed by performing edge detection on the entire image and calculating the average intensity of the edge gradient to measure the clarity of the image. If the image gradient value is lower than the set gradient value threshold, it is determined to be a blurred image;

[0181] The trunk area integrity score is determined by detecting whether there is a complete and continuous trunk structure of the fruit tree in the image. If there is a break, occlusion, or recognition failure in the structural fusion zone, the score will be reduced.

[0182] The occlusion rate score is calculated by counting the percentage of obscured or invalid information in the main structural area of ​​the image, such as overexposure, occlusion by branches and leaves, etc. The larger the proportion of obscured area, the lower the score;

[0183] Image frames with scores above the preset threshold are selected for the current stitching process, and image frames with scores below the threshold are discarded to improve stitching accuracy. Image frames that are not suitable for stitching due to occlusion, blur, overexposure, etc. are eliminated, reducing the post-processing burden.

[0184] The image cache pool is updated using a sliding time window: when a new image frame enters, the earliest frame is cleared from the cache pool to ensure that the cache capacity does not exceed the set upper limit;

[0185] For example, if the current speed of the robot is 0.4 m / s, then 5 time points are predicted in units of 0.1 seconds. There are currently 10 frames of images in the image cache pool. The system finds 3 matching frames in the predicted path.

[0186] The rating results are:

[0187] Image A: high clarity, complete backbone structure, low occlusion rate, score 95;

[0188] Image B: average clarity, partial occlusion of the main trunk, score 78;

[0189] Image C: Broken trunk and low clarity, score 62 (rejected);

[0190] Finally, image A and image B are used as the stitching input, and image C is excluded;

[0191] S5. For each pair of adjacent image frames, calculate the initial image registration transformation relationship based on the registered feature points in the structural fusion zone area;

[0192] In this embodiment, the affine transformation matrix estimated by the least squares method is used to calculate the image registration transformation relationship, including the three degrees of freedom of rotation, translation and scaling, to align the subsequent frame image to the coordinate system of the previous frame image and form a preliminary splicing docking;

[0193] During the transformation calculation process, a continuity constraint model of the fruit tree trunk structure is constructed to detect whether the position of the fruit tree trunk structure in the continuous frame images after the registration transformation is offset, bent or misplaced;

[0194] In this embodiment, the continuity constraint model extracts the key points of the central axis of the fruit tree trunk in adjacent image frames after splicing, and determines the position change trend of these points in the image coordinates; if it is found that the center point of the trunk between adjacent frames is offset too much, or the line formed by multiple consecutive center points is obviously bent or broken, it is determined to be structural discontinuity; and the continuity constraint model sets a column coordinate offset threshold and a structural angle change threshold to determine whether the spliced ​​structure is stable and linear;

[0195] If the continuity is not satisfied, the image registration relationship is readjusted to ensure the linear consistency and coherence of the fruit tree trunk structure in the final spliced ​​image;

[0196] If the main trunk structure of the fruit tree is detected to be discontinuous or the offset exceeds the preset range, the stitching parameters are automatically adjusted and the image registration relationship is recalculated to ensure that the main trunk structure line of the fruit tree in the stitched image is continuous and the linear features are consistent;

[0197] Specifically, the column coordinate difference between the positions of the central axis of the backbone in the two images is compared, which is defined as the backbone offset value. If the backbone offset value exceeds the set tolerance threshold, it is judged as structural misalignment or backbone discontinuity, and the current transformation matrix is ​​re-estimated: for example, the weights of feature points near the backbone center point are amplified, or unreliable edge points are removed and the registration relationship is recalculated until the structural lines are continuous and smooth.

[0198] If there are similar misalignments in three consecutive frames, the system will record the abnormal segment mark for post-processing and cleaning;

[0199] In the image fusion process, a weighted fusion algorithm is used, in which the structural fusion zone area is set as a high-weight fusion area, and other areas are set as low-weight fusion areas. Finally, a spliced ​​image with a continuous trunk structure of the fruit tree, clear image content, and uniform brightness distribution is generated.

[0200] By introducing a structural continuity detection mechanism into the stitching results, the stitching parameters are re-optimized if anomalies such as main structure dislocation or bending occur. In addition, by focusing on preserving the image information of the structural area during the fusion stage, the main image clarity and the naturalness of the image connection are improved.

[0201] By using the continuity of the target structure as the basis for judging the effectiveness of the splicing, it is possible to fundamentally suppress mis-splicing, frame skipping, and structural distortion, ensuring that the spliced ​​image can be used for navigation path recognition and control;

[0202] For example, before stitching two frames, the backbone center column numbers are 285 and 270 respectively, and the backbone offset value is 15, which is lower than the threshold of 20 → directly stitch;

[0203] The fusion region is set between columns 250–310, and the weights w1 = 0.8 and w2 = 0.2 are used;

[0204] The main structure of the fused image is coherent, the brightness of the edges of the two frames transitions naturally, and there are no obvious splicing marks.

[0205] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for stitching images of a robot operating in an orchard, characterized in that: include: S1: The visible light camera captures image sequences from the front and sides of the robot, covering the central axis of the fruit tree and the ground path area. The fruit tree structure recognition module is then used to extract the central axis of the fruit tree trunk in each frame. S2: Connect the central axes of the fruit tree trunks in each frame of the image sequence to generate a structural skeleton curve; S3: Taking the central axis of the fruit tree trunk as the central axis, the preset pixel distance is expanded in the image to form a structural fusion zone area, limiting the extraction and matching of feature points; S4: Based on the current position and posture data of the working robot and its historical posture data, the future viewing angle change range and the expected position of the fruit tree trunk structure in the image are predicted, and the required image frames are selected from the image buffer pool for stitching; S5: Based on the registered feature points in the structural fusion zone, the registration transformation relationship between adjacent image frames is calculated, and a continuity constraint model of the fruit tree trunk structure is constructed. A weighted fusion algorithm is used, with the structural fusion zone area as the high-weight area, to output the spliced ​​image.

2. The method for stitching images of an orchard operation robot according to claim 1, characterized in that: In step S1, the fruit tree structure recognition module analyzes the image sequence, specifically including the following steps: A. Analyze the vertical edge concentration zone in the trunk area of ​​the fruit tree by image gradient direction; B. Verify whether the candidate area meets the characteristics of the fruit tree trunk through symmetric morphology recognition; C. Through vertical area continuity judgment, confirm whether the features of the fruit tree trunk are continuous in the vertical direction of the grayscale image.

3. The method for stitching images of an orchard operation robot using visual navigation according to claim 1, characterized in that: In step S3, an infrared camera is configured for the working robot, and the viewing angle of the infrared camera is kept parallel to that of the visible light camera to collect infrared images under the same visual conditions; When the brightness of the structural fusion zone area in the image is not within a preset brightness threshold, the visible light image and the infrared image are registered in corresponding areas and a fused image is output; And the feature point extraction and structure fusion band matching steps are re-executed on the fused image.

4. The method for visual navigation image stitching of an orchard operation robot according to claim 1, characterized in that: In S2, the steps of constructing the structural skeleton curve of the fruit tree trunk include: Z1, collect the coordinates (Xt, Yt) of the bottom center point of the central axis of the fruit tree trunk in each frame of the visible light image in the image sequence, and form a time series arrangement of the center point sequence P = {P1, P2, ... Pn}; Z2. Calculate the column number difference between the bottom center points of the central axis of the main trunks of adjacent fruit trees. ; Z3, construct direction vectors for the center point sequence P, and calculate the change in angle between adjacent direction vectors; Z4, calculating the vector angle in step Z3, and comparing the vector angle with the preset turning angle threshold and redundant angle threshold; If the angle of a vector is greater than the preset angle threshold, it will be retained as a path feature point; If the angles of e consecutive vectors are smaller than the redundant angle threshold, the remaining vector angles except the first and last vector angles are considered redundant, and the corresponding line segments are considered straight line segments; Finally, the sequence of compressed path points is output as the structural skeleton curve in the image stitching module.

5. The method for visual navigation image stitching of an orchard operation robot according to claim 4, characterized in that: In the Z2 step, the weighted smoothing process is performed on the center point sequence (Pt-1, Pt, Pt+1) of three frames in a group, which is done as follows: Center point position = (0.25*Pt-1) + (0.5*Pt) + (0.25*Pt+1), and output the updated center point sequence .

6. The method for stitching images of an orchard operation robot using visual navigation according to claim 1, characterized in that: In S4, the method for predicting the position of the trunk structure of a fruit tree includes the following steps: Q1. Enter the current position coordinates of the operating robot , orientation angle , Linear speed , angular velocity and historical pose sequence ( , , ); Q2. Set the forecast time range , and with a fixed prediction step size Perform step-by-step prediction and get the number of prediction steps K, K= ÷ ; Q3. Calculate the angle difference through the continuous frame pose sequence and speed changes , determine the operating status of the robot; Q4, at each predicted location { On the top, construct the field of view range model, and use is the center direction, the opening angle is the field of view of the corresponding visible light camera, and the detection radius is the farthest effective imaging distance of the visible light camera; Within the scope of the field of view model, it is superimposed on the orchard layout map or the known fruit tree row model and the orientation angle is used to As a benchmark, mark the image area where the backbone structure is expected to appear in future image frames; The predicted path and the area where the main image appears at each step are spatially matched with the existing frames in the image buffer pool to determine whether they are "hit"; Q5. Output predicted path point sequence { , and the column interval of the structural fusion band corresponding to each predicted path point.

7. The method for stitching images of an orchard operation robot using visual navigation according to claim 6, characterized in that: In step Q3, the robot's operating status includes: Straight state ( <5°, <0.05m / s), using linear straight line model prediction, including: For the k-th step (k∈[1,K]) prediction: ； ； Predicted heading angle: = ; Turn state ( ≥5°), using a small-angle line segment iterative model prediction and setting it to continuously deflect at a fixed angular velocity, including: ； ； Predicted heading angle: = ; Last record { For the predicted location point, including the predicted path point { And the corresponding viewing direction .

8. The method for visual navigation image stitching of an orchard operation robot according to claim 2, characterized in that: Methods for determining the continuity of vertical regions include: c1. Divide the grayscale image into multiple judgment areas of equal height along the Y-axis from bottom to top, and perform the following judgment in each judgment area; c11. Count the column numbers of the stable high-gradient bands in the judgment area; c12. Compare the column numbers of the stable high-gradient bands in the judgment area with the column numbers of the stable high-gradient bands in the previous section to see if they overlap or intersect; c13. If two adjacent judgment areas both meet c12, they are marked as "structurally continuous"; c2. If the judgment area that occupies more than half of the height of the corresponding grayscale image is marked as "structurally continuous", it is considered that the vertical continuous structure of the fruit tree trunk exists in the grayscale image; c3. The fruit tree structure recognition module outputs the "fruit tree trunk structure exists" signal and provides the column coordinate values ​​of the trunk center points of all height segments.

Images