Multi-mode sensing fusion agricultural robot global positioning method

Abstract

The invention relates to a multi-modal perception fusion agricultural robot global positioning method, and the method comprises the steps: generating a sparse point cloud map and a local position posture of a key frame through improving a feature extraction and matching algorithm of a binocular SLAM system; wGS coordinates obtained through real-time difference are converted into plane coordinates, the poses of the key frames are matched with the plane coordinates through a timestamp difference threshold value synchronous alignment method, the poses of the key frames are converted into global plane coordinates through fusion, and meanwhile the local position poses of the key frames are corrected through the plane coordinates. A key frame map with global plane coordinates and local visual coordinates is imported into the improved SLAM system, and when a field agricultural robot carrying the improved SLAM system encounters real-time difference floating caused by tree shade shielding or passing through a viaduct, the improved SLAM system can provide plane coordinates of centimeter-level visual calculation until the real-time difference is restored to be fixed; the positioning accuracy of the field agricultural robot can be improved, and the problem of inaccurate positioning in related technologies is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent agricultural machinery, in particular to a multi-modal perception fusion agricultural robot global positioning method. BACKGROUND

[0002] Unmanned intelligent farm technology is gradually applied on a large scale. Agricultural robots are indispensable intelligent agricultural machinery in unmanned intelligent farms. Satellite positioning agricultural robots have reduced positioning accuracy when passing under tree shadows or viaducts due to multipath effects and other factors, affecting the operation safety and work efficiency of agricultural robots. Therefore, it is an urgent need to provide a reliable and accurate agricultural robot positioning perception method.

[0003] For the problem of inaccurate positioning in the related art, there is currently no effective solution. SUMMARY

[0004] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a multi-modal perception fusion agricultural robot global positioning method. The improved SLAM system outputs local visual positioning coordinates. When the real-time differential float of the agricultural robot fails, the visual positioning coordinates converted into plane coordinates are used for navigation.

[0005] In this embodiment, a multi-modal perception fusion agricultural robot global positioning method is provided to solve the problem of inaccurate positioning in the related art.

[0006] In a first aspect, a multi-modal perception fusion agricultural robot global positioning method is provided in this embodiment. The method comprises: obtaining images and satellite positioning information; the satellite positioning information includes real-time differential positioning information, and the real-time differential positioning information includes real-time differential world geodetic system coordinate information; converting the real-time differential world geodetic system coordinate information into plane coordinate information; extracting point features and line features from the images and performing rasterization processing to obtain rasterized point features and line features; sending the images and the rasterized point features and line features into an improved simultaneous localization and mapping (SLAM) system, outputting a key frame queue, and generating a key frame map pose; Fuse the plane coordinate information and the key frame map pose, calculate the global plane coordinate of the key frame, and correct the local visual coordinate by using the accurate global plane coordinate; Obtain a key frame map including global plane coordinates and local visual coordinates; Import the key frame map into the improved SLAM system, and perform global positioning.

[0007] In some embodiments, the image is a field image, and the image and satellite positioning information are obtained, including: Obtain a field image taken by a binocular camera and centimeter-level positioning information measured by real-time differential measurement.

[0008] In some embodiments, converting the real-time differential world geodetic system coordinate information into plane coordinate information includes: Obtain real-time differential information, obtain the real-time differential information to be converted from a real-time differential positioning module of an agricultural robot, and the real-time differential information includes centimeter-level longitude, latitude, height, relative height, and Euler angle data; Determine a plane coordinate system projection area, determine the plane coordinate system projection area according to the given longitude information; Calculate the central meridian, calculate the central meridian longitude according to the plane coordinate system projection area; Convert the world geodetic system coordinates of the real-time differential information into plane coordinates using a conversion formula of the plane coordinate system projection; Output the converted plane coordinates, and the plane coordinates include the horizontal Mercator coordinate and the vertical Mercator coordinate, and the plane coordinate system projection area and the central meridian longitude.

[0009] In some embodiments, the conversion formula of the plane coordinate system projection is, wherein, is the vertical coordinate of the plane coordinate system, is the horizontal coordinate of the plane coordinate system, is the latitude in radian, is the meridian arc length, is the radius of the prime vertical circle, is the longitude projection factor, is the square of the tangent of the latitude, is the curvature correction term related to the latitude, is the first eccentricity of the earth ellipsoid, is the projection scale factor.

[0010] In some embodiments, point features and line features are extracted from the image, and rasterization is performed to obtain rasterized point features and line features, including: A multi-layer pyramid model is constructed for the left and right views of the image, respectively; Line detection and corner detection are performed on each layer of the multi-layer pyramid model; For each line, a line band descriptor is generated by comparing the pixel intensities of the sub-regions around the line, and for each corner, a plurality of dimensions of binary robust independent basic feature descriptors are extracted as directional features; The line band descriptor and the directional feature descriptor are rasterized, and each line band descriptor and directional feature descriptor is assigned to a set of grid numbers; The Hamming distance is used to measure the similarity between two feature descriptors; The line band descriptor features and the directional features in the vicinity of a plurality of pixels are sorted based on the pixel distance between the two frames of images, and the feature points with the highest similarity are selected as the matching point pairs; The matching point pairs and the historical key point cloud of the two frames are randomly fitted to calculate the relative motion pose between the two frames of images; In the initialization stage, the 3D point and 3D line parameters are calculated by triangulation and line triangulation, and a point-line hybrid map is constructed; the pose is initialized, the key points and lines of the current frame are projected to update the sparse point cloud, and different weights are assigned to the points and lines, which are used jointly in the back-end optimization.

[0011] In some embodiments, the method further includes: visual odometry initialization of the SLAM system; The visual odometry initialization of the SLAM system includes: the visual odometry calculates the movement distance between the frame sequence, and when the distance exceeds a pre-set threshold, the current frame is marked as a key frame and stored.

[0012] In some embodiments, the image and the rasterized point features and line features are input into an improved SLAM system, and a key frame queue is output to generate a key frame map pose, including: The image, the rasterized LBD features and the ORB features are directly input into the SLAM system through an extended interface, and the SLAM system performs feature extraction, feature matching, pose estimation and map updating; The SLAM system automatically selects key frames according to the motion amplitude and the threshold of the change in viewing angle; the key frames serve as anchor points for map construction, provide temporal and spatial constraints in the back-end optimization, and ensure the stability of the pose estimation; Based on the feature matching relationship between adjacent key frames, the system calculates the three-dimensional coordinates of the feature points by using the principle of triangulation; by minimizing the re-projection error, a sparse point cloud map composed of spatial feature points is generated; The system constructs a nonlinear optimization problem with camera pose as a parameter by tracking the motion trajectory of feature points in the key frame sequence; the Gauss-Newton method or the Levenberg-Marquardt algorithm is used for iterative solution to realize the maximum likelihood estimation of the pose parameter; As the video stream continues to input, the SLAM system dynamically maintains the key frame database and the sparse point cloud map; the map structure and the camera trajectory are continuously optimized by local and global bundle adjustment, and new key frames are inserted according to the observation results.

[0013] In some embodiments, the plane coordinate information and the key frame map pose are fused to calculate the global plane coordinates of the key frame, and the accurate global plane coordinates are used to correct the local visual coordinates, including: Obtaining plane coordinate system information and calculating key frame map pose; Aligning the timestamps of the plane coordinate system information and the calculated key frame map pose timestamps, first, the timestamps of the two are subtracted, and second, the plane coordinate frames and the key frame map pose frames that are less than the pre-set threshold are taken as time-synchronized matching pairs; In the matched time-synchronized pairs, the plane coordinate frames are assigned to the key frames, a transformation model of the sparse map from the visual coordinate system to the global plane coordinate system is established, and the global plane coordinates of the key frame are obtained; The calculated key frame map pose is corrected by the global plane coordinates of the key frame.

[0014] In some embodiments, the transformation model of the sparse map from the visual coordinate system to the global plane coordinate system is: The camera coordinate system takes the right lens optical center of the binocular camera as the origin and follows the right-hand coordinate system; the horizontal coordinate X of the plane coordinate system points to the east, and the vertical coordinate Y points to the north, with units of meters; the rotation relationship between the camera and the GNSS antenna is: wherein, RGNSS→C represents the rotation matrix from the GNSS antenna coordinate system to the camera coordinate system, is a 3x3 unit matrix; The conversion of the local visual coordinates to the plane coordinates follows the following formula: ; ; wherein, Xp is the horizontal axis coordinate of the plane coordinate system, Yp is the vertical axis coordinate of the plane coordinate system, H is the heading angle, is the 3rd number of the translation vector, is the first number of the translation vector, is the calibrated visual origin coordinate X, is the calibrated visual origin coordinate Y.

[0015] In some embodiments, the calculated key frame map pose is corrected by the global plane coordinate of the key frame, including: The position and posture is agreed to be , , is the first key frame rotation matrix and translation vector from the camera coordinate system origin to the current local visual coordinate system, is the number of key frames; the origin coordinates of the local visual coordinate are obtained, the plane coordinates corresponding to the origin coordinates in the matching pair are obtained, which are marked as visual origin coordinates, and the visual coordinates are converted into plane coordinates, which are marked as , is the position coordinate of the first key frame in the local visual coordinate system, is the horizontal coordinate of the coordinate, is the vertical coordinate; the position coordinate in the global plane coordinate system is , is the position coordinate of the first key frame in the global plane coordinate system; the matching pair of the key frame global plane coordinate matched after the timestamp and the key frame local visual coordinate is ; the visual coordinates and global plane coordinates after coordinate conversion are calculated respectively , , , , is the change amount of the horizontal coordinate of the adjacent key frame in the local coordinate system; is the change amount of the vertical coordinate of the adjacent key frame in the local coordinate system; is the change amount of the horizontal coordinate of the adjacent key frame in the global plane coordinate system; is the change amount of the vertical coordinate of the adjacent key frame in the global plane coordinate system; the plane coordinates measured by real-time difference are taken as true values, , , , is the change amount of the horizontal and vertical coordinates of the corrected adjacent key frame in the plane coordinate system, and the corrected , ,

[0016] Compared with the related art, the multi-modal perception fusion agricultural robot global positioning method provided in the embodiment realizes high-precision real-time tracking and map construction of a field scene, generates a sparse point cloud map and a local position and posture of a key frame, by improving a feature extraction and matching algorithm of a binocular SLAM system, converts a WGS coordinate obtained by real-time difference into a plane coordinate, matches the position and posture of the key frame with the plane coordinate, and fuses the position and posture of the key frame into a global plane coordinate while correcting the local position and posture of the key frame by using the plane coordinate. The key frame map with the global plane coordinate and the local visual coordinate is imported into the improved SLAM system, when the field agricultural robot carrying the improved SLAM system encounters tree shade shielding or passes through an overpass to cause real-time difference floating, the improved SLAM system can provide a plane coordinate of centimeter-level visual calculation until the real-time difference is restored to be fixed, and the positioning accuracy of the field agricultural robot can be improved, and the problem of inaccurate positioning in the related art is solved.

[0017] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0018] The drawings described herein are intended to provide further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a hardware structure block diagram of a terminal of a multi-modal perception fusion agricultural robot global positioning method provided by an embodiment of the present application; Figure 2 is a flowchart of a multi-modal perception fusion agricultural robot global positioning method provided by an embodiment of the present application; Figure 3 is a flowchart of a multi-modal perception fusion agricultural robot global positioning method provided by an embodiment of the present application; Figure 4 is a point and line feature recognition and extraction image provided by an embodiment of the present application; Figure 5 is a sparse point cloud map of a city road scene provided by an embodiment of the present application; Figure 6 is a field local sparse point cloud map provided by an embodiment of the present application; Figure 7 is an error comparison diagram of positioning in an RTK floating solution state and multi-modal fusion positioning provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0020] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the general meaning understood by a person skilled in the art to which the present application belongs. In the present application, the terms "one", "a", "an", "the", "these", and similar words do not represent a quantitative limitation, but can be singular or plural. In the present application, the terms "include", "contain", "have" and any variants thereof are intended to cover non-exclusive inclusion; for example, a process, method and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. In the present application, the terms "connected", "connected", "coupled" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. In the present application, "multiple" means two or more. The term "and / or" describes the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. In general, the character " / " represents an "or" relationship between the associated objects. In the present application, the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0021] First, the terms related to one or more embodiments of the present specification are explained.

[0022] In the field of agriculture, the field generally refers to a large area, open terrain, cultivated land, which is distinguished from greenhouse, greenhouse and other facility agriculture. Field image refers to image data about large area farmland (i.e. "field") collected by various technical means. Field information refers to the sum of various information obtained and processed by various modern information technologies (such as remote sensing, Internet of Things, etc.) in the whole process of field planting.

[0023] LSD (Line Segment Detector), line segment detector, an algorithm for quickly and accurately detecting line structures from images.

[0024] FAST (Features from Accelerated Segment Test), accelerated segment test feature, a high-speed corner (point with sharp brightness change in image) detection algorithm.

[0025] LBD (Line Band Descriptor), Line Band Descriptor. A vector used to describe a straight line feature, which represents the appearance of the straight line by counting the gradient information of the band-shaped area around the straight line.

[0026] BRIEF (Binary Robust Independent Elementary Features), Binary Robust Independent Elementary Features. A method for generating a binary descriptor, which generates a string of binary codes by comparing the gray values of random point pairs in the neighborhood of the feature point, for describing the corner.

[0027] ORB (Oriented FAST and Rotated BRIEF), Oriented FAST and Rotated BRIEF. It combines the FAST corner detector and the BRIEF descriptor, and adds direction (rotation) invariance to it, which is a very efficient and commonly used image feature.

[0028] RANSAC (Random Sample Consensus), Random Sample Consensus, a robust estimation algorithm, which is used to filter out correct "inliers" from matching point pairs containing noise (false matches) and fit the optimal mathematical model.

[0029] The method embodiments provided in the embodiment can be executed in a terminal, a computer or a similar operation device. For example, it is run on a terminal, Figure 1 is a hardware structure block diagram of a terminal of a multi-modal perception fusion agricultural robot global positioning method provided by the embodiment of the present application. As shown in Figure 1 , the terminal can include one or more (only one is shown in Figure 1 ) processor 102 and memory 104 for storing data, wherein the processor 102 can include but not limited to processing devices such as microprocessor MCU or programmable logic device FPGA. The above terminal can also include a transmission device 106 for communication function and an input and output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above terminal. For example, the terminal can also include more or less components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0030] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program of a multi-modal perception fusion agricultural robot global positioning method in the embodiment. The processor 102 can execute various functional applications and data processing, i.e., implement the method described above, by running the computer program stored in the memory 104. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include memories remotely arranged with respect to the processor 102, which can be connected to the terminal through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0031] The transmission device 106 is used to receive or send data via a network. The network includes a wireless network provided by a communication provider of the terminal. In an example, the transmission device 106 includes a network interface controller (NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In an example, the transmission device 106 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0032] In the embodiment, a multi-modal perception fusion agricultural robot global positioning method is provided, Figure 2 is a flowchart of the multi-modal perception fusion agricultural robot global positioning method provided by the embodiment of the application, as shown in Figure 2 The flowchart includes the following steps: In step S210, an image and satellite positioning information are acquired. The satellite positioning information includes real-time differential measurement positioning information, and the real-time differential measurement positioning information includes real-time differential world geodetic system coordinate information.

[0033] Specifically, a field image photographed by a binocular camera and real-time differential measurement centimeter-level positioning information are acquired.

[0034] In step S220, the real-time differential world geodetic system coordinate information is converted into plane coordinate information.

[0035] Specifically, real-time differential world geodetic system (WGS) coordinate information is converted into plane coordinate system information.

[0036] Step S230, point features and line features are extracted from the image, and rasterization processing is performed to obtain rasterized point features and line features.

[0037] Specifically, the left and right views of step S210 are respectively constructed into multi-layer pyramid models, point and line features are extracted from each layer of the pyramid divided image, and the point and line features of each layer of the pyramid divided image are rasterized.

[0038] Step S240, the image and the rasterized point features and line features are sent into the improved SLAM system, and a key frame queue is output to generate a key frame map pose.

[0039] Specifically, the visual odometry is initialized, the visual odometry is run to generate a key frame map, the image of step S210 and the rasterized point and line features of step S230 are sent into the front tracking module of the SLAM system, a key frame queue is output, and a key frame map pose is generated.

[0040] Step S250, fuse the plane coordinate information and the key frame map pose to calculate the global plane coordinates of the key frame, and correct the local visual coordinates using the accurate global plane coordinates.

[0041] Specifically, the plane coordinate information and the key frame map pose of S240 are fused to calculate the global plane coordinates of the key frame, and the accurate global plane coordinates are used to correct the local visual coordinates.

[0042] Step S260, obtain a key frame map including global plane coordinates and local visual coordinates.

[0043] Step S270, import the key frame map into the improved SLAM system, and perform global positioning.

[0044] Specifically, the key frame map is imported into the improved SLAM system, and the agricultural robot equipped with the improved SLAM system can provide centimeter-level visual plane coordinate system positioning coordinates when encountering real-time differential floating, until the real-time differential is restored to fixed.

[0045] By improving the feature extraction and matching algorithm of the binocular SLAM system, high-precision real-time tracking and map construction of the field scene are realized, and a sparse point cloud map of key frames and a local position and posture are generated. The WGS coordinates obtained by real-time difference are converted into plane coordinates, and the pose of the key frame is matched with the plane coordinates by a time stamp difference threshold synchronization alignment method. The pose of the key frame is transformed to the global plane coordinate by fusion, and the local position and posture of the key frame are corrected using the plane coordinate. The key frame map with global plane coordinates and local visual coordinates is imported into the improved SLAM system. When the field agricultural robot carrying the improved SLAM system encounters tree shade obstruction or passes through an overpass causing real-time difference floating, the improved SLAM system can provide centimeter-level visual calculation of plane coordinates until the real-time difference is restored. The application can improve the positioning accuracy of the field agricultural robot and solve the problem of inaccurate positioning in the related art.

[0046] By integrating a binocular camera and a real-time difference positioning module on a flying agricultural robot platform, rich field information can be efficiently collected. After processing by the processing center, these data can construct a sparse field map, which is imported as prior information into the improved SLAM system. The system realizes real-time pose calculation of the agricultural robot in the field environment through feature point and line recognition extraction, image frame matching and repositioning algorithms. This visual perception-based positioning method is low in cost and high in adaptability, and can achieve accurate positioning and navigation in complex field environments. In addition, the introduction of a straight line feature detection and matching module in the SLAM system can effectively compensate for the single texture defect in the field scene, significantly improve the system's perception ability of linear features such as crop rows and ridges, and enhance the robustness of the algorithm in dynamic lighting and vegetation growth changes. The SLAM system that integrates prior maps and straight line feature enhancement can continuously output centimeter-level high-precision positioning results, ensuring the navigation continuity and trajectory accuracy of the agricultural robot in the planting and harvesting process, and providing reliable positioning information support for unmanned intelligent farm agricultural robots.

[0047] In some embodiments, converting the real-time difference world geodetic system coordinate information into plane coordinate information includes: obtaining real-time difference information, obtaining the real-time difference information to be converted from a real-time difference positioning module of the agricultural robot, the real-time difference information including centimeter-level longitude, latitude, height, relative height, and Euler angle data; determining a plane coordinate system projection area, determining the plane coordinate system projection area according to the given longitude information; calculating a central meridian, calculating the central meridian longitude according to the plane coordinate system projection area; converting the world geodetic system coordinates of the real-time difference information into plane coordinates using a conversion formula of the plane coordinate system projection; and outputting the converted plane coordinates, the plane coordinates including horizontal Mercator coordinates and vertical Mercator coordinates, and the plane coordinate system projection area and the central meridian longitude.

[0048] In some embodiments, the conversion formula of the plane coordinate system projection is, wherein, is a longitudinal axis coordinate of the plane coordinate system, is a transverse axis coordinate of the plane coordinate system, is a latitude in radian, is a meridian arc length, is a radius of curvature of the prime vertical, is a meridian difference projection factor, is a latitude tangent square, is a latitude related curvature correction term, is a first eccentricity of the earth ellipsoid, is a projection scale factor.

[0049] In some embodiments, point features and line features are extracted from the image and rasterized to obtain rasterized point features and line features, including: constructing a multi-layer pyramid model for left and right views of the image respectively; performing line detection and corner detection on each layer of the multi-layer pyramid model; for each line, generating a line band descriptor by comparing pixel intensities in a sub-region around the line, and for each corner, extracting a plurality of dimensions of binary robust independent basic feature descriptors as directional features; rasterizing the line band descriptor and the directional feature descriptor, and assigning each line band descriptor and directional feature descriptor to a set of grid numbers; using Hamming distance to measure the similarity between two feature descriptors; sorting the line band descriptor features and directional features in a nearby plurality of pixels based on the pixel distance between the two frames of images, and selecting the feature points with the highest similarity as matching point pairs; performing random fitting on the matching point pairs and the historical key point cloud of the two frames to calculate the relative motion pose between the two frames of images; initializing the 3D point and 3D line parameters using triangulation and line triangulation to construct a point-line hybrid map; initializing the pose, projecting and updating the sparse point cloud of the current frame of key points and lines, assigning different weights to the points and lines, and jointly using them in the back-end optimization.

[0050] In some embodiments, the directional features can be ORB features, the corners can be FAST corners, and the line detection can be LSD line detection.

[0051] In some embodiments, the method further includes: visual odometry initialization of the SLAM system; the visual odometry initialization of the SLAM system includes: the visual odometry calculates the moving distance between frame sequences, and when the distance exceeds a pre-set threshold, marks the current frame as a key frame and stores it.

[0052] In some embodiments, the image and the point features and line features of the rasterization processing are input into the improved SLAM system, and a key frame queue is output, and a key frame map pose is generated, including: the image, the rasterized LBD features and the ORB features are directly input into the SLAM system through an extended interface, the SLAM system performs feature extraction, feature matching, pose estimation and map updating; the SLAM system automatically filters key frames according to the motion amplitude and the threshold of the change of the angle of view; the key frames are used as anchor points for map construction, and provide spatiotemporal constraints in the back-end optimization to ensure the stability of the pose estimation; based on the feature matching relationship between adjacent key frames, the system calculates the three-dimensional coordinates of the feature points by using the principle of triangulation; by minimizing the re-projection error, a sparse point cloud map composed of spatial feature points is generated; the system constructs a nonlinear optimization problem with the camera pose as the parameter by tracking the motion trajectory of the feature points in the key frame sequence; the maximum likelihood estimation of the pose parameter is realized by using the Gauss-Newton method or the Levenberg-Marquardt algorithm for iterative solution; as the video stream continues to be input, the SLAM system dynamically maintains the key frame database and the sparse point cloud map; the map structure and the camera trajectory are continuously optimized through local and global bundle adjustment, and new key frames are inserted according to the observation results.

[0053] In some embodiments, the plane coordinate information and the key frame map pose are fused, the global plane coordinates of the key frame are calculated, and the local visual coordinates are corrected by using the accurate global plane coordinates, including: obtaining the plane coordinate system information and the calculated key frame map pose; aligning the time stamp of the plane coordinate system information and the time stamp of the calculated key frame map pose, first, the time stamps are subtracted, and second, the plane coordinate frames and the key frame map pose frames that are less than the pre-set threshold are taken as time-synchronized matching pairs; in the matched time-synchronized matching pairs, the plane coordinate frames are assigned to the key frames, a transformation model of the sparse map from the visual coordinate system to the global plane coordinate system is established, and the global plane coordinates of the key frame are obtained; the calculated key frame map pose is corrected by the global plane coordinates of the key frame.

[0054] In some embodiments, the transformation model of the sparse map from the visual coordinate system to the global plane coordinate system is: the camera coordinate system takes the right lens optical center of the binocular camera as the origin and follows the right-hand coordinate system; the horizontal coordinate X of the plane coordinate system points to the east, and the vertical coordinate Y points to the north, and the unit is meter; the rotation relationship between the camera and the GNSS antenna is: wherein, RGNSS→camera represents the rotation matrix from the GNSS antenna coordinate system to the camera coordinate system, I is a 3x3 unit matrix; The conversion of the local visual coordinates to the plane coordinates follows the following formula: ; ; wherein, is a horizontal axis coordinate of the plane coordinate system, is a vertical axis coordinate of the plane coordinate system, is a heading angle, is a third digit of the translation vector, is a first digit of the translation vector, is a calibrated visual origin coordinate X, is a calibrated visual origin coordinate Y.

[0055] In some embodiments, the calculated key frame map pose is corrected by global plane coordinates of the key frame, comprising: the position attitude is agreed to be wherein, , is a rotation matrix and a translation vector from the origin of the camera coordinate system to the current local visual coordinate system of the th key frame, is the number of key frames; the origin coordinates of the local visual coordinate are obtained, the plane coordinates corresponding to the origin coordinates are obtained in the matching pair, and are marked as visual origin coordinates; the visual coordinates are converted into plane coordinates, and are marked as wherein, is the position coordinates of the th key frame in the local visual coordinate system, is a horizontal coordinate of the coordinates, is a vertical coordinate; the position coordinates in the global plane coordinate system are wherein, is the position coordinates of the th key frame in the global plane coordinate system; the matching pair of the global plane coordinates of the key frame after the timestamp matching and the key frame local visual coordinates is ; the visual coordinates and the global plane coordinates after the coordinate conversion are calculated respectively , , , wherein: is a change amount of the horizontal coordinate of the adjacent key frame in the local coordinate system; is a change amount of the vertical coordinate of the adjacent key frame in the local coordinate system; is a change amount of the horizontal coordinate of the adjacent key frame in the global plane coordinate system; is a change amount of the vertical coordinate of the adjacent key frame in the global plane coordinate system; the plane coordinates of the real-time differential measurement are taken as true values, , wherein, , is a change amount of the horizontal and vertical coordinates of the corrected adjacent key frame in the plane coordinate system, and the corrected , Computing the local visual positioning coordinates.

[0056] The present embodiment is described and illustrated below by specific examples.

[0057] Figure 3 is a flow chart of a multi-modal perception fusion agricultural robot global positioning method provided by the present embodiment.

[0058] As shown in Figure 3 , the multi-modal perception fusion agricultural robot global positioning method provided by the present application comprises the following steps: Step 1: Obtain images and satellite positioning information, obtain field images taken by binocular cameras and centimeter-level positioning information measured by real-time differential measurement; Step 2: Convert the WGS coordinate information of the real-time differential into plane coordinate system information; Step 3: Image extraction point and line feature and rasterization processing: construct a multi-layer pyramid model for the left and right views of step 1 respectively, extract point and line features for each layer of pyramid layered image, and rasterize the point and line features of each layer of pyramid layered image; Step 4: Initialization of visual odometry; Step 5: Run the visual odometry to generate a key frame map: input the images of step 1, the rasterized point and line features of step 3 into the improved SLAM system, output a key frame queue, and generate a key frame map pose; Step 6: Fuse the plane coordinate information and the key frame map pose of step 5 to calculate the global plane coordinates of the key frame, and correct the local visual coordinates using the accurate global plane coordinates; Step 7: Obtain a key frame map with global plane coordinates and local visual coordinates; Step 8: Import the key frame map into the improved SLAM system, and the agricultural robot equipped with the improved SLAM system can provide centimeter-level visual plane coordinate system positioning coordinates when encountering real-time differential floating, until the real-time differential is restored to fixed.

[0059] Preferably, step 1 is specifically: the flying agricultural robot equipped with binocular cameras takes forward slightly downward images in a field 3m high, records the real-time differential information corresponding to each picture at the same time, after completion, the flying agricultural robot lands, and the collected data is imported into the processing center through the memory card, realizing the acquisition of image frames and real-time differential information.

[0060] The application discloses a multi-modal perception fusion agricultural robot global positioning method, which uses a flying agricultural robot load binocular camera to shoot a forward slightly downward image sequence and receives latitude and longitude coordinates through a real-time difference module, realizes high-precision real-time tracking and map construction of a field scene through improvement of a feature extraction and matching algorithm of a binocular SLAM system, generates a sparse point cloud map and a local position and posture of a key frame, converts WGS coordinates obtained by the real-time difference into plane coordinates, matches the position and posture of the key frame with the plane coordinates through a timestamp difference value threshold synchronization alignment method, and simultaneously fuses the position and posture of the key frame into global plane coordinates and corrects the local position and posture of the key frame through the plane coordinates. The key frame map with the global plane coordinates and the local visual coordinates is introduced into the improved SLAM system, when the field agricultural robot carrying the improved SLAM system encounters tree shade shielding or passes through a viaduct to cause real-time difference floating, the improved SLAM system can provide plane coordinates of centimeter-level visual calculation until the real-time difference is restored to be fixed, the application improves positioning accuracy, thereby improving robustness, safety and operation production efficiency of the field agricultural robot, and is suitable for production and operation requirements of four links of ploughing, planting, management and harvesting of unmanned intelligent farms.

[0061] The step 2 of converting the real-time difference information into plane coordinate system information is specifically as follows: Step 2-1: obtaining real-time difference information: obtaining real-time difference information to be converted from a real-time difference positioning module of the flying agricultural robot, wherein the real-time difference information includes centimeter-level longitude, latitude, height, relative height and Euler angle data; Step 2-2: determining a plane coordinate system projection area: determining the plane coordinate system projection area according to given longitude information; Step 2-3: calculating a central meridian: calculating the central meridian longitude according to the plane coordinate system projection area; for area 1, the central meridian longitude is 177 degrees, and for area 2-60, the calculation formula of the central meridian longitude is: , wherein, is the central meridian longitude; is a band number; Step 2-4: converting the WGS84 coordinates of the real-time difference information into plane coordinates by using a conversion formula of the plane coordinate system projection; The conversion formula of the plane coordinate system projection is as follows: , wherein, is a plane coordinate system longitudinal axis coordinate, is a plane coordinate system transverse axis coordinate, is an arc degree latitude, is a meridian arc length, Let be the radius of curvature of the circle. The difference in projection factor, The square of the latitude tangent. This is a latitude-dependent curvature correction term. The first eccentricity of the Earth's ellipsoid. This is the projection scaling factor; Step 2.5: Output the transformed planar coordinates, which include Mercator coordinates on the horizontal axis and Mercator coordinates on the vertical axis, as well as the projection area of ​​the planar coordinate system and the longitude of the central meridian.

[0062] In fields with simple textures, introducing the LSD line detection algorithm and LBD descriptors can significantly improve the system's ability to perceive linear features such as crop rows and field ridges, and enhance the algorithm's robustness under dynamic lighting and vegetation growth changes. Figure 4 This is a screenshot showing the results of point and line feature recognition and extraction. The detailed steps for ORB feature point and LSD line detection and matching are as follows: Step 3-1: Construct multi-level pyramid models for the left and right views from Step 1 respectively; Step 3-2: Perform LSD line detection and FAST corner detection on the layered images of each pyramid in the multi-layer pyramid model; Step 3-3: For each straight line, LBD descriptors are generated by comparing the pixel intensity of the sub-regions around the straight line, with the straight line as the center. At the same time, for each FAST corner point, 128-dimensional BRIEF descriptors are extracted as ORB features. Steps 3-4: Rasterize the line feature descriptors (LBD and ORB) and assign each LBD descriptor and ORB feature descriptor to a set raster number. Steps 3-5: Feature matching. Like ORB features, LBD features use Hamming distance to measure the similarity between two feature descriptors. Steps 3-6: Based on the pixel distance between the two frames, sort the LBD and ORB features within the nearest 20 pixels by similarity, and select the feature points with the highest similarity as the matching point pairs; Steps 3-7: Perform RANSAC random fitting on the matching point pairs and historical key point clouds of the two frames to calculate the relative motion pose between the two frames. Steps 3-8: In the initialization phase, triangulation and line triangulation are used to calculate the parameters of 3D points and 3D lines to construct a point-line hybrid map; the pose is initialized, the key points and lines of the current frame are projected to update the sparse point cloud, and different weights are assigned to points and lines for joint use in backend optimization.

[0063] During the operation of the SLAM system, the positions of the feature points and lines between adjacent image frames usually change within a small range due to the continuity of motion. Based on this characteristic, a grid-based local search matching strategy can be used: taking the feature points and lines detected in the previous frame as the center, a specific spatial range is defined around the center, and the region is divided into a plurality of grid units, and only the matching feature points and lines in the current frame are searched in these grid units. This optimization significantly improves the system performance in two aspects: on the one hand, by reducing the search range, the risk of false matching is greatly reduced, and the deviation of pose estimation caused by false correspondence is reduced; on the other hand, in the core modules such as system initialization, real-time tracking and sparse point cloud map construction, the adaptability of the algorithm to complex scenes (such as dynamic lighting and similar texture areas) is effectively improved, and the stability and reliability of the system as a whole are enhanced, providing a more solid technical guarantee for high-precision positioning and map construction.

[0064] The SLAM system supports multiple sensor inputs, and in this application, a binocular vision SLAM system mode is used for mapping. As shown in FIG. 1, a sparse point cloud map in an urban road scene is shown, and the map has high accuracy due to the characteristics of the urban road scene, such as many feature points and lines and rich texture. As shown in FIG. 2, a sparse point cloud map of a local field is shown, and the accuracy is significantly improved due to the enhanced SLAM system with straight line features. Figure 5 Figure 6 As shown in FIG. 3, a sparse point cloud map of a local field is shown, and the accuracy is significantly improved due to the enhanced SLAM system with straight line features.

[0065] Preferably, step 4 is specifically: visual odometry initialization of the SLAM system: before processing the image, system initialization operation needs to be completed, including loading camera parameters, configuring algorithm parameters and constructing data structures for tracking and mapping. The moving distance between frame sequences is calculated by visual odometry, and when the distance exceeds a pre-set threshold, the current frame is marked as a key frame and stored. In the initialization stage, the point cloud is updated based on the multiple key frame poses calculated by the SLAM system, thereby constructing the three-dimensional point cloud information required for tracking initialization.

[0066] Preferably, step 5 is specifically: Step 5-1: image processing: the image, grid-based LBD features and ORB features of step 1 are input into the SLAM system, and the ORBSLAM system 3 will perform feature extraction, feature matching, pose estimation and map updating; Step 5-2: key frame extraction: the SLAM system automatically selects key frames according to the motion amplitude and angle of view change threshold. These key frames serve as anchor points for map construction, providing temporal and spatial constraints in the back-end optimization to ensure the stability of the pose estimation; Step 5-3: based on the feature matching relationship between adjacent key frames, the system calculates the three-dimensional coordinates of the feature points using the principle of triangulation. By minimizing the re-projection error, a sparse point cloud map composed of spatial feature points is generated;​ Step 5-4: Position and pose estimation: The system constructs a nonlinear optimization problem parameterized by camera poses by tracking feature point motion trajectories in the sequence of keyframes. The Gauss-Newton method or Levenberg-Marquardt algorithm is used for iterative solution to achieve maximum likelihood estimation of pose parameters; Step 5-5: Real-time update: As the video stream continues to input, the system dynamically maintains the keyframe database and sparse point cloud map. Through local and global bundle adjustment (BA), the map structure and camera trajectory are continuously optimized, and new keyframes are inserted according to the latest observation results to ensure the consistency and real-time performance of the map.

[0067] Fuse the plane coordinate information of real-time difference conversion and the keyframe map pose, and correct the local visual coordinates using accurate global coordinates while calculating the global plane coordinates of the keyframe.

[0068] Step 6-1: Collect the plane coordinate system information obtained in step 2 and the keyframe map pose calculated by ORB SLAM system 3; Step 6-2: Align the timestamps of the plane coordinate system information and the keyframes map pose calculated by ORB SLAM system 3. First, subtract the timestamps of the two, and then take the plane coordinate frames and keyframe map pose frames that are less than the pre-set threshold as time-synchronized matching pairs; Step 6-3: In the matched time-synchronized matching pairs, assign the plane coordinate frames to the keyframes, establish a transformation model from the visual coordinate system to the global plane coordinate system for the sparse map, and obtain the global plane coordinates of the keyframes; Step 6-4: Correct the keyframe map pose calculated by ORB SLAM system 3 using the global plane coordinates of the keyframes.

[0069] Preferably, the transformation model from the visual coordinate system to the global plane coordinate system for the sparse map in step 6-3 is established as follows: The camera coordinate system takes the right lens optical center of the binocular camera as the origin and follows the right-hand coordinate system, where the X-axis points to the right along the horizontal direction of the camera, the Y-axis points downward along the vertical direction of the camera, and the Z-axis points forward along the optical axis direction, with units in meters (m); the horizontal coordinate X of the plane coordinate system points to the east, and the vertical coordinate Y points to the north, with units in meters (m); the camera and the GNSS antenna are rigidly connected in mechanical structure and the installation process ensures no relative rotation deviation, so the rotation relationship between them can be simplified as a unit matrix, i.e. wherein, RGNSS→camera represents the rotation matrix from the GNSS antenna coordinate system to the camera coordinate system, is a 3x3 unit matrix; the conversion of local visual coordinates to plane coordinates follows the following formula: ; ; wherein: is the horizontal axis coordinate of the plane coordinate system; is the vertical axis coordinate of the plane coordinate system; is the heading angle; is the 3rd digit of the translation vector; is the 1st digit of the translation vector; is the calibrated visual origin coordinate X; is the calibrated visual origin coordinate Y.

[0070] Preferably, after obtaining the key frame position and the corresponding plane coordinates by the fusion method in step 6-4, the plane coordinates are used to correct the local visual coordinates, which is specifically as follows: the position and posture is agreed to be wherein , is the rotation matrix and translation vector from the origin of the camera coordinate system to the current local visual coordinate system of the th key frame, is the number of key frames; the origin coordinate of the local visual coordinate is obtained, the plane coordinate corresponding to the origin coordinate in the matching pair is obtained, which is marked as the visual origin coordinate, the visual coordinate is converted into the plane coordinate, which is marked as wherein is the position coordinate of the th key frame in the local visual coordinate system, is the horizontal coordinate of the coordinate, is the vertical coordinate; the position coordinate in the global plane coordinate system is wherein is the position coordinate of the th key frame in the global plane coordinate system; the matching pair of the global plane coordinate of the key frame after the timestamp matching and the key frame local visual coordinate is ; the visual coordinate and the global plane coordinate after the coordinate conversion are calculated respectively , , , wherein: is the change amount of the horizontal coordinate of the adjacent key frame in the local coordinate system; is the change amount of the vertical coordinate of the adjacent key frame in the local coordinate system; is the change amount of the horizontal coordinate of the adjacent key frame in the global plane coordinate system; is a change amount of the ordinate of the adjacent key frame in the global plane coordinate system; the plane coordinate measured by the real-time differential measurement is taken as a true value, , wherein , The local visual positioning coordinate is calculated.

[0071] The SLAM system supports multiple sensor inputs, and in the present application, the binocular visual inertial odometry is used to estimate the pose of the agricultural robot in real time and perform navigation, and the specific implementation is as follows: The key frame map with global plane coordinates and local visual coordinates is imported into the improved SLAM system, the binocular visual odometry of the SLAM system is used to run in the existing key frame map scene, and the system determines the current position of the camera by using the pre-constructed key frame map, which not only greatly improves the positioning accuracy, but also reduces the computational overhead; the camera is installed on the agricultural robot, and when the improved SLAM system runs in real time, the visual positioning result is transmitted to the robot control system through the serial port, and the local visual coordinates are converted into global plane coordinates by using the coordinate transformation model in the above embodiment; when the agricultural robot drives under a tree canopy or an overpass, etc., the differential float is used to navigate until the real-time differential is restored. As shown in Figure 7 , the position deviation in the RTK float solution state is between 0.8 and 4.3 m, and the position deviation of the multi-modal fusion positioning is less than 10 cm within 10 s, and the positioning accuracy is obviously improved, which can be used for navigation operation of the agricultural robot in the RTK float solution.

[0072] The present application improves the production efficiency, safety and robustness of the agricultural robot in the unmanned intelligent farm through technical innovation. Specifically, a straight line feature detection and matching module is embedded in the SLAM system, and a sparse point cloud map with global plane coordinates is collected and constructed by the flying agricultural robot as prior information, and the binocular visual odometry pose output accuracy is significantly improved. This optimization enables the agricultural robot to maintain navigation continuity and ensure centimeter-level trajectory accuracy even in complex situations such as real-time differential signal floating and temporary failure during key operation links such as seedling planting and harvesting, providing reliable positioning information support for intelligent operation of unmanned intelligent farms.

[0073] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0074] In the embodiment, a multi-modal perception fusion agricultural robot global positioning device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. The terms "module", "unit", "sub-unit" and the like used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and contemplated.

[0075] The device comprises: An acquisition module is configured to acquire an image and satellite positioning information. The satellite positioning information comprises real-time differential positioning information, and the real-time differential positioning information comprises real-time differential World Geodetic System coordinate information. A conversion module is configured to convert the real-time differential World Geodetic System coordinate information into plane coordinate information. A feature extraction module is configured to extract point features and line features from the image and perform rasterization processing to obtain rasterized point features and line features. A generation module is configured to input the image and the rasterized point features and line features into an improved Simultaneous Localization and Mapping (SLAM) system, output a key frame queue, and generate a key frame map pose. A fusion module is configured to fuse the plane coordinate information and the key frame map pose, calculate a global plane coordinate of the key frame, and correct a local visual coordinate using the accurate global plane coordinate. An acquisition module is configured to acquire a key frame map comprising a global plane coordinate and a local visual coordinate. A positioning module is configured to import the key frame map into the improved SLAM system and perform global positioning.

[0076] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented by software or hardware. For modules implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can be located in different processors in any combination.

[0077] In the embodiment, an electronic device is also provided, which comprises a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0078] Optionally, the electronic device can further comprise a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0079] Optionally, in the embodiment, the processor can be configured to execute the following steps by a computer program: S1, acquiring an image and satellite positioning information; the satellite positioning information comprises real-time difference measurement positioning information, and the real-time difference measurement positioning information comprises real-time difference world geodetic system coordinate information; S2, converting the real-time difference world geodetic system coordinate information into plane coordinate information; S3, extracting point features and line features from the image and performing rasterization processing to obtain rasterized point features and line features; S4, inputting the image and the rasterized point features and line features into an improved simultaneous localization and mapping (SLAM) system to output a key frame queue and generate a key frame map pose; S5, fusing the plane coordinate information and the key frame map pose to calculate a global plane coordinate of the key frame, and correcting a local visual coordinate by using the accurate global plane coordinate; S6, acquiring a key frame map comprising the global plane coordinate and the local visual coordinate; S7, inputting the key frame map into the improved SLAM system and performing global positioning.

[0080] It should be noted that the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, and will not be described herein again.

[0081] In addition, in combination with the multi-modal perception fusion agricultural robot global positioning method provided in the above embodiments, a storage medium can also be provided to implement the method in the embodiment. The storage medium has a computer program stored thereon; and the computer program is executed by a processor to implement any one of the multi-modal perception fusion agricultural robot global positioning methods in the above embodiments.

[0082] It should be understood that the specific embodiments described herein are only used to explain the application, but not to limit it. According to the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0083] Obviously, the drawings are only some examples or embodiments of the present application, and those skilled in the art can also apply the present application to other similar situations without creative labor. In addition, it can be understood that although the work done in the development process may be complex and long, some design, manufacture or production changes made by those skilled in the art according to the technical content disclosed in the present application are only routine technical means and should not be regarded as insufficient disclosure of the present application.

[0084] The word "implementation" in this application refers to the specific features, structures, or characteristics described in connection with an implementation can be included in at least one implementation of the present application. The phrase appears in various places throughout the specification is not necessarily meant to refer to the same implementation, nor is it meant to imply that the implementation is preferred over other implementations. It will be apparent or implicit from the context that the implementations described in this application can be combined with other implementations without

[0085] The above-described implementations are merely some implementations of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of patent protection. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A global localization method for agricultural robots using multimodal perception fusion, characterized in that, The method includes: Acquire images and satellite positioning information; the satellite positioning information includes positioning information obtained through real-time differential measurement, which includes real-time differential Universal Geodetic System coordinate information; The real-time differential World Geodetic System coordinate information is converted into plane coordinate information; Point features and line features are extracted from the image, and then rasterized to obtain rasterized point features and line features. The image and the rasterized point and line features are fed into the improved real-time localization and mapping (SLAM) system, which outputs a keyframe queue and generates a keyframe map pose. By fusing planar coordinate information and keyframe map pose, the global planar coordinates of the keyframes are calculated, and the accurate global planar coordinates are used to correct the local visual coordinates. Obtain a keyframe map including global planar coordinates and local visual coordinates; The keyframe map is imported into the improved SLAM system and global localization is performed.

2. The multimodal perception fusion global localization method for agricultural robots according to claim 1, characterized in that, The image is a field image, and the acquisition of the image and satellite positioning information includes: Acquire field images captured by a binocular camera and centimeter-level positioning information obtained through real-time differential measurement.

3. The multimodal perception fusion-based global localization method for agricultural robots according to claim 1, characterized in that, The step of converting the real-time differential geodetic system coordinate information into plane coordinate information includes: Real-time differential information is obtained from the real-time differential positioning module of the agricultural robot. The real-time differential information includes centimeter-level longitude, latitude, altitude, relative height and Euler angle data. Determine the projection area of ​​the plane coordinate system. Based on the given longitude information, determine the projection area of ​​the plane coordinate system in which the object is located. Calculate the central meridian by calculating its longitude based on the projection area of ​​the plane coordinate system. The transformation formula for projection into a plane coordinate system is used to convert the real-time differential information of the World Geodetic System coordinates into plane coordinates. Output the transformed planar coordinates, which include Mercator coordinates on the horizontal axis and Mercator coordinates on the vertical axis, as well as the projection area of ​​the planar coordinate system and the longitude of the central meridian.

4. The multimodal perception fusion global localization method for agricultural robots according to claim 3, characterized in that, The transformation formula for projection onto a plane coordinate system is: in, The vertical axis coordinate of the plane coordinate system. The x-axis coordinates of the plane coordinate system. Latitude is measured in radians. The length of the meridian arc. Let be the radius of curvature of the circle. The difference in projection factor, The square of the latitude tangent. This is a latitude-dependent curvature correction term. The first eccentricity of the Earth's ellipsoid. This is the projection scaling factor.

5. The multimodal perception fusion global localization method for agricultural robots according to claim 1, characterized in that, The step of extracting point features and line features from the image and performing rasterization to obtain rasterized point features and line features includes: Construct multi-layered pyramid models for the left and right views of the image respectively; Line detection and corner detection are performed on the layered images of each pyramid in the multi-layered pyramid model; For each straight line, a line band descriptor is generated by comparing the pixel intensity of the sub-regions around the line, with the line as the center. At the same time, for each corner point, a multi-dimensional binary robust independent basic feature descriptor is extracted as a directional feature. Line feature descriptors, strip descriptors, and directional feature descriptors are rasterized, and each strip descriptor and directional feature descriptor is assigned to a set raster number. Hamming distance is used to measure the similarity between two feature descriptors; Based on the pixel distance between two consecutive frames, the similarity of line descriptor features and orientation features within multiple nearby pixels is sorted, and the feature points with the highest similarity are selected as matching point pairs. Randomly fit the matching point pairs and historical key point clouds of the two consecutive frames to calculate the relative motion pose between the two frames. In the initialization phase, triangulation and line triangulation are used to calculate the parameters of 3D points and 3D lines to construct a point-line hybrid map; the pose is initialized, and the key points and lines of the current frame are projected to update the sparse point cloud, assigning different weights to points and lines, which are used together in the backend optimization.

6. The multimodal perception fusion global localization method for agricultural robots according to claim 1, characterized in that, The method further includes: visual odometry initialization of the SLAM system; The visual odometry initialization of the SLAM system includes: the visual odometry calculates the distance traveled between frame sequences, and when the distance exceeds a preset threshold, the current frame is marked as a keyframe and stored.

7. The multimodal perception fusion global localization method for agricultural robots according to claim 1, characterized in that, The process of feeding the image and the rasterized point and line features into the improved SLAM system, outputting a keyframe queue, and generating keyframe map poses includes: Images, rasterized LBD features, and ORB features are directly input into the SLAM system through an extended interface. The SLAM system performs feature extraction, feature matching, pose estimation, and map updating. The SLAM system automatically selects key frames based on motion amplitude and viewpoint change thresholds; these key frames serve as anchor points for map construction, providing spatiotemporal constraints in backend optimization to ensure the stability of pose estimation. Based on the feature matching relationship between adjacent keyframes, the system uses the triangulation principle to calculate the three-dimensional coordinates of feature points; by minimizing the reprojection error, a sparse point cloud map composed of spatial feature points is generated. The system constructs a nonlinear optimization problem with camera pose as a parameter by tracking the motion trajectory of feature points in the keyframe sequence; it then uses the Gauss-Newton method or the Levenberg-Marquardt algorithm to iteratively solve the problem and achieve maximum likelihood estimation of the pose parameters. As the video stream continues to be input, the SLAM system dynamically maintains the keyframe database and sparse point cloud map; it continuously optimizes the map structure and camera trajectory through local and global bundle adjustment, while inserting new keyframes based on the observation results.

8. The multimodal perception fusion global localization method for agricultural robots according to claim 1, characterized in that, The process of fusing planar coordinate information and keyframe map pose to calculate the global planar coordinates of the keyframes, and using accurate global planar coordinates to correct local visual coordinates, includes: Obtain planar coordinate system information and calculate keyframe map pose; Align the timestamps of the plane coordinate system information with the timestamps of the calculated keyframe map poses. First, calculate the difference between the two timestamps. Then, use plane coordinate frames and keyframe map pose frames that are less than a pre-set threshold as time synchronization matching pairs. In the matched time-synchronized pair, the planar coordinate frame is assigned to the key frame, and a transformation model of the sparse map from the visual coordinate system to the global planar coordinate system is established to obtain the global planar coordinates of the key frame. The keyframe map pose is calculated using global planar coordinate correction of the keyframes.

9. The global localization method for agricultural robots based on multimodal perception fusion according to claim 8, characterized in that, The transformation model for establishing a sparse map from the visual coordinate system to the global planar coordinate system is as follows: The camera coordinate system has its origin at the optical center of the right lens of the binocular camera and follows a right-handed coordinate system; the horizontal coordinate X of the planar coordinate system points east, and the vertical coordinate Y points north, in meters; the rotation relationship between the camera and the GNSS antenna is as follows: ,in, This represents the rotation matrix from the GNSS antenna coordinate system to the camera coordinate system. It is a 3x3 identity matrix; The conversion of local visual coordinates to planar coordinates follows the formula: ； ； in, The x-axis coordinates of the plane coordinate system. The vertical axis coordinate of the plane coordinate system. For heading angle, It is the third number of the translation vector. The first digit of the translation vector. Let X be the coordinates of the calibrated visual origin. Y is the coordinate of the calibrated visual origin.

10. A global localization method for agricultural robots based on multimodal perception fusion according to claim 7, characterized in that, The keyframe map pose calculated through global planar coordinate correction of keyframes includes: The agreed position and orientation are ,in , For the first Each keyframe represents the rotation matrix and translation vector from the origin of the camera coordinate system to the current local visual coordinate system. The keyframe number is used; the origin coordinates of the local visual coordinates are obtained, and the corresponding planar coordinates are obtained from the matching pairs. These are marked as the visual origin coordinates, and the visual coordinates are converted into planar coordinates, denoted as... ,in It is the first The position coordinates of each keyframe in the local visual coordinate system This is the x-coordinate of the coordinate system. The vertical coordinate is y; the position coordinates in the global planar coordinate system are... ,in It is the first The position coordinates of each keyframe in the global plane coordinate system; the keyframe global plane coordinates and keyframe local visual coordinates matched by timestamp are... ; Calculate the transformed visual coordinates and global planar coordinates in adjacent keyframes respectively. , , , ,in: This represents the change in the x-coordinate of adjacent keyframes in the local coordinate system. This represents the change in the ordinate of adjacent keyframes in the local coordinate system. This represents the change in the x-coordinate of adjacent keyframes in the global plane coordinate system. This represents the change in the ordinate of adjacent keyframes in the global plane coordinate system; the plane coordinates measured in real-time differential measurement are used as the true values. , ,in , To measure the changes in the x and y coordinates of adjacent keyframes in the plane coordinate system after correction, the corrected values ​​are used. , Calculate the local visual positioning coordinates.

Images