A method, system, device and medium for tracking a target in a field environment with multiple obstacles

Abstract

The application discloses a kind of field environment multi-obstacle target tracking method, system, equipment and medium, it is related to multi-target tracking technical field, comprising: using Kalman filter to obtain predicted frame center pixel coordinate and BEV target prediction center coordinate;Using IMU data information is corrected to predicted frame center pixel coordinate;Obstacle target object identification is carried out to RGB image;Determine the point cloud center coordinate of bird's-eye view angle, and using IMU data information is corrected to point cloud center coordinate;Using OCSORT model carries out minimum cost matching, obtains initial matching information;If detection frame and predicted frame have been matched, then initial matching information is used as current frame successful matching information, otherwise based on depth point cloud information using LAPJV algorithm carries out minimum cost matching, obtains current frame successful matching information;According to current frame successful matching information, update effective trajectory.The application can improve the tracking performance of agricultural robot in field environment.

Description

Technical Field

[0001] This invention relates to the field of multi-target tracking technology, and in particular to a method, system, device, and medium for tracking multiple obstacles in a field environment. Background Technology

[0002] Multiple Object Tracking (MOT) refers to the detection and ID-based trajectory tracking of multiple targets such as pedestrians, cars, and animals in a video without prior knowledge of the number of targets. Different targets have different IDs to enable subsequent trajectory prediction and accurate location. MOT plays a crucial role in environmental perception for agricultural robots, and is mainly divided into appearance-based methods and motion-based methods. In practical applications, it has been found that appearance-based methods become less efficient when dealing with objects with similar appearances or occluded scenes, often performing worse than motion association algorithms. Furthermore, extracting appearance features is often time-consuming, resulting in poor real-time performance. However, in rugged and complex outdoor environments, robots often cannot move smoothly, and cameras may exhibit violent and irregular movements due to the rugged terrain. These drastic and irregular changes in the camera's field of view lead to frequent switching of the tracked object IDs, causing misjudgments by the tracking algorithm. This poses a significant challenge to motion-based MOT methods. Existing motion-based methods, such as Observation-Centric SORT (OCSORT) and Bytetrack, ignore the camera's own motion, resulting in inaccurate predictions in their Kalman filter frameworks and frequent switching of the tracked object ID, leading to incorrect tracking and matching results. Summary of the Invention

[0003] The purpose of this invention is to provide a method, system, device, and medium for tracking multiple obstacles in a field environment, so as to improve the tracking performance of agricultural robots in the field environment.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A method for tracking multiple obstacles in a wilderness environment includes:

[0006] Acquire successful matching information from previous frames and detection data from the current frame; the successful matching information from previous frames includes the obstacle center pixel coordinates and the BEV target center point coordinates from the previous frame; the detection data from the current frame includes the RGB image, depth point cloud information, and IMU data information from the current frame.

[0007] Kalman filtering is performed on the successful matching information of the previous frame to obtain the center pixel coordinates of the prediction box and the prediction center coordinates of the BEV target;

[0008] The center pixel coordinates of the prediction box are corrected using the IMU data information to obtain the center coordinates of the image prediction box;

[0009] Obstacle target object recognition is performed on the RGB image to obtain the center coordinates of the image detection box;

[0010] The point cloud center coordinates from the bird's-eye view are determined based on the depth point cloud information, and the point cloud center coordinates are corrected using the IMU data information to obtain the BEV target detection center coordinates.

[0011] The coordinates of the predicted center of the BEV target, the coordinates of the center of the predicted image box, the coordinates of the center of the image detection box, and the coordinates of the detected BEV target are input into the OCSORT model for minimum cost matching to obtain initial matching information, and to determine whether the detection box and the predicted box match.

[0012] If the detection box and the prediction box do not match, a cost matrix is ​​constructed based on the predicted center coordinates of the BEV target and the detection center coordinates of the BEV target. Based on the cost matrix, the LAPJV algorithm is used to perform minimum cost matching to obtain the successful matching information of the current frame. The successful matching information of the current frame includes the obstacle center pixel coordinates and the BEV target center point coordinates of the current frame.

[0013] If the detection box and the prediction box have matched, the initial matching information is used as the successful matching information for the current frame.

[0014] The steps are: update the valid trajectory based on the successful matching information of the current frame, and return to obtain the successful matching information of the previous frame and the detection data information of the current frame.

[0015] Optionally, the IMU data information is used to correct the center pixel coordinates of the prediction box to obtain the center coordinates of the image prediction box, specifically including:

[0016] The camera field of view of the previous frame, the camera field of view of the current frame, and the angle between the current frame and the previous frame are determined based on the IMU data information.

[0017] Based on the triangular relationship, the correction ratio is calculated according to the camera field of view of the previous frame, the camera field of view of the current frame, and the angle between the current frame and the previous frame.

[0018] The center pixel coordinates of the prediction box are corrected according to the correction ratio and the camera resolution to obtain the center coordinates of the image prediction box.

[0019] Optionally, obstacle target object recognition is performed on the RGB image to obtain the center coordinates of the image detection box, specifically including:

[0020] The YOLOv8 network model is used to identify obstacle targets in the RGB image, and the center coordinates of the image detection box are obtained.

[0021] Optionally, the IMU data information is used to correct the center coordinates of the point cloud to obtain the BEV target detection center coordinates, specifically including:

[0022] Determine the camera angle based on the IMU data information;

[0023] The point cloud center coordinates are corrected using the camera rotation angle to obtain the BEV target detection center coordinates.

[0024] Optionally, a cost matrix is ​​constructed based on the BEV target prediction center coordinates and the BEV target detection center coordinates, specifically including:

[0025] Calculate the horizontal and vertical coordinates of the BEV target prediction box center relative to the camera origin based on the BEV target prediction center coordinates, and use them as prediction box coordinate information;

[0026] Calculate the horizontal and vertical coordinates of the BEV target detection box center relative to the camera origin based on the BEV target detection center coordinates, and use them as the detection box coordinate information;

[0027] The Euclidean distance between the predicted bounding box and the detected bounding box is calculated based on the predicted bounding box coordinate information and the detected bounding box coordinate information.

[0028] Construct a cost matrix based on the Euclidean distances between all predicted bounding boxes and all detected bounding boxes.

[0029] Optionally, based on the cost matrix, the LAPJV algorithm is used for minimum cost matching to obtain successful matching information for the current frame, specifically including:

[0030] Based on the cost matrix, the LAPJV algorithm is used to perform minimum cost matching to obtain secondary matching information;

[0031] Compare the Euclidean distances between the predicted bounding boxes and the detected bounding boxes in the secondary matching information;

[0032] If the Euclidean distance is greater than a set threshold, the matching pair of the predicted box and the detected box will be discarded.

[0033] If the Euclidean distance is less than or equal to a set threshold, the matching pair of the predicted box and the detected box will be retained;

[0034] The processed secondary matching information is used as the successful matching information for the current frame.

[0035] Optionally, the step of updating the valid trajectory based on the current frame's successful matching information and returning to obtain the previous frame's successful matching information and the current frame's detection data information specifically includes:

[0036] Update the valid trajectory based on the successful matching information of the current frame, and determine whether the current frame is the last frame;

[0037] If the current frame is not the last frame, return to the steps of obtaining the successful matching information of the previous frame and the detection data information of the current frame;

[0038] If the current frame is the last frame, then the process ends.

[0039] A multi-obstacle target tracking system for outdoor environments includes:

[0040] The data acquisition module is used to acquire successful matching information of previous frames and detection data information of the current frame; the successful matching information of previous frames includes the obstacle center pixel coordinates and BEV target center point coordinates of the previous frame; the detection data information of the current frame includes the RGB image, depth point cloud information and IMU data information of the current frame.

[0041] The BEV target prediction center coordinate determination module is used to perform Kalman filtering on the previous frame successful matching information to obtain the prediction box center pixel coordinates and the BEV target prediction center coordinates.

[0042] The image prediction box center coordinate determination module is used to correct the center pixel coordinates of the prediction box using the IMU data information to obtain the center coordinates of the image prediction box.

[0043] The image detection box center coordinate determination module is used to identify obstacle targets in the RGB image and obtain the center coordinates of the image detection box.

[0044] The BEV target detection center coordinate determination module is used to determine the point cloud center coordinates from a bird's-eye view based on the depth point cloud information, and to correct the point cloud center coordinates using the IMU data information to obtain the BEV target detection center coordinates.

[0045] The OCSORT matching module is used to input the BEV target prediction center coordinates, the image prediction box center coordinates, the image detection box center coordinates, and the BEV target detection center coordinates into the OCSORT model to perform minimum cost matching, obtain initial matching information, and determine whether the detection box and the prediction box match.

[0046] The first matching information determination module is used to construct a cost matrix based on the predicted center coordinates of the BEV target and the detected center coordinates of the BEV target if the detection box and the predicted box do not match, and to perform minimum cost matching using the LAPJV algorithm based on the cost matrix to obtain the current frame successful matching information; the current frame successful matching information includes the obstacle center pixel coordinates and the BEV target center point coordinates of the current frame.

[0047] The second matching information determination module is used to take the initial matching information as the successful matching information of the current frame if the detection box and the prediction box have matched.

[0048] The effective trajectory update module is used to update the effective trajectory based on the successful matching information of the current frame and return it to the data acquisition module.

[0049] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to enable the electronic device to perform the above-described method for tracking multiple obstacles in a field environment.

[0050] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for tracking multiple obstacles in a field environment.

[0051] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0052] The multi-obstacle target tracking method in the field provided by this invention combines multi-sensor information including RGB images, depth point cloud information, and IMU data. Utilizing the ICMC (IMU-based Camera Motion Compensation) module, the center pixel coordinates of the predicted bounding box are corrected using IMU data to obtain the center coordinates of the image predicted bounding box. Similarly, the center coordinates of the depth point cloud information are corrected to obtain the BEV target detection center coordinates. This improves detection accuracy and significantly reduces false and missed matches. Based on the depth point cloud information, the DA (Depth-Aware) module performs secondary matching on unmatched detection and predicted bounding boxes in OCSORT, enabling cross-frame matching even in scenarios where frequent left and right camera rotation causes objects to briefly move out of the field of view, are temporarily occluded, or experience interruptions. Therefore, this invention improves the perception accuracy and anti-interference capability of agricultural robots in the field, thereby enhancing their tracking performance. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart of the multi-obstacle target tracking method in the field environment provided by the present invention;

[0055] Figure 2 This is a framework diagram of the multi-obstacle target tracking method in the field environment provided by the present invention;

[0056] Figure 3 A schematic diagram illustrating the correction of the center pixel coordinates of the prediction box using the ICMC module provided by this invention;

[0057] Figure 4 This is a schematic diagram of the IMU-corrected point cloud center coordinates provided by the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Existing motion detection-based methods such as OCSORT and Bytetrack often struggle in rugged and complex outdoor environments, where robots cannot navigate smoothly. Cameras exhibit drastic and irregular movements due to the rugged terrain, leading to frequent switching of the tracked object ID and resulting in mismatches and missed matches. To address these issues, this invention aims to provide a method, system, device, and medium for tracking multiple obstacles in outdoor environments, thereby improving the tracking performance of agricultural robots in such conditions.

[0060] Specifically, using the ICMC module proposed in this invention, the camera motion information recorded by the IMU can be used to correct and compensate for the predicted bounding box position, thereby improving detection accuracy and significantly reducing false and missed matches. When the camera rotates frequently left and right, causing the detected object to appear or disappear at the edge of the field of view, the predicted bounding box may extend beyond the actual field of view, which poses a problem for successfully matching the same target across frames. The DA module proposed in this invention uses the position of the detected object relative to the camera as the matching criterion and integrates this depth information into the tracking process to perform secondary matching on the detection boxes and predicted boxes that were not successfully matched in OCSORT, thus achieving cross-frame matching in this scenario.

[0061] In summary, this invention proposes two novel ICMC and DA modules, which, combined with IMU and point cloud data, are innovatively applied to the traditional OCSORT algorithm. This leads to the development of DA-OCSORT, a multi-target recognition and tracking method that can adapt to harsh outdoor environments, significantly improving the accuracy of agricultural robots' perception of outdoor environments and their anti-interference capabilities.

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] This invention provides a method for tracking multiple obstacles in a field environment, such as... Figure 1 and Figure 2 As shown, the method includes:

[0064] Step S1: Obtain successful matching information from the previous frame and detection data information from the current frame; the successful matching information from the previous frame includes the obstacle center pixel coordinates and the BEV target center point coordinates from the previous frame; the detection data information from the current frame includes the RGB image, depth point cloud information, and IMU data information from the current frame.

[0065] Here, BEV stands for Bird's-Eye View, a perspective from above when viewing an object or scene, much like a bird looking down at the ground from the air. In the fields of autonomous driving and robotics, data acquired through sensors (such as LiDAR and cameras) is often converted into BEV representations to better facilitate tasks such as object detection and path planning. BEV simplifies complex 3D environments into 2D images, which is particularly important for efficient computation in real-time systems.

[0066] In this embodiment, the RGB image of the obstacle in the current frame and the depth point cloud information are acquired by a depth binocular camera, and the IMU data information when the depth binocular camera is shooting is recorded as the detection data information of the current frame.

[0067] Step S2: Perform Kalman filtering on the successful matching information of the previous frame to obtain the center pixel coordinates of the prediction box and the prediction center coordinates of the BEV target.

[0068] In this embodiment, the information of successful matching in the previous frame (including the center pixel coordinates of the obstacle and the center point coordinates of the BEV target) is fed into the Kalman filter to obtain the center pixel coordinates of the predicted bounding box and the predicted coordinates of the BEV target center point output by the Kalman filter.

[0069] Step S3: Use the IMU data information to correct the center pixel coordinates of the prediction box to obtain the center coordinates of the image prediction box.

[0070] To reduce the bias caused by camera motion, the ICMC module proposed in this invention is used to correct the coordinates of the center pixel of the prediction box. The ICMC correction process is as follows: Figure 3 As shown, O is the camera origin, and point B is the predicted coordinate obtained through Kalman filtering, i.e., the coordinates of the center pixel of the prediction box. Solid lines correspond to the coordinates of the previous frame, and dashed lines describe the coordinates of the current frame. ∠AOC is the camera's field of view in the previous frame, and ∠DOE is the camera's field of view in the current frame. AC intersects DE, and the intersection point B is the center pixel of the prediction box. The angle of change in the x-direction between two frames is denoted by θ. x (∠AOD) represents the angle of view (z) in the x-direction of the camera. ∠AOC and ∠DOE are equal and both represent the angle of view (z) in the x-direction of the camera. x OF and OG represent the angle bisectors of the field of view between two frames, where AC is perpendicular to OF and OG is perpendicular to DE.

[0071] Regarding the compensation methods for the x-axis and y-axis: Since the established geometric calculation relationships are completely consistent, a calculation method taking the x-axis as an example is given. Because the measurement points maintain consistent absolute positions between frames, the self-motion compensation problem can be transformed into solving BD:DE under the given AB:AC condition. In this embodiment, the ratio BC:AB will be represented by r. x This indicates that the ratio BD:BE will be expressed using e. x This is indicated by... To simplify the formula, an intermediate calculation term 'a' is introduced. x The following formula can be obtained through trigonometric relationships:

[0072]

[0073]

[0074] Therefore, the x-axis correction ratio BD:DE can be calculated, that is... Given the known resolution of the camera, assume the number of pixels along the x-axis is k. x The corrected x-axis coordinate X is:

[0075]

[0076] The above process completes the correction of the x-axis coordinate of the center point of the prediction box in the image coordinate system. Similarly, the y-axis coordinate can be corrected. The corresponding geometric calculation relationship is completely consistent with the geometric calculation relationship of the x-axis correction, that is, the following formula can be obtained through trigonometric relationships:

[0077]

[0078]

[0079] Therefore, the corrected y-axis coordinate Y is:

[0080]

[0081] Since the calculation relationship remains unchanged, Figure 3 Similarly, as a schematic diagram for correction calculation in the y-axis direction, k y This represents the number of pixels along the y-axis. θ is the y-axis correction scale value. y The angle of change in the y-direction between two frames, z y e represents the field of view of the camera in the y-direction. y Indicates the ratio BD:BE, r y The ratio BC:AB is represented by a. y This represents an intermediate calculation term. Then (X, Y) are the center coordinates of the image prediction box.

[0082] In the above process, the camera field of view of the previous frame, the camera field of view of the current frame, and the angle between the current frame and the previous frame are all determined based on IMU data. Based on this process, the center of the prediction box can be corrected, thereby correcting the position of the corresponding prediction box, improving accuracy, and significantly reducing false matches and missed matches.

[0083] Step S4: Perform obstacle target object recognition on the RGB image to obtain the center coordinates of the image detection box.

[0084] Using the existing YOLOv8 network model, obstacle targets in RGB images are identified, and the coordinates of the center position of the detection box in the current frame, as well as the length and width of the detection box, are obtained.

[0085] Step S5: Determine the center coordinates of the point cloud from the bird's-eye view based on the depth point cloud information, and correct the center coordinates of the point cloud using the IMU data information to obtain the BEV target detection center coordinates.

[0086] Using IMU data recorded in each frame during shooting, the BEV target center point detection coordinates (i.e., the point cloud center coordinates from a bird's-eye view) are corrected, thus correcting the BEV coordinate changes caused by the camera's horizontal rotation. In the video stream, the BEV coordinate system is always fixed at the first frame, with the camera position as the origin O, the direction directly facing the camera as the positive y-axis, and the horizontal plane to the right of the camera as the positive x-axis. The BEV coordinate correction calculation method is as follows: Figure 4 As shown, since the camera's translational motion has little impact on the detection box position, the correction mainly considers the effect of the camera's horizontal rotation on the change in the detection box position. In the figure, O is the camera origin position, the solid line is the camera's position in the first frame image coordinate system, and the dashed line is the camera's position in the coordinate system after rotation. Point A' is the position of the center point of the detected obstacle. Since the shooting frame rate is above 15fps, the absolute position of point A' remains approximately unchanged between two frames. Assuming that the coordinates of point A' in the x'oy' coordinate system are (a', b'), the coordinates in the xoy coordinate system are (a”, b”), which are the coordinates of the BEV target detection center. The camera rotation angle is determined based on IMU data and is represented as γ(∠yoy'). The transformation formula is as follows:

[0087] a”=a'×cosγ+b'×sinγ

[0088] b”=b'×cosγ-a'×sinγ

[0089] Step S6: Input the BEV target prediction center coordinates, the image prediction box center coordinates, the image detection box center coordinates, and the BEV target detection center coordinates into the OCSORT model to perform minimum cost matching, obtain initial matching information, and determine whether the detection box and the prediction box match.

[0090] Step S7: If the detection box and the prediction box do not match, a cost matrix is ​​constructed based on the predicted center coordinates of the BEV target and the detection center coordinates of the BEV target. Based on the cost matrix, the LAPJV algorithm is used to perform minimum cost matching to obtain the successful matching information of the current frame. The successful matching information of the current frame includes the obstacle center pixel coordinates and the BEV target center point coordinates of the current frame.

[0091] Step S8: If the detection box and the prediction box have matched, then the initial matching information is used as the successful matching information of the current frame.

[0092] The center position data of all predicted bounding boxes in the previous frame and all detected bounding boxes in the current frame are fed into the OCSORT model. Successfully matched predicted bounding boxes and detected bounding boxes can be obtained through minimum cost matching.

[0093] For predicted and detected bounding boxes that failed to match in OCSORT, the DA module proposed in this invention performs secondary matching to obtain secondary matching information. During the trajectory prediction update process of DA-OCSORT, the DA module employs a Kalman filter algorithm to simultaneously predict and update the target spatial location information. Using the coordinates of the target center point in the current frame BEV determined based on depth point cloud information and the predicted position of the target center point in the BEV obtained by the Kalman filter, a cost matrix is ​​constructed for unmatched predicted and detected bounding boxes. The cost matrix records the Euclidean distance between all unmatched predicted and detected bounding boxes in the BEV. Based on depth point cloud information, this invention uses the LAPJV algorithm for minimum cost matching (minimizing the total matching cost). During the calculation process, a threshold h is set to filter matching pairs with an Euclidean distance greater than h; that is, matching pairs greater than h are discarded, while matching pairs less than or equal to h are retained. The processed secondary matching information can then be used as the successful matching information for the current frame. The DA module can maintain trajectory continuity when the object moves out of the field of view for a short time, is briefly occluded, or is interrupted, thereby improving overall tracking performance.

[0094]

[0095]

[0096] Where 'a' represents the center position of the unmatched bounding box (BEV), and 'b' represents the center position of the unmatched predicted bounding box (BEV). x and a y These represent the horizontal and vertical coordinates of the BEV target detection box center relative to the camera origin, calculated based on the BEV target detection center coordinates; b x and b y These represent the horizontal and vertical coordinates of the target prediction box center relative to the camera origin, calculated based on the BEV target prediction center coordinates; that is, the prediction box coordinate information. The distance between the two points is denoted as Dis. ab CM is the Euclidean distance between the predicted bounding box and the detected bounding box.

[0097] Step S9: Update the valid trajectory based on the successful matching information of the current frame, and return to step S1.

[0098] In this embodiment, the successfully matched detection boxes and prediction boxes are updated with the corresponding valid trajectories and used as the input of the Kalman filter in the next frame (including the center pixel coordinates of the obstacle detection box and the center point coordinates of the BEV target). Steps S1-S9 are repeated until the current frame is the last frame.

[0099] In summary, this invention proposes a DA-OCSORT with two new modules, ICMC and DA. ICMC records camera ego motion and performs position correction compensation on the predicted bounding boxes, ensuring accurate target tracking even under drastic camera ego motion. The DA module can perform secondary matching of unmatched current frame detection boxes from OCSORT and predicted boxes based on historical paths under BEV conditions, tracking the obstacle's BEV position relative to the camera. This maintains trajectory continuity even when the object briefly moves out of the field of view, is briefly occluded, or the tracking is interrupted, thereby improving overall tracking performance.

[0100] The traditional OCSORT matching process takes as input the predicted bounding box of each trajectory obtained by a Kalman filter based on previous historical frames in the image coordinate system, and the detection bounding box of the current frame. By matching the predicted bounding boxes and detection bounding boxes pairwise, based on the IOU information of each match (the higher the IOU, the higher the similarity and the easier the match), and setting a similarity threshold, if the IOU of the matched pair is higher than the threshold, it means that the detection bounding box and the trajectory are successfully matched, otherwise it fails. Therefore, the OCSORT output has two possibilities: ① successfully matched pairs ② unmatched predicted bounding boxes and unmatched detection bounding boxes.

[0101] Compared to the traditional OCSORT matching method, the ICMC module proposed in this invention is mainly linked to the Kalman filter in the input part of the OCSORT prediction end. It uses camera pose data obtained from the IMU to compensate for the center point position of the predicted box of the Kalman filter, thereby reducing the adverse effects of camera self-motion on MOT. The DA module is linked after the OCSORT matching detection box and predicted box image. During the trajectory prediction update process of DA-OCSORT, the Kalman filter is used to simultaneously predict and update the target spatial position information. Using depth point cloud data from the depth binocular camera system, the relative position information is integrated into the data association. Under the BEV view, the detection boxes and predicted boxes that were not successfully matched in OCSORT are matched again, so that the object can be successfully matched even if it moves out of the field of view for a short time, is briefly occluded, or is interrupted, thus maintaining the trajectory continuity. The DA-OCSORT proposed in this invention fully considers the camera's self-motion compensation and improves the tracking performance in outdoor environments.

[0102] To implement the above methods and achieve the corresponding functions and technical effects, a multi-obstacle target tracking system for outdoor environments is provided below. This system includes:

[0103] The data acquisition module is used to acquire successful matching information of previous frames and detection data information of the current frame; the successful matching information of previous frames includes the obstacle center pixel coordinates and BEV target center point coordinates of the previous frame; the detection data information of the current frame includes the RGB image, depth point cloud information and IMU data information of the current frame.

[0104] The BEV target prediction center coordinate determination module is used to perform Kalman filtering on the previous frame's successful matching information to obtain the prediction box center pixel coordinates and the BEV target prediction center coordinates.

[0105] The image prediction box center coordinate determination module is used to correct the center pixel coordinates of the prediction box using the IMU data information to obtain the center coordinates of the image prediction box.

[0106] The image detection box center coordinate determination module is used to identify obstacle targets in the RGB image and obtain the center coordinates of the image detection box.

[0107] The BEV target detection center coordinate determination module is used to determine the point cloud center coordinates from a bird's-eye view based on the depth point cloud information, and to correct the point cloud center coordinates using the IMU data information to obtain the BEV target detection center coordinates.

[0108] The OCSORT matching module is used to input the BEV target prediction center coordinates, the image prediction box center coordinates, the image detection box center coordinates, and the BEV target detection center coordinates into the OCSORT model for minimum cost matching, obtain initial matching information, and determine whether the detection box and the prediction box match.

[0109] The first matching information determination module is used to construct a cost matrix based on the predicted center coordinates of the BEV target and the detected center coordinates of the BEV target if the detection box and the predicted box do not match, and to perform minimum cost matching using the LAPJV algorithm based on the cost matrix to obtain the current frame successful matching information; the current frame successful matching information includes the obstacle center pixel coordinates and the BEV target center point coordinates of the current frame.

[0110] The second matching information determination module is used to take the initial matching information as the successful matching information of the current frame if the detection box and the prediction box have matched.

[0111] The effective trajectory update module is used to update the effective trajectory based on the successful matching information of the current frame and return it to the data acquisition module.

[0112] The present invention also provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the above-described method for tracking multiple obstacles in a field environment. The electronic device may be a server.

[0113] In addition, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for tracking multiple obstacles in a field environment.

[0114] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0115] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for tracking multiple obstacles in a wilderness environment, characterized in that, include: Obtain successful matching information from previous frames and detection data from the current frame; The successful matching information of the previous frame includes the obstacle center pixel coordinates and the BEV target center point coordinates of the previous frame; the detection data information of the current frame includes the RGB image, depth point cloud information and IMU data information of the current frame; Kalman filtering is performed on the successful matching information of the previous frame to obtain the center pixel coordinates of the prediction box and the prediction center coordinates of the BEV target; The center pixel coordinates of the prediction box are corrected using the IMU data information to obtain the center coordinates of the image prediction box; Obstacle target object recognition is performed on the RGB image to obtain the center coordinates of the image detection box; The point cloud center coordinates from the bird's-eye view are determined based on the depth point cloud information, and the point cloud center coordinates are corrected using the IMU data information to obtain the BEV target detection center coordinates. The coordinates of the predicted center of the BEV target, the coordinates of the center of the predicted image box, the coordinates of the center of the image detection box, and the coordinates of the detected BEV target are input into the OCSORT model for minimum cost matching to obtain initial matching information, and to determine whether the detection box and the predicted box match. If the detection box and the prediction box do not match, a cost matrix is ​​constructed based on the predicted center coordinates of the BEV target and the detection center coordinates of the BEV target. Based on the cost matrix, the LAPJV algorithm is used to perform minimum cost matching to obtain the successful matching information of the current frame. The successful matching information of the current frame includes the obstacle center pixel coordinates and the BEV target center point coordinates of the current frame. If the detection box and the prediction box have matched, the initial matching information is used as the successful matching information for the current frame. The steps are: update the valid trajectory based on the successful matching information of the current frame, and return to obtain the successful matching information of the previous frame and the detection data information of the current frame.

2. The method for tracking multiple obstacles in a field environment according to claim 1, characterized in that, The center pixel coordinates of the prediction box are corrected using the IMU data information to obtain the center coordinates of the image prediction box, specifically including: The camera field of view of the previous frame, the camera field of view of the current frame, and the angle between the current frame and the previous frame are determined based on the IMU data information. Based on the triangular relationship, the correction ratio is calculated according to the camera field of view of the previous frame, the camera field of view of the current frame, and the angle between the current frame and the previous frame. The center pixel coordinates of the prediction box are corrected according to the correction ratio and the camera resolution to obtain the center coordinates of the image prediction box.

3. The method for tracking multiple obstacles in a field environment according to claim 1, characterized in that, Obstacle target object recognition is performed on the RGB image to obtain the center coordinates of the image detection box, specifically including: The YOLOv8 network model is used to identify obstacle targets in the RGB image, and the center coordinates of the image detection box are obtained.

4. The method for tracking multiple obstacles in a field environment according to claim 1, characterized in that, The IMU data information is used to correct the center coordinates of the point cloud to obtain the BEV target detection center coordinates, specifically including: Determine the camera rotation angle based on the IMU data information; The point cloud center coordinates are corrected using the camera rotation angle to obtain the BEV target detection center coordinates.

5. The method for tracking multiple obstacles in a field environment according to claim 1, characterized in that, A cost matrix is ​​constructed based on the BEV target prediction center coordinates and the BEV target detection center coordinates, specifically including: Calculate the horizontal and vertical coordinates of the BEV target prediction box center relative to the camera origin based on the BEV target prediction center coordinates, and use them as prediction box coordinate information; Calculate the horizontal and vertical coordinates of the BEV target detection box center relative to the camera origin based on the BEV target detection center coordinates, and use them as the detection box coordinate information; The Euclidean distance between the predicted box and the detected box is calculated based on the predicted box coordinate information and the detected box coordinate information. Construct a cost matrix based on the Euclidean distances between all predicted bounding boxes and all detected bounding boxes.

6. The method for tracking multiple obstacles in a field environment according to claim 1, characterized in that, Based on the cost matrix, the LAPJV algorithm is used for minimum cost matching to obtain the successful matching information for the current frame, specifically including: Based on the cost matrix, the LAPJV algorithm is used to perform minimum cost matching to obtain secondary matching information; Compare the Euclidean distances between the predicted bounding boxes and the detected bounding boxes in the secondary matching information; If the Euclidean distance is greater than a set threshold, the matching pair of the predicted box and the detected box will be discarded. If the Euclidean distance is less than or equal to a set threshold, the matching pair of the predicted box and the detected box will be retained; The processed secondary matching information is used as the successful matching information for the current frame.

7. The method for tracking multiple obstacles in a field environment according to claim 1, characterized in that, The steps of updating the valid trajectory based on the current frame's successful matching information and returning to obtain the previous frame's successful matching information and the current frame's detection data information specifically include: Update the valid trajectory based on the successful matching information of the current frame, and determine whether the current frame is the last frame; If the current frame is not the last frame, return to the steps of obtaining the successful matching information of the previous frame and the detection data information of the current frame; If the current frame is the last frame, then the process ends.

8. A multi-obstacle target tracking system for outdoor environments, characterized in that, include: The data acquisition module is used to acquire successful matching information from previous frames and detection data from the current frame. The successful matching information of the previous frame includes the obstacle center pixel coordinates and the BEV target center point coordinates of the previous frame; the detection data information of the current frame includes the RGB image, depth point cloud information and IMU data information of the current frame; The BEV target prediction center coordinate determination module is used to perform Kalman filtering on the previous frame successful matching information to obtain the prediction box center pixel coordinates and the BEV target prediction center coordinates. The image prediction box center coordinate determination module is used to correct the center pixel coordinates of the prediction box using the IMU data information to obtain the center coordinates of the image prediction box. The image detection box center coordinate determination module is used to identify obstacle targets in the RGB image and obtain the center coordinates of the image detection box. The BEV target detection center coordinate determination module is used to determine the point cloud center coordinates from a bird's-eye view based on the depth point cloud information, and to correct the point cloud center coordinates using the IMU data information to obtain the BEV target detection center coordinates. The OCSORT matching module is used to input the BEV target prediction center coordinates, the image prediction box center coordinates, the image detection box center coordinates, and the BEV target detection center coordinates into the OCSORT model to perform minimum cost matching, obtain initial matching information, and determine whether the detection box and the prediction box match. The first matching information determination module is used to construct a cost matrix based on the predicted center coordinates of the BEV target and the detected center coordinates of the BEV target if the detection box and the predicted box do not match, and to perform minimum cost matching using the LAPJV algorithm based on the cost matrix to obtain the current frame successful matching information; the current frame successful matching information includes the obstacle center pixel coordinates and the BEV target center point coordinates of the current frame. The second matching information determination module is used to take the initial matching information as the successful matching information of the current frame if the detection box and the prediction box have matched. The effective trajectory update module is used to update the effective trajectory based on the successful matching information of the current frame and return it to the data acquisition module.

9. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the field environment multi-obstacle target tracking method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the multi-obstacle target tracking method in the field environment as described in any one of claims 1 to 7.

Images