A fast and accurate real-time moving target position estimation method and application

Abstract

The present application belongs to the computer vision technology, and mainly relates to the target positioning field, and particularly relates to a fast and accurate real-time moving target position estimation method and application. The present application can be used in the field of precision agriculture, and is used for pure visual moving target position estimation to provide moving target position information for precision fertilization, precision weeding and precision irrigation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, mainly involving the field of target localization, and particularly to a fast and accurate real-time moving target position estimation method and its application. Background Technology

[0002] In agriculture, automation and intelligentization are future development trends. In this process, determining the location of targets such as weeds and crops is fundamental to many applications, especially in precision agriculture. Various sensors are available for target localization, with cameras being the most widely used.

[0003] Camera-based visual object detection and tracking can estimate the position of a moving target. Among these techniques, deep learning-based detection models and Kalman filter-based motion models are currently the most widely used. The combination of these techniques can accurately estimate the position of a moving target.

[0004] Currently, numerous articles and patents have adapted and improved upon the aforementioned technical framework for different scenarios. However, the specific technologies used vary across different scenarios, especially in velocity and position estimation. Kalman filtering is the most widely used, but it is not essential. Furthermore, position estimation is affected not only by velocity estimation but also by time intervals, which differ across scenarios—a point rarely mentioned in patents.

[0005] In precision agriculture, the primary targets are crops and weeds, which are stationary, while the agricultural machinery is typically mobile during operation. This leads to the following two issues:

[0006] 1) Lag in target positioning: The target position obtained by processing the sampled data at time t changes significantly after a processing time Δt. This change and the resulting deviation is fatal to precision agriculture (for example, in laser weeding, the deviation may directly cause the laser to intersect on the crop).

[0007] 2) Short time window: Due to the movement of agricultural machinery, the processing time window for a single target is very short. Summary of the Invention

[0008] Therefore, this invention proposes a fast and accurate real-time moving target position estimation method and application to solve the above-mentioned problems.

[0009] To address the aforementioned technical issues, this paper proposes a fast and accurate real-time moving target position estimation method and its application.

[0010] This invention provides the following technical solution:

[0011] A fast and accurate real-time moving target position estimation method includes the following steps:

[0012] S100, Target Position Acquisition: Using a depth camera as a sensor, acquire the color image (img) of the i-th frame at time t. color and depth image (img) depth A deep learning-based object detector is used to obtain the target in the color image. color The pixel coordinates bbox(x0,y0,x1,y1) on the map are calculated, and the midpoint position p(x,y) is calculated.

[0013] Based on the interface provided by the depth camera, obtain the position c(x,y,z) of p(x,y) in the camera coordinate system, that is:

[0014] c(x,y,z)=F p→c (p(x,y),img depth (x,y))

[0015] Wherein: F p→c The function that maps pixel coordinates to camera coordinates, img depth (x,y) represents the depth information of the target midpoint p(x,y);

[0016] S200, Inter-frame target matching: Perform inter-frame target matching. For unmatched targets, treat them as new targets and wait for the next frame of data to be processed. For matched targets, proceed to S300.

[0017] S300, Dynamic State Estimation: For targets matched in consecutive frames, their center point positions are c respectively. i-1 (x,y,z) and c i (x,y,z), then the target velocity v corresponding to its i-th frame. i The velocities in the x, y, and z directions corresponding to (x, y, z) are:

[0018]

[0019] Where: t i and t i-1 These represent the time points in the i-th frame and the (i-1)-th frame, respectively.

[0020] S400, Target Position Estimation: Based on the motion physics model, the target has been in position for time d. i The following position is:

[0021]

[0022] Its time deviation from the actual scenario is:

[0023]

[0024] Where: d c The camera sampling delay is a constant and an inherent property of the camera. It is the calculation interval from when the camera samples the image to when the position estimation is completed;

[0025] Acquired This is the estimated position of the target relative to the camera at this moment;

[0026] S500. Update target position and velocity: Before processing the next frame, the target's position and velocity from the previous frame need to be updated using the information from the current i-th frame. The formula is:

[0027] bbox i-1 (x0, y0, x1, y1) = bbox i (x0,y0,x1,y1)

[0028] c i-1 (x,y,z)=c i (x,y,z)

[0029] v i-1 (x,y,z)=v i (x,y,z)

[0030] After S600 is completed, repeat the S100~S500 loop for the new i+1 frame data.

[0031] Preferably, in S100, the deep learning-based target detector is the YOLO series.

[0032] Preferably, in S200, the method for performing inter-frame target matching is either calculating the intersection-union ratio of the inter-frame target positions and determining whether it is greater than a threshold, or any one of the matching methods based on target appearance features.

[0033] Preferably, in step S300, a weighted filter is used to filter the velocity:

[0034] v i (x,y,z)=a*v i-1 (x,y,z)+(1-a)*v i (x,y,z)

[0035] Where: v i-1 (x,y,z) represents the target velocity corresponding to the (i-1)th frame, and a is a weighting factor with a range of (0,1), representing the influence of the previously estimated velocity and the current frame's estimated velocity.

[0036] An application of a fast and accurate real-time moving target position estimation method, which runs as described above, can be used in any scenario where the position of a moving target needs to be estimated.

[0037] A laser weeding robot includes:

[0038] Memory, used to store computer programs;

[0039] A processor, configured to implement, when executing the computer program, a fast and accurate real-time moving target position estimation method as described in any of the preceding claims.

[0040] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a fast and accurate real-time moving target position estimation method as described in any of the preceding claims.

[0041] Compared with the prior art, the beneficial effects of the present invention are: the present invention can be used in the field of precision agriculture for purely visual estimation of the position of moving targets, providing moving target position information for applications such as precision fertilization, precision weeding, and precision irrigation.

[0042] This invention is suitable for estimating the position of moving targets in low-speed (not exceeding 0.5 m / s) scenarios. Based on a pure vision algorithm, it combines target detection, target matching, motion state estimation, and motion duration estimation technologies to accurately provide the target's position information during its movement. Testing has shown a positioning accuracy within 5 mm, meeting the requirements of operational scenarios such as precision weeding, precision fertilization, and precision irrigation.

[0043] The contribution of this invention lies in proposing a fast and accurate real-time method for estimating the position of a moving target and its application. This method can be used not only in precision agriculture but also in any scenario requiring position estimation of a moving target. Although primarily applicable to low-speed scenarios, it can be applied to higher-speed scenarios by adjusting the matching threshold. Attached Figure Description

[0044] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0045] Figure 1 This is a flowchart illustrating the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] 1. Target location acquisition

[0048] This invention uses a depth camera as a sensor to acquire the color image (img) at time t in the i-th frame. color and depth image (img) depth The pixel coordinates (bbox(x0,y0,x1,y1)) of the target on the color image are obtained using a deep learning-based object detector (such as the YOLO series), and the midpoint position p(x,y) is calculated.

[0049] Based on the interface provided by the depth camera, obtain the position c(x,y,z) of p(x,y) in the camera coordinate system, that is:

[0050] c(x,y,z)=F p→c (p(x,y),img depth (x,y)) (1)

[0051] in:

[0052] ●F p→c This is a mapping function from pixel coordinates to camera coordinates.

[0053] ●img depth (x,y) represents the depth information of the target midpoint p(x,y).

[0054] Target matching between 2 frames

[0055] At low speeds, the position of the same target in the pixel coordinate system does not change much between consecutive frames. Based on this fact, inter-frame target matching can be completed simply by calculating the intersection over union (IoU) of the target positions between frames.

[0056] For unmatched targets, they are treated as new targets and processed in the next frame. For matched targets, motion estimation and position estimation are performed in steps 3 to 5.

[0057] 3. Motion State Estimation

[0058] For the target matched in the preceding and following frames, the center point positions are c respectively. i-1 (x,y,z) and c i(x,y,z), then the target velocity v corresponding to its i-th frame. i The velocities in the x, y, and z directions corresponding to (x, y, z) are:

[0059]

[0060] in:

[0061] ●t i and t i-1 These represent the time points in the i-th frame and the (i-1)-th frame, respectively.

[0062] Considering noise, this invention uses weighted filtering to filter the velocity, thereby improving the stability and accuracy of velocity estimation:

[0063] v i (x,y,z)=a*v i-1 (x,y,z)+(1-a)*v i (x,y,z) (3)

[0064] in:

[0065] ●a is a weighting factor, ranging from (0,1), which represents the impact of the previously estimated velocity and the current frame's estimated velocity.

[0066] 4 Target Location Estimation

[0067] Because the interval between consecutive frames is very short (33.3ms when the camera sampling frame rate is 30fps) and the target's movement speed is relatively slow, the target's motion between frames can be simplified to uniform linear motion. According to the motion physics model, the target's time d... i The following position is:

[0068]

[0069] Here d i Accurate estimation is crucial, and its time deviation from the actual scenario is as follows:

[0070]

[0071] in:

[0072] ●d c The camera sampling delay can be considered a constant, an inherent property of the camera, and varies from camera to camera. It's important to note that this can be the main delay component and is easily overlooked.

[0073] ● This is the calculation interval from when the camera samples an image to when the position estimation is completed. This interval can be directly obtained in the programming implementation. As long as it does not exceed the inter-frame interval, the overall position estimation process is real-time.

[0074] Acquired This is the estimated position of the target relative to the camera at this moment.

[0075] 5. Update target position and speed

[0076] Before processing the next frame, the target's position and velocity from the previous frame need to be updated using the information from the current i-th frame. The formula is:

[0077] bbox i-1 (x0, y0, x1, y1) = bbox i (x0,y0,x1,y1) (6.1)

[0078] c i-1 (x,y,z)=c i (x,y,z) (6.2)

[0079] v i-1 (x,y,z)=v i (x,y,z) (6.3)

[0080] After the update is complete, repeat the above loop for the new i+1 frame data.

[0081] During this process, as the agricultural machinery operates stably, the accuracy of its speed estimation will gradually increase. In addition, except for the target detection time, the other parts of the above calculation can be completed quickly, so the time interval estimation will become more and more stable, thus achieving a fast and accurate ideal state.

[0082] It should be noted that this method works well for low-speed moving targets, but for high-speed moving targets, a suitable "inter-frame target matching" method needs to be used instead, such as a matching method based on the target's appearance features. Other parts do not need to be modified.

[0083] The above detailed process can be summarized as follows:

[0084] This invention combines multiple technologies, including target detection, coordinate system transformation, and motion modeling, to solve the problem of position shift of moving targets from the start of data acquisition to the end of processing. It is described as follows:

[0085] Considering that this invention is mainly aimed at low-speed motion scenarios, therefore The transformation can be modeled using uniform linear motion:

[0086] in:

[0087] ●c i The position of the midpoint p(x,y) of the target in the i-th frame in the camera coordinate system is obtained by combining the target detector with the depth camera coordinate system transformation interface:

[0088] c(x,y,z)=F p→c (p(x,y),img depth (x,y)) (1)

[0089] ●v i The estimated target velocity for the i-th frame is based on the target midpoint position c in the i-th frame. i i-1 frame position c i-1 (x,y,z), and the target at frame i-1 velocity v i-1 (x,y,z) is used to calculate and obtain:

[0090]

[0091] v i (x,y,z)=a*v i-1 (x,y,z)+(1-a)*v i (x,y,z) (3)

[0092] in:

[0093] ■t i and t i-1 These represent the time points in the i-th frame and the (i-1)-th frame, respectively.

[0094] ■a is a weighting factor, ranging from (0,1).

[0095] ●d i To determine the time from data acquisition to processing, the camera sampling delay d needs to be considered. c and computing unit

[0096] Calculate time

[0097]

[0098] Finally, for the target bounding box (bbox) i-1 (x0, y0, x1, y1), coordinates of the center point c i-1 (x,y,z), velocity v i-1 Update (x,y,z).

[0099] Repeat the above process to continuously evaluate the target location.

[0100] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0101] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fast and accurate real-time moving target position estimation method, characterized in that, Includes the following steps: S100, Target Position Acquisition: Using a depth camera as a sensor, acquire the color image (img) of the i-th frame at time t. color and depth image (img) depth A deep learning-based object detector is used to obtain the target in the color image. color The pixel coordinates bbox(x0,y0,x1,y1) on the map are calculated, and the midpoint position p(x,y) is calculated. Based on the interface provided by the depth camera, obtain the position c(x,y,z) of p(x,y) in the camera coordinate system, that is: c(x,y,z)=F p→c (p(x,y),img depth (x,y)) Wherein: F p→c The function that maps pixel coordinates to camera coordinates, img depth (x,y) represents the depth information of the target midpoint p(x,y); S200, Inter-frame target matching: Perform inter-frame target matching. For unmatched targets, treat them as new targets and wait for the next frame of data to be processed. For matched targets, proceed to S300. S300, Dynamic State Estimation: For targets matched in consecutive frames, their center point positions are c respectively. i-1 (x,y,z) and c i (x,y,z), then the target velocity v corresponding to its i-th frame. i The velocities in the x, y, and z directions corresponding to (x, y, z) are: Where: t i and t i-1 These represent the time points in the i-th frame and the (i-1)-th frame, respectively. S400, Target Position Estimation: Based on the motion physics model, the target has been in position for time d. i The following position is: Its time deviation from the actual scenario is: Where: d c The camera sampling delay is a constant and an inherent property of the camera. It is the calculation interval from when the camera samples the image to when the position estimation is completed; Acquired This is the estimated position of the target relative to the camera at this moment; S500. Update target position and velocity: Before processing the next frame, the target's position and velocity from the previous frame need to be updated using the information from the current i-th frame. The formula is: bbox i-1 (x0,y0,x1,y1)=bbox i (x0,y0,x1,y1) c i-1 (x,y,z)=c i (x,y,z) v i-1 (x,y,z)=v i (x,y,z) After S600 is completed, repeat the S100~S500 loop for the new i+1 frame data.

2. The method for fast and accurate real-time moving target position estimation according to claim 1, characterized in that, In S100, the deep learning-based target detector is the YOLO series.

3. The method for fast and accurate real-time moving target position estimation according to claim 1, characterized in that, In S200, the method for inter-frame target matching is either to calculate the intersection-union ratio of the inter-frame target positions and determine whether it is greater than a threshold, or any one of the matching methods based on target appearance features.

4. The method for fast and accurate real-time moving target position estimation according to claim 1, characterized in that, In S300, weighted filtering is used to filter the velocity: v i (x,y,z)=a*v i-1 (x,y,z)+(1-a)*v i (x,y,z) Where: v i-1 (x,y,z) represents the target velocity corresponding to the (i-1)th frame, and a is a weighting factor with a range of (0,1), representing the influence of the previously estimated velocity and the current frame's estimated velocity.

5. An application of a fast and accurate real-time moving target position estimation method, characterized in that, The method for fast and accurate real-time moving target position estimation as described in any of claims 1 to 4 is applicable to any scenario requiring position estimation of a target during movement.

6. A laser weeding robot, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the fast and accurate real-time moving target position estimation method as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the fast and accurate real-time moving target position estimation method as described in any one of claims 1 to 4.

Images