Rapid and accurate real-time moving target position estimation method and application

Abstract

The invention belongs to the computer vision technology, mainly relates to the field of target positioning, and particularly relates to a quick and accurate real-time moving target position estimation method and application. The method can be used in the field of precision agriculture, is used for estimating the position of a pure visual moving target, and provides target position information in movement for precise fertilization, precise weeding and precise irrigation.

Description

Technical Field

[0001] The present invention belongs to computer vision technology, mainly relates to the field of target positioning, and in particular to a fast and accurate real-time moving target position estimation method and application. Background Art

[0002] In the agricultural field, automation and intelligence are the future development trends. In this process, the location of targets such as weeds and crops is the basis of many applications, especially in precision agriculture. In the process of achieving target positioning, there are many sensors to choose from, among which the most widely used is the camera.

[0003] Pure visual target detection and target tracking based on cameras can estimate the moving position of the target. Among them, the detection model based on deep learning and the motion model based on Kalman filtering are the most commonly used technologies. The combination of these technologies can well estimate the position of the moving target.

[0004] At present, there are many articles and patents that have adapted and improved different scenarios based on the above technical framework, but the specific technologies used in different scenarios are different, especially in speed estimation and position estimation. Kalman filtering is the most widely used one, but it is not necessary. In addition to being affected by speed estimation, position estimation is also affected by time intervals, and the time intervals are different in different scenarios, which is rarely mentioned in patents.

[0005] Considering the precision agriculture scenario, the main targets are crops and weeds. Although their positions are fixed, agricultural machinery is usually mobile during operation. This leads to the following two problems:

[0006] 1) Lag of target positioning: The target position obtained by processing the sampling data at time t has changed significantly after processing for a period of time Δt. The deviation caused by this change is fatal to precision agriculture (for example, in laser weeding, the deviation may directly cause the laser intersection to be on the crop);

[0007] 2) Short time window: Due to the movement of agricultural machinery, the processing time window for a single target is very short. Summary of the invention

[0008] Therefore, the present invention proposes a fast and accurate real-time moving target position estimation method and application to solve the above problems.

[0009] In order to solve the above technical problems, we propose a fast and accurate real-time moving target position estimation method and application.

[0010] The present invention provides the following technical solutions:

[0011] A fast and accurate real-time moving target position estimation method comprises the following steps:

[0012] S100, target position acquisition: Use the depth camera as a sensor to obtain the color image img of the i-th frame at time t color and depth image img depth , use the deep learning-based target detector to obtain the target in the color image img color The pixel coordinate position bbox(x0,y0,x1,y1) on the image is calculated, and the midpoint position p(x,y) is calculated.

[0013] According to the interface provided by the depth camera, get the position c(x,y,z) of p(x,y) in the camera coordinate system, that is:

[0014] c(x,y,z)=F p→c (p(x,y),img depth (x,y)

[0015] Among them: F p→c is the mapping function from pixel coordinates to camera coordinate system, img depth (x, y) is the depth information of the target midpoint p(x, y);

[0016] S200, inter-frame target matching: inter-frame target matching is performed, and unmatched targets are taken as new targets and wait for the next frame data to be processed. For matched targets, the process proceeds to S300;

[0017] S300, dynamic state estimation: The center points of the targets matched in the previous and next frames are c i-1 (x,y,z) and c i (x,y,z), then the target speed v corresponding to the i-th frame i The speeds in the x, y, and z directions corresponding to (x, y, z) are:

[0018]

[0019] Where: t i and t i-1 Represent the time points of the i-th frame and the i-1-th frame respectively;

[0020] S400, target position estimation: According to the motion physics model, the target passes through time d i The following positions are:

[0021]

[0022] The time deviation between it and the actual scene is:

[0023]

[0024] Where: d c is the camera sampling delay, which is a constant and is an intrinsic property of the camera. It is the calculation interval from when the camera samples the picture to when the position estimation is completed;

[0025] Obtained That is the estimated position of the target relative to the camera at this moment;

[0026] S500, update target position and speed: before processing the next frame, the position and speed of the target in the previous frame need to be updated, and the corresponding information of the current i-th frame is used for updating, and the formula is:

[0027] bbox i-1 (x0,y0,x1,y1)=bbox i (x0,y0,x1,y1)

[0028] c i-1 (x,y,z)=c i (x,y,z)

[0029] v i-1 (x,y,z)=v i (x,y,z)

[0030] S600: After the update is completed, the cycle of S100 to S500 is repeated for the new i+1 frame data.

[0031] Preferably, in the S100, the target detector based on deep learning is a YOLO series.

[0032] Preferably, in S200, the method for performing inter-frame target matching is to calculate the intersection-over-union ratio of the target positions between frames and determine whether it is greater than a threshold or any one of the matching methods based on target appearance features.

[0033] Preferably, in S300, weighted filtering is used to filter the speed:

[0034] v i (x,y,z)=a*v i-1 (x,y,z)+(1-a)*v i (x,y,z)

[0035] Where: v i-1 (x, y, z) is the target speed corresponding to the i-1th frame, a is the weight factor, ranging from (0, 1), which represents the influence of the previously estimated speed and the estimated speed of the current frame.

[0036] An application of a fast and accurate real-time moving target position estimation method runs any of the fast and accurate real-time moving target position estimation methods described above and is used in any scenario where the position of a moving target needs to be estimated.

[0037] A laser weeding robot, comprising:

[0038] Memory for storing computer programs;

[0039] A processor is used to implement the fast and accurate real-time moving target position estimation method as described in any of the above items when executing the computer program.

[0040] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for fast and accurate real-time moving target position estimation as described in any of the above items is implemented.

[0041] Compared with the prior art, the present invention has the following beneficial effects: the present invention can be used in the field of precision agriculture, for purely visual estimation of the position of moving targets, and for providing position information of moving targets for precision fertilization, precision weeding, and precision irrigation.

[0042] The present invention is suitable for the position estimation of moving targets in low-speed scenarios (no more than 0.5m / s). Based on pure visual algorithms, combined with target detection, target matching, motion state estimation, motion duration estimation and other technologies, it accurately provides the position information of the target during the motion process. According to the test, the positioning accuracy is within 5mm, which meets the requirements of operation scenarios such as precision weeding, precision fertilization, and precision irrigation.

[0043] The contribution of the present invention is that a fast and accurate real-time moving target position estimation method and application are proposed. The method can be used not only in the field of precision agriculture, but also in any scenario where the position of a moving target needs to be estimated. Although it is mainly applicable to low-speed scenarios, it can be applied to higher-speed scenarios by adjusting the matching threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0045] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] 1. Target location acquisition

[0048] The present invention uses a depth camera as a sensor to obtain the color image img at the i-th frame and time t color and depth image img depth . Use a deep learning-based target detector (such as the YOLO series) to obtain the pixel coordinate position bbox (x0, y0, x1, y1) of the target on the color image and calculate the midpoint position p (x, y).

[0049] According to the interface provided by the depth camera, get the position c(x,y,z) of p(x,y) in the camera coordinate system, that is:

[0050] c(x,y,z)=F p→c (p(x,y),img depth (x,y)) (1)

[0051] in:

[0052] ●F p→c is the mapping function from pixel coordinates to camera coordinates.

[0053] ●img depth (x,y) is the depth information of the target midpoint p(x,y).

[0054] Target matching between 2 frames

[0055] At low speeds, the position of the same target in the pixel coordinate system will not change much between the previous and next frames. Based on this fact, we only need to calculate the intersection over union (IoU) of the target position between frames to complete the target matching between frames.

[0056] For the unmatched targets, they are treated as new targets and wait for the next frame of data to be processed. For the matched targets, the motion estimation and position estimation of the following steps 3 to 5 are performed.

[0057] 3 Motion state estimation

[0058] For the targets matched in the previous and next frames, their center points are c i-1 (x,y,z) and c i(x,y,z), then the target speed v corresponding to the i-th frame i The speeds in the x, y, and z directions corresponding to (x, y, z) are:

[0059]

[0060] in:

[0061] ●t i and t i-1 Represent the time points of the i-th frame and the i-1-th frame respectively;

[0062] Taking noise into consideration, the present invention uses weighted filtering to filter the speed to improve the stability and accuracy of speed estimation:

[0063] v i (x,y,z)=a*v i-1 (x,y,z)+(1-a)*v i (x,y,z) (3)

[0064] in:

[0065] ●a is the weight factor, ranging from (0,1), which represents the influence of the previous estimated speed and the estimated speed of the current frame.

[0066] 4 Target Position Estimation

[0067] Since the interval between the previous and next frames is very short (when the camera sampling frame rate is 30fps, the interval between frames is 33.3ms), and the target moves slowly, the target's motion between frames can be simplified to uniform linear motion. According to the motion physics model, the target passes through time d i The following positions are:

[0068]

[0069] Here is the i It is very important to accurately estimate the time difference between the actual scenario and the actual scenario:

[0070]

[0071] in:

[0072] ●d c The camera sampling delay can be considered as a constant, which is an inherent property of the camera and varies from camera to camera. It should be noted that it may be the main delay part and is easy to be ignored;

[0073] ● It is the calculation interval from when the camera samples the picture to when the position estimation is completed. This interval can be directly obtained in the programming implementation. As long as it does not exceed the inter-frame interval, the overall position estimation process is real-time.

[0074] Obtained This is the estimated position of the target relative to the camera at this moment.

[0075] 5 Update target position and speed

[0076] Before processing the next frame, the position and speed of the target in the previous frame need to be updated using the corresponding information of the current i-th frame. The formula is:

[0077] bbox i-1 (x0,y0,x1,y1)=bbox i (x0,y0,x1,y1) (6.1)

[0078] c i-1 (x,y,z)=c i (x,y,z) (6.2)

[0079] v i-1 (x,y,z)=v i (x,y,z) (6.3)

[0080] After the update is completed, the above cycle is repeated for the new i+1 frame data.

[0081] In this process, with the stable operation of the agricultural machinery, the accuracy of its speed estimation will gradually increase, and except for the longer target detection time, the other parts of the above calculations can be completed quickly, so the estimation of the time interval will become more stable, thus achieving an ideal state of fast and accurate.

[0082] It should be noted that this method works well for slow-moving targets. For fast-moving targets, it needs to be replaced with a suitable "inter-frame target matching" method, such as a matching method based on target appearance features. Other parts do not need to be modified.

[0083] The above detailed process can be summarized as follows:

[0084] The present invention combines a variety of technologies, including target detection, coordinate system conversion, motion modeling and other technologies, to solve the problem of position offset of mobile targets from the beginning of data collection to the end of processing, which can be expressed as:

[0085] Considering that the present invention is mainly aimed at low-speed motion scenes, The transformation can be modeled using uniform linear motion:

[0086] in:

[0087] ●c i is the position of the midpoint p(x,y) of the target in the i-th frame in the camera coordinate system, which is obtained through the target detector combined with the depth camera coordinate system conversion interface:

[0088] c(x,y,z)=F p→c (p(x,y),img depth (x,y)) (1)

[0089] ●v i The estimated target motion speed for the i-th frame is based on the target midpoint position c in the i-th frame. i , i-1 frame position c i-1 (x, y, z), and the target speed v at frame i-1 i-1 (x,y,z) to calculate:

[0090]

[0091] v i (x,y,z)=a*v i-1 (x,y,z)+(1-a)*v i (x,y,z) (3)

[0092] in:

[0093] ■t i and t i-1 Represent the time points of the i-th frame and the i-1-th frame respectively;

[0094] ■a is the weight factor, ranging from (0,1).

[0095] ●d i The time from the start of data acquisition to the end of processing is the target, and the camera sampling delay d needs to be considered c and computing unit

[0096] Calculate time

[0097]

[0098] Finally, for the target box bbox i-1 (x0, y0, x1, y1), center point coordinates c i-1 (x,y,z), speed v i-1 (x,y,z) is updated.

[0099] The above process is repeated to continuously evaluate the target position.

[0100] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0101] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fast and accurate real-time moving target position estimation method, characterized in that: The following steps are involved: S100, target position acquisition: Use the depth camera as a sensor to obtain the color image img of the i-th frame at time t color and depth image img depth , use the deep learning-based target detector to obtain the target in the color image img color The pixel coordinate position bbox(x0,y0,x1,y1) on the image is calculated, and the midpoint position p(x,y) is calculated. According to the interface provided by the depth camera, get the position c(x,y,z) of p(x,y) in the camera coordinate system, that is: c(x,y,z)=F p→c (p(x,y),img depth (x,y)) Among them: F p→c is the mapping function from pixel coordinates to camera coordinate system, img depth (x, y) is the depth information of the target midpoint p(x, y); S200, inter-frame target matching: inter-frame target matching is performed, and unmatched targets are taken as new targets and wait for the next frame data to be processed. For matched targets, the process proceeds to S300; S300, dynamic state estimation: For the targets matched in the previous and next frames, their center points are c i-1 (x,y,z) and c i (x,y,z), then the target speed v corresponding to the i-th frame i The speeds in the x, y, and z directions corresponding to (x, y, z) are: Where: t i and t i-1 Represent the time points of the i-th frame and the i-1-th frame respectively; S400, target position estimation: According to the motion physics model, the target passes through time d i The following positions are: The time deviation between it and the actual scene is: Where: d c is the camera sampling delay, which is a constant and is an intrinsic property of the camera. It is the calculation interval from when the camera samples the picture to when the position estimation is completed; Obtained That is the estimated position of the target relative to the camera at this moment; S500, update target position and speed: before processing the next frame, the position and speed of the target in the previous frame need to be updated, and the corresponding information of the current i-th frame is used for updating, and the formula is: bbox i-1 (x0,y0,x1,y1)=bbox i (x0,y0,x1,y1) c i-1 (x,y,z)=c i (x,y,z) v i-1 (x,y,z)=v i (x,y,z) S600: After the update is completed, the cycle of S100 to S500 is repeated for the new i+1 frame data.

2. A fast and accurate real-time moving target position estimation method according to claim 1, characterized in that: In the S100, the target detector based on deep learning is the YOLO series.

3. A fast and accurate real-time moving target position estimation method according to claim 1, characterized in that: In the above S200, the method for performing inter-frame target matching is to calculate the intersection-over-union ratio of the target positions between frames and determine whether it is greater than a threshold or any one of the matching methods based on the target appearance features.

4. A fast and accurate real-time moving target position estimation method according to claim 1, characterized in that: In the above-mentioned S300, the speed is filtered using weighted filtering: v i (x,y,z)=a*v i-1 (x,y,z)+(1-a)*v i (x,y,z) Where: v i-1 (x, y, z) is the target speed corresponding to the i-1th frame, a is the weight factor, ranging from (0, 1), which represents the influence of the previously estimated speed and the estimated speed of the current frame.

5. Application of a fast and accurate real-time moving target position estimation method, characterized in that: A fast and accurate real-time moving target position estimation method as described in any one of claims 1 to 4 is used in any scenario where the position of a target in motion needs to be estimated.

6. A laser weeding robot, characterized in that: include: Memory for storing computer programs; A processor, used to implement the fast and accurate real-time moving target position estimation method as claimed in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for fast and accurate real-time moving target position estimation as claimed in any one of claims 1 to 4 is implemented.

Images