Target tracking method for rotating mirror system based on optical flow aided prediction

Abstract

The invention discloses a rotating mirror system target tracking method for optical flow auxiliary prediction, and the method comprises the steps: obtaining a target image in real time through a rotating mirror system, and dividing the image into a plurality of regions through uniform grid division; and based on the image gray values of the current frame and the previous frame, matching each region by adopting a mean shift algorithm to obtain an initial center coordinate and an optical flow vector. And the optical flow vector is iteratively optimized, so that the luminosity consistency error is minimized, and the accurate target position and motion information are obtained. And predicting the target position of the next frame according to the target position and the optical flow component thereof, and correcting the deflection angle in combination with the optical flow residual error. According to the method, efficient target tracking can be carried out in a complex dynamic environment, and continuous and accurate positioning of the rotating mirror system on the target is ensured. By adopting the method, the target tracking precision and the system response speed can be remarkably improved, the robustness and the real-time performance are high, and the method is suitable for high-precision target tracking application.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous target tracking, and in particular relates to a target tracking method of a rotating mirror system using optical flow-assisted prediction. Background Art

[0002] Target tracking through rotating mirrors is a core research area in computer vision, bionic perception and intelligent systems, and is widely used in scenarios such as autonomous driving, robot navigation, drone flight, and security monitoring. In traditional solutions, the realization of target tracking usually relies on mechanical motion systems or electronic scanning devices. Although these methods can complete the line of sight adjustment, they have slow response speeds. Due to the influence of inertia, the speed of adjusting the line of sight of the mechanical system is limited, and it is difficult to meet the needs of fast target tracking in dynamic scenes. The optical line of sight control technology based on the rotating mirror structure has gradually been popularized. The rotating mirror system adjusts the relative rotation angle of the two prisms to achieve rapid deflection of the line of sight, avoiding mechanical rotation or large-scale scanning. The speed of optical adjustment is much faster than mechanical movement, and it can adapt to rapidly changing scenes. However, the prism control accuracy directly affects the line of sight adjustment effect, and the prism angle change needs to be accurately calculated and controlled. Fast-moving targets and complex backgrounds will interfere with target positioning and line of sight adjustment, affecting the tracking performance of the system. A single optical perspective is difficult to provide information on the depth or motion trend of the target, which may cause tracking lag or loss. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a target tracking method of a rotating mirror system with optical flow assisted prediction.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] The present invention provides a method for tracking a target in a rotating mirror system using optical flow-assisted prediction, comprising the following steps:

[0006] Step S1: acquiring a target image in real time through a rotating mirror system, and dividing a current frame of the target image into a plurality of regions by using uniform grid division, wherein the rotating mirror system comprises a rotating double prism, and the rotating double prism comprises two prisms;

[0007] Step S2: For each region, the mean shift algorithm is used to match the regions of the current frame with the regions of the previous frame according to the image grayscale value of the current frame and the image grayscale value of the previous frame, so as to obtain the initial center coordinates of each region of the current frame;

[0008] Step S3: Calculate the initial optical flow vector of the initial center coordinates;

[0009] Step S4: iterating the center coordinates and the optical flow vectors of each region through the initial center coordinates and the initial optical flow vectors of each region until the photometric consistency error of the image is less than a first preset value, stopping the iteration, and obtaining the center coordinates and the optical flow vectors of each region of the current frame;

[0010] Step S5: Calculate the optical flow vectors of other coordinate points except the center coordinates in each area of ​​the current frame;

[0011] Step S6: judging the target area according to the optical flow vector of the center coordinates of each area of ​​the current frame, and calculating the target position coordinates of the current frame according to the target area;

[0012] Step S7: predicting the target position coordinates of the next frame by combining the target position coordinates of the current frame with the optical flow calculation prediction formula;

[0013] Step S8: Calculating the rotation angles of the two prisms in the rotating mirror system according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame;

[0014] Step S9: Calculate the optical flow residual according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and calculate the deflection angle correction amount of the rotating mirror system according to the optical flow residual;

[0015] Step S10: Calculating the rotation angle correction amount of the two prisms in the rotating mirror system according to the deflection angle correction amount of the rotating mirror system, and updating the deflection angle and rotation angle of the rotating mirror system according to the deflection angle correction amount and the rotation angle correction amount.

[0016] Furthermore, the step S2 includes the following steps:

[0017] For each small area, the mean shift algorithm is used to find the area in the previous frame that is most similar to the current area, and its center coordinates are used as the matching position of the current area. The mean shift algorithm formula is:

[0018]

[0019] Among them, (x gi ,y gi ) is the initial center coordinate of the i-th region, region i represents the pixel coordinate set in the ith region, F(x,y)=(u x,y ,v x,y ), F(x,y) is the optical flow vector obtained by the optical flow estimation formula of the pixel point (x,y), u x,y 、v x,y Respectively represent the horizontal motion component and vertical motion component at the pixel point (x, y);

[0020] The u x,y、v x,y The spatial gradient I of the image is calculated based on the image grayscale value I(x, y, t) of the current frame and the image grayscale value I(x, y, t-1) of the previous frame. xt ,I yt With time gradient I t It is obtained by the optical flow estimation formula, which is:

[0021] I xt u x,y +I yt v x,y +I t =0

[0022] Where I represents the image grayscale value of the dual prism imaging system, I(x,y,t) is the image grayscale value of the pixel point (x,y) at time t, i.e., the current time, I(x,y,t-1) is the image grayscale value of the pixel point (x,y) at time t-1, i.e., the previous time, and I xt ,I yt are the gradient values ​​of the image along x and y in space obtained at time t, I t The temporal gradient value obtained at time t.

[0023] Furthermore, the formula for calculating the initial optical flow vector of the initial center coordinate is:

[0024]

[0025] Among them, (u gi1 ,v gi1 ) is the initial center coordinate (x gi ,y gi ) is the initial optical flow vector, M is the region i The number of pixel coordinates in , N represents the region matching weight.

[0026] Furthermore, the center coordinates of each region and its optical flow vector are iterated, and the formula is:

[0027]

[0028] Among them, u gi (k) 、u gi (k) are the horizontal and vertical motion components of the center coordinates of the i-th region in the k-th iteration, respectively, u gi (k-1) 、v gi (k-1) are the horizontal and vertical motion components of the center coordinates of the ith region in the k-1th iteration, respectively. is the center coordinate of the i-th region in the k+l-th iteration, I xt ,I yt are the gradient values ​​of the image along x and y in space at time t, i.e., the current time, and I t It is the temporal gradient value obtained at time t, i.e. the current time.

[0029] Furthermore, the optical flow vectors of other coordinate points except the center coordinates in each area of ​​the current frame are calculated by the formula:

[0030]

[0031] Among them, u i,j 、v i,j They are respectively region i In addition to the center coordinates, the horizontal motion component and vertical motion component of the j-th pixel in the current frame, ω i is the brightness weight of the ith region, I xtj ,I ytj For region i The image grayscale gradient value of the j-th pixel in the current frame and the previous frame in the x and y directions except the center coordinates, I tj is the region in the current frame i The temporal gradient value of the j-th pixel except the center coordinate.

[0032] Further, the step S6 includes the following steps:

[0033] Calculate the optical flow vector (u gi ,v gi ) is greater than the threshold τ, the formula is:

[0034]

[0035] Among them, u gi 、v gi are the horizontal motion component and vertical motion component corresponding to the center coordinates of the i-th region respectively;

[0036] If it is greater than the threshold τ, the region i is marked as the target area. If it is less than or equal to the threshold τ, the region i is marked as the target is not in the area. The center coordinates of the areas marked as the target area are averaged and summed to obtain the target position coordinates of the current frame. The formula is:

[0037]

[0038] Among them, (x c ,y c ) is the target position coordinate of the current frame, (xgi ,y gi ) is the center coordinate of the area marked as the target area, and L is the number of areas marked as the target area.

[0039] Furthermore, the target position coordinates of the current frame are combined with the optical flow calculation prediction formula to predict the target position coordinates of the next frame, and the formula is:

[0040]

[0041] Among them, (x p ,y p ) is the predicted target position coordinate of the next frame, (x c ,y c ) is the target position coordinate of the current frame, u c 、v c are the optical flow components of the target position coordinates in the current frame, representing the target's movement speed in the x and y directions respectively, and are obtained by querying the optical flow vectors of other coordinate points except the center coordinates in each area of ​​the current frame in step S5, and Δt is the time difference between the current frame and the next frame.

[0042] Furthermore, the rotation angles of the two prisms in the rotating mirror system are calculated according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and the formula is:

[0043]

[0044] Among them, α and β are the rotation angles of the first and second prisms in the rotating mirror system. are the azimuths of the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and Δθ is the angle between the two prisms.

[0045] Furthermore, the optical flow residual is calculated based on the current frame target position coordinates and the predicted next frame target position coordinates, and the formula is:

[0046] Δx r =x p -x c

[0047] Δy r =y p -y c

[0048] Where Δx r , Δy r They are the optical flow residuals of the target in the horizontal and vertical directions of the image respectively;

[0049] The deflection angle correction amount of the rotating mirror system is calculated according to the optical flow residual, and the formula is:

[0050]

[0051] Among them, Δθ x , Δθ y are the visual axis deflection angle corrections in the x and y directions respectively, and f is the focal length of the industrial camera.

[0052] Furthermore, the rotation angle correction amount of the two prisms in the rotating mirror system is calculated according to the deflection angle correction amount of the rotating mirror system, and the formula is:

[0053]

[0054] Among them, Δα and Δβ are the rotation angle corrections of the first prism and the second prism respectively, and Δθ x , Δθ y are the correction values ​​of the visual axis deflection angle of the rotating mirror system in the x and y directions, K p1 , K i1 , K d1 They are the proportional, integral and differential gains of the inner loop feedback control respectively;

[0055] The deflection angle and rotation angle of the rotating mirror system are updated according to the deflection angle correction amount and the rotation angle correction amount, and the formula is:

[0056]

[0057] Among them, θ x ,θ y are the initial deflection angles of the mirror system in the x and y directions, θ x ′、θ y ′ are the deflection angles of the updated rotating mirror system in the x and y directions, Δθ x , Δθ y are the visual axis deflection angle corrections of the rotating mirror system in the x and y directions, respectively. α′ and β′ are the rotation angles of the first and second prisms in the updated double prism of the rotating mirror system, respectively.

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] (1) The mirror structure is used to achieve rapid control of the visual axis by optical means, shortening the response time of target tracking. The optical flow error correction method is introduced in the inverse solution process to perform vector calculation estimation. The characteristics of the motion vector can be predicted by optical flow, and the motion information of the next frame is provided to the mirror tracking system in the current frame, thereby improving the tracking response speed in conjunction with the mirror reverse ray tracing and prism angle solution theory.

[0060] (2) In dynamic scenes, the target’s motion trajectory may be disturbed by noise, occlusion or changes in illumination, resulting in a decrease in the accuracy of traditional tracking methods. The present invention combines optical flow information and dynamically corrects the mirror control parameters to achieve high-precision prediction of target motion and adjustment of the visual axis, thereby ensuring tracking accuracy. The rapid movement of targets in dynamic scenes, multi-target interaction or complex background changes often pose challenges to the robustness of the tracking system. The dual-drive target information processing framework fused with optical flow estimation enables the mirror tracking system to achieve stable tracking in complex environments.

[0061] (3) Make full use of image motion information to optimize target tracking, and use the scale fusion algorithm to make the rotation efficiency of the rotating mirror higher, avoiding problems such as spiral, jamming, and slowness during the rotating mirror tracking process, making the target tracking faster and smoother.

[0062] (4) The present invention uses a combination of optical flow estimation algorithm and mean shift algorithm to accurately match the target area of ​​the current frame with the previous frame, thereby obtaining the target's motion trajectory and position change. Through iterative optimization of optical flow vectors, the error in optical flow estimation can be effectively eliminated, thereby improving the accuracy of target positioning. This technical means can calculate the motion state of the target in each frame in real time, thereby achieving high-precision target tracking.

[0063] (5) The present invention iteratively optimizes the optical flow vectors of each region to ensure that each updated optical flow vector can better meet the photometric consistency requirements of the image. The iteration is stopped when the optical flow consistency error is less than a preset value, which further enhances the stability and robustness of the tracking process, and is particularly suitable for target tracking in complex backgrounds and dynamically changing environments.

[0064] (6) The present invention combines a rotating mirror system with a rotating dual prism, calculates the deviation between the target position and the predicted position, and adjusts the prism rotation angle in the rotating mirror system to achieve precise control of the target position. Through precise angle correction, the rotating mirror system can adjust the observation angle in real time to ensure that the target can be continuously and accurately tracked in complex motion scenes.

[0065] (7) The present invention introduces a PID control algorithm to perform real-time correction on the deflection angle in the rotating mirror system, and can quickly adjust when a target position prediction error occurs. PID control not only improves the response speed of the system, but also ensures continuous and stable tracking of the target in a dynamic environment, avoiding tracking failures caused by system hysteresis or inaccuracy.

[0066] (8) By dividing the target image into multiple uniform grid regions and using local optical flow estimation and region matching methods, the present invention can perform optical flow analysis in a smaller region, thereby reducing the computational complexity and improving the efficiency of target recognition and positioning. In addition, the calculation of optical flow vectors is combined with region determination, which effectively reduces the interference to non-target regions and improves the accuracy and real-time performance of target detection.

[0067] (9) In the target position prediction, the present invention optimizes the prediction of the target position by calculating the optical flow residual, ensuring that the system predicts the target more accurately in each frame. By correcting the prediction residual, the target position prediction error can be further reduced, ensuring the consistency between the rotation angle of the rotating mirror system and the actual position of the target, and improving the overall tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flow chart of the method of the present invention;

[0069] Figure 2 A schematic diagram of dual-mirror rapid target tracking using optical flow-assisted prediction of the present invention;

[0070] Figure 3 A schematic diagram of pixel motion and dual prism angle prediction and updating according to the present invention;

[0071] Figure 4 It is a tracking position comparison diagram of the present invention;

[0072] Figure 5 This is a tracking position error diagram of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0074] Embodiment 1:

[0075] This embodiment provides a method for tracking a target in a rotating mirror system using optical flow-assisted prediction. Figure 1 As shown, the following steps are included:

[0076] Step S1: acquiring a target image in real time through a rotating mirror system, and dividing a current frame of the target image into a plurality of regions by using uniform grid division, wherein the rotating mirror system includes a rotating double prism, and the rotating double prism includes two prisms;

[0077] Step S2: For each region, the mean shift algorithm is used to match the regions of the current frame with the regions of the previous frame according to the image grayscale value of the current frame and the image grayscale value of the previous frame, so as to obtain the initial center coordinates of each region of the current frame;

[0078] Step S3: Calculate the initial optical flow vector of the initial center coordinates;

[0079] Step S4: iterating the center coordinates and the optical flow vectors of each region through the initial center coordinates and the initial optical flow vectors of each region until the photometric consistency error of the image is less than a first preset value, stopping the iteration, and obtaining the center coordinates and the optical flow vectors of each region of the current frame;

[0080] Step S5: Calculate the optical flow vectors of other coordinate points except the center coordinates in each area of ​​the current frame;

[0081] Step S6: judging the target area according to the optical flow vector of the center coordinates of each area of ​​the current frame, and calculating the target position coordinates of the current frame according to the target area;

[0082] Step S7: predicting the target position coordinates of the next frame by combining the target position coordinates of the current frame with the optical flow calculation prediction formula;

[0083] Step S8: Calculating the rotation angles of the two prisms in the rotating mirror system according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame;

[0084] Step S9: Calculate the optical flow residual according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and calculate the deflection angle correction amount of the rotating mirror system according to the optical flow residual;

[0085] Step S10: Calculating the rotation angle correction amount of the two prisms in the rotating mirror system according to the deflection angle correction amount of the rotating mirror system, and updating the deflection angle and rotation angle of the rotating mirror system according to the deflection angle correction amount and the rotation angle correction amount.

[0086] Wherein, step S2 comprises the following steps:

[0087] For each small area, the mean shift algorithm is used to find the area in the previous frame that is most similar to the current area, and its center coordinates are used as the matching position of the current area. The mean shift algorithm formula is:

[0088]

[0089] Among them, (x gi ,y gi ) is the initial center coordinate of the i-th region, region i represents the pixel coordinate set in the ith region, F(x,y)=(u x,y ,vx,y ), F(x,y) is the optical flow vector obtained by the optical flow estimation formula of the pixel point (x,y), u x,y 、v x,y Respectively represent the horizontal motion component and vertical motion component at the pixel point (x, y);

[0090] u x,y 、v x,y The spatial gradient I of the image is calculated based on the image grayscale value I(x, y, t) of the current frame and the image grayscale value I(x, y, t-1) of the previous frame. xt ,I yt With time gradient I t It is obtained through the optical flow estimation formula, which is:

[0091] I xt u x,y +I yt v x,y +I t =0

[0092] Where I represents the image grayscale value of the dual prism imaging system, I(x,y,t) is the image grayscale value of the pixel point (x,y) at time t, i.e., the current time, I(x,y,t-1) is the image grayscale value of the pixel point (x,y) at time t-1, i.e., the previous time, and I xt ,I yt are the gradient values ​​of the image along x and y in space obtained at time t, I t The temporal gradient value obtained at time t.

[0093] Among them, the initial optical flow vector of the initial center coordinate is calculated as follows:

[0094]

[0095] Among them, (u gi1 ,v gi1 ) is the initial center coordinate (x gi ,y gi ) is the initial optical flow vector, M is the region i The number of pixel coordinates in , N represents the region matching weight.

[0096] Among them, the center coordinates of each area and its optical flow vector are iterated, and the formula is:

[0097]

[0098] Among them, u gi (k) 、u gi (k)are the horizontal and vertical motion components of the center coordinates of the i-th region in the k-th iteration, respectively, u gi (k-1) 、v gi (k-1) are the horizontal and vertical motion components of the center coordinates of the ith region in the k-1th iteration, respectively. is the center coordinate of the i-th region in the k+l-th iteration, I xt ,I yt are the gradient values ​​of the image along x and y in space at time t, i.e., the current time, and I t It is the temporal gradient value obtained at time t, i.e. the current time.

[0099] Among them, the optical flow vector of other coordinate points in each area of ​​the current frame except the central coordinate is calculated, and the formula is:

[0100]

[0101] Among them, u i,j 、v i,j They are respectively region i In addition to the center coordinates, the horizontal motion component and vertical motion component of the j-th pixel in the current frame, ω i is the brightness weight of the ith region, I xtj ,I ytj For region i The image grayscale gradient value of the j-th pixel in the current frame and the previous frame in the x and y directions except the center coordinates, I tj is the region in the current frame i The temporal gradient value of the j-th pixel except the center coordinate.

[0102] Wherein, step S6 comprises the following steps:

[0103] Calculate the optical flow vector (u gi ,v gi ) is greater than the threshold τ, the formula is:

[0104]

[0105] Among them, u gi 、v gi are the horizontal motion component and vertical motion component corresponding to the center coordinates of the i-th region respectively;

[0106] If it is greater than the threshold τ, the region i is marked as the target area. If it is less than or equal to the threshold τ, the region i is marked as the target is not in the area. The center coordinates of the areas marked as the target area are averaged and summed to obtain the target position coordinates of the current frame. The formula is:

[0107]

[0108] Among them, (x c ,y c ) is the target position coordinate of the current frame, (x gi ,y gi ) is the center coordinate of the area marked as the target area, and L is the number of areas marked as the target area.

[0109] Among them, the target position coordinates of the current frame are combined with the optical flow calculation prediction formula to predict the target position coordinates of the next frame. The formula is:

[0110]

[0111] Among them, (x p ,y p ) is the predicted target position coordinate of the next frame, (x c ,y c ) is the target position coordinate of the current frame, u c 、v c are the optical flow components of the target position coordinates in the current frame, representing the target's movement speed in the x and y directions respectively, and are obtained by querying the optical flow vectors of other coordinate points except the center coordinates in each area of ​​the current frame in step S5, and Δt is the time difference between the current frame and the next frame.

[0112] Among them, the rotation angle of the two prisms in the rotating mirror system is calculated according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame. The formula is:

[0113]

[0114] Among them, α and β are the rotation angles of the first and second prisms in the rotating mirror system. are the azimuths of the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and Δθ is the angle between the two prisms.

[0115] Among them, the optical flow residual is calculated based on the current frame target position coordinates and the predicted next frame target position coordinates. The formula is:

[0116] Δx r =x p -x c

[0117] Δy r =y p -y c

[0118] Where Δx r , Δy r They are the optical flow residuals of the target in the horizontal and vertical directions of the image respectively;

[0119] The deflection angle correction of the rotating mirror system is calculated based on the optical flow residual. The formula is:

[0120]

[0121] Among them, Δθ x , Δθ y are the visual axis deflection angle corrections in the x and y directions respectively, and f is the focal length of the industrial camera.

[0122] Among them, the rotation angle correction amount of the two prisms in the rotating mirror system is calculated according to the deflection angle correction amount of the rotating mirror system, and the formula is:

[0123]

[0124] Among them, Δα and Δβ are the rotation angle corrections of the first prism and the second prism respectively, and Δθ x , Δθ y are the correction values ​​of the visual axis deflection angle of the rotating mirror system in the x and y directions, K p1 , K i1 , K d1 They are the proportional, integral and differential gains of the inner loop feedback control respectively;

[0125] The deflection angle and rotation angle of the rotating mirror system are updated according to the deflection angle correction and the rotation angle correction. The formula is:

[0126]

[0127] Among them, θ x ,θ y are the initial deflection angles of the mirror system in the x and y directions, θ x ′、θ y ′ are the deflection angles of the updated rotating mirror system in the x and y directions, Δθ x , Δθ y are the visual axis deflection angle corrections of the rotating mirror system in the x and y directions, respectively. α′ and β′ are the rotation angles of the first and second prisms in the updated double prism of the rotating mirror system, respectively.

[0128] Embodiment 2:

[0129] The parts not mentioned in this embodiment are the same as those in Embodiment 1.

[0130] The target tracking system of this embodiment includes the following modules: a rotating mirror module, an imaging module, a computing unit, and a servo control system. As shown in the figure, the rotating mirror module is demonstrated by taking a rotating dual prism as an example. The rotating dual prism includes two prisms, which rotate along two independent axes perpendicular to the optical axis to achieve dynamic adjustment of the field of view. The rotation angle of each prism is represented by α and β respectively; the imaging module is composed of a CCD industrial camera for capturing targets; the computing unit includes an optical flow calculation module and a motion prediction module. The optical flow calculation is responsible for predicting the target motion vector, and the motion prediction module calculates the target position; the servo control system includes driving the dual prism to rotate, adjusting the visual axis in real time according to the predicted target position, and keeping the target in the imaging center. Figure 2 Figure 3 As shown in the figure, the high-precision target tracking algorithm based on optical flow prediction and correction includes the following steps:

[0131] Step 1: The brightness value of the pixel at coordinate (x, y) at time t can be expressed as I(x, y, t). After Δt time to the current frame, the brightness value of the pixel at (x+Δx, y+Δy) is I(x+Δx, y+Δy, t+Δt), so it can be expressed as and They represent the optical flow components of the target along the X and Y directions in the two-dimensional coordinate system of the image. At time t, the constraint equation of the optical flow is as follows:

[0132] I x u+I y v+I t =0

[0133] Where: I represents the image grayscale value of the dual prism imaging system; I x ,I y Indicates the gradient value of the image along x and y in space at time t; I t is the gradient value in time. In order to efficiently solve the above equation, the local area fitting method is used to solve the optical flow.

[0134] Step 2: In order to efficiently solve the above equation, the local area fitting method is used to solve the optical flow. First, the image is divided into several small areas, each of which contains multiple pixels. Uniform grid division is used, and for each small area, an optical flow vector needs to be calculated to represent the movement of this area.

[0135] Step 3: For each small area, it is necessary to match its position difference Δ(x i ,y i ). The mean shift algorithm can be used to find the area in the previous frame that is most similar to the current small area, and its center coordinates are used as the matching position of the current small area. The solution formula using the mean shift algorithm is:

[0136]

[0137] Among them, (x gi ,y gi ) is the initial center coordinate of the i-th region, region i represents the pixel coordinate set in the ith region, F(x,y)=(u x,y ,v x,y ), F(x,y) is the optical flow vector obtained by the optical flow estimation formula of the pixel point (x,y), u x,y 、v x,y Respectively represent the horizontal motion component and vertical motion component at the pixel point (x, y);

[0138] u x,y 、v x,y The spatial gradient I of the image is calculated based on the image grayscale value I(x, y, t) of the current frame and the image grayscale value I(x, y, t-1) of the previous frame. xt ,I yt With time gradient I t It is obtained through the optical flow estimation formula, which is:

[0139] I xt u x,y +I yt v x,y +I t =0

[0140] Where I represents the image grayscale value of the dual prism imaging system, I(x,y,t) is the image grayscale value of the pixel point (x,y) at time t, i.e., the current time, I(x,y,t-1) is the image grayscale value of the pixel point (x,y) at time t-1, i.e., the previous time, and I xt ,I yt are the gradient values ​​of the image along x and y in space obtained at time t, I t The temporal gradient value obtained at time t.

[0141] After completing the region matching, the pixel-based optical flow method is used to calculate the optical flow vector of each pixel, and then these vectors are averaged to obtain the (x gi ,y gi ) is the initial optical flow vector of the center coordinate:

[0142]

[0143] Among them, (u gi1 ,v gi1 ) is the initial center coordinate (x gi ,y gi) is the initial optical flow vector, M is the region i The number of pixel coordinates in , N represents the region matching weight.

[0144] Step 4: Once the motion vectors of each small area are obtained, they can be used to track the movement of the object. The position of the small area in the previous frame can be moved to the current frame according to its motion vector, and then the optical flow vector of the small area can be recalculated in the current frame. The position of each small area in the previous frame is moved to the current frame according to its motion vector, and then the optical flow vector of the small area is recalculated in the current frame. Through iterative calculation, the photometric consistency error of the image is Indicates convergence, the iterative formulas of u and v are as follows:

[0145] The center coordinates of each area and its optical flow vector are iterated, and the formula is:

[0146]

[0147] Among them, u gi (k) 、u gi (k) are the horizontal and vertical motion components of the center coordinates of the i-th region in the k-th iteration, respectively, u gi (k-1) 、v gi (k-1) are the horizontal and vertical motion components of the center coordinates of the ith region in the k-1th iteration, respectively. is the center coordinate of the i-th region in the k+l-th iteration, I xt ,I yt are the gradient values ​​of the image along x and y in space at time t, i.e., the current time, and I t It is the temporal gradient value obtained at time t, i.e. the current time.

[0148] Step 5: Combine the optical flow vectors of all small areas to obtain the optical flow field of the entire image. i ∈r i The optical flow vector (u i,j ,v i,j ), the movement speed of other coordinate points except the obtained central coordinate point can be calculated in matrix form:

[0149]

[0150] Among them, u i,j 、v i,j Respectively, the regional i In addition to the center coordinates, the horizontal motion component and vertical motion component of the j-th pixel in the current frame, ωi is the brightness weight of the ith region, I xtj ,I ytj For regional regiom i The image grayscale gradient value of the j-th pixel in the current frame and the previous frame in the x and y directions except the center coordinates, I tj is the region in the current frame i The temporal gradient value of the j-th pixel except the center coordinate.

[0151] The regional optical flow estimation method uses the local similarity in the image to improve the accuracy and stability of optical flow estimation. Combined with the pixel-based optical flow method, it can effectively reduce the noise and outliers in the optical flow estimation and has better robustness in processing complex scenes. After calculating V = (u, v) accurately, the optical flow field can be used to predict the target motion trajectory.

[0152] Step 6: Use the time integral of the optical flow field to predict the target's motion position in the next frame. The region whose optical flow motion vector value at the center coordinate point of the region is greater than the threshold τ is detected as the target, and the weighted average of the center points of all regions is determined as the current target position (x c ,y c ), as shown in the formula:

[0153]

[0154] Among them, (x c ,y c ) is the target position coordinate of the current frame, (x gi ,y gi ) is the center coordinate of the area marked as the target area, and L is the number of areas marked as the target area.

[0155] Combined with the optical flow calculation prediction formula, the position of the next frame (x p ,y p ) can be obtained through speed points:

[0156]

[0157] Among them, (x p ,y p ) is the predicted target position coordinate of the next frame, (x c ,y c ) is the target position coordinate of the current frame, u c 、v c are the optical flow components of the target position coordinates in the current frame, representing the target's movement speed in the x and y directions respectively, and are obtained by querying the optical flow vectors of other coordinate points except the center coordinates in each area of ​​the current frame in step S5, and Δt is the time difference between the current frame and the next frame.

[0158] Since the target motion in real scenes is complex, there is acceleration. By introducing the optical flow dynamic error model, the optical flow prediction trajectory can be combined with the target tracking system to achieve long-term prediction and correction of complex motion.

[0159] Step 7: The rotation angles α and β of the dual prism need to be adjusted according to the predicted position of the target so that the target is always in the center of the imaging field of view. The target point in the current frame is (x c ,y c ), the target prediction point is (x p ,y p ). The two-step method is used to solve the rotation angle solution, which is obtained by the following formula:

[0160]

[0161] Among them, α and β are the rotation angles of the first and second prisms in the rotating mirror system. are the azimuths of the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and Δθ is the angle between the two prisms.

[0162] Step 8: When the optical flow method predicts the target trajectory, due to the uncertainty of the prediction, there may be a deviation between the actual position of the target and the predicted position. This deviation is determined by the optical flow residual estimation formula. The optical flow residual error is directly converted into the visual axis deflection angle correction in the x and y directions:

[0163] The optical flow residual is calculated based on the current frame target position coordinates and the predicted next frame target position coordinates. The formula is:

[0164] Δx r =x p -x c

[0165] Δy r =y p -y c

[0166] Where Δx r , Δy r They are the optical flow residuals of the target in the horizontal and vertical directions of the image respectively;

[0167] The deflection angle correction of the rotating mirror system is calculated based on the optical flow residual. The formula is:

[0168]

[0169] Among them, Δθ x , Δθ yare the visual axis deflection angle corrections in the x and y directions respectively, and f is the focal length of the industrial camera.

[0170] Step 10: In order to ensure that the dual prism system and the optical flow method can be integrated with high precision and high dynamics for target tracking, a refined optimization control strategy needs to be designed. The fast inner loop feedback is centered on the prism angle adjustment and focuses on the rapid compensation of the instantaneous offset of the target. The real-time offset error Δx r ,Δy r Minimize. Use the following control strategy for the prism deflection angle (α, γ):

[0171]

[0172] Among them, Δα and Δβ are the rotation angle corrections of the first prism and the second prism respectively, and Δθ x , Δθ y are the correction values ​​of the visual axis deflection angle of the rotating mirror system in the x and y directions, K p1 , K i1 , K d1 They are the proportional, integral and differential gains of the inner loop feedback control respectively;

[0173] Then update the prism's deflection angle:

[0174]

[0175] Among them, θ x ,θ y are the initial deflection angles of the mirror system in the x and y directions, θ x ′、θ y ′ are the deflection angles of the updated rotating mirror system in the x and y directions, Δθ x , Δθ y are the visual axis deflection angle corrections of the rotating mirror system in the x and y directions, respectively. α′ and β′ are the rotation angles of the first and second prisms in the updated double prism of the rotating mirror system, respectively.

[0176] Step 12: The outer loop feedback predicts the global motion trajectory of the target based on the optical flow method, and forms a closed-loop control with the prism system for long-term trend correction and error compensation. The outer loop control adopts model predictive control (MPC) to dynamically optimize the prism angle solution:

[0177]

[0178] in represents the actual deflection angle at the kth moment; θ x [k],θ y[k] The desired target deflection angle. The prism trajectory is corrected at each time step based on the optimized prism angle adjustment (Δα, Δβ) output by MPC.

[0179] Step 13: The optical flow target state vector can be expressed as X = [x, y, u, v, a x ,a y ], so the motion model can be expressed as

[0180] X t+1 =AX t +BU t +W

[0181] Among them, A represents the state transfer matrix, U represents the control input vector, B represents the control input matrix, and W represents the noise matrix.

[0182] The measurement vector is the target optical flow offset and the actual feedback angle of the prism:

[0183] Z t =[Δx r ,Δy r ,α,β] T

[0184] Therefore, the optical flow prediction and update equation are performed by the following steps:

[0185]

[0186] Among them, P is the prediction vector and Q is the covariance matrix.

[0187] Update phase:

[0188]

[0189] Among them, H is the observation matrix and I is the identity matrix.

[0190] Step 14: Adaptively adjust the PID gain parameter K by analyzing the optical flow residual and prism response speed p ,K i ,K d , that is, K. If the target moves quickly, increase K d To improve the system response speed; if the error continues to accumulate, increase K i To compensate for the deviation; in the stable stage, reduce K p Prevent overshoot. Utilize the statistical characteristics of observation noise to adaptively adjust the noise covariance matrix Q and R. When the noise increases, increase R to reduce the impact of the measurement value on the state; when the noise decreases, reduce Q to increase the model prediction weight. Ultimately, the purpose of correcting the dual prism tracking effect is achieved.

[0191] This method is compared with the traditional single visual tracking method. The experimental results are as follows Figure 4 Figure 5 As shown in the figure, the experimental results show that the use of optical flow combined with dual prism angle control can achieve more accurate target prediction and adjustment. When the optical flow method is used to predict the target position, it can effectively improve the system's response speed to dynamic targets and reduce the error caused by the prism adjustment lag. In addition, by introducing the optical flow method to estimate the target trajectory, it can reduce interference in multi-target scenes and stably track the main target. Compared with the traditional dual prism feedback tracking method that does not use the optical flow method, the system using the optical flow method shows better stability and robustness under complex nonlinear trajectories and dynamic noise interference. In the experiment, by comparing the tracking results without the optical flow method, the optical flow method can more accurately predict the target's movement trend, greatly reducing the tracking error and hysteresis effect caused by static observation, especially when the target moves quickly or is blocked, the system shows higher tracking accuracy and response speed.

[0192] In general, compared with the traditional target tracking method based on visual feedback, the present invention can not only effectively improve the target tracking accuracy, but also optimize the system response time by introducing optical flow prediction, thus overcoming the computational burden and response delay problems in the traditional method. In addition, the present method can adaptively adjust the optical flow field in multi-target scenarios, further improving the robustness and accuracy in complex motion environments.

[0193] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0194] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for tracking a target in a rotating mirror system using optical flow-assisted prediction, characterized in that: The following steps are involved: Step S1: acquiring a target image in real time through a rotating mirror system, and dividing a current frame of the target image into a plurality of regions by using uniform grid division, wherein the rotating mirror system comprises a rotating double prism, and the rotating double prism comprises two prisms; Step S2: For each region, the mean shift algorithm is used to match the regions of the current frame with the regions of the previous frame according to the image grayscale value of the current frame and the image grayscale value of the previous frame, so as to obtain the initial center coordinates of each region of the current frame; Step S3: Calculate the initial optical flow vector of the initial center coordinates; Step S4: iterating the center coordinates and the optical flow vectors of each region through the initial center coordinates and the initial optical flow vectors of each region until the photometric consistency error of the image is less than a first preset value, stopping the iteration, and obtaining the center coordinates and the optical flow vectors of each region of the current frame; Step S5: Calculate the optical flow vectors of other coordinate points except the center coordinates in each area of ​​the current frame; Step S6: judging the target area according to the optical flow vector of the center coordinates of each area of ​​the current frame, and calculating the target position coordinates of the current frame according to the target area; Step S7: predicting the target position coordinates of the next frame by combining the target position coordinates of the current frame with the optical flow calculation prediction formula; Step S8: Calculating the rotation angles of the two prisms in the rotating mirror system according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame; Step S9: Calculate the optical flow residual according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and calculate the deflection angle correction amount of the rotating mirror system according to the optical flow residual; Step S10: Calculating the rotation angle correction amount of the two prisms in the rotating mirror system according to the deflection angle correction amount of the rotating mirror system, and updating the deflection angle and rotation angle of the rotating mirror system according to the deflection angle correction amount and the rotation angle correction amount.

2. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, characterized in that: The step S2 comprises the following steps: For each small area, the mean shift algorithm is used to find the area in the previous frame that is most similar to the current area, and its center coordinates are used as the matching position of the current area. The mean shift algorithm formula is: Among them, (x gi ,y gi ) is the initial center coordinate of the i-th region, region i represents the pixel coordinate set in the ith region, F(x,y)=(u x,y ,v x,y ), F(x,y) is the optical flow vector obtained by the optical flow estimation formula of the pixel point (x,y), u x,y 、v x,y Respectively represent the horizontal motion component and vertical motion component at the pixel point (x, y); The u x,y 、v x,y The spatial gradient I of the image is calculated based on the image grayscale value I(x, y, t) of the current frame and the image grayscale value I(x, y, t-1) of the previous frame. xt ,I yt With time gradient I t It is obtained by the optical flow estimation formula, which is: I xt u x,y +I yt v x,y +I t =0 Where I represents the image grayscale value of the dual prism imaging system, I(x,y,t) is the image grayscale value of the pixel point (x,y) at time t, i.e., the current time, I(x,y,t-1) is the image grayscale value of the pixel point (x,y) at time t-1, i.e., the previous time, and I xt ,I yt are the gradient values ​​of the image along x and y in space obtained at time t, I t The temporal gradient value obtained at time t.

3. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1 or 2, characterized in that: The formula for calculating the initial optical flow vector of the initial center coordinate is: Among them, (u gi1 ,v gi1 ) is the initial center coordinate (x gi ,y gi ) is the initial optical flow vector, M is the region i The number of pixel coordinates in , N represents the region matching weight.

4. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, characterized in that: The central coordinates of each region and its optical flow vector are iterated, and the formula is: in, are the horizontal and vertical motion components of the center coordinates of the ith region in the kth iteration, respectively. are the horizontal and vertical motion components of the center coordinates of the ith region in the k-1th iteration, respectively. is the center coordinate of the i-th region in the k+l-th iteration, I xt ,I yt are the gradient values ​​of the image along x and y in space at time t, i.e., the current time, and I t It is the temporal gradient value obtained at time t, i.e. the current time.

5. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, characterized in that: The formula for calculating the optical flow vector of other coordinate points in each area of ​​the current frame except the central coordinate is: Among them, u i,j 、v i,j They are respectively region i In addition to the center coordinates, the horizontal motion component and vertical motion component of the j-th pixel in the current frame, ω i is the brightness weight of the i-th region, I xtj ,I ytj For region i The image grayscale gradient value of the j-th pixel in the current frame and the previous frame in the x and y directions except the center coordinates, I tj The region in the current frame i The temporal gradient value of the j-th pixel except the center coordinate.

6. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, characterized in that: The step S6 comprises the following steps: Calculate the optical flow vector (y gi ,v gi ) is greater than the threshold τ, the formula is: Among them, y gi 、v gi are the horizontal motion component and vertical motion component corresponding to the center coordinates of the i-th region respectively; If it is greater than the threshold τ, the region i is marked as the target area. If it is less than or equal to the threshold τ, the region i is marked as the target is not in the area. The center coordinates of the areas marked as the target area are averaged and summed to obtain the target position coordinates of the current frame. The formula is: Among them, (x c ,y c ) is the target position coordinate of the current frame, (x gi ,y gi ) is the center coordinate of the area marked as the target area, and L is the number of areas marked as the target area.

7. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, characterized in that: The target position coordinates of the current frame are combined with the optical flow calculation prediction formula to predict the target position coordinates of the next frame. The formula is: Among them, (x p ,y p ) is the predicted target position coordinate of the next frame, (x c ,y c ) is the target position coordinate of the current frame, u c 、v c are the optical flow components of the target position coordinates in the current frame, representing the target's movement speed in the x and y directions respectively, and are obtained by querying the optical flow vectors of other coordinate points in each area of ​​the current frame except the center coordinates in step S5, and Δt is the time difference between the current frame and the next frame.

8. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, characterized in that: The rotation angles of the two prisms in the rotating mirror system are calculated according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and the formula is: Among them, α and β are the rotation angles of the first and second prisms in the rotating mirror system. are the azimuths of the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and Δθ is the angle between the two prisms.

9. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, characterized in that: The optical flow residual is calculated based on the current frame target position coordinates and the predicted next frame target position coordinates. The formula is: Δx r =x p -x c Δy r =y p -y c Where Δx r , Δy r They are the optical flow residuals of the target in the horizontal and vertical directions of the image respectively; The deflection angle correction amount of the rotating mirror system is calculated according to the optical flow residual, and the formula is: Among them, Δθ x , Δθ y are the visual axis deflection angle corrections in the x and y directions respectively, and f is the focal length of the industrial camera.

10. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, characterized in that: The rotation angle correction amount of the two prisms in the rotating mirror system is calculated according to the deflection angle correction amount of the rotating mirror system, and the formula is: Among them, Δα and Δβ are the rotation angle corrections of the first prism and the second prism respectively, and Δθ x , Δθ y are the visual axis deflection angle corrections of the rotating mirror system in the x and y directions, K p1 , K i1 , K d1 They are the proportional, integral and differential gains of the inner loop feedback control respectively; The deflection angle and rotation angle of the rotating mirror system are updated according to the deflection angle correction amount and the rotation angle correction amount, and the formula is: Among them, θ x ,θ y are the initial deflection angles of the mirror system in the x and y directions, θ x ′、θ y ′ are the deflection angles of the updated rotating mirror system in the x and y directions, Δθ x , Δθ y are the visual axis deflection angle corrections of the rotating mirror system in the x and y directions, respectively. α′ and β′ are the rotation angles of the first and second prisms in the updated double prism of the rotating mirror system, respectively.

Images