A target tracking method for rotating mirror system using optical flow assisted prediction

Abstract

The present invention discloses a target tracking method for a rotating mirror system with optical flow assisted prediction. The method obtains a target image in real time through a rotating mirror system, and divides the image into multiple regions using uniform grid division. Based on the image grayscale values ​​of the current frame and the previous frame, a mean shift algorithm is used to match each region to obtain the initial center coordinates and optical flow vectors. By iteratively optimizing the optical flow vector, the photometric consistency error is minimized, thereby obtaining accurate target position and motion information. Based on the target position and its optical flow component, the target position of the next frame is predicted, and the deflection angle is corrected in combination with the optical flow residual. This method can perform efficient target tracking in complex dynamic environments, ensuring that the rotating mirror system continuously and accurately positions the target. The method can significantly improve the target tracking accuracy and system response speed, has strong robustness and real-time performance, and is suitable for high-precision target tracking applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous target tracking, and in particular relates to a target tracking method for a rotating mirror system using optical flow-assisted prediction. Background Art

[0002] Target tracking using rotating mirrors is a core research area in computer vision, biomimetic perception, and intelligent systems, with widespread applications in scenarios such as autonomous driving, robot navigation, drone flight, and security surveillance. Traditional approaches typically rely on mechanical motion systems or electronic scanning devices. While these methods can achieve line of sight adjustment, they suffer from slow response speeds. Mechanical systems are limited by inertia, making them incapable of meeting the demands of rapid target tracking in dynamic scenarios. Optical line of sight control technology based on rotating mirror structures is gaining popularity. Rotating mirror systems achieve rapid line of sight deflection by adjusting the relative rotation angle of two prisms, avoiding mechanical rotation or large-scale scanning. Optical adjustment is much faster than mechanical motion and can adapt to rapidly changing scenes. However, prism adjustment accuracy directly affects line of sight adjustment, requiring precise calculation and control of prism angle changes. Fast-moving targets and complex backgrounds can interfere with target positioning and line of sight adjustment, impacting system tracking performance. A single optical field of view cannot provide information about the target's depth or motion trend, potentially leading to tracking lag or loss. Summary of the Invention

[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a target tracking method of a rotating mirror system with optical flow assisted prediction.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] The present invention provides a target tracking method for a rotating mirror system using optical flow-assisted prediction, comprising the following steps:

[0006] Step S1: acquiring a target image in real time through a rotating mirror system, and dividing a current frame of the target image into a plurality of regions using uniform grid division, wherein the rotating mirror system includes a rotating biprism, and the rotating biprism includes two prisms;

[0007] Step S2: For each region, use the mean shift algorithm to match the regions of the current frame with those of the previous frame based on the image grayscale values ​​of the current frame and the image grayscale values ​​of the previous frame to obtain the initial center coordinates of each region of the current frame;

[0008] Step S3: Calculate the initial optical flow vector of the initial center coordinates;

[0009] Step S4: iterating the center coordinates and optical flow vectors of each region using the initial center coordinates and initial optical flow vectors of each region until the photometric consistency error of the image is less than a first preset value, stopping the iteration, and obtaining the center coordinates and optical flow vectors of each region in the current frame;

[0010] Step S5: Calculate the optical flow vectors of all coordinate points except the center coordinate in each area of ​​the current frame;

[0011] Step S6: judging the target area based on the optical flow vectors of the center coordinates of each area in the current frame, and calculating the target position coordinates in the current frame based on the target area;

[0012] Step S7: Predict the target position coordinates of the next frame by combining the target position coordinates of the current frame with the optical flow calculation prediction formula;

[0013] Step S8: Calculating the rotation angles of the two prisms in the rotating mirror system according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame;

[0014] Step S9: Calculating the optical flow residual according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and calculating the deflection angle correction amount of the rotating mirror system according to the optical flow residual;

[0015] Step S10: Calculating the rotation angle correction values ​​of the two prisms in the rotating mirror system according to the deflection angle correction value of the rotating mirror system, and updating the deflection angle and rotation angle of the rotating mirror system according to the deflection angle correction value and the rotation angle correction value.

[0016] Furthermore, step S2 includes the following steps:

[0017] For each small area, the mean shift algorithm is used to find the area in the previous frame that is most similar to the current area, and its center coordinates are used as the matching position of the current area. The mean shift algorithm formula is:

[0018]

[0019] Among them, (x gi ,y gi ) is the initial center coordinate of the i-th region, region i represents the pixel coordinate set in the ith region, F(x,y)=(u x,y ,v x,y ), F(x,y) is the optical flow vector obtained by the optical flow estimation formula of the pixel point (x,y), u x,y 、v x,y Represent the horizontal motion component and vertical motion component at the pixel point (x, y) respectively;

[0020] The u x,y、v x,y The spatial gradient I of the image is calculated based on the image grayscale value I(x, y, t) of the current frame and the image grayscale value I(x, y, t-1) of the previous frame. xt , I yt With time gradient I t It is obtained through the optical flow estimation formula, which is:

[0021] I xt u x,y +I yt v x,y +I t =0

[0022] Where I represents the image grayscale value of the dual-prism imaging system, I(x,y,t) is the image grayscale value of the pixel (x,y) at time t, that is, the current time, I(x,y,t-1) is the image grayscale value of the pixel (x,y) at time t-1, that is, the previous time, I xt , I yt are the gradient values ​​of the image along x and y in space obtained at time t, I t The temporal gradient value obtained at time t.

[0023] Furthermore, the formula for calculating the initial optical flow vector of the initial center coordinate is:

[0024]

[0025] Among them, (u gi1 ,v gi1 ) is the initial center coordinate (x gi ,y gi )’s initial optical flow vector, M is the region i The number of pixel coordinates in , N represents the region matching weight.

[0026] Furthermore, the center coordinates of each region and its optical flow vector are iterated, and the formula is:

[0027]

[0028] Among them, u gi (k) 、u gi (k) are the horizontal and vertical motion components of the center coordinates of the i-th region in the k-th iteration, respectively, u gi (k-1) 、v gi (k-1) are the horizontal and vertical motion components of the center coordinates of the i-th region in the k-1-th iteration, respectively. is the center coordinate of the i-th region in the k+l-th iteration, I xt , I yt They are the gradient values ​​of the image along x and y in space at time t, that is, the current time, and I t is the temporal gradient value obtained at time t, i.e. the current time.

[0029] Furthermore, the optical flow vectors of the coordinate points other than the center coordinates in each area of ​​the current frame are calculated using the formula:

[0030]

[0031] Among them, u i,j 、v i,j They are respectively region i In addition to the center coordinates, the horizontal motion component and vertical motion component of the j-th pixel in the current frame, ω i is the brightness weight of the i-th region, I xtj , I ytj For region i The image grayscale gradient value of the j-th pixel in the current frame and the previous frame in the x and y directions except the center coordinates, I tj Region in the current frame i The temporal gradient value of the j-th pixel except the center coordinate.

[0032] Furthermore, step S6 includes the following steps:

[0033] Calculate the optical flow vector (u gi ,v gi ) is greater than the threshold τ, the formula is:

[0034]

[0035] Among them, u gi 、v gi are the horizontal motion component and vertical motion component corresponding to the center coordinates of the i-th region respectively;

[0036] If it is greater than the threshold τ, then the region i is marked as the target area. If it is less than or equal to the threshold τ, then the region i is marked as the target is not in the area. The center coordinates of the areas marked as the target area are averaged and summed to obtain the target position coordinates of the current frame. The formula is:

[0037]

[0038] Among them, (x c ,y c ) is the target position coordinate of the current frame, (xgi ,y gi ) is the center coordinate of the area marked as the target area, and L is the number of areas marked as the target area.

[0039] Furthermore, the target position coordinates of the current frame are combined with the optical flow calculation prediction formula to predict the target position coordinates of the next frame. The formula is:

[0040]

[0041] Among them, (x p ,y p ) is the predicted target position coordinate of the next frame, (x c ,y c ) is the target position coordinate of the current frame, u c 、v c are the optical flow components of the target position coordinates in the current frame, representing the target's movement speed in the x and y directions respectively. They are obtained by querying the optical flow vectors of other coordinate points except the center coordinates in each area of ​​the current frame in step S5. Δt is the time difference between the current frame and the next frame.

[0042] Furthermore, the rotation angles of the two prisms in the rotating mirror system are calculated based on the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and the formula is:

[0043]

[0044] Among them, α and β are the rotation angles of the first and second prisms in the rotating mirror system. are the azimuths of the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and Δθ is the angle between the two prisms.

[0045] Furthermore, the optical flow residual is calculated based on the current frame target position coordinates and the predicted next frame target position coordinates, and the formula is:

[0046] Δx r =x p -x c

[0047] Δy r =y p -y c

[0048] Where Δx r , Δy r are the optical flow residuals of the target in the horizontal and vertical directions of the image respectively;

[0049] The deflection angle correction amount of the rotating mirror system is calculated based on the optical flow residual, and the formula is:

[0050]

[0051] Where Δθ x , Δθ y are the visual axis deflection angle corrections in the x and y directions respectively, and f is the focal length of the industrial camera.

[0052] Furthermore, the rotation angle correction amount of the two prisms in the rotating mirror system is calculated according to the deflection angle correction amount of the rotating mirror system, and the formula is:

[0053]

[0054] Among them, Δα and Δβ are the rotation angle corrections of the first prism and the second prism respectively, and Δθ x , Δθ y are the visual axis deflection angle corrections of the rotating mirror system in the x and y directions, K p1 , K i1 , K d1 are the proportional, integral and differential gains of the inner loop feedback control respectively;

[0055] The deflection angle and rotation angle of the rotating mirror system are updated according to the deflection angle correction value and the rotation angle correction value. The formula is:

[0056]

[0057] Among them, θ x ,θ y are the initial deflection angles of the mirror system in the x and y directions, θ x ′、θ y ′ are the deflection angles of the updated rotating mirror system in the x and y directions, Δθ x , Δθ y are the correction values ​​of the visual axis deflection angle of the rotating mirror system in the x and y directions, respectively. α′ and β′ are the rotation angles of the first and second prisms in the updated double prism of the rotating mirror system, respectively.

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] (1) Through the rotating mirror structure, the visual axis can be quickly adjusted by optical means to shorten the response time of target tracking. The optical flow error correction method is introduced in the inverse solution process to perform vector calculation estimation. The characteristics of the motion vector can be predicted by optical flow, and the motion information of the next frame can be provided to the rotating mirror tracking system in the current frame, thereby improving the tracking response speed in conjunction with the rotating mirror reverse ray tracing and prism angle solution theory.

[0060] (2) In dynamic scenes, the target's motion trajectory may be disturbed by noise, occlusion or changes in illumination, resulting in a decrease in the accuracy of traditional tracking methods. The present invention combines optical flow information and dynamically corrects the rotation mirror control parameters to achieve high-precision prediction of target motion and adjustment of the visual axis, thereby ensuring tracking accuracy. The rapid movement of targets in dynamic scenes, multi-target interaction or complex background changes often pose challenges to the robustness of the tracking system. The dual-drive target information processing framework fused with optical flow estimation enables the rotation mirror tracking system to achieve stable tracking in complex environments.

[0061] (3) Make full use of image motion information to optimize target tracking, and make the rotation efficiency of the rotating mirror higher through the scale fusion algorithm, avoiding problems such as spiral, jamming, and slowness during the rotating mirror tracking process, making the target tracking faster and smoother.

[0062] (4) The present invention utilizes a combination of an optical flow estimation algorithm and a mean shift algorithm to accurately match the target area between the current frame and the previous frame, thereby obtaining the target's motion trajectory and position change. Through iterative optimization of the optical flow vector, errors in optical flow estimation can be effectively eliminated, improving the accuracy of target positioning. This technical approach can calculate the target's motion state in each frame in real time, thereby achieving high-precision target tracking.

[0063] (5) The present invention iteratively optimizes the optical flow vectors of each region, ensuring that each updated optical flow vector better meets the photometric consistency requirements of the image. Iteration is terminated when the optical flow consistency error is less than a preset value, further enhancing the stability and robustness of the tracking process. This is particularly suitable for tracking targets in complex backgrounds and dynamically changing environments.

[0064] (6) The present invention combines a rotating dual-prism system with a rotating mirror system. By calculating the deviation between the target position and the predicted position, the system adjusts the prism rotation angle, achieving precise control of the target position. Through precise angle correction, the system can adjust the observation angle in real time, ensuring that the target can be continuously and accurately tracked in complex motion scenes.

[0065] (7) This invention introduces a PID control algorithm to perform real-time correction of the deflection angle in the rotating mirror system, enabling rapid adjustment when target position prediction errors occur. PID control not only improves the system's response speed but also ensures continuous and stable target tracking in dynamic environments, avoiding tracking failures caused by system hysteresis or inaccuracies.

[0066] (8) By dividing the target image into multiple uniform grid regions and employing local optical flow estimation and region matching methods, the present invention is able to perform optical flow analysis within a smaller region, thereby reducing computational complexity and improving the efficiency of target recognition and positioning. Furthermore, the combination of optical flow vector calculation and region determination effectively reduces interference with non-target regions, improving the accuracy and real-time performance of target detection.

[0067] (9) In target position prediction, the present invention optimizes target position prediction by calculating optical flow residuals, ensuring more accurate target prediction in each frame. By correcting the prediction residuals, the target position prediction error can be further reduced, ensuring consistency between the rotation angle of the rotating mirror system and the actual target position, and improving overall tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flow chart of the method of the present invention;

[0069] Figure 2 Schematic diagram of dual-mirror fast target tracking using optical flow-assisted prediction according to the present invention;

[0070] Figure 3 Schematic diagram of pixel motion and dual prism angle prediction update according to the present invention;

[0071] Figure 4 This is a tracking position comparison diagram of the present invention;

[0072] Figure 5 This is the tracking position error diagram of the present invention. DETAILED DESCRIPTION

[0073] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0074] Example 1:

[0075] This embodiment provides a method for tracking a target in a rotating mirror system using optical flow-assisted prediction. Figure 1 As shown, the following steps are included:

[0076] Step S1: acquiring a target image in real time through a rotating mirror system, and dividing a current frame of the target image into a plurality of regions using a uniform grid division, wherein the rotating mirror system includes a rotating biprism, and the rotating biprism includes two prisms;

[0077] Step S2: For each region, use the mean shift algorithm to match the regions of the current frame with those of the previous frame based on the image grayscale values ​​of the current frame and the image grayscale values ​​of the previous frame to obtain the initial center coordinates of each region of the current frame;

[0078] Step S3: Calculate the initial optical flow vector of the initial center coordinates;

[0079] Step S4: iterating the center coordinates and optical flow vectors of each region using the initial center coordinates and initial optical flow vectors of each region until the photometric consistency error of the image is less than a first preset value, stopping the iteration, and obtaining the center coordinates and optical flow vectors of each region in the current frame;

[0080] Step S5: Calculate the optical flow vectors of all coordinate points except the center coordinate in each area of ​​the current frame;

[0081] Step S6: judging the target area based on the optical flow vectors of the center coordinates of each area in the current frame, and calculating the target position coordinates in the current frame based on the target area;

[0082] Step S7: Predict the target position coordinates of the next frame by combining the target position coordinates of the current frame with the optical flow calculation prediction formula;

[0083] Step S8: Calculating the rotation angles of the two prisms in the rotating mirror system according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame;

[0084] Step S9: Calculating the optical flow residual according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and calculating the deflection angle correction amount of the rotating mirror system according to the optical flow residual;

[0085] Step S10: Calculating the rotation angle correction values ​​of the two prisms in the rotating mirror system according to the deflection angle correction value of the rotating mirror system, and updating the deflection angle and rotation angle of the rotating mirror system according to the deflection angle correction value and the rotation angle correction value.

[0086] Wherein, step S2 includes the following steps:

[0087] For each small area, the mean shift algorithm is used to find the area in the previous frame that is most similar to the current area, and its center coordinates are used as the matching position of the current area. The mean shift algorithm formula is:

[0088]

[0089] Among them, (x gi ,y gi ) is the initial center coordinate of the i-th region, region i represents the pixel coordinate set in the ith region, F(x,y)=(u x,y ,vx,y ), F(x,y) is the optical flow vector obtained by the optical flow estimation formula of the pixel point (x,y), u x,y 、v x,y Represent the horizontal motion component and vertical motion component at the pixel point (x, y) respectively;

[0090] u x,y 、v x,y The spatial gradient I of the image is calculated based on the image grayscale value I(x, y, t) of the current frame and the image grayscale value I(x, y, t-1) of the previous frame. xt , I yt With time gradient I t It is obtained through the optical flow estimation formula, which is:

[0091] I xt u x,y +I yt v x,y +I t =0

[0092] Where I represents the image grayscale value of the dual-prism imaging system, I(x,y,t) is the image grayscale value of the pixel (x,y) at time t, that is, the current time, I(x,y,t-1) is the image grayscale value of the pixel (x,y) at time t-1, that is, the previous time, I xt , I yt are the gradient values ​​of the image along x and y in space obtained at time t, I t The temporal gradient value obtained at time t.

[0093] Among them, the initial optical flow vector of the initial center coordinate is calculated as follows:

[0094]

[0095] Among them, (u gi1 ,v gi1 ) is the initial center coordinate (x gi ,y gi )’s initial optical flow vector, M is the region i The number of pixel coordinates in , N represents the region matching weight.

[0096] Among them, the center coordinates of each area and its optical flow vector are iterated, and the formula is:

[0097]

[0098] Among them, u gi (k) 、u gi (k)are the horizontal and vertical motion components of the center coordinates of the i-th region in the k-th iteration, u gi (k-1) 、v gi (k-1) are the horizontal and vertical motion components of the center coordinates of the i-th region in the k-1-th iteration, respectively. is the center coordinate of the i-th region in the k+l-th iteration, I xt , I yt They are the gradient values ​​of the image along x and y in space at time t, that is, the current time, and I t is the temporal gradient value obtained at time t, i.e. the current time.

[0099] Among them, the optical flow vector of other coordinate points except the center coordinate in each area of ​​the current frame is calculated using the formula:

[0100]

[0101] Among them, u i,j 、v i,j They are regions i In addition to the center coordinates, the horizontal motion component and vertical motion component of the j-th pixel in the current frame, ω i is the brightness weight of the i-th region, I xtj , I ytj For region i The image grayscale gradient value of the j-th pixel in the current frame and the previous frame in the x and y directions except the center coordinates, I tj Region in the current frame i The temporal gradient value of the j-th pixel except the center coordinate.

[0102] Wherein, step S6 includes the following steps:

[0103] Calculate the optical flow vector (u gi ,v gi ) is greater than the threshold τ, the formula is:

[0104]

[0105] Among them, u gi 、v gi are the horizontal motion component and vertical motion component corresponding to the center coordinates of the i-th region respectively;

[0106] If it is greater than the threshold τ, then the region i is marked as the target area. If it is less than or equal to the threshold τ, then the region i is marked as the target is not in the area. The center coordinates of the areas marked as the target area are averaged and summed to obtain the target position coordinates of the current frame. The formula is:

[0107]

[0108] Among them, (x c ,y c ) is the target position coordinate of the current frame, (x gi ,y gi ) is the center coordinate of the area marked as the target area, and L is the number of areas marked as the target area.

[0109] Among them, the target position coordinates of the current frame are combined with the optical flow calculation prediction formula to predict the target position coordinates of the next frame. The formula is:

[0110]

[0111] Among them, (x p ,y p ) is the predicted target position coordinate of the next frame, (x c ,y c ) is the target position coordinate of the current frame, u c 、v c are the optical flow components of the target position coordinates in the current frame, representing the target's movement speed in the x and y directions respectively. They are obtained by querying the optical flow vectors of other coordinate points except the center coordinates in each area of ​​the current frame in step S5. Δt is the time difference between the current frame and the next frame.

[0112] The rotation angles of the two prisms in the rotating mirror system are calculated based on the target position coordinates of the current frame and the predicted target position coordinates of the next frame. The formula is:

[0113]

[0114] Among them, α and β are the rotation angles of the first and second prisms in the rotating mirror system. are the azimuths of the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and Δθ is the angle between the two prisms.

[0115] Among them, the optical flow residual is calculated based on the current frame target position coordinates and the predicted next frame target position coordinates. The formula is:

[0116] Δx r =x p -x c

[0117] Δy r =y p -y c

[0118] Where Δx r , Δy r are the optical flow residuals of the target in the horizontal and vertical directions of the image respectively;

[0119] The deflection angle correction of the rotating mirror system is calculated based on the optical flow residual. The formula is:

[0120]

[0121] Where Δθ x , Δθ y are the visual axis deflection angle corrections in the x and y directions respectively, and f is the focal length of the industrial camera.

[0122] The rotation angle correction of the two prisms in the rotating mirror system is calculated based on the deflection angle correction of the rotating mirror system. The formula is:

[0123]

[0124] Among them, Δα and Δβ are the rotation angle corrections of the first prism and the second prism respectively, and Δθ x , Δθ y are the visual axis deflection angle corrections of the rotating mirror system in the x and y directions, K p1 , K i1 , K d1 are the proportional, integral and differential gains of the inner loop feedback control respectively;

[0125] The deflection angle and rotation angle of the rotating mirror system are updated according to the deflection angle correction value and the rotation angle correction value. The formula is:

[0126]

[0127] Among them, θ x ,θ y are the initial deflection angles of the mirror system in the x and y directions, θ x ′、θ y ′ are the deflection angles of the updated rotating mirror system in the x and y directions, Δθ x , Δθ y are the correction values ​​of the visual axis deflection angle of the rotating mirror system in the x and y directions, respectively. α′ and β′ are the rotation angles of the first and second prisms in the updated double prism of the rotating mirror system, respectively.

[0128] Example 2:

[0129] The parts not mentioned in this embodiment are the same as those in embodiment 1.

[0130] The target tracking system of this embodiment includes the following modules: a rotating mirror module, an imaging module, a computing unit, and a servo control system. As shown in the figure, the rotating mirror module is demonstrated by taking a rotating biprism as an example. The rotating biprism includes two prisms, which rotate along two independent axes perpendicular to the optical axis to achieve dynamic adjustment of the field of view. The rotation angle of each prism is represented by α and β respectively; the imaging module is composed of a CCD industrial camera for capturing the target; the computing unit includes an optical flow calculation module and a motion prediction module. The optical flow calculation is responsible for predicting the target motion vector, and the motion prediction module calculates the target position; the servo control system includes driving the biprism to rotate, adjusting the visual axis in real time according to the predicted target position, and keeping the target in the imaging center. As shown in the figure, Figure 2 Figure 3 As shown in Figure 1, the high-precision target tracking algorithm based on optical flow prediction and correction includes the following steps:

[0131] Step 1: The brightness value of the pixel at coordinate (x, y) at time t can be expressed as I(x, y, t). After time Δt to the current frame, the brightness value of the pixel at (x+Δx, y+Δy) is I(x+Δx, y+Δy, t+Δt), so it can be expressed as and They represent the optical flow components of the target along the X and Y directions in the two-dimensional coordinate system of the image. At time t, the constraint equation of the optical flow is as follows:

[0132] I x u+I y v+I t =0

[0133] Where: I represents the image grayscale value of the dual prism imaging system; I x , I y Indicates the gradient value of the image along x and y in space at time t; I t is the gradient value in time. In order to solve the above equations efficiently, the local area fitting method is used to solve the optical flow.

[0134] Step 2: To efficiently solve the above equation, we use the local region fitting method to calculate optical flow. First, the image is divided into several small regions, each containing multiple pixels. A uniform grid is used for the division. For each small region, an optical flow vector is calculated to represent the motion of that region.

[0135] Step 3: For each small area, it is necessary to match its position difference Δ(x i ,y i ). The mean shift algorithm can be used to find the area in the previous frame that is most similar to the current small area, and its center coordinates are used as the matching position of the current small area. The solution formula using the mean shift algorithm is:

[0136]

[0137] Among them, (x gi ,y gi ) is the initial center coordinate of the i-th region, region i represents the pixel coordinate set in the ith region, F(x,y)=(u x,y ,v x,y ), F(x,y) is the optical flow vector obtained by the optical flow estimation formula of the pixel point (x,y), u x,y 、v x,y Represent the horizontal motion component and vertical motion component at the pixel point (x, y) respectively;

[0138] u x,y 、v x,y The spatial gradient I of the image is calculated based on the image grayscale value I(x, y, t) of the current frame and the image grayscale value I(x, y, t-1) of the previous frame. xt , I yt With time gradient I t It is obtained through the optical flow estimation formula, which is:

[0139] I xt u x,y +I yt v x,y +I t =0

[0140] Where I represents the image grayscale value of the dual-prism imaging system, I(x,y,t) is the image grayscale value of the pixel (x,y) at time t, that is, the current time, I(x,y,t-1) is the image grayscale value of the pixel (x,y) at time t-1, that is, the previous time, I xt , I yt are the gradient values ​​of the image along x and y in space obtained at time t, I t The temporal gradient value obtained at time t.

[0141] After completing the region matching, the pixel-based optical flow method is used to calculate the optical flow vector of each pixel, and then these vectors are averaged to obtain the optical flow vector (x gi ,y gi ) is the initial optical flow vector of the center coordinate:

[0142]

[0143] Among them, (u gi1 ,v gi1 ) is the initial center coordinate (x gi ,y gi)’s initial optical flow vector, M is the region i The number of pixel coordinates in , N represents the region matching weight.

[0144] Step 4: Once the motion vectors of each small area are obtained, they can be used to track the movement of the object. The position of the small area in the previous frame can be moved to the current frame according to its motion vector, and then the optical flow vector of the small area can be recalculated in the current frame. The position of each small area in the previous frame can be moved to the current frame according to its motion vector, and then the optical flow vector of the small area can be recalculated in the current frame. Through iterative calculation, the photometric consistency error of the image is Indicates convergence, the iterative formulas of u and v are as follows:

[0145] Iterate the center coordinates of each area and its optical flow vector, the formula is:

[0146]

[0147] Among them, u gi (k) 、u gi (k) are the horizontal and vertical motion components of the center coordinates of the i-th region in the k-th iteration, respectively, u gi (k-1) 、v gi (k-1) are the horizontal and vertical motion components of the center coordinates of the i-th region in the k-1-th iteration, respectively. is the center coordinate of the i-th region in the k+l-th iteration, I xt , I yt They are the gradient values ​​of the image along x and y in space at time t, that is, the current time, and I t is the temporal gradient value obtained at time t, i.e. the current time.

[0148] Step 5: Combine the optical flow vectors of all small areas to obtain the optical flow field of the entire image. i ∈r i The optical flow vector (u i,j ,v i,j ), the movement speed of other coordinate points except the obtained central coordinate point can be calculated in matrix form:

[0149]

[0150] Among them, u i,j 、v i,j Respectively, regional i In addition to the center coordinates, the horizontal motion component and vertical motion component of the j-th pixel in the current frame, ωi is the brightness weight of the i-th region, I xtj , I ytj For regional regiom i The image grayscale gradient value of the j-th pixel in the current frame and the previous frame in the x and y directions except the center coordinates, I tj Region in the current frame i The temporal gradient value of the j-th pixel except the center coordinate.

[0151] Regional optical flow estimation methods utilize local similarities in images to improve the accuracy and stability of optical flow estimation. Combined with pixel-based optical flow methods, they can effectively reduce noise and outliers in optical flow estimation and provide better robustness in complex scenes. After accurately calculating V = (u, v), the optical flow field can be used to predict the target's motion trajectory.

[0152] Step 6: Use the time integral of the optical flow field to predict the target's motion position in the next frame. The region whose optical flow motion vector value of the center coordinate point of the region is greater than the threshold τ is detected as the target, and the weighted average of the center points of all regions is determined as the current target position (x c ,y c ), as shown in the formula:

[0153]

[0154] Among them, (x c ,y c ) is the target position coordinate of the current frame, (x gi ,y gi ) is the center coordinate of the area marked as the target area, and L is the number of areas marked as the target area.

[0155] Combined with the optical flow calculation prediction formula, the position of the next frame (x p ,y p ) can be obtained through speed points:

[0156]

[0157] Among them, (x p ,y p ) is the predicted target position coordinate of the next frame, (x c ,y c ) is the target position coordinate of the current frame, u c 、v c are the optical flow components of the target position coordinates in the current frame, representing the target's movement speed in the x and y directions respectively. They are obtained by querying the optical flow vectors of other coordinate points except the center coordinates in each area of ​​the current frame in step S5. Δt is the time difference between the current frame and the next frame.

[0158] Since the target motion in real scenes is complex, there is acceleration. By introducing the optical flow dynamic error model, the optical flow prediction trajectory can be combined with the target tracking system to achieve long-term prediction and correction of complex motion.

[0159] Step 7: The rotation angles α and β of the dual prism need to be adjusted according to the predicted target position so that the target is always in the center of the imaging field of view. The target point in the current frame is (x c ,y c ), the target prediction point is (x p ,y p ). The two-step method is used to solve the rotation angle solution, which is obtained by the following formula:

[0160]

[0161] Among them, α and β are the rotation angles of the first and second prisms in the rotating mirror system. are the azimuths of the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and Δθ is the angle between the two prisms.

[0162] Step 8: When the optical flow method predicts the target trajectory, due to the uncertainty of the prediction, there may be a deviation between the actual target position and the predicted position. This deviation is determined by the optical flow residual estimation formula. The optical flow residual error is directly converted into the correction value of the visual axis deflection angle in the x and y directions:

[0163] The optical flow residual is calculated based on the current frame target position coordinates and the predicted next frame target position coordinates. The formula is:

[0164] Δx r =x p -x c

[0165] Δy r =y p -y c

[0166] Where Δx r , Δy r are the optical flow residuals of the target in the horizontal and vertical directions of the image respectively;

[0167] The deflection angle correction of the rotating mirror system is calculated based on the optical flow residual. The formula is:

[0168]

[0169] Where Δθ x , Δθ yare the visual axis deflection angle corrections in the x and y directions respectively, and f is the focal length of the industrial camera.

[0170] Step 10: To ensure that the dual prism system and the optical flow method can be integrated with high precision and high dynamics for target tracking, a refined optimization control strategy needs to be designed. The fast inner loop feedback is centered on the prism angle adjustment and focuses on the rapid compensation of the instantaneous offset of the target. The real-time offset error Δx r ,Δy r Minimize. Use the following control strategy for the prism deflection angle (α, γ):

[0171]

[0172] Among them, Δα and Δβ are the rotation angle corrections of the first prism and the second prism respectively, and Δθ x , Δθ y are the visual axis deflection angle corrections of the rotating mirror system in the x and y directions, K p1 , K i1 , K d1 are the proportional, integral and differential gains of the inner loop feedback control respectively;

[0173] Then update the prism's deflection angle:

[0174]

[0175] Among them, θ x ,θ y are the initial deflection angles of the mirror system in the x and y directions, θ x ′、θ y ′ are the deflection angles of the updated rotating mirror system in the x and y directions, Δθ x , Δθ y are the correction values ​​of the visual axis deflection angle of the rotating mirror system in the x and y directions, respectively. α′ and β′ are the rotation angles of the first and second prisms in the updated double prism of the rotating mirror system, respectively.

[0176] Step 12: The outer loop feedback predicts the global motion trajectory of the target based on the optical flow method, and forms a closed-loop control with the prism system for long-term trend correction and error compensation. The outer loop control adopts model predictive control (MPC) to dynamically optimize the prism angle solution:

[0177]

[0178] in represents the actual deflection angle at the kth moment; θ x [k],θ y[k] The desired target deflection angle. Based on the optimized prism angle adjustment (Δα, Δβ) output by MPC, the prism trajectory is corrected at each time step.

[0179] Step 13: The optical flow target state vector can be expressed as X = [x, y, u, v, a x ,a y ], so the motion model can be expressed as

[0180] X t+1 =AX t +BU t +W

[0181] Where A represents the state transfer matrix, U represents the control input vector, B represents the control input matrix, and W represents the noise matrix.

[0182] The measurement vector is the target optical flow offset and the actual feedback angle of the prism:

[0183] Z t =[Δx r ,Δy r ,α,β] T

[0184] Therefore, the optical flow prediction and update equation are performed by the following steps:

[0185]

[0186] Among them, P is the prediction vector and Q is the covariance matrix.

[0187] Update phase:

[0188]

[0189] Among them, H is the observation matrix and I is the identity matrix.

[0190] Step 14: Adaptively adjust the PID gain parameter K by analyzing the optical flow residual and prism response speed p ,K i ,K d , that is, K. If the target moves quickly, increase K d To improve the system response speed; if the error continues to accumulate, increase K i To compensate for the deviation; in the stable stage, reduce K p Prevent overshoot. Leveraging the statistical properties of observation noise, the noise covariance matrices Q and R are adaptively adjusted. When noise increases, R is increased to reduce the impact of the measured value on the state; when noise decreases, Q is reduced to increase the model's prediction weight. Ultimately, this achieves the goal of correcting the dual-prism tracking effect.

[0191] This method is compared with the traditional single visual tracking method. The experimental results are as follows Figure 4 Figure 5 As shown, the experimental results show that the use of optical flow method combined with dual prism angle control can achieve more accurate target prediction and adjustment. When the optical flow method is used to predict the target position, it can effectively improve the system's response speed to dynamic targets and reduce the error caused by prism adjustment lag. In addition, by introducing the optical flow method to estimate the target trajectory, it can reduce interference in multi-target scenes and stably track the main target. Compared with the traditional dual prism feedback tracking method that does not use the optical flow method, the system using the optical flow method shows better stability and robustness under complex nonlinear trajectories and dynamic noise interference. In the experiment, by comparing the tracking results without the optical flow method, the optical flow method can more accurately predict the target's motion trend, greatly reducing the tracking error and lag effect caused by static observation, especially when the target moves quickly or is occluded, the system shows higher tracking accuracy and response speed.

[0192] Overall, compared to traditional object tracking methods based on visual feedback, this method not only effectively improves target tracking accuracy but also optimizes system response time by introducing optical flow prediction, overcoming the computational burden and response delay issues inherent in traditional methods. Furthermore, in multi-target scenarios, this method can adaptively adjust the optical flow field, further enhancing robustness and accuracy in complex motion environments.

[0193] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0194] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A target tracking method for a rotating mirror system using optical flow-assisted prediction, characterized in that: The following steps are involved: Step S1: acquiring a target image in real time through a rotating mirror system, and dividing a current frame of the target image into a plurality of regions using uniform grid division, wherein the rotating mirror system includes a rotating biprism, and the rotating biprism includes two prisms; Step S2: For each region, use the mean shift algorithm to match the regions of the current frame with those of the previous frame based on the image grayscale values ​​of the current frame and the image grayscale values ​​of the previous frame to obtain the initial center coordinates of each region of the current frame; Step S3: Calculate the initial optical flow vector of the initial center coordinates; Step S4: iterating the center coordinates and optical flow vectors of each region using the initial center coordinates and initial optical flow vectors of each region until the photometric consistency error of the image is less than a first preset value, stopping the iteration, and obtaining the center coordinates and optical flow vectors of each region in the current frame; Step S5: Calculate the optical flow vectors of all coordinate points except the center coordinate in each area of ​​the current frame; Step S6: judging the target area based on the optical flow vectors of the center coordinates of each area in the current frame, and calculating the target position coordinates in the current frame based on the target area; Step S7: Predict the target position coordinates of the next frame by combining the target position coordinates of the current frame with the optical flow calculation prediction formula; Step S8: Calculating the rotation angles of the two prisms in the rotating mirror system according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame; Step S9: Calculating the optical flow residual according to the target position coordinates of the current frame and the predicted target position coordinates of the next frame, and calculating the deflection angle correction amount of the rotating mirror system according to the optical flow residual; Step S10: Calculating the rotation angle correction values ​​of the two prisms in the rotating mirror system according to the deflection angle correction value of the rotating mirror system, and updating the deflection angle and rotation angle of the rotating mirror system according to the deflection angle correction value and the rotation angle correction value.

2. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, wherein: The step S2 comprises the following steps: For each small area, the mean shift algorithm is used to find the area in the previous frame that is most similar to the current area, and its center coordinates are used as the matching position of the current area. The mean shift algorithm formula is: in, For the i The initial center coordinates of the region, Indicates the i The set of pixel coordinates within the area, , Pixel The optical flow vector obtained based on the pixel-based optical flow estimation formula is: 、 Represents pixel points Horizontal motion component and vertical motion component; described 、 According to the image gray value of the current frame The grayscale value of the image of the previous frame Calculate the spatial gradient of the image With time gradient It is obtained through the optical flow estimation formula, which is: in, represents the image grayscale value of the dual-prism imaging system, Pixel At time t, that is, the grayscale value of the image at the current moment, Pixel The grayscale value of the image at time t-1, which is the previous moment, 、 are the gradient values ​​of the image along x and y in space obtained at time t, The temporal gradient value obtained at time t.

3. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 2, wherein: Calculate the initial optical flow vector of the initial center coordinates. The formula is: in, is the initial center coordinate The initial optical flow vector, for The number of pixel coordinates in , N Indicates the region matching weight.

4. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 2, wherein: Iterate the center coordinates of each area and its optical flow vector, the formula is: in, 、 Respectively k In the iteration i The horizontal and vertical motion components of the center coordinates of the region, 、 Respectively In the iteration i The horizontal and vertical motion components of the center coordinates of the region, For the In the iteration i The center coordinates of the region, 、 They are the gradient values ​​of the image along x and y in space at time t, i.e. the current time, is the temporal gradient value obtained at time t, i.e. the current time.

5. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 2, wherein: Calculate the optical flow vector of all coordinate points except the center coordinate in each area of ​​the current frame. The formula is: in, Respectively for regions In addition to the center coordinates, the horizontal motion component and vertical motion component of the j-th pixel in the current frame are: is the brightness weight of the i-th region, 、 For the region The image grayscale gradient value in the x and y directions of the j-th pixel point in the current frame and the previous frame except the center coordinates, The area in the current frame The temporal gradient value of the j-th pixel except the center coordinate.

6. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, wherein: The step S6 comprises the following steps: Calculate the optical flow vector corresponding to the center coordinates of each area of ​​the current frame Is it greater than the threshold? , the formula is: in, are the horizontal motion component and vertical motion component corresponding to the center coordinates of the i-th region respectively; If it is greater than the threshold , then mark area i as the target area, if it is less than or equal to the threshold , then mark area i as the target not in the area, average and sum the center coordinates of the area marked as the target, and obtain the target position coordinates of the current frame. The formula is: in, is the target position coordinate of the current frame, are the center coordinates of the area marked as the target, is the number of areas marked as targets.

7. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, wherein: The target position coordinates of the next frame are predicted by combining the target position coordinates of the current frame with the optical flow calculation prediction formula. The formula is: in, is the predicted target position coordinate of the next frame, is the target position coordinate of the current frame, 、 is the optical flow component of the target position coordinates in the current frame, which represents the target's movement speed in the x and y directions respectively. It is obtained by querying the optical flow vectors of other coordinate points in each area of ​​the current frame except the center coordinate in step S5. is the time difference between the current frame and the next frame.

8. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, wherein: The rotation angles of the two prisms in the rotating mirror system are calculated based on the current frame target position coordinates and the predicted next frame target position coordinates. The formula is: in, 、 are the rotation angles of the first and second prisms in the rotating mirror system, are the azimuths of the target position coordinates of the current frame and the predicted target position coordinates of the next frame, is the angle between the two prisms.

9. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, wherein: The optical flow residual is calculated based on the current frame target position coordinates and the predicted next frame target position coordinates. The formula is: in, 、 are the optical flow residuals of the target in the horizontal and vertical directions of the image respectively; The deflection angle correction amount of the rotating mirror system is calculated based on the optical flow residual, and the formula is: in, 、 are the visual axis deflection angle corrections in the x and y directions, is the focal length of an industrial camera.

10. The method for tracking a target in a rotating mirror system using optical flow-assisted prediction according to claim 1, wherein: The rotation angle correction amount of the two prisms in the rotating mirror system is calculated according to the deflection angle correction amount of the rotating mirror system, and the formula is: in, are the rotation angle corrections of the first prism and the second prism respectively, 、 are the visual axis deflection angle corrections of the rotating mirror system in the x and y directions, are the proportional, integral and differential gains of the inner loop feedback control respectively; The deflection angle and rotation angle of the rotating mirror system are updated according to the deflection angle correction value and the rotation angle correction value. The formula is: in, are the initial deflection angles of the mirror system in the x and y directions, are the deflection angles of the updated mirror system in the x and y directions, 、 are the visual axis deflection angle corrections of the rotating mirror system in the x and y directions, 、 They are the rotation angles of the first and second prisms in the updated rotating mirror system double prism.

Images