A micro-motion trajectory tracking method based on optical flow method and spatial filtering amplification

Abstract

This invention relates to the field of computer vision and image processing technology, and provides a method for tracking minute motion trajectories based on optical flow and spatial filtering amplification. The aim is to solve the problems of low trajectory tracking accuracy, poor real-time performance, and insufficient environmental adaptability caused by weak minute motion signals in existing shooting training video analysis techniques. The method includes the following steps: receiving a video frame sequence; performing brightness normalization and image denoising preprocessing on each frame; dynamically locating the region of interest (ROI) of the muzzle using a matching method based on a predefined reference template; applying spatial filtering amplification to the ROI, decomposing the image frequency layers on a spatial scale, amplifying the changing signals within a specific frequency range on a temporal scale, and reconstructing an enhanced image to highlight minute motions; applying optical flow methods, including dense optical flow algorithms or sparse optical flow algorithms, to the enhanced ROI region to estimate motion vectors; constructing a trajectory point sequence based on the motion vectors, performing smoothing processing, and outputting smoothed trajectory data.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and provides a method for tracking minute motion trajectories based on optical flow and spatial filtering amplification. Background Technology

[0002] In the field of shooting training, accurate tracking and analysis of muzzle trajectories are crucial for assessing shooters' skill levels and optimizing training effectiveness. Traditional methods primarily rely on hardware devices for trajectory recording, such as laser marking systems, high-precision inertial measurement units (IMUs), or infrared trackers. However, these solutions have significant drawbacks: firstly, the hardware is expensive and complex to deploy, making large-scale deployment difficult in ordinary training scenarios; secondly, they are greatly affected by environmental factors (such as changes in lighting and obstructions), resulting in insufficient stability; and thirdly, they are highly dependent on the equipment, limiting the system's flexibility and applicability.

[0003] For software-based motion detection, existing technologies mostly employ frame difference or traditional optical flow algorithms. Frame difference estimates motion by comparing pixel values ​​between adjacent frames, but this method is insensitive to minute jitter and is easily affected by environmental noise and lighting changes, leading to a high false detection rate. While traditional optical flow algorithms can provide more refined motion vector estimates, they typically assume the target region of interest (ROI) remains fixed in the video, failing to account for the dynamic drift of the muzzle caused by shooter movements or camera shake, thus affecting tracking stability. Furthermore, existing technologies generally lack effective methods for enhancing minute motion signals, making it difficult to capture the high-frequency, low-amplitude vibration characteristics of the muzzle during firing (such as the subtle displacement at the moment the trigger is pulled).

[0004] Furthermore, some studies have attempted to improve trajectory extraction accuracy by combining image processing techniques, but due to limitations in algorithm design and computational efficiency, the following problems still exist:

[0005] 1. Insufficient dynamic ROI positioning capability: Existing methods mostly rely on preset fixed areas or manual marking, which is difficult to adapt to scenarios where the gun position changes over time, resulting in target loss or tracking deviation;

[0006] 2. Weak perception of minute movements: Traditional image processing workflows have limited sensitivity to low-amplitude displacements and cannot effectively distinguish between the real movement of the gun muzzle and noise interference;

[0007] 3. Insufficient real-time performance and robustness: Some algorithms have high computational complexity, making it difficult to meet the real-time feedback requirements in shooting training, and are prone to misjudgment in complex backgrounds or dynamic environments.

[0008] In summary, existing technologies have significant shortcomings in terms of accuracy, stability, and environmental adaptability in muzzle trajectory tracking. There is an urgent need for a solution that does not rely on external hardware, can adapt to dynamic ROI, and enhance minute motion signals to meet the needs of high-precision, real-time shooting training analysis. Summary of the Invention

[0009] The purpose of this invention is to solve the problems of low trajectory tracking accuracy, poor real-time performance, and insufficient environmental adaptability in existing shooting training video analysis technology due to weak small motion signals, unstable dynamic ROI positioning, and reliance on external hardware.

[0010] To achieve the above objectives, the present invention employs the following technical means:

[0011] This invention provides a method for tracking minute motion trajectories based on optical flow and spatial filtering amplification, comprising the following steps:

[0012] Step 1: Receive the video frame sequence and preprocess each frame image, including brightness normalization and image denoising;

[0013] Step 2: In each frame of the preprocessed image, based on a predefined reference template, the muzzle region is dynamically located using a matching method to obtain the region of interest (ROI);

[0014] Step 3: Apply spatial filtering and amplification processing to the ROI region, including: decomposing the image frequency layer on a spatial scale, amplifying the changing signal within a specific frequency range on a temporal scale, and reconstructing it into an enhanced image to highlight minute movements;

[0015] Step 4: On the enhanced ROI region, apply optical flow to estimate the motion vector. The optical flow method is either a dense optical flow algorithm or a sparse optical flow algorithm.

[0016] Step 5: Construct a trajectory point sequence based on the motion vector, and smooth the trajectory point sequence to output the smoothed trajectory data.

[0017] In the above scheme, step 1 includes the following steps:

[0018] Receive video stream data, wherein the video stream data is a real-time stream or offline recorded content;

[0019] Perform basic image processing operations on each acquired frame, including:

[0020] Convert the image to grayscale space or the luminance channel, and unify the luminance dynamic range using one of the following methods:

[0021] Linear transformation, including histogram stretching or normalization to a fixed brightness range, is specifically formulated as follows:

[0022]

[0023] Where I(x,y) represents the pixel brightness value of the original image at coordinates (x,y), I min Minimum brightness value in the image, I max Maximum brightness value in the image;

[0024] Alternatively, adaptive histogram equalization, including the CLAHE method, can be used, with the following formula:

[0025]

[0026] Among them, s k This represents the pixel value after equalization, where L represents the number of gray levels, and n... j This represents the number of pixels at gray level j, where M and N represent the width and height of the image, respectively.

[0027] via s k Or I norm After obtaining a brightness-normalized image (x, y), high-frequency noise and compression artifacts are eliminated using one of the following methods:

[0028] Gaussian filtering, the specific formula is:

[0029]

[0030] Where G(x,y) represents the weight value of the Gaussian kernel at coordinate (x,y), and σ represents the standard deviation of the Gaussian distribution;

[0031] Image Gaussian filtering operation

[0032]

[0033] Among them, I filtered (x,y) represents the pixel value of the filtered image at coordinates (x,y), k represents the radius of the filter kernel, G(i,j) is the weight function of the Gaussian filter at coordinates (i,j), and i,j represent the offset of the convolution kernel in the horizontal and vertical directions.

[0034] Median filtering

[0035] I filtered (x,y)=median{I(x+i,y+j)|-k≤i,j≤k}

[0036] Where median represents the median operation;

[0037] bilateral filtering

[0038]

[0039] Among them, I filtered (x) represents the pixel value at coordinate x in the filtered image, W p f represents the normalization factor, Ω represents the neighborhood window, and f r Let f represent the gray-level similarity function. s Let x represent the spatial similarity function. i Indicates the pixel coordinates within the neighborhood;

[0040] Through I filtered (x) After noise and artifact removal, it is possible to eliminate the overall jitter between video frames, including calculating the global transform between adjacent frames, processing the transform parameters with a smoothing filter, and applying the inverse transform, where the global transform adopts an affine transform model:

[0041]

[0042] Where x and y represent the transformed coordinates, and x and y represent the original coordinates, a 11 a 12 a 21 a 22 t represents the rotation and scaling parameters. x , t y This represents the translation parameter.

[0043] In the above scheme, step 2 includes the following steps:

[0044] In each frame of the image, a predefined reference template is searched using a matching method to determine the area where the muzzle is located. The matching method includes, but is not limited to, one of the following three forms:

[0045] Template matching: The reference template is slid across the current video frame image for comparison, the normalized cross-correlation coefficient (NCC) between each sliding window position and the template is calculated, and the position with the largest correlation coefficient is selected as the ROI of the current frame;

[0046] Feature matching: Local invariant feature points and their descriptors are extracted from the reference template and the current frame image, respectively, and matched using nearest neighbor or Hamming distance. After removing mismatches using the RANSAC algorithm, the coordinates of the ROI region are calculated based on the location of the matching points.

[0047] Similarity calculation: The reference template is compared with each candidate window region in the current video frame. The perceptual similarity between each window and the template is calculated using the Structural Similarity Index (SSIM), and the window with the highest SSIM score is selected as the ROI of this frame.

[0048] In the above scheme, step 3 includes the following steps:

[0049] The region of interest (ROI) is subjected to spatial multi-scale decomposition, specifically including extracting specific frequency bands of the image using a Laplacian pyramid or a high-pass filter at the spatial scale.

[0050] The construction process of the Laplace pyramid is as follows:

[0051] G0=I,G l+1 =downsample(G i *g)

[0052] Where I represents the original image, g represents the Gaussian kernel, * represents the convolution operation, and G... l This represents the Gaussian pyramid image of the l-th layer, and downsample represents the downsampling operation;

[0053] Then, the Laplace pyramid layers are constructed using differences:

[0054] L l =G l -upsample(G l+1 )

[0055] Where upsample represents the upsampling operation, L l This represents an image of the Laplace pyramid at level l.

[0056] Alternatively, a high-pass filter can be used to directly extract high-frequency components from the image to enhance local detail changes;

[0057] The above filtering process yields an image for a specific frequency band. The pixel value changes at each spatial frequency level are analyzed over time, and the target frequency band is selected for amplification. Specifically, this includes:

[0058] Model the value of each pixel over time as a one-dimensional time series I l (x,y,t), where t=1,2,...,T,I l This represents the pixel value of the image after spatial frequency decomposition at time t;

[0059] The time series is filtered using a bandpass filter to select a target frequency band; the filtered signal is then multiplied by an amplification factor α to obtain an enhanced pixel value change signal.

[0060] The processed image frequency layer is merged with the original pyramid structure or image structure and reconstructed into an enhanced image sequence, specifically including:

[0061]

[0062] in, This represents the reconstructed layers of the Pyramid of Laplace. This indicates the change in pixel value after magnification.

[0063] In the above scheme, step 4 includes the following steps:

[0064] In the image enhancement sequence, the motion vectors of pixels or feature points are estimated according to the fundamental optical flow constraint equations:

[0065]

[0066] Where I(x,y,t) represents the brightness value of the image at coordinates (x,y) and time t, and u and v represent the motion velocity components in the x and y directions, respectively; this is an equation with two unknowns (u,v) that cannot be solved directly and is called the "aperture problem" in optical flow estimation;

[0067] One of the following two optical flow algorithms is used for motion estimation to approximate the solution:

[0068] A. Dense optical flow algorithm, which provides an optical flow estimate for each pixel in the image, i.e., u and v have estimates for all pixels:

[0069] A1. Constructing an image pyramid: Perform multi-scale pyramid decomposition on two consecutive frames of ROI images;

[0070] A2. In each pyramid layer, approximate the grayscale changes of the image patch using polynomial expansion:

[0071]

[0072] in, Let A represent pixel coordinates, b be a symmetric matrix, and c be a constant term.

[0073] B. Sparse optical flow algorithm aims to estimate optical flow only for a few feature points (such as corner points) in the image, that is, to estimate only the u and v of these key points, rather than estimating for all pixels;

[0074] B1. Extract corner features from the ROI in the first frame;

[0075] B2. In subsequent frames, a local window is constructed centered on each feature point. Assuming that the pixel motion within the window is consistent, the displacement is solved by minimizing the error:

[0076]

[0077] Among them, I x I y I t denoted as the partial derivatives of the image in the x, y, and t directions, respectively, and u and v represent the displacement components in the x and y directions, respectively.

[0078] By solving the system of linear equations Obtain the displacement vector in:

[0079]

[0080] The muzzle motion trajectory is constructed based on the displacement vectors between consecutive frames.

[0081] In the above scheme, step 5 includes the following steps:

[0082] The original muzzle motion trajectory estimated by the optical flow method is smoothed to remove random jitter, jumps, or drift deviations. The smoothing process employs one of the following methods:

[0083] At each time point t, a weighted average is calculated using the trajectory coordinates of that point and several frames before and after it, and the smoothed point coordinates are output. The specific formula is as follows:

[0084]

[0085] Where, x t The x-coordinate and y-coordinate of the original trajectory at time t represent the coordinates of the original trajectory. t The y-coordinate of the original trajectory at time t represents the value of the original trajectory, and k represents the radius of the filter kernel. This represents the smoothed x-coordinate value. This represents the smoothed y-coordinate value;

[0086] Kalman filter: consists of two stages: prediction and update. The specific formula is as follows:

[0087] The state variable is defined as:

[0088]

[0089] Where, x t The state vector at time t, x t and y t Let v represent the position coordinates of the two-dimensional muzzle trajectory estimated by the optical flow method. x,t and v y，t These represent the velocity components in the x and y directions, respectively;

[0090] The state transition equation is:

[0091] x t =Fx t-1 +w t

[0092] Where F is the state transition matrix, w t The system noise is represented by the observation equation:

[0093] z t =Hx t +v t

[0094] Among them, z t Let v be the observed value, H be the observation matrix, and v be the observed value. t Indicates observation noise;

[0095] Kalman filtering recursion is used to dynamically estimate the target state. The calculation process includes:

[0096]

[0097] in This is the prior state estimate at the current time t. It requires manually set initial state estimates. x0 and y0 are extracted from the first frame image. The velocity can be set to 0 or estimated using the difference between the first and second frames. The initial values ​​have a significant impact on the first few frames, but their influence gradually weakens with subsequent filtering. F is the state transition matrix. This is the state estimate from the previous step. It is the prior covariance matrix, P t-1 Here, Q is the covariance matrix of the previous time step, H is the process noise covariance matrix, R is the observation matrix, and K is the observation noise covariance. t It is the Kalman gain, z t These are actual observed values. It is the updated state estimate, where I is the identity matrix;

[0098] After smoothing, the optimized trajectory data will be output in two ways:

[0099] Graphical visualization: Overlaying smooth trajectories as continuous curves or trajectory points onto the original video frames;

[0100] Structured coordinate output: Exports trajectory data as structured data in the form of (x, y, t) triples.

[0101] Because the present invention employs the above-mentioned technical means, it has the following beneficial effects:

[0102] 1. The present invention solves the technical problem of weak perception of minute motion in the background art by using spatial filtering and amplification processing in step 3, including decomposing the image frequency layer on a spatial scale (such as using a Laplacian pyramid or a high-pass filter to extract specific frequency bands) and amplifying the changing signal within a specific frequency range on a time scale (such as modeling pixel time series and applying a band-pass filter), and achieves the effect of significantly enhancing the signal of minute motion of the gun muzzle.

[0103] In shooting training scenarios, minute muzzle vibrations (such as the high-frequency, low-amplitude displacement at the moment of trigger pull) are easily masked by environmental noise or changes in lighting due to their small amplitude, making them difficult to accurately detect using traditional frame difference or optical flow methods. This invention addresses this by using spatial multi-scale decomposition (constructing a Laplace pyramid layer L). l =G l -upsample(G l+1 This process decouples the image into different frequency levels, separating high-frequency detail information; subsequently, in the time dimension, the signal in the target frequency band is amplified (for time series I). l Filter the image using (x, y, t) and multiply by an amplification factor α to reconstruct and enhance the image. This process highlights previously inconspicuous minute movements at both the visual and computational levels, thereby enhancing the sensitivity of subsequent optical flow methods to low-amplitude displacements. As a result, the system can reliably capture the true micro-motion characteristics of the muzzle, avoiding false or missed detections due to weak signals, and improving the accuracy and robustness of trajectory tracking.

[0104] 2. The present invention solves the technical problem of insufficient dynamic ROI positioning capability in the background technology by using the technical means of dynamically locating the muzzle region based on a predefined reference template in step 2, including template matching (such as calculating the normalized cross-correlation coefficient NCC), feature matching (such as extracting SIFT or ORB feature points and removing mismatches through RANSAC) or similarity calculation (such as using the structural similarity index SSIM), and achieves the effect of adaptively tracking changes in muzzle position.

[0105] Traditional methods rely on fixed preset areas or manual marking, which cannot adapt to dynamic drift of the muzzle caused by shooter movements or camera shake, easily leading to target loss or tracking deviation. This invention, in each frame, uses a matching mechanism (SSIM calculation) to... The system dynamically searches for the window region most similar to the reference template and automatically determines the ROI coordinates. This method comprehensively considers image brightness, contrast, and structural information, maintaining high matching robustness even under changes in lighting, partial occlusion, or background interference. Therefore, the system can adapt to changes in the muzzle position in real time without manual intervention, solving the tracking instability problem caused by ROI drift and improving environmental adaptability and automation.

[0106] 3. This invention uses the optical flow method to estimate motion vectors in step 4, including dense optical flow algorithms (such as the Farneback method using polynomial expansion). (Estimating pixel-level displacement) or sparse optical flow algorithms (such as the Lucas-Kanade method to minimize error) By combining the enhanced image from step 3, the technical problem of low tracking accuracy in the background technology is solved, and the effect of high-precision reconstruction of the muzzle motion trajectory is achieved.

[0107] Existing frame difference methods or traditional optical flow methods are insensitive to minute motions and are easily affected by noise, leading to large trajectory estimation errors. This invention applies optical flow to the spatially filtered and amplified Region of Interest (ROI). Since the minute motion signal has been enhanced (step 3), the optical flow constraint equation... It can solve for displacement vectors more accurately (e.g., dense optical flow provides full-field motion estimation, while sparse optical flow focuses on key points). For example, the Farneback method optimizes the displacement field through image pyramid iteration, making small displacement features more obvious in enhanced images and reducing the impact of aperture issues; the Lucas-Kanade method assumes motion consistency within the feature point window and combines RANSAC to improve robustness. Therefore, the system can continuously and accurately estimate muzzle displacement, constructing a realistic motion path, solving the problem of low accuracy, and providing a reliable data foundation for shooter action evaluation.

[0108] 4. The present invention employs a smoothing technique for the motion vector in step 5, including a moving average (such as calculating...). ) or Kalman filtering (e.g., defining state variables) By updating the estimate through recursive equations, the technical problems of random jitter, jumps or drift deviations in trajectory data are solved, and the effect of outputting a stable and smooth trajectory is achieved.

[0109] Optical flow estimation is susceptible to sensor noise or image artifacts, causing discontinuities and jitters in the original trajectory, affecting the accuracy of subsequent analysis. This invention introduces a smoothing mechanism: moving average suppresses high-frequency noise by weighting the coordinates of neighboring frames; Kalman filtering models the system state (such as position and velocity) and uses a prediction-update loop (e.g., calculating the Kalman gain). By integrating historical trajectories and current observations, errors are dynamically corrected. This method effectively suppresses short-term interference (such as single-frame matching errors) and long-term drift, making the trajectory curve more consistent. As a result, the system outputs a more reliable and smooth trajectory, solving the problem of insufficient robustness, facilitating visualization or structured export (such as (x,y,t) triples), and improving the practicality of training analysis.

[0110] 5. The present invention, through the design of the entire method, including preprocessing in step 1 (brightness normalization and image denoising), dynamic ROI positioning in step 2, spatial filtering and amplification in step 3, optical flow estimation in step 4, and smoothing in step 5, solves the technical problems of insufficient real-time performance and robustness and dependence on external hardware in the background technology, and achieves the effect of efficient real-time processing of video streams without the need for additional equipment. Attached Figure Description

[0111] Figure 1 Here is the algorithm flowchart;

[0112] Figure 2 The muzzle magnified by motion;

[0113] Figure 3 The center of the gun muzzle;

[0114] Figure 4 This is a diagram showing the reconstructed muzzle trajectory. Detailed Implementation

[0115] The embodiments of the present invention will be described in detail below. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, any modifications or equivalent substitutions made to the present invention should be covered within the scope of the claims of the present invention.

[0116] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without these specific details.

[0117] This invention proposes a muzzle trajectory tracking method combining spatial filtering amplification and optical flow. This method performs high-precision, real-time motion analysis on shooting training videos, effectively extracting subtle muzzle trajectory information. This technology is particularly suitable for post-production auxiliary analysis, athlete action evaluation, and system feedback design in shooting training scenarios. The method includes the following main steps:

[0118] Step 1: Video Input and Preprocessing

[0119] First, the system acquires video stream data generated during shooting training from the recording equipment. The video can be a real-time stream or offline recording. Next, a series of basic image processing operations are performed on each acquired frame, including:

[0120] 1. Brightness normalization processing

[0121] To minimize the impact of image brightness fluctuations caused by factors such as changes in lighting, uneven exposure, or variations in the shooting environment on subsequent image analysis steps (such as ROI localization and optical flow estimation), this step performs brightness normalization processing on each frame of the video. Specific methods include:

[0122] ① Convert the image to grayscale space or luminance channel (such as the Y or V component in YUV or HSV);

[0123] ② Unify the dynamic range of brightness through linear transformations (such as histogram stretching and normalization to a fixed brightness range);

[0124] ③ Alternatively, adaptive histogram equalization (such as CLAHE) can be used to enhance local contrast while controlling artifacts caused by over-enhancement. These methods can effectively improve the overall brightness consistency of the image and enhance the stability and robustness of subsequent feature extraction, template matching, and motion estimation.

[0125] RGB to grayscale conversion (luminance channel extraction):

[0126] Y=0.299·R+0.587·G+0.114·B

[0127] Linear brightness normalization (normalized to [0, 255]):

[0128]

[0129] Where I(x, y) represents the pixel brightness value of the original image at coordinates (x, y), I min Minimum brightness value in the image, I max Maximum brightness value in the image;

[0130] Histogram equalization formula (discrete form):

[0131]

[0132] Among them, s k This represents the pixel value after equalization, where L represents the number of gray levels, and n... j This represents the number of pixels at gray level j, where M and N represent the width and height of the image, respectively.

[0133] 2. Image denoising processing

[0134] To eliminate interference such as high-frequency noise generated during sensor photosensing and block artifacts produced during compression, this invention incorporates image denoising operations in the image preprocessing stage. Common methods include, but are not limited to: Gaussian Blur: using a two-dimensional Gaussian kernel to perform low-pass filtering on the image, smoothing high-frequency regions and effectively suppressing random noise; Median Filter: taking the median pixel value of a local region of the image, especially suitable for removing salt-and-pepper noise; Bilateral Filter: smoothing noisy regions while preserving edge details, suitable for edge-sensitive scenes; Non-Local Means: performing global modeling based on image similarity principles to retain more details. Image denoising can significantly improve the consistency of image texture in subsequent steps and reduce mismatches and optical flow estimation deviations caused by noise.

[0135] Gaussian filter definition:

[0136]

[0137] Where G(x,y) represents the weight value of the Gaussian kernel at coordinate (x,y), and σ represents the standard deviation of the Gaussian distribution;

[0138] Image Gaussian filtering operation

[0139]

[0140] Among them, I filtered (x,y) represents the pixel value of the filtered image at coordinates (x,y), and k represents the radius of the filtering kernel;

[0141] Median filtering

[0142] I filtered (x,y)=median{I(x+i,y+j)|-k≤i,j≤k}

[0143] Where median represents the median operation;

[0144] bilateral filtering

[0145]

[0146] Among them, I filtered (x) represents the pixel value at coordinate x in the filtered image, W p f represents the normalization factor, Ω represents the neighborhood window, and f r Let f represent the gray-level similarity function. s Let x represent the spatial similarity function. i Indicates the pixel coordinates within the neighborhood;

[0147] 3. Inter-frame stabilization (optional)

[0148] In some application scenarios, slight overall jitter may occur between video frames due to unstable camera mounting or external interference (such as wind or vibration). This global displacement can interfere with the accurate positioning of the ROI and the extraction of the true trajectory of the gun's movement. Therefore, this invention optionally adds inter-frame video stabilization processing during the preprocessing stage. Specific methods include:

[0149] 3.1 Calculate the global transformation (such as affine or perspective transformation) between adjacent frames, which can be estimated using feature point matching + RANSAC method;

[0150] 3.2 Use smoothing filters (such as moving average or Kalman filtering) to process inter-frame transform parameters;

[0151] 3.3 Apply an inverse transform to the current frame to stabilize it onto the previous frame or a global reference frame, eliminating interference caused by non-target motion. This process keeps the background relatively still in the video sequence, helping to highlight the actual motion changes in the muzzle area and improving the accuracy of ROI extraction and trajectory estimation.

[0152] Affine transformation model:

[0153]

[0154] Where x and y represent the transformed coordinates, and x and y represent the original coordinates, a 11 a 12 a 21 a 22 t represents the rotation and scaling parameters. x , t y This represents the translation parameter.

[0155] Step 2: Dynamic ROI Location

[0156] In each frame, to accurately locate the muzzle area, a content-based matching method is used to search the sample image. Optional matching methods include:

[0157] 1. Template Matching

[0158] Template matching is a classic image matching method that finds the most similar regions by sliding a predefined template image (i.e., a sample) across a target frame image. The specific steps are as follows: Perform a sliding window operation between the sample template image and the current video frame; calculate the NCC value between the template and the region at each sliding window position to measure the similarity; find the position with the highest correlation coefficient as a candidate ROI for the muzzle region in the current frame. Template matching is suitable for scenes where the muzzle region has relatively stable texture and minimal scale changes. It is simple to implement and has strong real-time performance, but it is sensitive to scale transformations, rotations, and occlusions.

[0159] 2. Feature Matching

[0160] To enhance the robustness of matching in complex scenarios, this invention can also introduce a matching method based on locally invariant features. Commonly used feature descriptors include:

[0161] ①SIFT (Scale Invariant Feature Transform): Extracts key points and their feature descriptors that are scale and rotation invariant in an image. It is suitable for complex transformations such as image scaling and rotation.

[0162] ②ORB (Oriented Fast and Rotated BRIEF): An efficient feature extraction and description method suitable for real-time processing, especially for resource-constrained embedded systems.

[0163] The matching process includes: extracting feature points and their descriptors from the sample and the current frame; matching point pairs using nearest neighbor or Hamming distance metrics; eliminating incorrect matches using methods such as RANSAC to determine robust matching regions; and calculating the coordinates of the ROI region based on the location of the matched points. This method has stronger adaptability to changes in viewpoint, partial occlusion, and noise interference, and is especially suitable for precise gun muzzle area localization in dynamic scenes.

[0164] 3. Similarity Measurement

[0165] In scenarios where template or feature matching is unsuitable, this invention can also employ a similarity measurement method based on overall image structural information, such as the Structural Similarity Index (SSIM). SSIM comprehensively considers the brightness, contrast, and structural information of an image to calculate the perceptual similarity between two image patches, expressing a sensitivity to image differences that is closer to the human visual system. The method of use is as follows: a sliding comparison is performed between the sample image and each candidate window region in the frame image; the SSIM score of each candidate region is calculated relative to the sample; the region with the highest score is selected as the ROI of the current frame. SSIM strongly preserves texture details and structural information, helping to achieve good ROI localization results even with degraded image quality and complex noisy backgrounds.

[0166] Structural Similarity Index (SSIM)

[0167]

[0168] Where, μ x and μ y These are the average values ​​of the x and y values ​​of the image, respectively, reflecting brightness information; and σ represents the variance of each component, indicating contrast. xy C1 is the covariance of the two images, measuring structural similarity; C2 and C1 are constants used to avoid zero denominators, and are usually set depending on the range of pixel values. The SSIM value ranges from -1 to 1, with the closer to 1 indicating higher image similarity. It is commonly used in image quality assessment and compression distortion detection.

[0169] Step 3: Spatial filtering and amplification processing

[0170] To extract and amplify subtle motion changes of the gun muzzle within the Region of Interest (ROI), this invention introduces spatial filtering amplification technology, referencing the Eulerian Video Magnification (EVM) approach, to enhance minute motion and frequency variation signals within the ROI. The processing steps include:

[0171] 1. Extract specific frequency bands from an image using a Laplacian pyramid or high-pass filter on a spatial scale.

[0172] To effectively capture minute spatial displacement changes in local areas (such as the muzzle of a gun) in an image, the first step is to...

[0173] The Region of Interest (ROI) image undergoes spatial multi-scale decomposition. Specific methods include: Laplacian Pyramid: decomposing the original image into multiple spatial frequency levels, representing the low-frequency structure and high-frequency details of the image. This method is implemented using Gaussian pyramid differencing, facilitating independent operations on motion at different scales; alternatively, high-pass filters (such as band-pass filters or edge enhancement filters) can be used to directly extract high-frequency components such as edges and details from the image, aiding in the detection of subtle local changes.

[0174] The purpose of this step is to decouple the details and structural information in the image, providing a clearer target frequency band for subsequent signal processing in the time dimension.

[0175] Construction of the Pyramid of Gauss

[0176] G0=I,G l+1 =downsample(G l *g)

[0177] Laplace Pyramid Levels

[0178] L l =G l -upsample(G l+1 )

[0179] in:

[0180] I: Original image, g: Gaussian kernel, *: Convolution operation, upsample: Downsampling and upsampling operations, L-layer Laplacian pyramid image

[0181] 2. Magnify the changing signal within a specific frequency range on a time scale to highlight minute vibrations with low amplitude and high frequency.

[0182] After spatial decomposition, the pixel value changes at each spatial frequency level are analyzed in the time dimension, and the dynamic signal within a specific frequency range is extracted and amplified. This process includes:

[0183] ① Perform one-dimensional time series modeling on the value of each pixel over time; ② Use a bandpass filter (such as an IIR filter or an ideal filter) to select the target frequency range (such as 0.4-1Hz), which usually corresponds to human movements or micro-movements of equipment.

[0184] ③ Multiply the filtered signal by an amplification factor α (such as α = 10 or α = 20) to enhance its amplitude.

[0185] This processing can significantly enhance subtle displacements that are difficult to detect with the naked eye (such as the shaking of the gun muzzle in a short period of time), making it easier to accurately track them using optical flow methods.

[0186] Pixel time series modeling and filtering

[0187] I l (x,y,t), t=1,2,...,T

[0188] 3. Reconstruct and enhance the image, preserving the original texture while making micro-motions more apparent.

[0189] After completing the spatial-temporal magnification process, the processed signals need to be resynthesized into an image sequence for optical flow algorithm processing or direct visualization. The image reconstruction steps are as follows: ① Merge the frequency layer of the temporally magnified image with the original pyramid structure or image structure; ② Perform reverse reconstruction of the pyramid structure to restore the image frame with full resolution; ③ Or, overlay the original image on top of the high-pass enhancement to ensure that the enhanced area still conforms to the original visual background.

[0190] The reconstructed image has two characteristics: it retains the original texture and background information, ensuring the integrity of the overall image structure; and the small motion features of local areas (such as the muzzle) are significantly enhanced, providing more obvious displacement features for subsequent optical flow tracing algorithms.

[0191] Spatial filtering amplification can effectively enhance the optical flow method's ability to detect subtle muzzle tremors, thereby improving the robustness and accuracy of subsequent motion estimation.

[0192] Pyramid image layer synthesis

[0193]

[0194] Step 4: Motion estimation based on optical flow method

[0195] Within the magnified Region of Interest (ROI), optical flow is used to estimate the motion of pixels or feature points in the image. This invention can be selected according to actual needs:

[0196] Fundamental constraint equations for optical flow:

[0197] I(x,y,t)=I(x+Δx,y+Δy,t+Δt)

[0198] Performing a Taylor expansion, we obtain the optical flow constraint equation:

[0199]

[0200] Dense optical flow algorithms (such as Farneback optical flow)

[0201] Dense optical flow algorithms aim to estimate the motion displacement vector of each pixel in an image between adjacent frames, i.e., to construct a complete optical flow field. In this invention, the Farneback optical flow method is selected as a representative method, characterized by efficient computation and sensitivity to small-scale motion, making it suitable for tasks involving minute motion detection.

[0202] The specific processing steps are as follows: construct a multi-scale image pyramid for two consecutive frames in the ROI region; in each layer, use the Farneback method to approximate the grayscale changes of the image patch through polynomial expansion to estimate local motion; iterate and optimize in the pyramid from top to bottom, and finally obtain the displacement vector field (OpticalFlow Field) of each pixel on the original resolution layer.

[0203] The Farneback method approximates the local grayscale values ​​of an image as a quadratic polynomial:

[0204]

[0205] This pixel-level motion estimation can depict the overall micro-motion trends of the muzzle area in detail, such as jitter patterns, direction of motion, and diffusion areas, which is of great value for constructing continuous trajectories and motion energy fields. Its drawback is that it is more sensitive to occlusion, rapid changes, or areas with monotonous textures, but after spatial filtering and amplification, the minute jitters are enhanced, which helps to overcome these limitations.

[0206] Sparse optical flow algorithms (such as the Lucas-Kanade method)

[0207] Sparse optical flow algorithms estimate the motion path of a set of key points in an image across consecutive frames by tracking that set. Compared to dense optical flow, it requires less computation and is more suitable for real-time processing or resource-constrained systems. In this invention, the classic Lucas-Kanade optical flow method is chosen, which is based on the assumptions of grayscale preservation and local window consistency.

[0208] The algorithm process includes: using corner detection (such as the Shi-Tomasi algorithm) to extract feature points with significant local changes in the ROI of the first frame; in the second frame, constructing a small window centered on each feature point, assuming that all pixels within the window have the same motion; solving for the displacement vector of each feature point by minimizing the error of the optical flow constraint equation; optionally introducing an image pyramid structure to handle large-scale displacements and improve robustness.

[0209] The core formula of the sparse optical flow method—the Lucas-Kanade method—constructs an error minimization problem within the neighborhood Ω of each key point:

[0210]

[0211] The Lucas-Kanade method is suitable for scenes with clear features and rich textures, and is especially suitable for tracking local structures such as muzzle edges and aiming points, making it easy to form sparse trajectory lines with physical meaning.

[0212] In training scenarios, its output can be used to evaluate the stability of a shooter's movements, analyze the drift range of specific key points, or extract feature paths for subsequent recognition.

[0213] By calculating the displacement vectors of feature points between the current frame and the previous frame, the motion trajectory estimation between consecutive frames is obtained, and a dynamic trajectory line of the gun muzzle on the time axis is constructed. This motion trajectory will serve as the core data foundation for the system output.

[0214] Step 5: Trajectory Smoothing and Output

[0215] In actual video acquisition and processing, various factors often influence the data, such as sensor noise, optical flow estimation errors, image compression artifacts, background interference, or matching offsets. These factors cause random jitter, discontinuous jumps, or slight drift deviations in the muzzle motion trajectory obtained directly through optical flow algorithms. These errors not only reduce the readability of the trajectory data but may also affect the accuracy of subsequent analysis algorithms.

[0216] To improve the stability and practical value of the trajectory, this step introduces a trajectory smoothing mechanism to denoise, fit, and predict the original motion path. Common methods include, but are not limited to:

[0217] Moving Average Filter

[0218] Moving average filtering is a simple and effective linear smoothing method suitable for removing high-frequency jitter noise from trajectory data.

[0219] The processing method is as follows: at each time t, the trajectory coordinates of the point and several frames before and after it (e.g., 3 to 7 frames) are weighted and averaged; the smoothed point coordinates are output as the trajectory position of the frame; the filter window size is adjustable to balance the smoothing effect and response delay.

[0220] Moving average filter:

[0221]

[0222] This method can effectively suppress abnormal fluctuations caused by single-frame errors, making the trajectory curve more consistent and smooth, and suitable for later visualization.

[0223] The Kalman filter, as a recursive estimation algorithm, performs real-time denoising, smoothing, and prediction on the initial motion trajectory extracted by optical flow or spatial filtering amplification. Based on the system's state transition model and observation model, it updates the current estimate and covariance in each frame, effectively suppressing errors caused by image noise or local jitter, and enhancing the continuity and reliability of the trajectory. The smoothed trajectory after Kalman filtering serves as the final trajectory output or as a stable input basis for subsequent modules such as action recognition, pose analysis, or statistical measurement.

[0224] Kalman filter:

[0225]

[0226] x t =Fx i-1 +w t

[0227] In this invention, a Kalman filter is used to estimate the state of the muzzle trajectory, and the model includes:

[0228] State variables (such as position, velocity);

[0229] The observation model refers to the trajectory measurements obtained from optical flow estimation.

[0230] The system noise and observation noise covariance matrix is ​​used to control the filter response sensitivity.

[0231] Each frame undergoes two steps using Kalman filtering: predicting the position of the trajectory point in the current frame; updating the deviation between the predicted value and the actual observation value; and outputting the optimal estimate.

[0232] This method can effectively integrate historical trajectory information with current observation results to achieve trajectory smoothing, completion and error correction, and has good robustness, especially in the case of short-term occlusion or missing data.

[0233] Recursive calculation process:

[0234]

[0235]

[0236] in It is the prior state estimate at the current time t, and F is the state transition matrix. This is the state estimate from the previous step. It is the prior covariance matrix, P t-1 Here, Q is the covariance matrix of the previous time step, H is the process noise covariance matrix, R is the observation matrix, and K is the observation noise covariance. t It is the Kalman gain, z t These are actual observed values. It is the updated state estimate, where I is the identity matrix;

[0237] Trajectory Output and Visualization

[0238] After smoothing, the system can output the optimized trajectory data in two ways:

[0239] Graphical visualization output: The smooth trajectory is overlaid on the original video frame as a continuous curve or trajectory points;

[0240] Data output in coordinate format: The trajectory is exported as a (x,y,t) triplet as structured data, such as CSV, JSON or database records; it can be further used for behavior analysis (such as stability assessment, trajectory clustering), motion feature extraction (such as vibration frequency, shaking amplitude) or machine learning model training.

[0241] This technical proposal has the following characteristics:

[0242] 1. Possesses real-time processing capabilities

[0243] The muzzle trajectory tracking method designed in this invention has high operating efficiency, enabling rapid processing and analysis of each frame of an image in a video stream. Through optimization of the algorithm flow (such as using a lightweight image processing module, local ROI processing, and adapting to an efficient optical flow algorithm), the system can achieve near real-time operation, meeting the real-time feedback requirements during shooting training. This method has low dependence on computing resources and can be deployed and run on ordinary PCs, embedded devices, and even mobile platforms, demonstrating good practicality and promising prospects for widespread application, especially suitable for on-site teaching, intelligent target machine systems, or portable auxiliary training equipment.

[0244] 2. Automatic matching of dynamic ROI regions

[0245] In each frame, the system automatically locates the Region of Interest (ROI) containing the gun muzzle by matching a preset sample image with the current image content, eliminating the need for manual pre-labeling, selection, or configuration of fixed coordinates. This process is based on techniques such as template matching, feature matching, or similarity analysis, exhibiting strong adaptability and high generalization ability, capable of adapting to changes in conditions such as different gun types, backgrounds, and shooting angles. This feature significantly improves the system's automation level and robustness, providing a stable and accurate initial input for subsequent processing.

[0246] 3. Introduce spatial filtering to amplify and enhance minute motion signals.

[0247] To effectively extract the minute muzzle shakes and subtle displacement changes that occur during firing, this invention uses spatial filtering and amplification as a pre-processing step for optical flow analysis. Drawing inspiration from Eulerian Video Magnification, the ROI region image is first subjected to joint filtering and amplitude amplification (e.g., ...) at both spatial and temporal scales. Figure 2 The image shows a magnified muzzle image, which makes previously imperceptible high-frequency, low-amplitude movements more prominent in both visual and computational processing. This processing greatly enhances the optical flow algorithm's sensitivity to micro-motion signals, improving the accuracy and stability of subsequent motion estimation, and is one of the key innovations in achieving high-precision trajectory tracking.

[0248] 4. Achieving high-precision trajectory estimation by combining optical flow method.

[0249] refer to Figure 3 Using the muzzle center as the reference point, within a magnified Region of Interest (ROI), optical flow methods (including dense or sparse optical flow algorithms) are applied to estimate the motion of pixels or feature points between consecutive frames, thereby constructing the inter-frame muzzle trajectory. Optical flow methods can meticulously capture the instantaneous displacement trend of the target, and are particularly suitable for reflecting continuous motion states over a short period of time. This method can realistically and accurately reproduce the dynamic behavior of the muzzle in space during firing, assisting in evaluating the shooter's stability, aiming consistency, and instantaneous control ability. It is the core technical component of this invention for achieving high-precision spatiotemporal tracking.

[0250] 5. No need to rely on additional hardware sensors

[0251] Unlike traditional methods that require external sensing devices such as laser targets, infrared trackers, inertial measurement units (IMUs), or high-frame-rate industrial cameras, the muzzle trajectory tracking method used in this invention can achieve high-resolution motion trajectory analysis entirely based on ordinary video data input. This advantage not only significantly reduces the economic cost of system deployment and operation but also enhances application flexibility, making it widely adaptable to various training scenarios and teaching environments. It provides a practical and feasible solution for the promotion and popularization of shooting assistance systems.

[0252] 6. Combines visualization and data output

[0253] refer to Figure 4 This invention supports dual output modes: on the one hand, the system can overlay the muzzle motion trajectory onto the original video image in real time as a curve, trajectory point, or dynamic line segment, providing intuitive visual feedback of the motion process; on the other hand, the trajectory data can also be exported as a structured coordinate sequence, combined with derived features such as timestamps, velocity vectors, and direction changes, for subsequent training analysis, motion modeling, or data mining. This design concept of visual display + numerical analysis helps coaches and shooters deeply understand the characteristics of the movements, enabling data-driven precision training optimization.

[0254] The core innovations of this application are as follows:

[0255] 1. Spatial filtering and amplification are introduced into the muzzle trajectory tracking process to enhance minute motion signals;

[0256] 2. Dynamic ROI matching mechanism based on samples for each frame;

[0257] 3. A method for the combined processing of spatial filtering and optical flow tracing;

[0258] 4. It can output trajectory data and visualization results in real time;

[0259] 5. The algorithm achieves high-precision trajectory tracking without the need for additional hardware support.

[0260] What are the technical advantages of this application proposal compared to existing technologies?

[0261] 1. Enhanced detection capabilities:

[0262] This application significantly enhances the visibility of minute motion signals in video images by introducing a spatial filtering amplification mechanism into the traditional image processing workflow. It is particularly effective at amplifying low-amplitude, high-frequency jitter that is extremely difficult to detect during shooting. Compared to existing methods with limited sensitivity to image motion (such as detection based directly on frame differences or feature changes), this invention significantly enhances subtle movements while preserving the original texture and structure of the image, thereby improving the identifiability and interpretability of micro-motion events (such as aiming stability and displacement at the moment of trigger pull).

[0263] 2. Improved tracking accuracy:

[0264] Existing methods often employ frame difference or background modeling to estimate the motion of target regions, which are easily affected by brightness fluctuations, background interference, or occlusion, leading to significant trajectory estimation errors. This application adopts a motion estimation strategy based on optical flow, which accurately reconstructs the true motion trajectory of the target object by analyzing the continuous displacement vectors of pixels or feature points between adjacent frames. Especially in images after magnification, subtle motion features are more prominent, giving the optical flow algorithm higher response sensitivity and robustness, significantly improving the stability and accuracy of trajectory tracking.

[0265] 3. Strong automatic adaptability:

[0266] Unlike methods that rely on static preset regions or manual annotation, this invention proposes a dynamic ROI automatic matching mechanism. This mechanism can automatically locate the muzzle area in each frame of the image based on techniques such as template matching, feature matching, or similarity calculation. This mechanism can adapt to interference factors such as changes in shooting angle, slight target displacement, and differences in lighting, avoiding the problem of target loss or misjudgment due to fixed area limitations. This enables robust tracking of the muzzle trajectory in dynamic scenes, significantly improving the system's generalization ability and robustness.

[0267] 4. No external devices required:

[0268] Traditional high-precision muzzle motion analysis systems often rely on external devices such as laser targets, IMU sensors, infrared markers, or high-frame-rate industrial cameras, increasing system cost, maintenance complexity, and deployment barriers. This proposed solution requires only standard-resolution video as input to achieve high-precision muzzle trajectory tracking without relying on any external sensors or additional markers. This pure vision solution significantly reduces the difficulty of technology integration and usage costs, and has good portability and scalability.

[0269] 5. Can be deployed in real time:

[0270] This invention fully considers the computational efficiency of the algorithm and the system response latency in its design. By optimizing and trimming modules such as video preprocessing, ROI matching, magnification and enhancement, and optical flow estimation, a lightweight, real-time motion analysis workflow is constructed. This system can run stably on standard hardware platforms (such as ordinary laptops, edge computing devices, and embedded systems), achieving online real-time feedback analysis. It is particularly suitable for practical application scenarios with high requirements for response speed and ease of deployment, such as intelligent shooting range systems, shooting teaching aid software, and portable shooting training terminals.

Claims

1. A method for tracking micro-motion trajectory based on optical flow method and spatial filtering amplification, characterized in that, Includes the following steps: Step 1: Receive the video frame sequence and preprocess each frame image, including brightness normalization and image denoising; Step 2: In each preprocessed frame of the image, based on a predefined reference template, the muzzle region is dynamically located using a matching method to obtain the region of interest; Step 3: Apply spatial filtering and amplification processing to the region of interest, including: decomposing the image frequency layer on a spatial scale, amplifying the changing signal within a specific frequency range on a temporal scale, and reconstructing it into an enhanced image to highlight minute movements; Step 4: On the enhanced region of interest, apply optical flow to estimate the motion vector, wherein the optical flow method is either a dense optical flow algorithm or a sparse optical flow algorithm; Step 5: Construct a trajectory point sequence based on the motion vector, smooth the trajectory point sequence, and output the smoothed trajectory data; The region of interest (ROI) is subjected to spatial multi-scale decomposition, specifically including the extraction of specific frequency bands of the image using a Laplacian pyramid or a high-pass filter at the spatial scale. The construction process of the Pyramid of Laplace is as follows: in, Represents the original image. Represents the Gaussian kernel. This represents the convolution operation. Indicates the first A layered Gaussian pyramid image. Indicates a downsampling operation; Then, the Laplace pyramid layers are constructed using differences: wherein, denotes an up-sampling operation, denotes the first layer of Laplacian pyramid images; Alternatively, a high-pass filter can be used to directly extract high-frequency components from the image to enhance local detail changes; The above filtering process yields an image for a specific frequency band. The pixel value changes at each spatial frequency level are analyzed over time, and the target frequency band is selected for amplification. Specifically, this includes: Model the value of each pixel as a one-dimensional time series. , indicating the first The image after spatial frequency decomposition in time The pixel value at time t, where ; The time series is filtered using a bandpass filter to select a target frequency band; the filtered signal is then multiplied by an amplification factor. This yields an enhanced pixel value change signal. ; The processed image frequency layer is merged with the original pyramid structure or image structure and reconstructed into an enhanced image sequence, specifically including: wherein, represents a reconstructed Laplacian pyramid layer, represents an enlarged pixel value variation signal.

2. The method of claim 1, wherein, Step 1 includes: Receive video stream data, wherein the video stream data is a real-time stream or offline recorded content; Perform basic image processing operations on each acquired frame, including: Convert the image to grayscale space or the luminance channel, and unify the luminance dynamic range using one of the following methods: Linear transformation, including histogram stretching or normalization to a fixed brightness range, is specifically formulated as follows: wherein, represents a pixel intensity value of the original image at coordinates represents a pixel intensity value of the original image at coordinates represents a minimum intensity value in the image, represents a maximum intensity value in the image; Alternatively, adaptive histogram equalization, including the CLAHE method, can be used, with the following formula: in, This represents the pixel value after equalization. Represents the number of gray levels. Represents grayscale level The number of pixels, and These represent the width and height of the image, respectively. pass or After obtaining a brightness-normalized image, high-frequency noise and compression artifacts are eliminated using one of the following methods: Gaussian filtering, the specific formula is: wherein, denotes the weight value of the Gaussian kernel at the coordinate denotes the weight value of the Gaussian kernel at the coordinate denotes the standard deviation of the Gaussian distribution; Image Gaussian filtering operation in, Indicates the filtered image in coordinates Pixel value at that location, Indicates the radius of the filter kernel. The weighting function of the Gaussian filter in coordinates The value at that location, This indicates the offset of the convolution kernel in the horizontal and vertical directions; Median filtering wherein denotes the median operation; bilateral filtering in, Indicates the filtered image in coordinates Pixel value at that location, Represents the normalization factor. Represents the neighborhood window, Represents the gray-level similarity function. Represents the spatial similarity function, Indicates the pixel coordinates within the neighborhood; By After eliminating noise and artifacts, the overall jitter between video frames is selected to be eliminated, including computing global transformations between adjacent frames, processing transformation parameters using a smoothing filter, and applying an inverse transformation, wherein the global transformation adopts an affine transformation model: in, and Represents the transformed coordinates. and Represents the original coordinates Indicates rotation and scaling parameters, This represents the translation parameter.

3. The method of claim 1, wherein, Step 2 includes the following steps: In each frame of the image, a predefined reference template is searched using a matching method to determine the area where the gun muzzle is located. The matching method includes one of the following three forms: Template matching: The reference template is slid across the current video frame image for comparison, the normalized cross-correlation coefficient between each sliding window position and the template is calculated, and the position with the largest correlation coefficient is selected as the ROI of the current frame; Feature matching: Local invariant feature points and their descriptors are extracted from the reference template and the current frame image, respectively, and matched using nearest neighbor or Hamming distance. After removing mismatches using the RANSAC algorithm, the coordinates of the ROI region are calculated based on the location of the matching points. Similarity calculation: The reference template is compared with each candidate window region in the current video frame. The perceptual similarity between each window and the template is calculated using the structural similarity index. The window with the highest SSIM score is selected as the ROI of this frame.

4. The method of claim 1, wherein, Step 4 includes the following steps: In the image enhancement sequence, the motion vectors of pixels or feature points are estimated according to the fundamental optical flow constraint equations: in, Indicates the image in coordinates and time brightness value, and They represent and The velocity component in the direction of motion; this is an equation with two unknowns. The problem that cannot be solved directly is called the "aperture problem" in optical flow estimation. One of the following two optical flow algorithms is used for motion estimation to approximate the solution: A. Dense optical flow algorithm, which gives an estimate of the optical flow for each pixel in the image, i.e. There is an estimate for all pixels: A1. Constructing an image pyramid: Perform multi-scale pyramid decomposition on two consecutive frames of ROI images; A2. In each pyramid layer, approximate the grayscale changes of the image patch using polynomial expansion: wherein denotes the pixel coordinate, is a symmetric matrix, is a vector, c is a constant term; B. Sparse optical flow algorithms, which aim to estimate the optical flow only for a few feature points in the image, that is, only for these key points and not for all pixels; B1. Extract corner features from the ROI in the first frame; B2. In subsequent frames, a local window is constructed centered on each feature point. Assuming that the pixels within the window move uniformly, the displacement is solved by minimizing the error: wherein respectively denote the partial derivatives of the image in directions, respectively denote the displacement components in directions; by solving the linear equation system the displacement vector wherein: The muzzle motion trajectory is constructed based on the displacement vectors between consecutive frames.

5. The method of claim 1, wherein, Step 5 includes the following steps: The original muzzle motion trajectory estimated by the optical flow method is smoothed to remove random jitter, jumps, or drift deviations. The smoothing process employs one of the following methods: At each time point The smoothed point coordinates are output by weighted averaging of the point and the trajectory coordinate values in several frames before and after the point, and the specific formula is: in, Indicates time The original trajectory Coordinate values Indicates time The original trajectory Coordinate values Indicates the radius of the filter kernel. Indicates smoothed Coordinate values Represents the smoothed Coordinate values; Kalman filter: consists of two stages: prediction and update. The specific formula is as follows: The state variable is defined as: in, Indicates time The state vector, and Let represent the position coordinates of the two-dimensional muzzle trajectory estimated by the optical flow method. and They represent and The velocity component in the direction; The state transition equation is: wherein is the state transition matrix, represents the system noise; and the observation equation is wherein is the actual observation, is the observation matrix, denotes the observation noise; Kalman filtering recursion is used to dynamically estimate the target state. The calculation process includes: in It is the current moment. Prior state estimation requires manually set initial state estimates, extracted from the first frame image. , The speed is set to 0 or estimated using the difference between the first and second frames. The initial value has a significant impact on the first few frames, but its influence gradually weakens as filtering progresses. F is the state transition matrix. This is the state estimate from the previous step. It is the prior covariance matrix. It is the covariance matrix of the previous time step. It is the process noise covariance matrix. It is the observation matrix. It is the observation noise covariance. It is Kalman gain. These are actual observed values. This is the updated state estimate. It is the identity matrix; After smoothing, the optimized trajectory data will be output in two ways: Graphical visualization: Overlaying smooth trajectories as continuous curves or trajectory points onto the original video frames; Structured coordinate output: The trajectory data is exported as structured data in the form of triples. triples.

Images