Object tracking method based on spatial transformer network and convolutional network

Abstract

The present invention provides a target tracking method based on a spatial transformation network and a convolutional network, belonging to the field of machine vision technology. The method comprises the following steps: moving a designated target according to different motion trajectories in a complex scene and capturing a video, and frame-splitting the captured video; extracting features from the frame image using a convolutional network to obtain a first feature map; spatially transforming the first feature map and focusing on the target area based on the spatial transformation network to obtain a second feature map; extracting advanced features from the second feature map based on the convolutional network to predict target locking; obtaining predicted trajectories based on the locked positions of all predicted targets and sorting them in frame order, and performing comparative analysis with the corresponding movement trajectories to obtain a moving boundary difference set; and optimizing and adjusting the target tracking edge based on all moving boundary difference sets. In complex scenes, even if the target is subject to various interferences and changes, the optimized and adjusted target tracking edge can more stably track the target.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a target tracking method based on a spatial transformation network and a convolutional network. Background Art

[0002] In the field of target tracking, traditional methods suffer from low tracking accuracy and poor robustness when dealing with fast-moving targets and similar background interference. While methods based on convolutional neural networks have made some progress with the development of deep learning, they still face challenges such as spatial perception bias and the difficulty in balancing algorithm complexity and speed. For example, in complex scenarios, the rapid movement of targets can lead to inaccurate spatial position predictions, while similar backgrounds can easily cause target misjudgment, compromising tracking effectiveness.

[0003] Therefore, the present invention proposes a target tracking method based on spatial transformation network and convolutional network. Summary of the Invention

[0004] The present invention provides a target tracking method based on a spatial transformation network and a convolutional network to solve the technical problems raised by the above background technology.

[0005] The present invention provides a target tracking method based on a spatial transformation network and a convolutional network, comprising:

[0006] Step 1: Move the designated target along different motion trajectories in a complex scene and shoot video, then split the captured video into frames;

[0007] Step 2: Using a convolutional network to extract features from the frame image to obtain a first feature map, and performing spatial transformation and target area focusing on the first feature map based on the spatial transformation network to obtain a second feature map;

[0008] Step 3: extracting high-level features from the second feature map based on the convolutional network to predict target locking;

[0009] Step 4: Based on the locked positions of all predicted targets and sorted in frame order, the predicted trajectories are obtained and compared with the corresponding moving trajectories to obtain a moving boundary difference set, wherein the moving boundary difference set includes the boundary difference area shape of each frame image;

[0010] Step 5: Optimize and adjust the target tracking edge based on all moving boundary difference sets.

[0011] Preferably, the complex scenes are deployed in advance, and there are N1 complex scenes, and each complex scene has N2 pre-set motion trajectories.

[0012] Preferably, the frame splitting of the captured video includes:

[0013] Determine the duration of each motion track, and obtain the unit split duration consistent with the motion duration from the duration-split comparison table. Perform corresponding splitting of blank tracks for complex scenes without motion tracks to obtain several original frames.

[0014] Analyzing the distribution of colors close to the target color of the designated target in each raw frame and setting a priority to each raw frame;

[0015] Splitting the video captured under the movement trajectory according to the unit splitting time to obtain comparison frames, and analyzing the visual effect and coherence of each comparison frame;

[0016] Determine the number of refinements for the corresponding frame based on the priority, visual effect, and coherence of the same frame;

[0017] According to the sum of all the refined quantities, the unit splitting duration is readjusted, and the captured video is frame split according to the adjusted duration.

[0018] Preferably, the number of refinements of the corresponding frame is determined based on the priority, visual effect, and coherence of the same frame, including:

[0019]

[0020] in, represents the number of refinements for the corresponding frame, Indicates the priority of the corresponding frame, Indicates the preset level of the corresponding frame, Indicates the preset effect coefficient of the corresponding frame; Represents the visual effect coefficient of the corresponding frame; Indicates the preset nature of the corresponding frame; Indicates the coherence of the corresponding frames; Indicates the rounding symbol; represents a constant, whose value is 2.7; ln represents the sign of the logarithmic function.

[0021] Preferably, performing high-level feature extraction on the second feature map based on the convolutional network includes:

[0022] Construct a deep residual convolution module, where the deep residual convolution module contains multiple residual blocks, and each residual block fuses features from different layers through skip connections;

[0023] At the same time, a multi-scale convolution kernel is introduced to process the second feature map in parallel through convolution kernels of different sizes to obtain feature information at different scales;

[0024] Using a time series analysis method, combined with the position, velocity, and acceleration of the designated target in the previous frame, the position and state of the designated target in the current frame are predicted;

[0025] The extracted high-level features are combined with the predicted target location information to lock the predicted target.

[0026] Preferably, the target tracking edge is optimized and adjusted based on all moving boundary difference sets, including:

[0027] Based on the moving boundary difference set, the actual boundary and the standard boundary of the locked position of each image frame are respectively mapped into a two-dimensional coordinate system, and the shape of the boundary difference region between the actual boundary and the standard boundary is determined, and the difference area Si, the maximum difference distance L1i, and the minimum difference distance L2i of the boundary difference region shape are determined, wherein the actual boundary is the boundary of the predicted target;

[0028] like , at this time, assign 0 to the corresponding image frame, otherwise, assign 1 to the image frame, where, represents the area difference threshold; Indicates the radius of the target object; Indicates the boundary recognition accuracy; Indicates the number of image frames in the corresponding captured video;

[0029] The image frame assigned 1 is regarded as the first frame, and the background corresponding to the boundary difference area of the first frame is extracted from the complex background and regarded as the blurred background;

[0030] If the blurred background is of the expansion type, the blurred background and the designated target image are converted from the RGB color space to the HSV color space, and the target edge is locked;

[0031] If the blurred background is of the indented type, then the texture features of the blurred background and the designated target image are extracted based on the gray level co-occurrence matrix, and the target edge is locked according to the texture features;

[0032] If the blurred background is both an expanding and a shrinking type, then the edge is self-drawn based on the known shape features of the designated target and the standard trajectory center point under the corresponding image frame to lock the target edge;

[0033] Performing another difference analysis based on the target edge and the standard edge in the corresponding image frame to obtain an updated difference vector;

[0034] Project the non-zero elements in the updated difference image vector into the two-dimensional coordinate system respectively. If the difference area of each non-zero element is less than or equal to and the variance of the difference areas of all non-zero elements in the updated difference vector is less than a1, in which case the target edge of each image frame is retained;

[0035] Otherwise, the target edge of each first frame is manually selected and the difference distance set of the first frame is constructed, and the labeled image of the first frame is obtained, and the neural network is optimized and trained until the analysis of all moving boundary difference sets is completed.

[0036] Preferably, manually selecting the target edge of each first frame, constructing a difference distance set of the first frame, and obtaining the annotated image of the first frame include:

[0037] The edge of the manually selected target is displayed prominently in the first frame using the first color;

[0038] Distance lines are prominently displayed using a second color display on the first frame according to the difference distance set, wherein each distance line starts from a corresponding point on the edge of the target and ends at the other end of the corresponding difference distance;

[0039] The labeled image is obtained based on the saliency display results.

[0040] Preferably, the updated difference vectors correspond one-to-one to the image frames in the corresponding moving boundary difference set, and the non-zero elements corresponding to the first frame image are assigned 0 elements corresponding to the image that is assigned 0.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] By moving a designated target along multiple motion trajectories in complex scenes and performing frame splitting, a large amount of frame image data containing information about different target motion states and complex environments can be obtained. A convolutional network is used to extract basic features from the frame images to generate a first feature map, providing preliminary feature information for subsequent analysis. The first feature map is then processed using a spatial transformer network to generate a second feature map, making the target's representation more prominent and standardized. Advanced feature extraction from the second feature map and predictive target lock are performed to accurately identify the target's category and location in frames of complex scenes. Comparative analysis of the predicted and moving trajectories yields a moving boundary difference set, which can be used to quantitatively assess the accuracy and reliability of the predicted target lock. Optimizing the target tracking edge based on the moving boundary difference set allows the target tracking edge to more accurately reflect the target's actual position and shape, reducing errors and drift during the target tracking process. In complex scenes, even when the target is subject to various interferences and changes, the optimized target tracking edge can more stably track the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is a flowchart of a target tracking method based on a spatial transformation network and a convolutional network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0046] The present invention provides a target tracking method based on spatial transformation network and convolutional network, such as Figure 1 Shown, including:

[0047] Step 1: Move the designated target along different motion trajectories in a complex scene and shoot video, then split the captured video into frames;

[0048] Step 2: Using a convolutional network to extract features from the frame image to obtain a first feature map, and performing spatial transformation and target area focusing on the first feature map based on the spatial transformation network to obtain a second feature map;

[0049] Step 3: extracting high-level features from the second feature map based on the convolutional network to predict target locking;

[0050] Step 4: Based on the locked positions of all predicted targets and sorted in frame order, the predicted trajectories are obtained and compared with the corresponding moving trajectories to obtain a moving boundary difference set, wherein the moving boundary difference set includes the boundary difference area shape of each frame image;

[0051] Step 5: Optimize and adjust the target tracking edge based on all moving boundary difference sets.

[0052] Preferably, the complex scenes are deployed in advance, and there are N1 complex scenes, and each complex scene has N2 pre-set motion trajectories.

[0053] In this embodiment, a complex scene is a pre-planned and arranged environment with high complexity and interference, used to simulate various real-world conditions that are unfavorable to target recognition and tracking. For example, a complex scene simulated for security surveillance may contain a large number of similar objects, complex background patterns, varying lighting conditions, and dynamic interference. Here, N1 different complex scenes are assumed, each with a unique combination of environmental layout and interference factors.

[0054] In this embodiment, the designated target is an object that needs to be paid attention to and locked in research or application, such as a suspicious person in security monitoring, a specific vehicle in autonomous driving, etc.

[0055] In this embodiment, a motion trajectory is the path that a specified target follows in a complex scene. N2 motion trajectories are pre-set. For example, in a simulated security surveillance scenario, a person might move through a crowd along an S-shaped path, circle an area, or zigzag between multiple obstacles.

[0056] In this embodiment, frame splitting involves breaking down the captured video into a series of independent static image frames, each representing a specific moment in the video. For example, a video with 25 frames per second, after frame splitting, will have 25 independent images per second. Using a high-definition camera, the target's movement is captured from different angles, ensuring a complete record of the target's motion in complex scenarios. After filming is complete, video processing software or algorithms are used to split the captured video into frames, resulting in a series of continuous frame images, which provide basic data for subsequent analysis and processing.

[0057] In this embodiment, the first feature map is the result of a convolutional network performing preliminary feature extraction on a frame image. It contains basic image feature information, such as lines, simple shapes, and local textures. The second feature map is the feature map obtained after the spatial transformation network spatially transforms the first feature map and focuses on the target area. Compared to the first feature map, the target features in the second feature map are more prominent, and background interference is further reduced, which is more conducive to subsequent high-level feature extraction and targeting of the target. The convolutional network extracts features from the frame image through its internal convolutional layers, pooling layers, and other structures. After multiple layers of calculations, the first feature map is generated. For example, using a ResNet-50 network, the image passes through multiple residual blocks and pooling layers in sequence to extract the basic image features. The first feature map is then input into the spatial transformation network. The spatial transformation network learns the position and posture information of the target in the image and performs spatial transformation on the first feature map, such as rotating the target to an upright orientation and enlarging the target area. At the same time, it uses an attention mechanism to focus on the target area and suppress background noise and irrelevant information, thereby generating the second feature map.

[0058] In this embodiment, advanced features are more abstract and semantic than basic features, reflecting the essential attributes and category information of the target. For example, for pedestrian targets, advanced features may include the overall structure of the human body, posture, and clothing style; for vehicle targets, advanced features may include the vehicle's outline, headlight shape, wheel structure, etc.

[0059] In this embodiment, target lock prediction is achieved by using an algorithm to determine the target's position in the image and its category based on the extracted high-level features. The target is then marked in the image with a bounding box or other means to achieve target lock. The second feature map is then fed back into the convolutional network. The network's deeper layers, such as those containing more convolutional layers and fully connected layers, further extract and abstract the second feature map to reveal the target's high-level semantic features. For example, in the deep layers of the network, through the stacking of multiple convolutional layers and the application of nonlinear activation functions, high-level features such as the target's overall structure and functional components are gradually extracted. Based on these extracted high-level features, a classifier (such as a Softmax classifier) is then used to determine the target's category. Simultaneously, a regressor (such as a Bounding-Box regressor) is used to predict the target's position in the image and determine its bounding box, achieving target lock prediction.

[0060] In this embodiment, the specific coordinate position of the target in the frame image obtained by predicting target lock is typically represented by the coordinates of the upper left and lower right corners of the bounding box. The predicted trajectory is the target motion trajectory formed by connecting the target lock positions in all frames in the order of the frames. The movement trajectory is the actual motion path of the designated target in the complex scene, which was pre-set in step 1.

[0061] In this embodiment, the moving boundary difference set is a set of results obtained after comparative analysis, and contains the shape information of the boundary difference area between the predicted trajectory and the moving trajectory in each frame image. For example, in a certain frame, there may be an irregular difference area between the predicted target bounding box and the bounding box corresponding to the actual target movement path. The shape information of this area is included in the moving boundary difference set.

[0062] The target tracking edge is the boundary line or bounding box used to define the target range and track the target's motion during the target tracking process. It is updated and adjusted across different frames as the target moves. Optimization and adjustment modify and refine the position, shape, size, and other parameters of the target tracking edge based on the information provided by the moving boundary difference set to improve the accuracy and stability of target tracking. For example, if the bounding box of the predicted trajectory in a certain frame is smaller than the bounding box of the actual movement trajectory, and the difference area is concentrated to the right of the target, the target tracking edge can be appropriately expanded to the right. If the difference area is irregular in shape, it indicates that the target's posture or position has changed significantly, and more detailed adjustments to the target tracking edge are required, such as adjusting the angle and shape of the bounding box.

[0063] The beneficial effects of the above technical solution are as follows: by moving a designated target along multiple motion trajectories in complex scenes and performing frame splitting, a large amount of frame image data containing different target motion states and complex environmental information can be obtained. The basic features of the frame image are extracted through a convolutional network to obtain a first feature map, which provides preliminary feature information for subsequent analysis. The first feature map is then processed using a spatial transformation network to obtain a second feature map, making the target's representation in the feature map more prominent and standardized. By performing advanced feature extraction on the second feature map and achieving predicted target locking, the target's category and position can be accurately identified in the frame image of the complex scene. By comparing and analyzing the predicted trajectory and the moving trajectory to obtain a moving boundary difference set, the accuracy and reliability of the predicted target locking can be quantitatively evaluated. After optimizing and adjusting the target tracking edge based on the moving boundary difference set, the target tracking edge can more accurately reflect the actual position and shape of the target, reducing errors and drift during the target tracking process. In complex scenes, even if the target is subject to various interferences and changes, the optimized and adjusted target tracking edge can more stably track the target.

[0064] The present invention provides a target tracking method based on a spatial transformation network and a convolutional network, which performs frame splitting on a captured video, including:

[0065] Determine the duration of each motion track, and obtain the unit split duration consistent with the motion duration from the duration-split comparison table. Perform corresponding splitting of blank tracks for complex scenes without motion tracks to obtain several original frames.

[0066] Analyzing the distribution of colors close to the target color of the designated target in each raw frame and setting a priority to each raw frame;

[0067] Splitting the video captured under the movement trajectory according to the unit splitting time to obtain comparison frames, and analyzing the visual effect and coherence of each comparison frame;

[0068] Determine the number of refinements for the corresponding frame based on the priority, visual effect, and coherence of the same frame;

[0069] According to the sum of all the refined quantities, the unit splitting duration is readjusted, and the captured video is frame split according to the adjusted duration.

[0070] Preferably, the number of refinements of the corresponding frame is determined based on the priority, visual effect, and coherence of the same frame, including:

[0071]

[0072] in, represents the number of refinements for the corresponding frame, Indicates the priority of the corresponding frame, Indicates the preset level of the corresponding frame, Indicates the preset effect coefficient of the corresponding frame; Represents the visual effect coefficient of the corresponding frame; Indicates the preset nature of the corresponding frame; Indicates the coherence of the corresponding frames; Indicates the rounding symbol; represents a constant, whose value is 2.7; ln represents the sign of the logarithmic function.

[0073] In this embodiment, the values of both the presetness and preset effect coefficients are 1, and the value of the preset level is 0.5. In video frame processing, different frames have varying importance and quality. Relying solely on a single factor (such as priority or visual effect alone) to determine the number of frame refinements cannot fully and accurately reflect the actual frame processing requirements. This formula comprehensively considers multiple key factors, including priority (Yx), preset level (R0), preset effect coefficient (G0), visual effect coefficient (Gx), presetness (U0), and coherence (Ux): Priority: Reflects the importance of a frame in the overall video processing; important frames often require more attention and processing. Visual effect coefficient and preset effect coefficient: Compare the actual visual effect with the expected one, measuring the quality of the visual effect. Frames with poor visual quality require more refinement to improve quality. Coherence and presetness: Reflect the continuity of a frame in the video sequence and the difference from the expected continuity. Frames with poor coherence also require more processing to ensure video smoothness. This formula, which takes multiple factors into consideration, can more accurately and comprehensively determine the number of refinements for each frame, making video frame processing more targeted and reasonable, avoiding over-processing or under-processing, and improving the overall video processing effect and efficiency.

[0074] Visual effect coefficient (Gx) = current clarity / optimal clarity, where the optimal clarity is set in advance.

[0075] The coherence coefficient (Ux) = 1-((distance between matching feature points-minimum distance) / (maximum distance-minimum distance)).

[0076] In this embodiment, the duration-segmentation comparison table is a pre-established mapping table that records the correspondence between different motion durations and unit segmentation durations. For example, a motion duration of 1-5 seconds corresponds to a unit segmentation duration of 0.1 seconds, and a motion duration of 5-10 seconds corresponds to a unit segmentation duration of 0.2 seconds, etc. This is used to guide subsequent video segmentation operations.

[0077] In this embodiment, the unit splitting duration is a basic unit of time interval when splitting a video into frames.

[0078] In this embodiment, a blank track refers to a virtual track concept set in a complex scene where there is no actual movement track of a specified target. It is used to uniformly process video segmentation operations and is the initial frame image obtained after segmenting the complex scene video. For example, in a simulated shopping mall surveillance scene, the movement track of pedestrian A lasts for 8 seconds. The unit segmentation time is determined by the reference table to be 0.2 seconds. The video recording pedestrian A's movement is segmented according to this time length to obtain the original frame. For scene areas without pedestrian movement, the video under the blank track is also segmented according to the unit segmentation time of 0.2 seconds.

[0079] In this embodiment, the target color specifies the color attribute of the target. For example, for a red car, the target color is red, and similar colors such as dark red and light red affect the color of the red car itself.

[0080] In this embodiment, priority=area of regions occupied by similar colors in the original frame / total area of regions in the original frame.

[0081] In this embodiment, the comparison frames are frame images for comparison and analysis obtained by splitting the captured video under the movement trajectory according to the unit splitting time length.

[0082] In this embodiment, the refinement number is a quantitative indicator of the degree of further processing (e.g., adding details, optimizing the display, etc.) to be performed on the frame, determined based on factors such as the priority, visual quality, and coherence of the original frame. For example, a refinement number of 3 indicates that the frame needs to be refined at three levels or steps.

[0083] In this embodiment, if the total number of refinements is large, it indicates that the frame as a whole requires more detailed processing, and the unit splitting duration may be appropriately shortened; otherwise, it may be appropriately extended. Then, based on the adjusted unit splitting duration, the captured video is frame split again to obtain the final frame image for subsequent analysis or application.

[0084] The beneficial effects of the above technical solution are: splitting videos in complex scenes into original frames at reasonable time intervals, providing basic materials for subsequent in-depth analysis and processing of frame images, and by setting priorities for original frames, being able to distinguish the importance of different frames in subsequent processing, and by analyzing the visual effects and coherence of the contrasting frames, being able to fully understand the quality and content connection of the video frames, providing an important basis for subsequently determining the number of frame refinements, and by calculating the number of refinements, being able to determine the personalized degree of further processing for each frame, making the subsequent processing of the frames more targeted, and by adjusting the unit splitting duration according to the total number of refinements and re-splitting the frames, being able to dynamically adapt to the actual processing requirements of the frames in the video and optimize the granularity of the frame splitting.

[0085] The present invention provides a target tracking method based on a spatial transformer network and a convolutional network, wherein high-level feature extraction is performed on the second feature map based on the convolutional network, comprising:

[0086] Construct a deep residual convolution module, where the deep residual convolution module contains multiple residual blocks, and each residual block fuses features from different layers through skip connections;

[0087] At the same time, a multi-scale convolution kernel is introduced to process the second feature map in parallel through convolution kernels of different sizes to obtain feature information at different scales;

[0088] Using a time series analysis method, combined with the position, velocity, and acceleration of the designated target in the previous frame, the position and state of the designated target in the current frame are predicted;

[0089] The extracted high-level features are combined with the predicted target location information to lock the predicted target.

[0090] In this embodiment, the second feature map generation module processes the basic feature map, introduces a spatial attention mechanism, and weights different areas of the basic feature map according to the position and importance of the target in the image, highlighting the features of the target area to generate a second feature map.

[0091] The high-level feature extraction module processes the second feature map using a built-in deep residual convolution module and multi-scale convolution kernels. Multiple residual blocks in the deep residual convolution module fuse features from different layers through skip connections to learn more complex high-level semantic features. Multi-scale convolution kernels use convolution kernels of different sizes, such as 3×3, 5×5, and 7×7, to process the second feature map in parallel, acquiring feature information at different scales.

[0092] The target motion prediction module collects the position information of the target in the historical frames and inputs it into the LSTM network. The LSTM network predicts the position and motion state of the target in the next few frames by learning the motion patterns in the historical data.

[0093] The predictive target lock module combines the extracted high-level features with the target motion prediction results. A classifier determines the target's category, such as pedestrian or vehicle, and a regressor accurately regresses the target's position to determine its specific coordinates. When the target's classification confidence exceeds a set threshold, such as 0.8, the system successfully locks on the target and marks it with a bounding box in the image.

[0094] The beneficial effect of the above technical solution is: by extracting high-level semantic features of the target and combining it with target motion prediction, more accurate and timely target locking can be achieved.

[0095] The present invention provides a target tracking method based on a spatial transformer network and a convolutional network, which optimizes and adjusts the target tracking edge based on all moving boundary difference sets, including:

[0096] Based on the moving boundary difference set, the actual boundary and the standard boundary of the locked position of each image frame are respectively mapped into a two-dimensional coordinate system, and the shape of the boundary difference region between the actual boundary and the standard boundary is determined, and the difference area Si, the maximum difference distance L1i, and the minimum difference distance L2i of the boundary difference region shape are determined, wherein the actual boundary is the boundary of the predicted target;

[0097] like , at this time, assign 0 to the corresponding image frame, otherwise, assign 1 to the image frame, where, represents the area difference threshold; Indicates the radius of the target object; Indicates the boundary recognition accuracy; Indicates the number of image frames in the corresponding captured video;

[0098] The image frame assigned 1 is regarded as the first frame, and the background corresponding to the boundary difference area of the first frame is extracted from the complex background and regarded as the blurred background;

[0099] If the blurred background is of the expansion type, the blurred background and the designated target image are converted from the RGB color space to the HSV color space, and the target edge is locked;

[0100] If the blurred background is of the indented type, then the texture features of the blurred background and the designated target image are extracted based on the gray level co-occurrence matrix, and the target edge is locked according to the texture features;

[0101] If the blurred background is both an expanding and a shrinking type, then the edge is self-drawn based on the known shape features of the designated target and the standard trajectory center point under the corresponding image frame to lock the target edge;

[0102] Performing another difference analysis based on the target edge and the standard edge in the corresponding image frame to obtain an updated difference vector;

[0103] Project the non-zero elements in the updated difference image vector into the two-dimensional coordinate system respectively. If the difference area of each non-zero element is less than or equal to and the variance of the difference areas of all non-zero elements in the updated difference vector is less than a1, in which case the target edge of each image frame is retained;

[0104] Otherwise, the target edge of each first frame is manually selected and the difference distance set of the first frame is constructed, and the labeled image of the first frame is obtained, and the neural network is optimized and trained until the analysis of all moving boundary difference sets is completed.

[0105] Preferably, manually selecting the target edge of each first frame, constructing a difference distance set of the first frame, and obtaining the annotated image of the first frame include:

[0106] The edge of the manually selected target is displayed prominently in the first frame using the first color;

[0107] Distance lines are prominently displayed using a second color display on the first frame according to the difference distance set, wherein each distance line starts from a corresponding point on the edge of the target and ends at the other end of the corresponding difference distance;

[0108] The labeled image is obtained based on the saliency display results.

[0109] Preferably, the updated difference vectors correspond one-to-one to the image frames in the corresponding moving boundary difference set, and the non-zero elements corresponding to the first frame image are assigned 0 elements corresponding to the image that is assigned 0.

[0110] In this embodiment, the actual boundary refers to the boundary range of the target in the image frame, obtained by the predictive target locking algorithm, and is usually represented by a rectangular box, polygon, etc. to represent the position and size of the target in the image. The standard boundary is a pre-set ideal boundary that represents the actual position and shape of the target. It can be determined based on manual annotation, high-precision sensor data, or other reliable references, and is used for comparison and evaluation with the actual boundary.

[0111] In this embodiment, the geometric shape of the area where there is a difference between the actual boundary and the standard boundary may be an irregular polygon, an arc, or other shapes, reflecting the spatial inconsistency between the predicted boundary and the actual boundary. The difference area Si is the area of the boundary difference area, which is used to quantify the degree of difference between the actual boundary and the standard boundary. The larger the difference area, the greater the deviation in target locking. The maximum difference distance L1i is the maximum distance from a point on the actual boundary to the corresponding point on the standard boundary within the boundary difference area, reflecting the maximum deviation in distance between the two. The minimum difference distance L2i is the minimum distance from a point on the actual boundary to the corresponding point on the standard boundary within the boundary difference area, reflecting the minimum deviation in distance between the two.

[0112] In this embodiment, the area difference threshold is a pre-set critical value for determining whether the area of the boundary difference region is acceptable, and is set to 5 square centimeters. The boundary recognition accuracy is set to 0.9.

[0113] In this embodiment, the blurred background is the background portion extracted from the complex background and related to the target boundary difference area. The background may be difficult to distinguish from the target due to inaccurate target locking, background interference, etc., and further processing is required to achieve accurate target edge locking.

[0114] In this embodiment, an expanding background blur refers to a blur whose range extends outward relative to the actual target range. This means that the blurred background includes areas that are not part of the target but have similar color, texture, and other features to the target, making it difficult to accurately define the target edge. A shrinking background blur, in contrast to the expanding type, shrinks inward relative to the actual target range, causing portions of the target edge to be mistaken for the background, similarly making it difficult to locate the target edge. The type of background blur is determined. If it is an expanding type, the blurred background and the designated target images are converted from RGB color space to HSV color space. In HSV space, the differences in hue, saturation, and lightness between the target and background are analyzed. A color segmentation algorithm (such as threshold segmentation or cluster segmentation) is used to locate the target edge based on these differences. If the blurred background is a shrinking type, texture features of the blurred background and designated target images are extracted based on the gray-level co-occurrence matrix. The eigenvalues of the gray-level co-occurrence matrix, such as contrast and correlation, are calculated. By comparing the differences in texture features between the target and background, a texture analysis algorithm (such as an eigenvalue-based classification algorithm) is used to locate the target edge. If the blurred background is both expanding and contracting, the edge is self-drawn using a graphics drawing algorithm (such as Bezier curve drawing and polygon drawing) based on the known shape features of the specified target (such as circle, rectangle, etc.) and the standard trajectory center point under the corresponding image frame, thereby locking the target edge.

[0115] In this embodiment, when the analysis results of the updated difference vectors do not meet the requirements, a manual frame is created on each first frame image to accurately identify the target edge. Using a specific drawing tool or software, the framed target edge is prominently displayed on the image in a first color (e.g., red). Then, based on the difference distance set, a distance line is drawn in a second color (e.g., blue) on the first frame image, starting from the corresponding point on the target edge and ending at the other end of the corresponding difference distance, to clearly indicate the difference distance. Based on these saliency display results, an annotated image is generated. Using the annotated images as training data, the neural network is optimized and trained, with the neural network parameters continuously adjusted until all moving boundary difference sets are analyzed and the target edge locking results for all image frames achieve satisfactory accuracy.

[0116] The above technical solution has the following beneficial effects: by determining the shape and related parameters of the boundary difference area, the accuracy of target lock can be intuitively and quantitatively evaluated. By assigning numerical values to image frames, image frames with large target lock deviations can be quickly screened out, and the image frames can be divided into two categories, laying the foundation for subsequent targeted processing of image frames in different situations. By focusing on the blurred background, the problem of similar colors and difficulty in distinguishing the target and background can be more effectively solved, creating conditions for subsequent accurate target edge lock. Using appropriate methods to lock the target edge according to different blurred background types can fully utilize the characteristic differences between the target and background in color, texture, shape, etc., effectively solving the problem of difficulty in distinguishing the target from the blurred background. Through re-difference analysis and processing of the updated difference vector, the re-locked target edge can be comprehensively evaluated, ensuring the high accuracy of the final locked target edge. Manual selection and construction of annotated images can leverage human subjective judgment and experience to provide accurate training samples for the neural network. Through optimized training, the neural network can learn more accurate target edge lock methods, improving its performance in dealing with complex backgrounds and target lock deviation.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A target tracking method based on a spatial transformer network and a convolutional network, characterized in that: include: Step 1: Move the designated target along different motion trajectories in a complex scene and shoot video, then split the captured video into frames; Step 2: Using a convolutional network to extract features from the frame image to obtain a first feature map, and performing spatial transformation and target area focusing on the first feature map based on the spatial transformation network to obtain a second feature map; Step 3: extracting high-level features from the second feature map based on the convolutional network to predict target locking; Step 4: Based on the locked positions of all predicted targets and sorted in frame order, the predicted trajectories are obtained and compared with the corresponding moving trajectories to obtain a moving boundary difference set, wherein the moving boundary difference set includes the boundary difference area shape of each frame image; Step 5: Optimize and adjust the target tracking edge based on all moving boundary difference sets; Optimize the target tracking edge based on the difference set of all moving boundaries, including: Based on the moving boundary difference set, the actual boundary and the standard boundary of the locked position of each image frame are respectively mapped into a two-dimensional coordinate system, and the shape of the boundary difference region between the actual boundary and the standard boundary is determined, and the difference area Si, the maximum difference distance L1i, and the minimum difference distance L2i of the boundary difference region shape are determined, wherein the actual boundary is the boundary of the predicted target; like , at this time, assign 0 to the corresponding image frame, otherwise, assign 1 to the image frame, where, represents the area difference threshold; Indicates the radius of the target object; Indicates the boundary recognition accuracy; Indicates the number of image frames in the corresponding captured video; The image frame assigned 1 is regarded as the first frame, and the background corresponding to the boundary difference area of the first frame is extracted from the complex background and regarded as the blurred background; If the blurred background is of the expansion type, the blurred background and the designated target image are converted from the RGB color space to the HSV color space, and the target edge is locked; If the blurred background is of the indented type, then the texture features of the blurred background and the designated target image are extracted based on the gray level co-occurrence matrix, and the target edge is locked according to the texture features; If the blurred background is both an expanding and a shrinking type, then the edge is self-drawn based on the known shape features of the designated target and the standard trajectory center point under the corresponding image frame to lock the target edge; Performing another difference analysis based on the target edge and the standard edge in the corresponding image frame to obtain an updated difference vector; Project the non-zero elements in the updated difference image vector into the two-dimensional coordinate system respectively. If the difference area of each non-zero element is less than or equal to and the variance of the difference areas of all non-zero elements in the updated difference vector is less than a1, in which case the target edge of each image frame is retained; Otherwise, the target edge of each first frame is manually selected and the difference distance set of the first frame is constructed, and the labeled image of the first frame is obtained, and the neural network is optimized and trained until the analysis of all moving boundary difference sets is completed.

2. The target tracking method based on spatial transformer network and convolutional network according to claim 1, characterized in that: The complex scenes are deployed in advance, and there are N1 complex scenes, and each complex scene has N2 pre-set motion trajectories.

3. The target tracking method based on spatial transformer network and convolutional network according to claim 1, characterized in that: Frame splitting of captured video, including: Determine the duration of each motion track, and obtain the unit split duration consistent with the motion duration from the duration-split comparison table. Perform corresponding splitting of blank tracks for complex scenes without motion tracks to obtain several original frames. Analyzing the distribution of colors close to the target color of the designated target in each raw frame and setting a priority to each raw frame; Splitting the video captured under the movement trajectory according to the unit splitting time to obtain comparison frames, and analyzing the visual effect and coherence of each comparison frame; Determine the number of refinements for the corresponding frame based on the priority, visual effect, and coherence of the same frame; According to the sum of all the refined quantities, the unit splitting duration is readjusted, and the captured video is frame split according to the adjusted duration.

4. The target tracking method based on spatial transformer network and convolutional network according to claim 3, characterized in that: The number of refinements for the corresponding frame is determined based on the priority, visual effect, and coherence of the same frame, including: in, represents the number of refinements for the corresponding frame, Indicates the priority of the corresponding frame, Indicates the preset level of the corresponding frame, Indicates the preset effect coefficient of the corresponding frame; Represents the visual effect coefficient of the corresponding frame; Indicates the preset nature of the corresponding frame; Indicates the coherence of the corresponding frames; Indicates the rounding symbol; represents a constant, whose value is 2.7; ln represents the sign of the logarithmic function.

5. The target tracking method based on spatial transformer network and convolutional network according to claim 1, characterized in that: Performing high-level feature extraction on the second feature map based on the convolutional network, including: Construct a deep residual convolution module, where the deep residual convolution module contains multiple residual blocks, and each residual block fuses features from different layers through skip connections; At the same time, a multi-scale convolution kernel is introduced to process the second feature map in parallel through convolution kernels of different sizes to obtain feature information at different scales; Using a time series analysis method, combined with the position, velocity, and acceleration of the designated target in the previous frame, the position and state of the designated target in the current frame are predicted; The extracted high-level features are combined with the predicted target location information to lock the predicted target.

6. The target tracking method based on spatial transformer network and convolutional network according to claim 1, characterized in that: Manually selecting the target edge of each first frame and constructing a difference distance set of the first frame, and obtaining an annotated image of the first frame, including: The edge of the manually selected target is displayed saliency in the first frame using the first color; Distance lines are prominently displayed using a second color display on the first frame according to the difference distance set, wherein each distance line starts from a corresponding point on the edge of the target and ends at the other end of the corresponding difference distance; The labeled image is obtained based on the saliency display results.

7. The target tracking method based on spatial transformer network and convolutional network according to claim 1, characterized in that: The updated difference vector corresponds one-to-one to the image frames in the corresponding moving boundary difference set, and the non-zero elements corresponding to the first frame image are assigned 0 and the zero elements corresponding to the image are assigned 0.

Images