A multi-target trajectory tracking and semantic binding method and system based on improved KCF

Abstract

This invention proposes a multi-target trajectory tracking and semantic binding method and system based on an improved KCF (Kinetic Cross-Functional Filtering). The method includes: acquiring a video frame sequence, performing target detection and establishing trajectory objects; updating trajectory positions using frequency domain correlation filtering; performing semantic segmentation to obtain real-world object instances, calculating the spatial consistency score and temporal continuity preference score between the trajectory bounding box and the instance mask, and generating candidate records; calculating a stability score based on the average score within a sliding window and a displacement penalty term, and solidifying a binding effective interval containing start and end frames and instance identifiers; and generating a rendered image with superimposed semantic information based on the current frame's hit interval by reading the position and label. This invention effectively solves the problems of identity switching and semantic information loss in multi-target tracking by fusing gradient direction histogram features with semantic segmentation results and combining spatiotemporal consistency constraints, thereby improving the stability of trajectory binding and the accuracy of rendering output.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method and system for multi-target trajectory tracking and semantic binding based on an improved KCF. Background Technology

[0002] With the continuous development of applications such as video surveillance, intelligent transportation, industrial site perception, and augmented reality, vision-based multi-target detection and trajectory tracking technologies have become an important foundation for realizing behavior analysis, situational awareness, and real-world information overlay. Existing systems typically first detect and identify targets in video sequences, and then use correlation filtering-based tracking algorithms to obtain continuous trajectories. Among these, kernel correlation filtering is widely used for real-time multi-target tracking due to its low computational cost and mature implementation path. To further obtain semantic interpretation of the targets, some solutions introduce semantic segmentation results, associating the trajectory with object categories or instances in the image, thereby forming a binding relationship between the trajectory and real-world objects, which can then be used for real-time overlay display or post-event backtracking. However, in complex real-world scenarios, occlusion, intersection, and proximity between targets can cause appearance features to become similar in local areas. Even if the tracking algorithm outputs a continuous trajectory, object drift may occur, making the trajectory visually "coherent" but actually corresponding to the wrong entity. Moreover, such errors are often difficult to detect in a timely manner. At the same time, semantic segmentation is a computationally expensive processing link, and engineering systems often output it asynchronously. Its results are inconsistent with the trajectory in time. If direct association is performed only on a frame-by-frame basis, semantic annotation is prone to lag, jumps, or jitter, resulting in a misalignment between the trajectory and the semantics of the real scene at the display level. More importantly, many existing implementations treat the correspondence between trajectory and semantics as a one-time determined result, lacking stable judgment and interval expression for time continuity. This means that short-term occlusion or local mismatch can bind and solidify the error into subsequent frames and even historical data, thereby affecting the credibility and interpretability of playback analysis. In applications that require simultaneous multi-target synchronous tracking, real-time synchronous display of trajectory and real-world scene, and consistency of historical backtracking, the above problems will be amplified into system-level bottlenecks, manifesting as unstable real-time overlay, unreliable backtracking results, and difficulty in auditing or reviewing the results, thus limiting the long-term stable operation of related technologies in engineering scenarios. Summary of the Invention

[0003] The purpose of this invention is to design a multi-target trajectory tracking and semantic binding method and system based on an improved KCF, which can effectively reduce the propagation risk of trajectory mismatch with real-world entities, mitigate display misalignment caused by semantic jitter, and improve the credibility and engineering usability of backtracking results.

[0004] To achieve the above objectives, a first aspect of the present invention provides a multi-target trajectory tracking and semantic binding method based on an improved KCF, the method comprising: The video frame sequence is obtained, and target detection is performed on each frame to obtain a set of detection results. In the first frame, a trajectory object containing a unique trajectory identifier, position sequence and frame number set is created for each detection box. In subsequent frames, the search area is determined based on the trajectory position of the previous frame, the gradient direction histogram features of the search area are extracted and discrete Fourier transform is performed, the response map is calculated using frequency domain correlation filtering, the position with the maximum response value is selected to update the trajectory position of the current frame, and the updated position and frame number are appended to the trajectory object. Semantic segmentation is performed on each frame of the image to obtain a set of real-world object instances. The spatial consistency score between the bounding box of the trajectory object in the current frame and the mask region of each instance in the set of real-world object instances is calculated. The candidate priority score is calculated by combining the temporal continuity preference of the preferred instance identifier in the previous frame. The instance identifier and category label with the highest ranking are written as candidate records into the trajectory object. Traverse the frame sequence number of the trajectory object, calculate the stability score based on the average candidate priority score within the sliding window and the trajectory center displacement penalty term, and solidify the frame segments that continuously meet the threshold condition as the binding effective interval. The binding effective interval includes the start frame, the end frame and the corresponding binding instance identifier. Based on the current frame number hitting the binding effective interval, the position sequence in the trajectory object and the category label corresponding to the binding instance identifier are read to generate a rendered image with superimposed semantic information.

[0005] Furthermore, the calculation of the response map using frequency domain correlation filtering includes: The gradient orientation histogram features are mapped to the frequency domain space, and training samples are constructed by cyclic shifting and the relevant filter weights are calculated. In the trajectory initialization stage, the initial filter weights are obtained by solving based on the features of the first frame. In the running stage, the current frame features and filter weights are used to perform convolution operation in the frequency domain, and the response map is obtained by inverse transformation.

[0006] Further, calculating the spatial consistency score between the bounding box of the trajectory object in the current frame and the mask regions of each instance in the set of real-world object instances includes: Calculate the number of pixels in the intersection region between the bounding box region of the trajectory object and the mask regions of each instance in the current frame; Calculate the number of pixels in the union region of the bounding box region of the trajectory object and the mask regions of each instance in the current frame; The ratio of the number of pixels in the intersection region to the number of pixels in the union region is used as the spatial consistency score.

[0007] Furthermore, the calculation of the candidate priority score by combining the temporal continuity preference of the preferred instance identifier of the previous frame includes: Get the preferred instance identifier of the previous frame record; Determine whether the instance identifier of the current frame is consistent with the preferred instance identifier recorded in the previous frame. If they are consistent, generate a time continuity indicator value. The sum of the spatial consistency score, the preset weight parameter, and the temporal continuity indicator value is used as the candidate priority score.

[0008] Furthermore, the calculation of the stability score based on the average candidate priority score within the sliding window and the trajectory center displacement penalty term includes: Set the sliding window length, extract the candidate priority scores corresponding to the preferred instance identifiers of each frame within the window, and calculate the average value; Calculate the displacement norm of the center coordinates of adjacent frames within the window, and divide the displacement norm by the square root of the area of ​​the trajectory bounding box to obtain the relative displacement intensity. The average relative displacement intensity within the window is calculated, and the stability score is obtained by subtracting the product of the preset penalty weight and the average relative displacement intensity from the average candidate priority score.

[0009] Furthermore, the step of solidifying consecutively satisfying frame segments as binding effective intervals includes: Traverse the frames in ascending order of their sequence numbers. When the stability score is greater than or equal to the preset threshold for the first time, record it as the starting frame of the interval. Continue expanding backward until a frame with a stability score lower than a preset threshold appears, and record the frame preceding that frame as the interval termination frame. The preferred instance identifier that appears most frequently within the statistical interval is used as the binding instance identifier for the binding effective interval.

[0010] Further, the step of hitting the binding effective interval based on the current frame sequence number includes: Search the target interval entry that covers the current frame number in the binding effective interval record of the trajectory object; If there are multiple target interval entries that cover the current frame number, select the target interval entry with the highest interval confidence as the hit interval; If no target interval entry covering the current frame number is found, it is considered a miss.

[0011] Furthermore, the process of generating a rendered image with overlaid semantic information includes: Calculate the distance of the current frame number relative to the start and end frames of the hit interval; Based on the distance and the preset boundary transition width, the transparency weight is calculated using piecewise linear interpolation. A bounding box is drawn with the position of the trajectory object in the current frame as the geometric center, and a text label is generated with the category label corresponding to the bound instance identifier; The transparency weight is applied to the rendering parameters of the bounding box and the text label to generate a rendered image with superimposed semantic information.

[0012] Furthermore, the method also includes a backtracking output step: Receive the range of playback frames specified by the user; Filter the interval entries that intersect with the playback frame range from the binding effective interval records of the trajectory object; For each frame within the playback frame range, perform the steps of hitting the binding effective interval and generating a rendered image with superimposed semantic information, and output the historical replay video stream.

[0013] In a second aspect of the invention, a multi-target trajectory tracking and semantic binding system based on an improved KCF is provided, the system comprising: The video acquisition and trajectory initialization module is used to acquire video frame sequences, perform target detection on each frame to obtain a set of detection results, and establish a trajectory object containing a unique trajectory identifier, position sequence, and frame number set for each detection box in the first frame; in subsequent frames, the search area is determined based on the trajectory position of the previous frame, the gradient direction histogram features of the search area are extracted and discrete Fourier transform is performed, the response map is calculated using frequency domain correlation filtering, the position with the maximum response value is selected to update the trajectory position of the current frame, and the updated position and frame number are appended to the trajectory object; The semantic binding candidate calculation module is used to perform semantic segmentation on each frame of the image to obtain a set of real-world object instances, calculate the spatial consistency score between the bounding box of the trajectory object in the current frame and the mask region of each instance in the set of real-world object instances, calculate the candidate priority score by combining the temporal continuity preference of the preferred instance identifier in the previous frame, and write the instance identifier and category label of the top-ranked instance as candidate records into the trajectory object. The binding interval solidification module is used to traverse the frame number sequence of the trajectory object, calculate the stability score based on the average candidate priority score in the sliding window and the trajectory center displacement penalty term, and solidify the frame segments that continuously meet the threshold condition as the binding effective interval. The binding effective interval includes the start frame, the end frame and the corresponding binding instance identifier. The rendering output module is used to read the position sequence in the trajectory object and the category label corresponding to the binding instance identifier based on the current frame number hitting the binding effective interval, and generate a rendered image with superimposed semantic information.

[0014] The beneficial technical effects of the present invention are at least as follows: To address the aforementioned issues, this invention provides a multi-target trajectory tracking and semantic binding method and system based on an improved KCF (Kinetic Cross-Functional Flow). Using multi-target trajectories formed by correlation filtering as a continuous skeleton, candidate real-world object instances obtained from semantic segmentation are introduced into each frame. A stable candidate sequence is constructed through spatial consistency and cross-frame continuous preference. Furthermore, a joint stability evaluation is performed on the candidate sequence and trajectory motion features in the temporal dimension, elevating frame-by-frame association to an effective interval record with start and end boundaries. This ensures that the binding relationship naturally possesses temporal consistency and resistance to short-term fluctuations. At the output level, the system drives real-time overlay and retrospective reproduction based on the effective interval, transforming interval hit determination into executable rendering and event output actions for each frame. This ensures that real-time presentation and historical presentation follow the same binding logic and remain consistent. Through this integrated process of candidate construction, interval effective generation, and interval-driven output centered on trajectory objects, this invention effectively reduces the propagation risk of trajectory-real-scene entity mismatch under common conditions of multi-target occlusion interaction and semantic asynchronous output, mitigates display misalignment caused by semantic jitter, and improves the credibility and engineering usability of backtracking results, thereby better meeting the practical application needs of multi-target synchronous tracking, trajectory-real-scene real-time synchronous display, and historical backtracking. Attached Figure Description

[0015] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0016] Figure 1 This is a flowchart of a multi-target trajectory tracking and semantic binding method based on an improved KCF according to the present invention.

[0017] Figure 2 This is a system framework diagram for multi-target trajectory tracking and semantic binding based on an improved KCF according to the present invention. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0019] In one or more embodiments, such as Figure 1 As shown, a multi-target trajectory tracking and semantic binding method based on improved KCF is disclosed, the method comprising the following: S1: Obtain the video frame sequence, perform target detection on each frame to obtain a set of detection results, and establish a trajectory object containing a unique trajectory identifier, position sequence and frame number set for each detection box in the first frame; in subsequent frames, determine the search area based on the trajectory position of the previous frame, extract the gradient direction histogram features of the search area and perform discrete Fourier transform, calculate the response map using frequency domain correlation filtering, select the position with the maximum response value to update the trajectory position of the current frame, and append the updated position and frame number to the trajectory object; Specifically, this step revolves around the complete construction process of video data into multi-target trajectory objects, the core of which lies in transforming continuous video streams into a structured collection of trajectory objects. This provides a stable, continuous, and indexable data foundation for subsequent semantic processing. The system first acquires video frame sequences through the video acquisition module. This video acquisition module can be a fixed-position network camera, an industrial line scanner, or an embedded terminal camera. The video stream is output through a hardware encoder, decoded, and then arranged into a frame sequence in chronological order. Each frame image... This is a two-dimensional pixel matrix containing complete spatial information of the current scene, such as vehicles and pedestrians in a traffic scene, or equipment and operators in a factory scene. Frame number It is automatically generated by the system when reading the video stream to ensure consistency in temporal order.

[0020] Obtaining each frame image Then, the system calls the preceding target detection network to generate a set of detection results. This detection network consists of a multi-layer convolutional feature extraction structure. First, it extracts basic spatial features through several convolutional layers and non-linear activation layers. Then, it enhances the expressive power of targets of different sizes through a multi-scale feature fusion structure. Finally, the bounding box regression branch outputs the target location, and the classification branch outputs the category label. Each detection result... It should include at least the target bounding box location (represented by center coordinates and width / height), class label, and confidence score. The system performs [the following] in the first frame: A trajectory object is created for each detection box within the area. And assign a unique trajectory identifier to it. Trajectory object Internal maintenance location sequence With frame sequence number set , where the initial position Directly from The bounding box position, and the frame sequence set records the time index of the target's first appearance.

[0021] In the second and subsequent frames, the system uses the trajectory position from the previous frame. Perform a local search on the current frame. Specifically, this involves searching the image... China and Israel A search region is cropped around the center, with its width and height expanded proportionally to the current width and height of the target (e.g., 1.5 times the current width and height) to ensure coverage of the true target location even with slight shifts between frames. When extracting features from the cropped image patch, the image is first converted to grayscale, then the gradient direction histogram is calculated to form a multi-channel feature tensor. This feature tensor is then normalized to ensure consistent representation of targets at different scales in the feature space. Subsequently, the feature tensor undergoes a discrete Fourier transform, mapping it to the frequency domain. Training samples are constructed through cyclic shifting, and relevant filter weights are calculated. The filter weights are obtained during trajectory initialization based on the features of the first frame, and during runtime, the current frame features are used to generate the response map. .

[0022] The trajectory position is updated in the following form: ; in, Indicates the first The first frame The location of the trajectory within the search area The response value at the specified location is obtained by inverse transformation of the result of convolution of the current frame features and filter weights in the frequency domain. The center position of the bounding box corresponding to the maximum value in the response graph; This indicates that the coordinate point with the largest response value is selected. The response value reflects the similarity between the current location and the historical target appearance features; therefore, selecting the location with the largest response ensures the stable continuation of the trajectory across consecutive frames. For example, when the target undergoes a small displacement between two frames, a significant peak will form in the response map near the actual location, and the system updates accordingly. This creates a smooth trajectory.

[0023] In multi-object scenarios, the system performs analysis on each trajectory object. A fixed-length historical position buffer is maintained to limit the center position of the search area in the next frame, ensuring that the search range expands around the historical motion trajectory. If the detection module outputs a new detection result in a certain frame... The system calculates its bounding box relative to the existing trajectory position. The system matches the spatial overlap; when the overlap reaches a set ratio, the detection result is used to correct the current trajectory position; otherwise, a new trajectory object is created and a new trajectory is assigned. In this way, the system sequentially performs search, response calculation, and position update for all trajectory objects in each frame, and then updates the new position. With the current frame number Added to the data structure of the trajectory object.

[0024] After processing consecutive frames, a complete set of multi-target trajectory objects is formed. Each trajectory object contains a unique identifier. Complete location sequence and the corresponding set of frame numbers This structure is continuous in the time dimension and unified in data organization, enabling subsequent steps to directly attach semantic candidate information and binding states to the same trajectory object without changing the existing data structure, thus ensuring the consistency and continuity of the entire solution in engineering implementation.

[0025] S2: Perform semantic segmentation on each frame of the image to obtain a set of real-world object instances. Calculate the spatial consistency score between the bounding box of the trajectory object in the current frame and the mask region of each instance in the set of real-world object instances. Combine the temporal continuity preference of the preferred instance identifier in the previous frame to calculate the candidate priority score. Write the instance identifier and category label with the highest ranking as candidate records into the trajectory object. Specifically, the set of multi-target trajectory objects has already been obtained in step one. Based on this, this step sequences the geometric positions of each trajectory in each frame. A correspondence is established with the segmentation results of real-world object instances in the same frame, and this correspondence is written back to the same trajectory object in the form of "candidate instance records." The key engineering constraint here is that multiple adjacent targets, multiple instances of the same type, and local occlusion may exist within the same frame, leading to multiple solutions for the trajectory bounding box coverage area and the instance mask area. To address this, this step first establishes a computable spatial consistency metric within a single frame, and then introduces a preference for "candidate continuity" at the cross-frame level. This makes the candidate records more stable over time, thus providing a directly usable candidate sequence for the next step of "binding effective interval determination," rather than scattered frame-by-frame matching results.

[0026] The semantic segmentation result is obtained by a convolutional neural network with an encoder-decoder structure in frame sequence number. The results are obtained through inference from the corresponding images. The encoder consists of multiple convolutional layers and downsampling to extract semantic features; the decoder recovers spatial details through upsampling and skip connections; and the pixel-by-pixel classification layer outputs a category label map. Instance-level results are obtained by performing connectivity analysis on regions of the same category: spatially disconnected regions are split into different instances, and each instance is assigned an instance identifier, forming a set of real-world object instances for that frame. Each instance It must at least include the instance mask region (pixel set), the instance bounding box, and the category label; the instance identifier is denoted as... Its source is the numbering result of the connected component instantiation process. Trajectory object The geometric position of this frame is determined by Give, Derived from the improvement of KCF in step one in frame The update of the peak position of the response indicates that the trajectory is in frame The bounding box.

[0027] The spatial consistency metric within a single frame adopts the Intersection over Union (IoU), widely used in classical object detection and segmentation. Its original source is the mathematical definition of "intersection over union" in set theory and measure theory, and it is standardized as a "region overlap ratio" metric in computer vision. This step directly adopts this definition, replacing the "sets" with "trajectory bounding box regions" and "instance mask regions" to obtain the trajectory. In frame With examples Spatial consistency score : ; in, The initial source of this is the classic definition of IoU; This represents the intersection region between the trajectory bounding box and the instance mask region. This represents the union region of the two sets; This represents the area value obtained by counting the set of pixels within a region. Both sides of the formula are ratios, making it a dimensionless quantity with a numerical range of... This aligns with the common-sense explanation of "overlap ratio." The system in frames... Inner traversal of all instances calculate Instances that meet the preset ratio threshold are included in the candidate set; when there are multiple candidates in the same frame, this step continues to sort the candidates so as to write a stable candidate sequence to the trajectory object.

[0028] The cross-frame candidate ranking is derived from a combination of classic probabilistic graphical models and the "likelihood term + prior term" approach commonly used in dynamic programming: single-frame IoU provides the likelihood of spatial matching, while cross-frame consistency provides the prior preference that "the same trajectory tends to correspond to the same instance." This step implements the "prior preference" as a simple, computable indicator and combines it with... Linear superposition forms candidate priority scores The derivation process is as follows: The candidate priority score is considered as a weighted combination of two factors: "spatial consistency" and "temporal continuity," where spatial consistency is directly taken as... Temporal continuity is determined using a binary method based on whether the current instance identifier matches the preferred instance identifier of the trajectory in the previous frame. The weights are determined by preset parameters. Control, obtain: ; in, Representing the trajectory In frame For example Candidate priority score; The spatial consistency score is calculated using the previous formula; These are preset weight parameters used to control the impact of time-continuous preferences on the ranking. This is an indicator function, derived from the mathematical definition of an indicator function. It takes the value 1 when the condition inside the parentheses is true, and takes the value 0 otherwise. For example Instance identifier; For trajectory objects The preferred instance identifier recorded in the previous frame, which comes from the first candidate after the previous frame's candidate sorting, is stored in an internal field of the trajectory object. Both sides of the formula are dimensionless quantities: It is a dimensionless ratio. It is a dimensionless binary value. The weights are dimensionless, satisfying the consistency requirements. The logical relationship between the two equations is as follows: first, the spatial consistency of a single frame is calculated using the IoU formula. Then Substituting into the candidate priority score formula, and synthesizing with the time-continuous preference term. Finally press Sort the candidate instances in the same frame and write them to the trajectory object.

[0029] The candidate writing process unfolds centered around the trajectory object: for each trajectory Every frame The system first starts from the semantic segmentation instance set Retrieve the instance list from the data and calculate all instances. Then calculate all Then, sort the instances in descending order, select the top-ranked instances and write them into the "Frame Number → Candidate List" field of the trajectory object. Simultaneously, write the instance identifier that ranks first into the preferred instance identifier field of the trajectory object. Used for the next frame The value of . To facilitate direct use in subsequent steps, this step records the instance identifier in the candidate list entries. Category tags and their corresponding This ensures that subsequent binding effectiveness determinations can read candidate sequences of the same trajectory along the time axis and perform stability determinations.

[0030] The following is a computational example for engineering debugging, demonstrating the feasibility of the above calculations in typical occlusion / interlacing scenarios. Assume a certain frame... Inside, trajectory The bounding box region and two instances , There is significant overlap between the regions, as determined by pixel counting: , ,as well as , Then there is , Let the preferred instance identifier of the trajectory in the previous frame be... The identifiers of the two instances in this frame are respectively , and set The indicators are respectively , Substituting the candidate priority score, we get , Sort results selection As the preferred candidate and write it to the trajectory object, while still being able to It is retained in the list as a secondary candidate. The common practical implications of this example are: when two adjacent instances highly overlap with the trajectory bounding box, the continuity preference makes the trajectory more likely to continue the real-world entity corresponding to the previous frame, thus obtaining a more stable candidate sequence at dense intersections.

[0031] The output of this step is the updated collection of multi-target trajectory objects. Each trajectory object in the original , , Based on this, the candidate record field of "Frame Sequence Number → Candidate Instance List" and the preferred instance identifier sequence for cross-frame continuity are added and populated. This information can be directly used in the next step to determine whether "the same trajectory stably corresponds to the same real-world instance in consecutive frames", thereby naturally advancing the geometric trajectory generated in step one to the state of "the trajectory has real-world entity candidates", forming a computable input oriented towards semantic binding.

[0032] S3: Traverse the frame sequence number of the trajectory object, calculate the stability score based on the average candidate priority score in the sliding window and the trajectory center displacement penalty term, and solidify the frame segments that continuously meet the threshold condition as the binding effective interval. The binding effective interval includes the start frame, the end frame and the corresponding binding instance identifier. Specifically, in step two, each trajectory object has already been... After the frame-by-frame candidate information is written into the trajectory object, this step elevates "frame-by-frame candidate" to "interval-level binding". (Track object) Step one provides a position sequence organized by frame number. Step two supplements the candidate instance priority score (in order of priority) at the same frame sequence number. The format is recorded in the candidate list for each frame, and in the preferred instance identifier. The corresponding entry can be retrieved directly. In real-world scenarios, target occlusion, parallel approach, and intersection can cause brief jumps in candidate instances within a few frames, and semantic segmentation boundaries can also experience local jitter. Therefore, interval generation cannot rely on single-frame judgments; instead, it needs to combine the "persistence of candidate priority scores" and the "smoothness of trajectory geometric motion" into a window score, and then solidify the frame segments that continuously meet the conditions into effective intervals. The construction idea of ​​the window score comes from the sliding window averaging (local smoothing of the signal) in classical time series processing and the regularization idea in classical optimization (adding a penalty term to constrain undesirable changes in addition to the principal term); based on this, the "principal term" is taken as the interval within the window. The average value is used to obtain a stability score suitable for the "trajectory-real-scene object binding" scenario. The "penalty item" is taken as the normalized average value of the center displacement of the trajectory of adjacent frames within the window.

[0033] For trajectory objects any frame Take a length of sliding window Within this window, the stability score is defined as: ; in, The window ends at frame Stability score at that time; The window length is given by the system configuration option; For trajectory In frame Preferred instance identifier The corresponding candidate priority score is directly read from the candidate list entries written into the trajectory object in step two; The penalty weight is given by the system configuration item; For bounding box The calculated center coordinates are obtained by using the following method: The geometric field takes the center point; For bounding box The area is obtained by multiplying the width and height. Both the left and right sides of the formula are scoring quantities: the first term is the window average of the candidate priority score, and the second term is the window average of the relative displacement intensity multiplied by a weight to form a penalty; the relative displacement intensity is proportionally determined by "removing the bit and taking the square root of the area," making the displacement intensity comparable across different target scales, thus maintaining numerical consistency. Subtracting the two terms reflects the engineering constraint that "the stronger the candidate persistence, the higher the score; the greater the trajectory anomaly, the lower the score," which can suppress geometrical abrupt changes caused by accidental jumps to adjacent targets after occlusion at the scoring level, thereby reducing interval false triggers.

[0034] based on Generating the effective interval can be completed with a single linear scan. For each trajectory... The system traverses the frames in ascending order of their sequence numbers. And calculate in each frame ;when The threshold condition is met continuously. ( When configuring system settings, the frame that first meets the threshold is recorded as the starting frame of the interval. The interval continues to expand until the first frame that does not meet the threshold is encountered, and the frame preceding that frame is designated as the interval termination frame. The binding instance identifier corresponding to this interval is taken as the preferred instance identifier that appears most frequently within the interval (from the trajectory object). (obtained directly from statistics), thus allowing a small number of frames within the interval to experience brief jitter in the preferred identifier while still maintaining semantic consistency within the interval. The interval confidence level is taken as the interval within... The average value is used as a reference for the strength of subsequent output stages. The generated interval entries are appended to the trajectory object in chronological order. The "Binding Effective Interval Record Field" requires each entry to contain at least an instance identifier, start frame, end frame, and interval confidence level; the original trajectory object... , , And the candidate records remain unchanged, thus ensuring that the next step can directly use the "range record field" to drive real-time output and historical backtracking.

[0035] The following is a debugging calculation example demonstrating the operability of the above window scoring and interval generation. Using a certain trajectory... Taking the window from frame 10 to frame 14 as an example, let's assume... , Step two involves writing the preferred score sequence (i.e. If the values ​​in frames 10.14 are 0.72, 0.75, 0.70, 0.74, and 0.73 respectively, then the average value of the first window is... The bounding box sequence written in step one. Computable center coordinate sequence Assume that the center displacement norms of adjacent frames in frames 11 to 14 are 6, 5, 6, and 5 respectively, and the corresponding... In frames 10..13, the values ​​are 30, 30, 29, and 29 respectively, so the relative displacement intensities are respectively , , , The second window average is multiplied by The punishment is approximately ,therefore If the threshold is set to If frame 14 satisfies the condition, and subsequent frames continue to satisfy the condition, the scanning process will set the starting frame to the frame that first satisfies the threshold and expand it to form... When a frame's score decreases or the trajectory exhibits significant geometric shifts leading to an increase in the penalty term, At that point, the scanning process truncates the interval and writes it into the interval record. This embodiment simultaneously demonstrates the collaboration of two types of input: step two... Determining "candidate persistence", step one The "motion smoothness" is determined, and the two are combined in the same score. The final interval record is directly stored inside the trajectory object for use in the next step.

[0036] S4: Based on the current frame number hitting the binding effective interval, read the position sequence in the trajectory object and the category label corresponding to the binding instance identifier, and generate a rendered image with superimposed semantic information.

[0037] Specifically, this step uses the set of multi-target trajectory objects output in step three. As the sole input, it directly executes the real-time output and historical backtracking output of trajectory-real-scene semantics. Each trajectory object The position sequence indexed by frame number is already available in step one. With frame sequence number set In step two, candidate records with instance identifiers and category labels are already available. In step three, the interval-level binding results have been solidified into interval record entries (each entry contains an instance identifier). , start frame Termination frame (And interval reliability). This step converts the "interval entry" into an executable output action for each frame: in real-time mode, the system follows the video frame sequence number; in retrospective mode, the system replays the video according to the frame sequence number range specified by the user. Both modes use the same execution chain of "interval hit - read trajectory position - read instance semantics - render overlay", ensuring that real-time display and playback display are logically consistent.

[0038] The interval hit determination adopts the classic definition of interval membership in set theory and expresses it as a directly computable Boolean quantity using indicator functions in mathematics. For any trajectory object... With the current frame number First Find the interval record with The corresponding interval entry (if only one entry is allowed to be active at any given time, then the time-based overwrite will be used). The entries; if multiple entries exist, select the one with the highest priority based on interval confidence), and then calculate the interval hit mark. : ; in, Represents a trajectory object In frame Does it hit a certain binding effective range? These are indicator functions, derived from the standard mathematical representation of set membership relations; and This is the start and end frame number for the trajectory object interval record entry written in step three. Both sides of this equation are dimensionless Boolean quantities. , , All are frame sequence indexes within the same sequence, satisfying consistency and common sense requirements. After the interval is hit, the system reads from the trajectory object. Used as the geometric rendering location for this frame, and the instance identifier is read from the interval entry. ; Instance identifier The corresponding category label comes from the instance record in step two (the category label is written into the instance entry after the category label map output by the semantic segmentation pixel-by-pixel classification layer is instantiated by the connected component), so that a superimposed element of "bounding box + category text / color label" can be generated in the rendering stage.

[0039] To make the visual presentation at the boundary of the interval more consistent with the perception of actual monitoring and industrial scenarios, this step introduces a commonly used linear interpolation technique from computer graphics (derived from the classic alphablending transparency control) during rendering: linearly fade-in / fade-out is applied to the annotation transparency within several frames at the beginning and end of the interval, allowing the annotations to smoothly change from weak to strong and from strong to weak. Let the boundary transition width be... (Given by system configuration options), the transparency weight of the frame is calculated when the hit interval is reached. : ; in, The transparency weight used for rendering the overlay in this frame; The boundary transition width; The use of this term originates from the common "saturation clipping" approach in piecewise linear interpolation implementations, ensuring that the fade-in / fade-out remains stable after reaching 1. The formula consists of the multiplication of three parts: It is guaranteed that superposition only occurs when the interval is hit; the first term Describes the gradual ingress process, increasing frame by frame from the start of the interval; the second term This describes the fade-out process that decreases frame by frame before reaching the end of the interval. Both the numerator and denominator are the frame number difference and the transition width, which are ratios of the same type of quantity; therefore, the entire... The weights are dimensionless, consistent with the general principle of rendering transparency. The logical relationship between the two equations is as follows: first, it is obtained from the interval membership relationship. , and then Substituting this as a gating term into the transparency weight formula, the final result is... Controls the display strength of overlaid elements; when When the value is 0, the overlay elements are invisible; when the value is 1, the overlay elements are displayed at full intensity.

[0040] The execution process in real-time mode is as follows: The system reads the current frame number... At that time, traverse the collection of trajectory objects. Perform an interval hit query and calculate for each trajectory object. and Then in the rendering engine Draw a bounding box for the geometric position, and with The corresponding category labels generate text or color identifiers, and finally the transparency weights are applied. The parameters are passed to the overlay drawing interface as rendering parameters to achieve synchronized output of trajectory and real-world semantics. The execution process in backtracking mode uses the same per-frame calculation chain: when the user specifies the playback frame range... At that time, the system filters the interval entries that intersect with the range from the interval records of the trajectory object, and performs... arrive Perform the same operation frame by frame and The calculation and rendering actions achieve the same historical reproduction effect as the real-time mode; during the replay process, the semantic instance identifier and trajectory position superimposed on each frame are all from the recorded fields inside the trajectory object, avoiding the recalculation burden in the playback stage.

[0041] A computational example is provided to demonstrate the operability of interval hit and transparency weights. Suppose a trajectory object... The interval entries are instance identifiers. , start frame Terminate frame The transition width is set to When playing back to frame When, satisfy ,therefore Transparency weight is calculated as follows: This manifests as a weak display of the annotation in the initial stage. When hour, The annotation achieves stable display at full intensity. hour, This manifests as a gradual exit before the end of the interval. When When the interval condition is not met, ,thereby The semantic annotation for this instance will not be rendered in this frame. Rendering occurs in each frame. Draw a bounding box for the location and identify it with an instance. The retrieved category labels complete the semantic annotation overlay, and the transparency is adjusted from... Direct control produces output results that can be directly observed and adjusted.

[0042] In one or more embodiments, such as Figure 2 As shown, a multi-target trajectory tracking and semantic binding system based on an improved KCF is disclosed, the system comprising: The video acquisition and trajectory initialization module is used to acquire video frame sequences, perform target detection on each frame to obtain a set of detection results, and establish a trajectory object containing a unique trajectory identifier, position sequence, and frame number set for each detection box in the first frame; in subsequent frames, the search area is determined based on the trajectory position of the previous frame, the gradient direction histogram features of the search area are extracted and discrete Fourier transform is performed, the response map is calculated using frequency domain correlation filtering, the position with the maximum response value is selected to update the trajectory position of the current frame, and the updated position and frame number are appended to the trajectory object; The semantic binding candidate calculation module is used to perform semantic segmentation on each frame of the image to obtain a set of real-world object instances, calculate the spatial consistency score between the bounding box of the trajectory object in the current frame and the mask region of each instance in the set of real-world object instances, calculate the candidate priority score by combining the temporal continuity preference of the preferred instance identifier in the previous frame, and write the instance identifier and category label of the top-ranked instance as candidate records into the trajectory object. The binding interval solidification module is used to traverse the frame number sequence of the trajectory object, calculate the stability score based on the average candidate priority score in the sliding window and the trajectory center displacement penalty term, and solidify the frame segments that continuously meet the threshold condition as the binding effective interval. The binding effective interval includes the start frame, the end frame and the corresponding binding instance identifier. The rendering output module is used to read the position sequence in the trajectory object and the category label corresponding to the binding instance identifier based on the current frame number hitting the binding effective interval, and generate a rendered image with superimposed semantic information.

[0043] It is worth noting that the specific workflow of the multi-target trajectory tracking and semantic binding system based on improved KCF provided in this embodiment of the invention is the same as that of the multi-target trajectory tracking and semantic binding method based on improved KCF described in the above embodiment, and will not be repeated here.

[0044] This invention also provides a multi-target trajectory tracking and semantic binding device based on an improved KCF, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps described in the above embodiment of a multi-target trajectory tracking and semantic binding method based on an improved KCF, for example... Figure 1 The steps S1 to S4 described above; or, when the processor executes the computer program, it implements the functions of each module in the above system embodiments.

[0045] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the improved KCF-based multi-target trajectory tracking and semantic binding device.

[0046] The aforementioned multi-target trajectory tracking and semantic binding device based on improved KCF can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the device may also include input / output devices, network access devices, buses, etc.

[0047] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the improved KCF-based multi-target trajectory tracking and semantic binding device, connecting various parts of the device via various interfaces and lines.

[0048] The memory can be used to store the computer program and / or modules. The processor implements various functions of the multi-target trajectory tracking and semantic binding device based on the improved KCF by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created according to the operation of the controller, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0049] The module integrating multi-target trajectory tracking and semantic binding devices based on improved KCF, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0050] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0051] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-target trajectory tracking and semantic binding method based on improved KCF, characterized in that, The method includes: The video frame sequence is obtained, and target detection is performed on each frame to obtain a set of detection results. In the first frame, a trajectory object containing a unique trajectory identifier, position sequence and frame number set is created for each detection box. In subsequent frames, the search area is determined based on the trajectory position of the previous frame, the gradient direction histogram features of the search area are extracted and discrete Fourier transform is performed, the response map is calculated using frequency domain correlation filtering, the position with the maximum response value is selected to update the trajectory position of the current frame, and the updated position and frame number are appended to the trajectory object. Semantic segmentation is performed on each frame of the image to obtain a set of real-world object instances. The spatial consistency score between the bounding box of the trajectory object in the current frame and the mask region of each instance in the set of real-world object instances is calculated. The candidate priority score is calculated by combining the temporal continuity preference of the preferred instance identifier in the previous frame. The instance identifier and category label with the highest ranking are written as candidate records into the trajectory object. Traverse the frame sequence number of the trajectory object, calculate the stability score based on the average candidate priority score within the sliding window and the trajectory center displacement penalty term, and solidify the frame segments that continuously meet the threshold condition as the binding effective interval. The binding effective interval includes the start frame, the end frame and the corresponding binding instance identifier. Based on the current frame number hitting the binding effective interval, the position sequence in the trajectory object and the category label corresponding to the binding instance identifier are read to generate a rendered image with superimposed semantic information.

2. The multi-target trajectory tracking and semantic binding method based on improved KCF according to claim 1, characterized in that, The calculation of the response map using frequency domain correlation filtering includes: The gradient orientation histogram features are mapped to the frequency domain space, and training samples are constructed by cyclic shifting and the relevant filter weights are calculated. In the trajectory initialization stage, the initial filter weights are obtained by solving based on the features of the first frame. In the running stage, the current frame features and filter weights are used to perform convolution operation in the frequency domain, and the response map is obtained by inverse transformation.

3. The multi-target trajectory tracking and semantic binding method based on improved KCF according to claim 1, characterized in that, The calculation of the spatial consistency score between the bounding box of the trajectory object in the current frame and the mask regions of each instance in the set of real-world object instances includes: Calculate the number of pixels in the intersection region between the bounding box region of the trajectory object and the mask regions of each instance in the current frame; Calculate the number of pixels in the union region of the bounding box region of the trajectory object and the mask regions of each instance in the current frame; The ratio of the number of pixels in the intersection region to the number of pixels in the union region is used as the spatial consistency score.

4. The multi-target trajectory tracking and semantic binding method based on improved KCF according to claim 3, characterized in that, The calculation of candidate priority scores by combining the temporal continuity preference of the preferred instance identifier of the previous frame includes: Get the preferred instance identifier of the previous frame record; Determine whether the instance identifier of the current frame is consistent with the preferred instance identifier recorded in the previous frame. If they are consistent, generate a time continuity indicator value. The sum of the spatial consistency score, the preset weight parameter, and the temporal continuity indicator value is used as the candidate priority score.

5. The multi-target trajectory tracking and semantic binding method based on improved KCF according to claim 1, characterized in that, The stability score is calculated based on the average candidate priority score within the sliding window and the trajectory center displacement penalty term, including: Set the sliding window length, extract the candidate priority scores corresponding to the preferred instance identifiers of each frame within the window, and calculate the average value; Calculate the displacement norm of the center coordinates of adjacent frames within the window, and divide the displacement norm by the square root of the area of ​​the trajectory bounding box to obtain the relative displacement intensity. The average relative displacement intensity within the window is calculated, and the stability score is obtained by subtracting the product of the preset penalty weight and the average relative displacement intensity from the average candidate priority score.

6. The multi-target trajectory tracking and semantic binding method based on improved KCF according to claim 5, characterized in that, The step of solidifying consecutively satisfying frame segments into a binding effective interval includes: Traverse the frames in ascending order of their sequence numbers. When the stability score is greater than or equal to the preset threshold for the first time, record it as the starting frame of the interval. Continue expanding backward until a frame with a stability score lower than a preset threshold appears, and record the frame preceding that frame as the interval termination frame. The preferred instance identifier that appears most frequently within the statistical interval is used as the binding instance identifier for the binding effective interval.

7. The multi-target trajectory tracking and semantic binding method based on improved KCF according to claim 1, characterized in that, The step of hitting the binding effective interval based on the current frame sequence number includes: Search the target interval entry that covers the current frame number in the binding effective interval record of the trajectory object; If there are multiple target interval entries that cover the current frame number, select the target interval entry with the highest interval confidence as the hit interval; If no target interval entry covering the current frame number is found, it is considered a miss.

8. The multi-target trajectory tracking and semantic binding method based on improved KCF according to claim 7, characterized in that, The generated rendered image with overlaid semantic information includes: Calculate the distance of the current frame number relative to the start and end frames of the hit interval; Based on the distance and the preset boundary transition width, the transparency weight is calculated using piecewise linear interpolation. A bounding box is drawn with the position of the trajectory object in the current frame as the geometric center, and a text label is generated with the category label corresponding to the bound instance identifier; The transparency weight is applied to the rendering parameters of the bounding box and the text label to generate a rendered image with superimposed semantic information.

9. The multi-target trajectory tracking and semantic binding method based on improved KCF according to claim 1, characterized in that, The method also includes a backtracking output step: Receive the range of playback frames specified by the user; Filter the interval entries that intersect with the playback frame range from the binding effective interval records of the trajectory object; For each frame within the playback frame range, perform the steps of hitting the binding effective interval and generating a rendered image with superimposed semantic information, and output the historical replay video stream.

10. A multi-target trajectory tracking and semantic binding system based on improved KCF, characterized in that, The system includes: The video acquisition and trajectory initialization module is used to acquire video frame sequences, perform target detection on each frame to obtain a set of detection results, and establish a trajectory object containing a unique trajectory identifier, position sequence, and frame number set for each detection box in the first frame; in subsequent frames, the search area is determined based on the trajectory position of the previous frame, the gradient direction histogram features of the search area are extracted and discrete Fourier transform is performed, the response map is calculated using frequency domain correlation filtering, the position with the maximum response value is selected to update the trajectory position of the current frame, and the updated position and frame number are appended to the trajectory object; The semantic binding candidate calculation module is used to perform semantic segmentation on each frame of the image to obtain a set of real-world object instances, calculate the spatial consistency score between the bounding box of the trajectory object in the current frame and the mask region of each instance in the set of real-world object instances, calculate the candidate priority score by combining the temporal continuity preference of the preferred instance identifier in the previous frame, and write the instance identifier and category label of the top-ranked instance as candidate records into the trajectory object. The binding interval solidification module is used to traverse the frame number sequence of the trajectory object, calculate the stability score based on the average candidate priority score in the sliding window and the trajectory center displacement penalty term, and solidify the frame segments that continuously meet the threshold condition as the binding effective interval. The binding effective interval includes the start frame, the end frame and the corresponding binding instance identifier. The rendering output module is used to read the position sequence in the trajectory object and the category label corresponding to the binding instance identifier based on the current frame number hitting the binding effective interval, and generate a rendered image with superimposed semantic information.

Images