Time-frequency domain kalman fusion multi-target tracking method for group dance live broadcast

Abstract

This invention provides a time-frequency domain Kalman fusion multi-target tracking method for live group dance, involving the fields of image processing, target detection, and tracking. This invention integrates a method for rhythm-aware weighted fusion of time-frequency domain one-step motion extrapolation and Kalman prediction into a lightweight tracking pipeline of detection-prediction-two-stage association-update-birth / termination. Musical rhythm is used as a dynamic prior; regular segments are dominated by Kalman, while time-frequency extrapolation weights are increased for dance beats and sudden movements. Simultaneously, general, non-transferable scene constraints are removed, while two-stage association and occlusion maintenance are retained. This allows for a balance of real-time performance, robustness, and ID stability under conditions of simultaneous multi-person dance, frequent mutual occlusion, and slight camera movement. It can seamlessly adapt to mainstream human detectors and actual live streaming systems, solving the problem of balancing real-time performance and ID stability in live group dance under conditions of strong occlusion, strong synchronization, and low latency.

Description

Technical Field

[0001] This invention relates to the fields of image processing, target detection and tracking, and in particular to a time-frequency domain Kalman fusion multi-target tracking method for live group dance broadcasts. Background Technology

[0002] In terms of dance movement presentation, group dance performances often require multiple people to perform synchronized rising and falling movements, and to precisely trigger sudden turns and acceleration / deceleration changes according to the music. However, traditional motion prediction methods are mostly based on the assumptions of smooth motion + Kalman mechanics. These assumptions frequently fail when faced with the aforementioned complex and sudden dance movements, making it difficult to accurately capture and predict the dancers' movement trajectories.

[0003] Stage environment factors also pose significant challenges to motion prediction. Stage lighting constantly changes according to performance needs, and camera positions may shift slightly, leading to highly unstable background prior information. Scene-constrained motion prediction strategies rely heavily on stable background information. In this unstable context, such strategies are difficult to effectively transfer and apply to different stage scenarios, failing to meet the diverse needs of actual live streaming.

[0004] While existing deep trajectory prediction networks can fit complex motion patterns to some extent, they have significant drawbacks. Their model structures are typically bloated, increasing computational resource consumption and leading to high latency, failing to meet the stringent real-time requirements of live streaming. Furthermore, deep trajectory prediction networks require substantial investment in data collection, model training, and deployment, further limiting their widespread application in actual live streaming.

[0005] In view of the above-mentioned pain points, in order to achieve real-time motion prediction and stable identity (ID) recognition under the conditions of strong occlusion, strong synchronization and low latency in live streaming, this invention proposes a time-frequency domain Kalman fusion multi-target tracking method for group dance live streaming to solve the above problems. Summary of the Invention

[0006] To address the aforementioned technical problems in related technologies, this invention proposes a time-frequency domain Kalman fusion multi-target tracking method for live group dance broadcasts.

[0007] This invention provides a time-frequency domain Kalman fusion multi-target tracking method for live group dance streaming, comprising the following steps:

[0008] S1. Object Detection and Input Alignment: Audio and video synchronization sampling is performed on the live video stream of the group dance to obtain the sampling time. video frames and audio blocks And calculate audio blocks rhythm probability at the current sampling time Then in the current video frame The target detector is called to obtain the detection candidate set and decompose it into a high-confidence candidate set according to a preset confidence threshold. With low-confidence candidate set ;

[0009] S2. Trajectory State Prediction: Perform constant-velocity Kalman one-step prior prediction on the on-orbit trajectory in parallel to obtain the prior predicted state. and prior covariance Thus, the prior predicted location is obtained. and prior predicted velocity components The next velocity extrapolation is obtained by one-step extrapolation based on the wavelet trend and details of the recent velocity sequence. and the next step of extrapolation position Then, based on rhythm and probability The fusion prediction position and fusion prediction velocity are obtained by combining the fusion weights with the end detail energy generation, and the fusion prediction state is constructed based on the two. Fusion prediction box With fusion prediction covariance ;

[0010] S3. Two-stage data association: using fusion prediction frames As a priori, based on the set of on-orbit trajectories at the previous sampling time. With high confidence candidate set Perform a global optimal match, and then compare the remaining unmatched trajectories with the low-confidence candidate set. Complementation, outputting the final set of matching pairs. Unmatched trajectory set With the set of unmatched detections ;

[0011] S4. Observation Update and Occlusion Maintenance: For the final set of matched pairs The corresponding trajectory is updated using standard Kalman metric to obtain the posterior state at the k-th sampling time. With posterior covariance For the set of unmatched low-confidence detections Unmatched trajectories are preserved and fused priors are incremented to increase the loss count. ;

[0012] S5. Birth and Termination of Trajectories: First, based on the termination rules, long-term lost trajectories are removed from the existing trajectory set to obtain the retained set. Then, based on the birth rule, the never-matched detection set High-confidence detection candidates are selected from the pool to generate new trajectories, which are then aggregated into a new set. Then retain the set Gathering with new students The updated set of on-orbit trajectories is obtained by merging. Used for the next frame.

[0013] Specifically, step S1 includes the following steps:

[0014] S11. Perform audio and video synchronization sampling on the live video stream of the group dance to obtain the sampling time. video frames and audio blocks Then according to the audio blocks The rhythm probability at the current sampling time is obtained by performing rhythm probability calculation. And record the time step. And the wavelet extrapolation window length M; k is the sampling index;

[0015] S12, in the current video frame Call the target detector Get from bounding box With confidence level The candidate set of detection ;in, For video frames The number of candidates detected in the sample;

[0016] S13, Based on the preset threshold value Detect candidate set Decompose into a high-confidence candidate set With low-confidence candidate set .

[0017] Specifically, step S2 includes the following steps:

[0018] S21. At the k-th sampling time, the on-orbit trajectory set from the previous sampling time... Each trajectory i uses its posterior state from the previous time step. and the covariance at the previous time step The prior predicted state is obtained by performing a one-step prior prediction using a constant velocity discrete linear model. with prior covariance and prior predicted location and prior predicted velocity components ;

[0019] S22, Predicting the velocity component from prior knowledge Perform a one-level time-frequency decomposition based on recent velocity sequences and a one-step extrapolation to obtain the next velocity extrapolation. and the next step of extrapolation position ;

[0020] S23, Utilizing end-detail energy and rhythmic probability Constructing rhythm-aware fusion weights And based on rhythm perception, the fusion weights Extrapolation of the next speed and the next step of extrapolation position The fusion is performed to obtain the fusion prediction location. With fusion prediction speed And based on the fusion prediction location With fusion prediction speed Constructing fusion prediction state Fusion prediction box With fusion prediction covariance .

[0021] Specifically, step S3 includes the following steps:

[0022] S31, High-confidence candidate priority: Based on the on-orbit trajectory set at the previous sampling time. With high confidence candidate set Construct the first gating cost matrix Then the first gating cost matrix The first-stage matching pair set is obtained using the Hungarian algorithm. And based on the first stage matching set Calculate the set of unmatched high-confidence trajectories With unmatched high-confidence detection sets ;

[0023] S32. Supplementing low-confidence candidates: Based on the set of unmatched high-confidence trajectories With low-confidence candidate set Constructing the second gating cost matrix Then the second gating cost matrix The second-stage supplementary matching pair set is obtained using the Hungarian algorithm. And supplement the matching pair set according to the second stage. Calculate the set of unmatched trajectories With unmatched low-confidence detection set ;

[0024] S33, Set the first-stage matching pairs With the second phase supplementary matching pair set At the same time k, merging is performed to obtain the final set of matching pairs. and its corresponding trajectory measurement alignment list And based on the unmatched high-confidence detection set With unmatched low-confidence detection set Summarized into a set of unmatched detections .

[0025] Specifically, step S4 includes the following steps:

[0026] S41. For the final set of matching pairs The corresponding trajectory is updated using standard Kalman measurements, updating the posterior state of the matched trajectory at the k-th sampling time using a three-step process of residual-gain-correction. With posterior covariance and will count the loss Reset to zero and update the posterior state of the matched trajectory at time k. ;

[0027] S42. For the set of unmatched low-confidence detections The unmatched trajectories are preserved and fused prior as the posterior to update the posterior state of the unmatched trajectories at the k-th sampling time. With posterior covariance and count the number of lost items. Increment the posterior update result of the unmatched trajectory at time k. .

[0028] Specifically, the termination rule in step S5 is as follows:

[0029] At the k-th sampling time, update the result for each trajectory i that survived in the previous time step using its posterior. Loss count Perform a judgment; if the count is lost... Greater than or equal to the preset maximum loss threshold The trajectory is removed from the set of in-orbit trajectories to form a retained set. And add the trajectory to the set of terminated trajectories.

[0030] Specifically, the birth rule mentioned in step S5 is the confidence level for detecting candidates. ; The confidence level for the trajectory's birth.

[0031] Specifically, step S5 also includes: simultaneously calling the unique number generation function when generating the new trajectory. Assign a trajectory ID to the new trajectory, then obtain the state of the new trajectory with the detection box initialization state, and set the initial covariance of the new trajectory.

[0032] Specifically, step S23 utilizes end-detail energy and rhythm probability. Constructing rhythm-aware fusion weights Specifically, it includes:

[0033] The end is defined as the last sample of the most recent M-frame sliding window at time k, and the end detail energy is constructed accordingly. :

[0034] ,

[0035] in, Indicates the detail component in the x-direction at the end; Indicates the detail component in the y-direction at the end;

[0036] rhythm probability With end detail energy Linear combination, then through Mapped to (0,1) and restricted to [0,1] using the clipping function clip(), the rhythm-aware fusion weights are obtained. :

[0037] ,

[0038] in For the Sigmoid function; It is the first calibrable coefficient; This is the second calibrable coefficient; This is the third calibrable coefficient.

[0039] Specifically, in step S31, the on-orbit trajectory set from the previous sampling time is used... With high confidence candidate set Construct the first gating cost matrix Specifically:

[0040] At the k-th sampling time, input the set of on-orbit trajectories from the previous sampling time. With the high-confidence candidate set of the current frame For each matching pair Calculate the fusion prediction box of trajectory i High-confidence candidates detection box The intersection-union ratio is calculated, and gating is performed using a high-scoring matching threshold to obtain the first gating cost matrix. :

[0041]

[0042] in, This indicates that the matching pair is considered unmatchable; is the high-scoring matching threshold; IoU() is the intersection-union ratio function; i represents the index of the trajectory; j represents the index of the high-confidence candidate; Indicates the number of high-confidence candidates; This represents the number of on-orbit trajectories at the previous sampling time. The confidence level of high-confidence candidates.

[0043] This invention provides a time-frequency domain Kalman fusion multi-target tracking method for live group dance, centered on lightweight, transferable, and rhythm-sensitive features. First, it aligns audio and video time in each frame, calls any human detector to obtain candidate boxes, and layers them according to confidence levels, forming an input compatible with real-time association. Then, it performs parallel two-way one-step prediction for each on-orbit target: one path uses a constant-velocity Kalman to obtain physically consistent priors; the other path performs a wavelet decomposition from recent velocity sequences, extrapolating trends in low frequencies and preserving bursts in high frequencies, obtaining time-frequency domain extrapolation results. Next, it maps the rhythm intensity and high-frequency detail energy obtained from beat detection as fusion weights: Kalman dominates in regular segments, while increasing the proportion of time-frequency extrapolation during music timing or abrupt changes in direction, thus adaptively covering both stable and burst-type movements. Finally, it uses the fused prediction boxes to enter a two-stage data association process: first, it performs a global optimal match with high-confidence detectors. The remaining trajectories are then used to supplement low-confidence detections, maximizing recall without sacrificing stability. Standard Kalman updates are performed on matched trajectories. For unmatched trajectories, only the prior is maintained and the loss count is incremented to withstand short-term occlusion. Finally, new trajectories are initialized according to a threshold from unmatched detections, and old trajectories with loss counts exceeding the threshold are terminated. This invention removes scene constraints dependent on the stage environment and consists of only linear operators and a single wavelet layer. It ensures low latency and simple implementation, while maintaining ID stability and robustness under strong synchronization, strong occlusion, and sudden movements in group dance live streaming. It has good cross-stage and camera position portability and can be seamlessly adapted to mainstream human detectors and actual live streaming systems. It effectively solves the problems of traditional methods in group dance live streaming, such as assumption failure due to complex movements, difficulty in transferring scene constraints, and difficulty in balancing real-time performance and ID stability under strong occlusion, strong synchronization, and low latency. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of a time-frequency domain Kalman fusion multi-target tracking method for live group dance provided in an embodiment of the present invention. Detailed Implementation

[0046] The invention will be explained in detail through the following embodiments. The purpose of this invention is to protect all technical improvements within its scope. In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "multiple" means two or more, unless otherwise explicitly specified.

[0047] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0048] Example 1

[0049] refer to Figure 1 This embodiment provides a time-frequency domain Kalman fusion multi-target tracking method for live group dance broadcasts, including the following steps:

[0050] S1. Object Detection and Input Alignment: Audio and video synchronization sampling is performed on the live video stream of the group dance to obtain the sampling time. video frames and audio blocks And calculate audio blocks rhythm probability at the current sampling time Then in the current video frame The above calls the object detector to obtain the detection candidate set. And based on the preset threshold Detect candidate set Decompose into a high-confidence candidate set With low-confidence candidate set Then, the candidate set for detection is determined through identity mapping. All bounding boxes are uniformly mapped to the observation candidate set. ;

[0051] S11. Perform audio and video synchronization sampling on the live video stream of the group dance to obtain the sampling time. video frames and audio blocks Then according to the audio blocks The rhythm probability at the current sampling time is obtained by performing rhythm probability calculation. And record the time step. And the wavelet extrapolation window length M; k is the sampling index;

[0052] Audio and video synchronous sampling: Synchronously decode the live video stream of the group dance according to the preset video frame rate, and obtain the current sampling time at the k-th sampling time. Corresponding RGB video frames ,in Here, represents the pixel height and pixel width (integers) of the video frame, and 3 represents the number of RGB channels; and an audio signal aligned with the video frame's timestamp is extracted from the audio track as an audio block according to a fixed-length time window, denoted as . This time window can be set to a frame interval or an integer multiple thereof, thereby ensuring... With video frames Strictly synchronized in time;

[0053] Rhythm probability calculation: audio blocks Input rhythm estimation function To obtain the rhythm probability at the current sampling time. : ,

[0054] in As a scalar, a larger value indicates a stronger rhythm or timing at the current moment. Rhythm estimation function. This can be achieved using existing beat / rhythm detection algorithms (such as a rhythm perception module based on short-time Fourier transform, start point detection, and beat tracking). This invention only uses it as a unified rhythm feature extraction interface, which is existing technology and will not be elaborated here.

[0055] Time step calculation: Record the current sampling time. (Unit: seconds, can be decimal), and compared with the previous sampling time. Define a unified time step together : ,

[0056] in, It is usually equal to the video frame interval (e.g., 24 fps). (seconds), which will be used as a unified time scale in the Kalman time-frequency prediction and subsequent steps of S2.

[0057] Wavelet extrapolation window settings: Set the length of the wavelet extrapolation window. In this embodiment, M is set to 16;

[0058] This step is used to collect the velocity sequence of the most recent M frames in S2 and perform wavelet time-frequency analysis and one-step extrapolation. This step only completes the setting and recording of the fixed parameters.

[0059] The set of basic parameters required for subsequent processing of this frame This means that the rhythm probability, time step, and extrapolation window length have been aligned with the current frame video, providing a unified time base and rhythm prior for subsequent object detection and hierarchical processing.

[0060] It is understood that in this embodiment, the sampling index k is used to identify the discrete time. This refers to the k-th sampling time. Sampling time The corresponding video frame, in this embodiment, both the k-th frame and time k refer to the sampling time. The data.

[0061] S12, in the current video frame Call the target detector Get from bounding box With confidence level The candidate set of detection ;

[0062] In the current video frame Call the predefined target detector (This function name is only used as a mapping notation and does not limit the specific network structure), resulting in the bounding box. With confidence level The candidate set for detection is as follows:

[0063]

[0064] in For video frames The number of candidates detected (integers), bounding boxes Let the bounding box vector of the j-th candidate be written as:

[0065]

[0066] in, The pixel coordinates of the bounding box center point in the image (unit: px, range approximately). ), These are the pixel width and height of the bounding box, respectively; This represents the corresponding confidence level (scalar).

[0067] Through the above processing, the original video frames Convert into a set of structured detection candidates The detection candidate set of the current frame and the bounding box vector of each candidate. With confidence level Still keeping up with the times timestamp Rhythm Probability and time step Alignment provides a foundation for subsequent confidence level stratification and observation vector construction.

[0068] S13, Based on the preset threshold value Detect candidate set Decompose into a high-confidence candidate set With low-confidence candidate set And the candidate set for detection is determined by identity mapping. All bounding boxes are uniformly mapped to the observation candidate set. ;

[0069] Candidate stratification based on confidence level:

[0070] Based on preset threshold Detect candidate set Decompose into a high-confidence candidate set With low-confidence candidate set Formal definition:

[0071]

[0072]

[0073] in They will be given priority in participating in data association during the subsequent high-confidence phase. This is used to supplement the recall; the preset information threshold Select according to actual needs;

[0074] To better distinguish high-confidence candidate sets With low-confidence candidate set To distinguish between the two sets, add superscripts to the elements in each set while keeping the original values. This results in the updated high-confidence candidate set. With low-confidence candidate set :

[0075]

[0076]

[0077] in, Indicates the number of high-confidence candidates; Indicates the number of low-confidence candidates;

[0078] Construction of observation vectors and set representation:

[0079] To directly reuse the detection results in subsequent Kalman updates, a definition is defined from the bounding box. Identity mapping to the observation vector:

[0080]

[0081] Therefore, all detection boxes in the current frame can be uniformly represented as an observation candidate set. :

[0082]

[0083] Each element in this set is related to The candidates in the input form are matched one-to-one, which prepares the measurement input for subsequent Kalman filtering.

[0084] Uniform packaging of aligned data packets:

[0085] To ensure that subsequent steps can uniformly access rhythm, time, and detection information at the same timestamp, the above quantities are packaged into an aligned data packet for the k-th sampling time:

[0086]

[0087] Among them, time step It will be used as a unified time step in subsequent Kalman prediction and time-frequency extrapolation; It will be used as a weighting factor in rhythm-aware fusion; and They participate in matching in both phases of the association process; It is then used as the input to the measurement vector set in the Kalman update.

[0088] S2. Trajectory State Prediction: Perform constant-velocity Kalman one-step prior prediction on the on-orbit trajectory in parallel to obtain the prior predicted state. and prior covariance Thus, the prior predicted location is obtained. and prior predicted velocity components The next velocity extrapolation is obtained by one-step extrapolation based on the wavelet trend and details of the recent velocity sequence. and the next step of extrapolation position Then, based on rhythm and probability The fusion prediction position and fusion prediction velocity are obtained by combining the fusion weights with the end detail energy generation, and the fusion prediction state is constructed based on the two. Fusion prediction box With fusion prediction covariance ;

[0089] S21. At the k-th sampling time, the on-orbit trajectory set from the previous sampling time... Each trajectory i uses its posterior state from the previous time step. and the covariance at the previous time step The prior predicted state is obtained by performing a one-step prior prediction using a constant velocity discrete linear model. with prior covariance and prior predicted location and prior predicted velocity components ;

[0090] The trajectory is generated by the S5 birth mechanism; at the k-th sampling time, the set of on-orbit trajectories at the previous sampling time. ; trajectory in It consists of the surviving trajectories that were updated by S4 and retained after trajectory management by S5 in the previous moment; The number of on-orbit trajectories at the previous sampling time is the number of trajectories that were still alive after the end of the previous frame, i.e., the trajectory set. cardinality:

[0091] The constant velocity discrete linear model belongs to the existing standard CV model, and its structure is shown below;

[0092] Firstly by Construct the state transition matrix This allows for the extrapolation of position using the first-order integral of velocity, while maintaining constant velocity and unchanged width and height; simultaneously, an observation matrix is ​​constructed. Used to map states to the observation space ,

[0093]

[0094] And set the process noise covariance Covariance of measurement noise (These are all preset positive definite matrices, used to characterize model uncertainty and detection measurement error.)

[0095] At this moment, the posterior state of the previous time step is and the covariance at the previous time step is , representing the confirmed posterior state vector and covariance of trajectory i at the previous time step (the (k-1)th sampling time step) (this posterior is obtained from the measurement update / occlusion maintenance of the previous frame and is retained as the prediction prior library for the next frame after birth / termination); where the state vector is uniformly written as:

[0096] ,

[0097] Where x, y represent the two-dimensional position of the target box, which are the center pixel coordinates (px); w, h are the width and height (px) of the target box. The image plane velocity component (px / s); both width and height are in a 6-dimensional state, evolving on the same time axis as position / velocity;

[0098] The prior prediction state is then obtained according to the standard Kalman prediction formula. with prior covariance :

[0099] ,

[0100] Obtain the prior prediction position of the Kalman one-step prior prediction state and prior predicted velocity components Width and height components Predicting state based on prior knowledge Read directly;

[0101] It is understandable that the prior state is predicted in one step by the Kalman filter. It is essentially a joint estimation of the target's position and velocity at the next moment; under the constant velocity state definition used in this method, The first two dimensions are the predicted position components, so their position sub-vectors can be directly used to construct the prediction. Similarly, the velocity components are directly extracted from the velocity sub-block to form the velocity structure. This involves extracting / indexing the prior state vector to obtain the position and velocity, which are then used for subsequent time-frequency extrapolation and correlation gating.

[0102] S22, Predicting the velocity component from prior knowledge Perform a one-level time-frequency decomposition based on recent velocity sequences and a one-step extrapolation to obtain the next velocity extrapolation. and the next step of extrapolation position ;

[0103] Wavelet time-frequency one-step extrapolation: Performing only one level of time-frequency decomposition and one-step extrapolation on the velocity component yields the next velocity and position that are sensitive to rhythm / motion.

[0104] First, construct a single-layer Haar Discrete Wavelet Decomposition (DWT): Set the Haar analysis filter length L=2, and let... For each velocity component sequence Perform the convolution solution for downsampling 2:

[0105]

[0106] Obtain low-frequency coefficients (Trend) and High Frequency Coefficients (Details), where n is the coefficient index. ;

[0107] Then perform Trend / Detail Reconstruction (IDWT) and take the end samples: with a synthetic filter (Haar case is acceptable) )right , Upsampling and convolution reconstructing back to the original time axis yields the trend component. With detail ;

[0108] The sample at the end of the window is recorded as the latest trend. Latest details The term "end" refers to the value taken at the end of the most recent M-frame window at time k.

[0109] Based on recent velocity sequences (most recent) (Frame rate) is then extrapolated one more step (trend difference + detail index preservation): the latest time-to-time trend is... Perform a difference extrapolation for the latest time details. Hold the index to obtain the trend in the next moment. With details :

[0110]

[0111] in, To preserve the coefficients for details;

[0112] This leads to the next step of velocity prediction. (Time-frequency extrapolation):

[0113]

[0114] Among them, direction indicators ;

[0115] And assemble to obtain the next speed extrapolation : ;

[0116] And with the known position at the current time k With time step By performing a first-order integral on the extrapolated velocity, we obtain the extrapolated position for the next step. : ;

[0117] Wherein, the known position at the current time k is input. Taken from Kalman one-step prediction positional components ,Right now = ; = ;

[0118] S23, Utilizing end-detail energy and rhythmic probability Constructing rhythm-aware fusion weights And based on rhythm perception, the fusion weights Extrapolation of the next speed and the next step of extrapolation position The fusion is performed to obtain the fusion prediction location. With fusion prediction speed And based on the fusion prediction location With fusion prediction speed Constructing fusion prediction state Fusion prediction box With fusion prediction covariance ;

[0119] The end is defined as the last sample of the most recent M-frame sliding window at time k (i.e., the detail value corresponding to the latest frame), and the end detail energy is constructed accordingly. :

[0120]

[0121] in, Indicates the detail component in the x-direction at the end; Indicates the detail component in the y-direction at the end;

[0122] This quantity characterizes the intensity of motion abrupt changes / jitter at the end of the current window, and is used to adjust whether to rely more on TF (rhythm / motion sensitivity) or KF (physical smoothing stability).

[0123] rhythm probability With end detail energy Linear combination, then through Mapped to (0,1) and restricted to [0,1] using the clipping function clip(), the rhythm-aware fusion weights are obtained. :

[0124]

[0125] in For the Sigmoid function, ; These are calibrable coefficients;

[0126] The Sigmoid function is a commonly used sigmoid activation function that maps any real number to the interval (0,1).

[0127] The physical meaning is the adaptive confidence allocation coefficient of the two-way prediction: The larger the value, the more it leans towards TF (rhythm / motion sensitivity), and the smaller the value, the more it leans towards KF (physical smoothness and stability).

[0128] Calibration coefficients Generally, the calibration process proceeds by first applying a neutral value, then taking small steps to determine the values; the first calibrable coefficient... Used as a bias for control The baseline level is usually set to 0; the second calibrable coefficient >0 is used to amplify the probability of rhythm. Impact on weights; third calibrable coefficient >0 is used to amplify end detail energy. To mitigate the impact, small positive numbers of the same magnitude (e.g., 1) are typically used initially to ensure a balance between the contributions of the two factors. Subsequently, fine-tuning using a grid search / Bayesian optimization on the validation set, targeting a tracking metric (e.g., MOTA), is performed to achieve the desired balance. It avoids long-term saturation (always remaining close to 0 or 1), thus achieving a stable adaptive switching between placing greater trust in TF and greater trust in KF.

[0129] Position / velocity convex combination fusion (width and height follow Kalman):

[0130] The fused predicted position is obtained by performing convex combination fusion of two-dimensional position and two-dimensional velocity. With fusion prediction speed :

[0131]

[0132] Meanwhile, to maintain the geometric stability of the frame scale and avoid redundant predictions, the width and height components adopt the Kalman prior:

[0133]

[0134] in, For prior prediction of location; For prior prediction of velocity components; To extrapolate the speed for the next step; This is the next step for extrapolation; For the fusion width; For high integration;

[0135] The fusion position after fusion With fusion speed quantification, and with Together they are assembled into a 6-dimensional fusion prediction state. With fusion prediction box :

[0136] ,

[0137]

[0138] Covariance follows the Kalman prior:

[0139]

[0140] in, To integrate predicted covariance; For prior covariance;

[0141] It is understandable that the above operations are performed on trajectory i. In order to better distinguish the different prediction results of each trajectory i, the prediction results are marked with 'i' to indicate their correspondence with trajectory i:

[0142] Fusion prediction state of trajectory i Transform into : ;

[0143] get

[0144] Fusion prediction box of trajectory i Transform into : ;

[0145] get:

[0146] Fusion prediction covariance of trajectory i Transform into ;

[0147] in As a unified prior used for gating / matching in S3 and Kalman measurement updates in S4, it ensures that rhythm sensitivity and physical consistency are connected in a closed loop within the same state space.

[0148] This step first involves performing a Kalman one-step prior prediction using a constant-velocity discrete linear model to obtain a physically consistent estimate for the next frame; subsequently, based on the most recent... The wavelet time-frequency decomposition and minimum extrapolation of frame rate give the rhythm / movement sensitive next step speed and position; finally, the fusion weights are constructed using rhythm intensity and high-frequency detail energy, and a convex combination is performed between the two predictions to obtain a fusion prediction state that is more in line with the live streaming scene of group dance.

[0149] S3. Two-stage data association: using fusion prediction frames As a priori, based on the set of on-orbit trajectories at the previous sampling time. With high confidence candidate set Perform a global optimal match, and then compare the remaining unmatched trajectories with the low-confidence candidate set. Complementation, outputting the final set of matching pairs. and its corresponding trajectory measurement alignment list and the set of unmatched trajectories With the set of unmatched detections ;

[0150] S31, High-confidence candidate priority: Based on the on-orbit trajectory set at the previous sampling time. With high confidence candidate set Construct the first gating cost matrix Then the first gating cost matrix The first-stage matching pair set is obtained using the Hungarian algorithm. And based on the first stage matching set Calculate the set of unmatched high-confidence trajectories With unmatched high-confidence detection sets ;

[0151] At the k-th sampling time, input the set of on-orbit trajectories from the previous sampling time. With the high-confidence candidate set of the current frame For each matching pair Calculate the fusion prediction box of trajectory i High-confidence candidates detection box The intersection-union ratio is calculated, and gating is performed using a high-scoring matching threshold to obtain the first gating cost matrix. :

[0152]

[0153] in, This indicates that the pair is considered unmatched (in engineering implementations, a sufficiently large constant cost can be used). (Instead, to ensure a stable solution).

[0154] in, This represents the high-scoring matching threshold; where IoU is the intersection-union ratio function. (Returns the intersection-union ratio of the two boxes, scalar); ; i represents the index of the trajectory; j represents the index of the high-confidence candidate;

[0155] IoU (Intersection over Union) is a commonly used evaluation metric that measures the degree of matching by calculating the ratio of the intersection to the union between the predicted bounding box and the ground truth bounding box.

[0156] The first gated cost matrix enter The first-stage matching pair set is obtained by performing Hungarian one-to-one minimum cost matching. :

[0157]

[0158] in, This is the Hungarian algorithm, used to solve for minimum cost matching under one-to-one constraints;

[0159] Then, based on the first stage matching set Calculate the set of unmatched high-confidence trajectories With unmatched high-confidence detection sets :

[0160]

[0161]

[0162] in, This is a set difference operation, which removes the index that has already been matched from the entire set.

[0163] S32. Supplementing low-confidence candidates: Based on the set of unmatched high-confidence trajectories With low-confidence candidate set Constructing the second gating cost matrix Then the second gating cost matrix The second-stage supplementary matching pair set is obtained using the Hungarian algorithm. And supplement the matching pair set according to the second stage. Calculate the set of unmatched trajectories With unmatched low-confidence detection set ;

[0164] At the k-th sampling time, input the set of unmatched high-confidence trajectories from the first stage (S31). With low-confidence candidate set Furthermore, by continuing to use fused prior predicted bounding boxes as the matching benchmark and supplementing low-confidence candidates, a gated cost matrix is ​​constructed for the remaining trajectories and low-confidence detections from the first stage, and Hungarian matching is performed, thereby improving recall without significantly introducing false matches.

[0165] For each matching pair Calculate the fusion prediction box of trajectory i With low-confidence candidates detection box The intersection-union ratio is calculated, and gating is performed using a low-scoring matching threshold to obtain the second gating cost matrix. :

[0166]

[0167] in, The low-scoring matching threshold is defined as follows: IoU() is the intersection-union ratio function; i represents the set of unmatched high-confidence trajectories. The trajectory in the data; j represents the index of the low-confidence candidate;

[0168] The second gating cost matrix enter The second-stage supplementary matching pair set is obtained by performing Hungarian one-to-one minimum cost matching. :

[0169]

[0170] in, This is the Hungarian algorithm, used to solve for minimum cost matching under one-to-one constraints;

[0171] Then, based on the supplementary matching pair set from the second phase... Calculate the set of unmatched trajectories With unmatched low-confidence detection set ;

[0172]

[0173]

[0174] in, This is a set difference operation, which means removing the index that has already been matched from the current candidate set.

[0175] S33, Set the first-stage matching pairs With the second phase supplementary matching pair set At the same time k, merging is performed to obtain the final set of matching pairs. and its corresponding trajectory measurement alignment list And based on the unmatched high-confidence detection set With unmatched low-confidence detection set Summarized into a set of unmatched detections ;

[0176] The first phase matching pair set With the second phase supplementary matching pair set At the same time k, merging is performed to obtain the final set of matching pairs. :

[0177]

[0178] in, This represents a one-to-one correspondence between the trajectory index and the detection index at time k.

[0179] Construct observation vectors to update input (select corresponding detection boxes based on matching pairs):

[0180] For each matching pair :

[0181] like Then the measurement of trajectory i is taken from the high-confidence detection box. ;

[0182] like Then the measurement of trajectory i is taken from the low-confidence detection box. .

[0183] This results in a trajectory measurement alignment list. Among them, measurement Automatic press choose or This ensures that S4 can directly perform Kalman measurement updates for each matched trajectory;

[0184] Unmatched high-confidence detection set With unmatched low-confidence detection set Summarized into a set of unmatched detections :

[0185] ,

[0186] Unmatched detection set Used for new trajectory birth determination or candidate accumulation in S5;

[0187] S4. Observation Update and Occlusion Maintenance: For the final set of matched pairs The corresponding trajectory is updated using standard Kalman metric to obtain the posterior state at the k-th sampling time. With posterior covariance For the set of unmatched low-confidence detections Unmatched trajectories are preserved and fused priors are incremented to increase the loss count. The posterior state of the trajectory at time k is obtained. ;

[0188] S41. For the final set of matching pairs The corresponding trajectory is updated using standard Kalman measurements, updating the posterior state of the matched trajectory at the k-th sampling time using a three-step process of residual-gain-correction. With posterior covariance and will count the loss Reset to zero and update the posterior state of the matched trajectory at time k. ;

[0189] Matched trajectories can be derived from the final set of matching pairs. The corresponding trajectory measurement alignment list Extract trajectory i and corresponding measurements Then, according to the identity mapping in step S1: Measurement Convert to measurement vector ;

[0190] Perform standard Kalman measurement updates to obtain the posterior optimal linear unbiased estimate of time k in a three-step process of residual-gain-correction, and then reset the missing count to zero:

[0191] a) Residual (Innovation) Calculation:

[0192]

[0193] in This represents the difference between the measurement and the prior prediction in the observation space; where express

[0194] b) Update covariance With Kalman gain :

[0195]

[0196]

[0197] in Kalman gain is used for adaptive balancing of belief in measurement and belief in prediction; observation matrix and measurement noise covariance: H is the observation matrix, R is the measurement noise covariance, derived from step S2;

[0198] c) Posterior state With posterior covariance renew:

[0199]

[0200]

[0201] in for Identity matrix.

[0202] d) Reset the loss count: Loss count Setting it to 0 indicates that the trajectory has been successfully observed and corrected in the current frame.

[0203] For each Output the posterior state of the matched trajectory at time k:

[0204] It is then written into the trajectory library for subsequent S5 survival management and next moment prediction.

[0205] S42. For the set of unmatched low-confidence detections The unmatched trajectories are preserved and fused prior as the posterior to update the posterior state of the unmatched trajectories at the k-th sampling time. With posterior covariance and count the number of lost items. Increment the posterior update result of the unmatched trajectory at time k. ;

[0206] When a trajectory does not match any measurement in the current frame, it is determined that it may be in an occlusion / missed detection state. A maintenance strategy without measurement updates is adopted to maintain trajectory continuity and update the loss count:

[0207] Due to the lack of measurement, the fused prior is directly used as the current posterior output:

[0208]

[0209] That is, no Kalman measurement update is performed in this frame, only the prediction result is retained to achieve occlusion continuation across frames.

[0210] Increment the loss count: This is used to record the number of consecutive frames for which the trajectory has not been observed and corrected, and to provide S5 with a threshold for executing termination rules.

[0211] For each Output the posterior update result after occlusion is maintained, that is, the posterior update result of the unmatched trajectory at time k. ;

[0212] in The value has been incremented according to the unmatched rule;

[0213] By using steps S41 and S42, the posterior state results of the matched and unmatched trajectories at time k are obtained. The posterior update result is written into the trajectory library and passed to S5 for survival / termination management and next moment prediction input.

[0214] S5. Birth and Termination of Trajectories: First, based on the termination rules, long-term lost trajectories are removed from the existing trajectory set to obtain the retained set. Then, based on the birth rule, the never-matched detection set High-confidence detection candidates are selected from the pool to generate new trajectories, which are then aggregated into a new set. Then retain the set Gathering with new students The updated set of on-orbit trajectories is obtained by merging. Used for the next frame.

[0215] a) Termination rule (delete long-lost trajectories): At the k-th sampling time, update the result of each surviving trajectory i from the previous time using its posterior. Loss count Perform a judgment; if the count is lost... Greater than or equal to the preset maximum loss threshold The trajectory is removed from the set of in-orbit trajectories to form a retained set. And add the trajectory to the set of terminated trajectories;

[0216] For each trajectory Update the results from its posterior. Get the loss count If its loss count satisfies If a trajectory has not been observed and corrected for a long time (due to continuous occlusion / drift / out of view), it is removed from the trajectory set and denoted as the terminated trajectory set. The remaining trajectories that did not meet the termination condition form a retention set. And retain its posterior update results As the input library for predicting the next time step.

[0217] b) Birth rules (new trajectories generated from unmatched detections):

[0218] For the unmatched detection set Each detection candidate If the birth rules are met, a new trajectory is generated:

[0219] The birth rule is the confidence level for detecting candidates. ; Generate a confidence score for the trajectory, selected based on actual needs, which is a real number greater than 0 and less than 1; and call the unique identifier generation function. Assign a trajectory ID to the new trajectory: ,

[0220] NewID() is a unique ID generation function, a globally auto-incrementing / non-repeating ID generation function that guarantees... ; and initialize the state with the detection box (the position and scale of the detection box come from ; (The velocity is initialized to 0 or a small value) to obtain the state of the new trajectory. : And set the initial covariance of the new trajectory. ;

[0221] Represents the initial covariance, and the relationship between the initial covariance and the newborn state. The six-dimensional components correspond one-to-one. Where diag() is a diagonal matrix. Used to characterize the position (x, y) and scale (w, h) uncertainties initialized by the detection box. Used to characterize the uncertainty corresponding to the initial velocity (set to 0 or a small value);

[0222] In this embodiment, settings can be preset based on experience. The unit is pixels (px).

[0223] Then initialize the loss count. Set to 0; aggregate all new trajectory IDs that meet the birth criteria into a newborn set. .

[0224] c) Set merging and index updating (forming the on-orbit set for the next time step)

[0225] Keep the collection Gathering with new students The updated set of on-orbit trajectories is obtained by merging. :

[0226] ,

[0227] Update the results after the a posteriori Write the trajectory into the trajectory library as input for the prediction (S2) and association (S3) of the next time step k+1.

[0228] Finally, the updated trajectory management result at the k-th sampling time is obtained, which is used for prediction and association in the next frame (k+1): where the termination trajectory set It can be used for logging / statistics; on-orbit trajectory collections The trajectory state prediction of S2 will be entered as the on-orbit trajectory set in the next moment, thereby realizing cross-frame closed-loop tracking.

[0229] This embodiment provides a time-frequency domain Kalman fusion multi-target tracking method for live group dance, focusing on lightweight, transferability, and rhythm sensitivity. First, it aligns audio and video time in each frame, calls any human detector to obtain candidate boxes, and layers them according to confidence level, forming an input compatible with real-time association. Then, it performs two-way one-step prediction in parallel for each on-orbit target: one path uses a constant-velocity Kalman to obtain physically consistent priors; the other path performs a wavelet decomposition from the recent velocity sequence, extrapolating low-frequency trends and preserving high-frequency bursts to obtain time-frequency domain extrapolation results. Next, it maps the rhythm intensity and high-frequency detail energy obtained from beat detection as fusion weights: Kalman dominates in regular segments, while increasing the proportion of time-frequency extrapolation during music timing or sharp changes in direction, thus adaptively covering both stable and burst motion types. Finally, it uses the fused prediction boxes to enter a two-stage data association process: first, it performs a global optimal match with high-confidence detectors. The remaining trajectories are then used to supplement low-confidence detections, maximizing recall without sacrificing stability. Standard Kalman updates are performed on matched trajectories. For unmatched trajectories, only the prior is maintained and the loss count is incremented to withstand short-term occlusion. Finally, new trajectories are initialized according to a threshold from unmatched detections, and old trajectories with loss counts exceeding the threshold are terminated. This invention removes scene constraints dependent on the stage environment and consists of only linear operators and a single wavelet layer. It ensures low latency and simple implementation, while maintaining ID stability and robustness under strong synchronization, strong occlusion, and sudden movements in group dance live streaming. It has good cross-stage and camera position portability and can be seamlessly adapted to mainstream human detectors and actual live streaming systems. It effectively solves the problems of traditional methods in group dance live streaming, such as assumption failure due to complex movements, difficulty in transferring scene constraints, and difficulty in balancing real-time performance and ID stability under strong occlusion, strong synchronization, and low latency.

[0230] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 A process, multiple processes, and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0231] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0232] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0233] The parts of this invention not described in detail are prior art. It will be apparent to those skilled in the art that this invention is not limited to the details of the above exemplary embodiments, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and are intended to encompass all changes falling within the meaning and scope of equivalents within this invention.

Claims

1. A time-frequency domain Kalman fusion multi-target tracking method for live group dance streaming, characterized in that, Includes the following steps: S1. Object Detection and Input Alignment: Audio and video synchronization sampling is performed on the live video stream of the group dance to obtain the sampling time. video frames and audio blocks And calculate audio blocks rhythm probability at the current sampling time Then in the current video frame The target detector is called to obtain the detection candidate set and decompose it into a high-confidence candidate set according to a preset confidence threshold. With low-confidence candidate set The target detector is a human body detector. Calculate audio blocks rhythm probability at the current sampling time Specifically, this includes: audio blocks Input rhythm estimation function To obtain the rhythm probability at the current sampling time : ; S2. Trajectory State Prediction: Perform constant-velocity Kalman one-step prior prediction on the on-orbit trajectory in parallel to obtain the prior predicted state. and prior covariance Thus, the prior predicted location is obtained. and prior predicted velocity components The next velocity extrapolation is obtained by one-step extrapolation based on the wavelet trend and details of the recent velocity sequence. and the next step of extrapolation position Then, based on rhythm and probability The fusion prediction position and fusion prediction velocity are obtained by combining the fusion weights with the end detail energy generation, and the fusion prediction state is constructed based on the two. Fusion prediction box With fusion prediction covariance ; With rhythm probability The specific weights for fusion with end-detail energy generation include: The end is defined as the last sample of the most recent M-frame sliding window at time k, and the end detail energy is constructed accordingly. : ， in, Indicates the detail component in the x-direction at the end; Indicates the detail component in the y-direction at the end; rhythm probability With end detail energy Linear combination, then through Mapped to (0,1) and restricted to [0,1] using the clipping function clip(), the rhythm-aware fusion weights are obtained. : ， in For the Sigmoid function; It is the first calibrable coefficient; This is the second calibrable coefficient; This is the third calibrable coefficient; S3. Two-stage data association: using fusion prediction frames As a priori, based on the set of on-orbit trajectories at the previous sampling time. With high confidence candidate set Perform a global optimal match, and then compare the remaining unmatched trajectories with the low-confidence candidate set. Complementation, outputting the final set of matching pairs. Unmatched trajectory set With the set of unmatched detections ; S4. Observation Update and Occlusion Maintenance: For the final set of matched pairs The corresponding trajectory is updated using standard Kalman metric to obtain the posterior state at the k-th sampling time. With posterior covariance For the unmatched low-confidence detection set Unmatched trajectories are preserved and fused priors are incremented to increase the loss count. ; S5. Birth and Termination of Trajectories: First, based on the termination rules, long-term lost trajectories are removed from the existing trajectory set to obtain the retained set. Then, based on the birth rule, the never-matched detection set High-confidence detection candidates are selected from the pool to generate new trajectories, which are then aggregated into a new set. Then retain the set Gathering with new students The updated set of on-orbit trajectories is obtained by merging. The trajectory state prediction will be included in the S2 trajectory set at the next moment.

2. The method according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11. Perform audio and video synchronization sampling on the live video stream of the group dance to obtain the sampling time. video frames and audio blocks Then according to the audio blocks The rhythm probability at the current sampling time is obtained by performing rhythm probability calculation. And record the time step. And the wavelet extrapolation window length M; k is the sampling index; S12, in the current video frame Call the target detector Get from bounding box With confidence level The candidate set of detection ;in, For video frames The number of candidates detected in the sample; S13, Based on the preset threshold value Detect candidate set Decompose into a high-confidence candidate set With low-confidence candidate set .

3. The method according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21. At the k-th sampling time, the on-orbit trajectory set from the previous sampling time... Each trajectory i uses its posterior state from the previous time step. and the covariance at the previous time step The prior predicted state is obtained by performing a one-step prior prediction using a constant velocity discrete linear model. with prior covariance and prior prediction location and prior predicted velocity components ; S22, Predicting the velocity component from prior knowledge Perform a one-level time-frequency decomposition based on recent velocity sequences and a one-step extrapolation to obtain the next velocity extrapolation. and the next step of extrapolation position ; S23, Utilizing end-detail energy and rhythmic probability Constructing rhythm-aware fusion weights And based on rhythm perception, the fusion weights Extrapolation of the next speed and the next step of extrapolation position The fusion is performed to obtain the fusion prediction location. With fusion prediction speed And based on the fusion prediction location With fusion prediction speed Constructing fusion prediction state Fusion prediction box With fusion prediction covariance .

4. The method according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31, High-confidence candidate priority: Based on the on-orbit trajectory set at the previous sampling time. With high confidence candidate set Construct the first gating cost matrix Then the first gating cost matrix The first-stage matching pair set is obtained using the Hungarian algorithm. And based on the first stage matching set Calculate the set of unmatched high-confidence trajectories With unmatched high-confidence detection sets ; S32. Supplementing low-confidence candidates: Based on the set of unmatched high-confidence trajectories With low-confidence candidate set Constructing the second gating cost matrix Then the second gating cost matrix The second-stage supplementary matching pair set is obtained using the Hungarian algorithm. And supplement the matching pair set according to the second stage. Calculate the set of unmatched trajectories With unmatched low-confidence detection set ; S33, Set the first-stage matching pairs With the second phase supplementary matching pair set At the same time k, merging is performed to obtain the final set of matching pairs. and its corresponding trajectory measurement alignment list And based on the unmatched high-confidence detection set With unmatched low-confidence detection set Summarized into a set of unmatched detections .

5. The method according to claim 1, characterized in that, Step S4 specifically includes the following steps: S41. For the final set of matching pairs The corresponding trajectory is updated using standard Kalman measurements, updating the posterior state of the matched trajectory at the k-th sampling time using a three-step process of residual-gain-correction. With posterior covariance and will count the loss Reset to zero and update the posterior state of the matched trajectory at time k. ; S42. For the set of unmatched low-confidence detections The unmatched trajectories are preserved and fused prior as the posterior to update the posterior state of the unmatched trajectories at the k-th sampling time. With posterior covariance and count the number of lost items. Increment the posterior update result of the unmatched trajectory at time k. .

6. The method according to claim 1, characterized in that, The termination rule mentioned in step S5 is as follows: At the k-th sampling time, update the result for each trajectory i that survived in the previous time step using its posterior. Loss count Perform a judgment; if the count is lost... Greater than or equal to the preset maximum loss threshold The trajectory is removed from the set of in-orbit trajectories to form a retained set. And add the trajectory to the set of terminated trajectories.

7. The method according to claim 1, characterized in that, The birth rule mentioned in step S5 is the confidence level for detecting candidates. ; The confidence level for the trajectory's birth.

8. The method according to claim 1, characterized in that, Step S5 also includes: simultaneously calling the unique number generation function when generating the new trajectory. Assign a trajectory ID to the new trajectory, then obtain the state of the new trajectory with the detection box initialization state, and set the initial covariance of the new trajectory.

9. The method according to claim 4, characterized in that, In step S31, the on-orbit trajectory set from the previous sampling time is used as a basis. With high confidence candidate set Construct the first gating cost matrix Specifically: At the k-th sampling time, input the set of on-orbit trajectories from the previous sampling time. With the high-confidence candidate set of the current frame For each matching pair Calculate the fusion prediction box of trajectory i High-confidence candidates detection box The intersection-union ratio is calculated, and gating is performed using a high-scoring matching threshold to obtain the first gating cost matrix. : ， in, This indicates that the matching pair is considered unmatchable; is the high-scoring matching threshold; IoU() is the intersection-union ratio function; i represents the index of the trajectory; j represents the index of the high-confidence candidate; Indicates the number of high-confidence candidates; This represents the number of on-orbit trajectories at the previous sampling time. The confidence level of high-confidence candidates.

Images