Multi-target tracking method and device based on scene-specific long-term prediction

Abstract

This application provides a multi-target tracking method and device based on scene-specific long-term prediction, relating to the field of multi-camera tracking. The method determines first and second duration thresholds based on the historical spatial distribution of different targets in the target space and spatiotemporal measurement errors; selects time points that satisfy a historical duration not less than the first duration threshold and an interval not less than the second duration threshold between the selected time point and a second time point in a second trajectory segment; inputs the historical trajectory segment and the interval into a scene-specific model trained without labels, and outputs a multimodal probability distribution to determine whether the two trajectories represent the same target. This application addresses the problem of insufficient signal-to-noise ratio in short-term association judgments in real-world deployment scenarios due to spatial positioning and temporal synchronization errors, a problem not recognized by existing technologies. It determines the long-term timescale through error statistics and directly establishes a mapping from historical trajectories to future locations using a scene-specific model, eliminating the need for progressively predicting intermediate paths, reducing error accumulation, and improving the robustness and tracking continuity of public safety monitoring.

Description

Technical Field

[0001] This application relates to the field of multi-camera tracking technology, and in particular to a multi-target tracking method and device based on scene-specific long-term prediction. Background Technology

[0002] Multi-Object Tracking (MOT) in indoor spaces is one of the core tasks in computer vision, widely applied in scenarios such as public safety. In public safety scenarios, to maintain order and ensure the safety of people in public places, necessary image acquisition and trajectory analysis of pedestrian targets can be performed. Its task is to correctly match and merge pedestrian targets detected in video frames acquired by a multi-camera system deployed in a specific indoor space according to their identities, forming a complete and continuous trajectory sequence for each target. This provides trajectory data support for security monitoring, abnormal behavior detection, and personnel flow analysis in public places. In typical deployment environments such as commercial stores and office parks, dozens to hundreds of cameras typically cover indoor spaces ranging from hundreds to thousands of square meters. The cameras are connected via a local area network, and the spatial positioning and time synchronization accuracy of the system is limited by engineering implementation conditions.

[0003] Currently, existing multi-target tracking methods mainly fall into three categories. The first category is similarity matching methods without explicit spatiotemporal constraints. These methods weight the apparent feature distance of the detection box sequence within a trajectory segment with the spatial distance of the short-term predicted position, and compare this weighted distance with a manually set similarity threshold to determine whether two trajectory segments belong to the same target. The second category is explicit spatiotemporal constraint matching methods based on manually preset parameter models. These methods attempt to replace the fixed threshold directly set in the first category with a similarity threshold function that varies with spatial distance and time interval, constructed using a manually selected function form (such as a bivariate sigmoid function) and a set of hyperparameters. The third category is trajectory prediction-based association methods. These methods use Kalman filtering, Long Short-Term Memory (LSTM) networks, or Transformer models to predict the future position of the target and use the distance between the predicted position and the actual observed position as motion similarity for trajectory association. In addition, non-Markovian methods that utilize behavioral patterns to impose global consistency constraints on trajectories have emerged in the prior art. These methods associate behavioral patterns with trajectories through energy functions to improve the global consistency of tracking.

[0004] None of the methods mentioned above adequately address the long-term robust tracking problem in real-world deployment scenarios. On one hand, the spatiotemporal constraints or prediction models relied upon by existing methods are typically trained on short-term inertial extrapolation or general datasets, with prediction timescales mostly on the order of seconds. Beyond 10 seconds, there are no universal spatiotemporal constraints that hold true independently of the specific indoor space, making it impossible to reliably predict the target's position tens of seconds later. On the other hand, the spatiotemporal constraints or prediction models in existing methods are not learned from the specific indoor space where deployment takes place, but rather from manually set or general scene training. This fails to capture the unique long-term movement patterns of pedestrians specific to that space, resulting in poor adaptability when deployed to new scenarios. More importantly, under typical commercial deployment conditions, the spatial positioning error of multi-camera systems can reach the meter level (e.g., 1 to 2 meters, with lens distortion areas reaching 3 to 4 meters), and the time synchronization error can reach the second level (e.g., 1 to 3 seconds). This means that the predicted position difference and measurement error on which short-term association judgments rely are on the same order of magnitude, leading to insufficient signal-to-noise ratio for association judgments. Any solution relying on short-term, fine-grained spatiotemporal constraints may fail in practical use due to error contamination. However, existing technologies have not recognized the fundamental dilemma of insufficient signal-to-noise ratio in short-term association judgment, nor do they have the technical means to reliably predict the future position of a target on a time scale of tens of seconds. As a result, they cannot reconstruct the signal-to-noise ratio required for association judgment, leading to insufficient association reliability in long-term occlusion or trajectory breakage scenarios, and prominent issues of trajectory fragmentation and identity switching. Summary of the Invention

[0005] In view of this, embodiments of this application provide a multi-target tracking method and device based on scene-specific long-term prediction to eliminate or improve one or more defects existing in the prior art.

[0006] The first aspect of this application provides a multi-target tracking method based on scene-specific long-term prediction, including: The method is based on the following conditions: the duration of the historical trajectory of the first trajectory segment collected from the target monitoring space before the first time point is not less than the first duration threshold corresponding to the target monitoring space, and the time interval between the first time point and the second time point in the second trajectory segment collected from the target monitoring space is not less than the second duration threshold. The method then selects the first time point from the first trajectory segment and the second time point from the second trajectory segment. The first and second duration thresholds are predetermined based on the ratio of historical spatial location dispersion to system spatiotemporal measurement error being not less than a preset multiple. The system spatiotemporal measurement error is the composite value of the spatial positioning error and time synchronization error of the multi-camera system covering the target monitoring space. The historical spatial location dispersion is the arithmetic mean of the spatial location differences between different targets within the target monitoring space under the candidate time span used to determine the first and second duration thresholds. Historical trajectory segments with a duration of not less than the first duration threshold before the first time point are extracted from the first trajectory segment. The historical trajectory segments and the time interval are input into a long-term location prediction model specific to the target monitoring space, so that the model outputs a multimodal spatial location probability distribution of the target monitoring space corresponding to the second time point. The long-term location prediction model is pre-trained based on unlabeled historical trajectories accumulated by the target monitoring space itself. The probability value of the actual spatial position of the second time point in the second trajectory segment in the multimodal spatial position probability distribution is used as the association matching degree. Based on the association matching degree, a multi-target tracking result is obtained to indicate whether the first trajectory segment and the second trajectory segment belong to the same target.

[0007] In some embodiments of this application, before selecting the first time point from the first trajectory segment and the second time point from the second trajectory segment, the method further includes: Obtain statistical values ​​of spatial positioning error and time synchronization error of the multi-camera system within the target monitoring space; Multiply the statistical value of the time synchronization error by the preset pedestrian walking speed to obtain the equivalent spatial error; The larger value between the statistical value of the spatial positioning error and the equivalent spatial error is taken as the system spatiotemporal measurement error; Within each candidate time span in the preset candidate time span set, the historical spatial location distribution of different targets within the target monitoring space is statistically analyzed. From each of the candidate time spans, the shortest time span in which the ratio of the historical spatial location distribution to the system spatiotemporal measurement error is not less than the preset multiple is selected as the second duration threshold. The first duration threshold is defined as the shortest duration of a typical pedestrian movement process that is not less than the second duration threshold and covers the target monitoring space. The first duration threshold is not less than the second duration threshold.

[0008] In some embodiments of this application, before selecting the first time point from the first trajectory segment and the second time point from the second trajectory segment, the method further includes: The upper bound of the time synchronization error of the multi-camera system in the target monitoring space is multiplied by the preset pedestrian walking speed to obtain the upper bound of the equivalent spatial error. The larger of the preset upper bound of the spatial positioning error of the multi-camera system within the target monitoring space and the upper bound of the equivalent spatial error is taken as the spatiotemporal measurement error of the system. Based on the principle that the historical spatial location distribution of different targets grows superlinearly with time intervals, while the spatiotemporal measurement error of the system does not accumulate with time intervals, the shortest time span with a ratio of the historical spatial location distribution to the spatiotemporal measurement error of the system not less than the preset multiple is selected from each candidate time span in the preset candidate time span set as the second duration threshold. The first duration threshold is defined as the shortest duration of a typical pedestrian movement process that is not less than the second duration threshold and covers the target monitoring space. The first duration threshold is not less than the second duration threshold.

[0009] In some embodiments of this application, before selecting the first time point from the first trajectory segment and the second time point from the second trajectory segment, the method further includes: Obtain the unlabeled historical trajectory of the target monitoring space; wherein, the unlabeled historical trajectory is obtained by processing the historical video data of the target monitoring space based on a multi-target tracking algorithm; Select trajectories from the unlabeled historical trajectories whose duration is not less than the first duration threshold; Each trajectory with a duration not less than the first duration threshold after filtering is sampled at a preset sampling interval to obtain multiple sampling time points; For each sampling time point, extract historical trajectory segments that occurred before that sampling time point and whose duration is not less than the first duration threshold; For each historical trajectory segment, a training sample is formed by combining each of a plurality of future time intervals not less than the second duration threshold with the historical trajectory segment, and the actual spatial location corresponding to the future time interval is used as the label of the training sample, thereby obtaining a training sample set composed of each training sample with the label respectively. Based on the training sample set, a preset sequence model is trained using the cross-entropy loss function to obtain a long-term location prediction model specific to the target monitoring space; wherein, the long-term location prediction model is used to output the multimodal spatial location probability distribution of the target monitoring space according to the input historical trajectory fragment and the time interval.

[0010] In some embodiments of this application, the step of training a preset sequence model using a cross-entropy loss function based on the training sample set to obtain a long-term location prediction model specific to the target monitoring space includes: The training steps for the current iteration round are executed, wherein the training steps include: inputting historical trajectory fragments and future time intervals from the training samples into the sequence model to obtain the multimodal spatial location probability distribution of the target monitoring space output by the sequence model; using the actual spatial location corresponding to the label of the training sample as the supervision signal, calculating the negative log-likelihood of the output unit corresponding to the actual spatial location in the multimodal spatial location probability distribution as the cross-entropy loss of the training sample, and not imposing additional penalty terms on other output units in the multimodal spatial location probability distribution; and updating the parameters of the sequence model based on the cross-entropy loss of the training samples in the current iteration round. If the preset convergence condition is not met at present, the training steps of the next iteration round are executed; If the preset convergence condition is met, a long-term location prediction model specific to the target monitoring space is obtained based on the parameters of the latest updated sequence model.

[0011] In some embodiments of this application, the long-term location prediction model includes a sequence encoding module and a probability output module; The sequence encoding module includes a Transformer encoder, a recurrent neural network with an attention mechanism, a temporal convolutional network, or a neural ODE network, used to encode the historical trajectory fragment and the time interval as a whole input, and output a contextual representation of the target's long-term motion pattern within the first duration threshold. The probability output module is used to map the context representation to multiple grids into which the target monitoring space is divided, and output the probability of the target appearing in each grid at the second time point as a discrete multimodal spatial location probability distribution; or it is used to map the context representation to a multimodal spatial location probability distribution in the form of a Gaussian mixture distribution or a heatmap.

[0012] In some embodiments of this application, the multimodal spatial location probability distribution is a discrete probability distribution, which includes multiple spatial locations and a probability value corresponding to each spatial location, and each spatial location corresponds to a preset spatial region; Different first time points and different second time points constitute different candidate point pairs; wherein, the first trajectory segment has a continuous historical trajectory before the first time point with a duration not less than the first duration threshold; The step of using the probability value of the actual spatial location of the second time point in the second trajectory segment in the multimodal spatial location probability distribution as the association matching degree includes: For each candidate point pair, perform a matching degree determination step separately; The matching degree determination step includes: The multiple spatial locations are sorted in descending order of probability value, and the top K spatial locations after sorting are taken as candidate locations. Determine whether the actual spatial location of the second time point in the candidate point pair falls into a preset spatial region corresponding to any of the candidate locations; If it falls into the range, the probability value of the candidate position in the multimodal spatial position probability distribution is used as the association matching degree of the candidate point pair. If the candidate point does not fall into any of the preset spatial regions corresponding to the candidate positions, then zero is used as the correlation matching degree corresponding to the candidate point pair; where K is a preset positive integer.

[0013] In some embodiments of this application, obtaining a multi-target tracking result based on the association matching degree to indicate whether the first trajectory segment and the second trajectory segment belong to the same target includes: The correlation matching degree corresponding to each candidate point is statistically aggregated, and the aggregation result is used as the target correlation matching degree. The target association matching degree is compared with a preset score threshold. When the target association matching degree is not less than the preset score threshold, the first trajectory segment and the second trajectory segment are determined to belong to the same target. When the target association matching degree is less than the preset score threshold, the first trajectory segment and the second trajectory segment are determined to not belong to the same target. Generate and output a multi-target tracking result that includes the determination result, the target association matching degree, and trajectory data composed of the first trajectory segment and the second trajectory segment.

[0014] In some embodiments of this application, before comparing the target correlation matching degree with a preset score threshold, the method further includes: Obtain the appearance features of the first trajectory segment and the second trajectory segment respectively; wherein, the appearance features are appearance features extracted based on the image information of the detection boxes in the first trajectory segment and the second trajectory segment; The current target association matching degree is compared with a preset coarse screening threshold; wherein the preset coarse screening threshold is less than the preset score threshold; If the target association matching degree is not less than the preset coarse screening threshold, then calculate the apparent feature distance between the apparent features of the first trajectory segment and the second trajectory segment respectively; The target association matching degree and the apparent feature distance are weighted and fused to obtain the fusion result; The target association matching degree is updated using the fusion result; The method further includes: If the target association matching degree is less than the preset coarse screening threshold, the determination result that the first trajectory segment and the second trajectory segment do not belong to the same target is directly obtained.

[0015] In some embodiments of this application, the multi-target tracking method based on scene-specific long-term prediction further includes: The trajectory data in the multi-target tracking results within the current update cycle is continuously accumulated; If the update triggering conditions are met, the current long-term location prediction model specific to the target monitoring space is updated and trained to obtain the updated long-term location prediction model specific to the target monitoring space. The update triggering conditions include at least one of the following: the trajectory data reaches a preset data volume threshold, the drift detection result of the association confidence distribution determined by multiple target association matching degrees generated in the current update cycle exceeds a preset drift threshold, and a scene layout change event is detected in the target monitoring space.

[0016] In some embodiments of this application, the update training is any one of incremental training, full retraining, sliding window retraining, or knowledge distillation using the current long-term location prediction model as the teacher model; wherein, the training samples used for the update training are constructed based on the trajectory data corresponding to the update training, using the same sample construction method as when training the long-term location prediction model specific to the target monitoring space. In the incremental training and knowledge distillation, the trajectory data corresponding to the updated training is the trajectory data accumulated in the current update cycle; in the full retraining, the trajectory data corresponding to the updated training is all the trajectory data accumulated in the target monitoring space; in the sliding window retraining, the trajectory data corresponding to the updated training is the trajectory data accumulated in multiple update cycles.

[0017] In some embodiments of this application, the first duration threshold ranges from 30 seconds to 120 seconds, the second duration threshold ranges from 10 seconds to 120 seconds, and the first duration threshold is not less than the second duration threshold. The preset multiple ranges from 5 to 20 times.

[0018] A second aspect of this application provides a multi-target tracking device based on scene-specific long-term prediction, comprising: The candidate point selection module is used to select the first time point from the first trajectory segment and the second time point from the second trajectory segment, based on the following conditions: the duration of the historical trajectory of the first trajectory segment collected from the target monitoring space before the first time point is not less than the first duration threshold corresponding to the target monitoring space, and the time interval between the first time point and the second time point in the second trajectory segment collected from the target monitoring space is not less than the second duration threshold. The first duration threshold and the second duration threshold are predetermined based on the ratio of historical spatial location dispersion to system spatiotemporal measurement error being not less than a preset multiple. The system spatiotemporal measurement error is the composite value of the spatial positioning error and time synchronization error of the multi-camera system covering the target monitoring space. The historical spatial location dispersion is the arithmetic mean of the spatial location differences between different targets in the target monitoring space under the candidate time span used to determine the first duration threshold and the second duration threshold. The probability prediction module is used to extract historical trajectory segments with a duration of not less than the first duration threshold before the first time point from the first trajectory segment, and input the historical trajectory segments and the time interval into a long-term position prediction model specific to the target monitoring space, so that the model outputs a multimodal spatial position probability distribution of the target monitoring space corresponding to the second time point; wherein, the long-term position prediction model is pre-trained based on the unlabeled historical trajectories accumulated by the target monitoring space itself. The association decision module is used to take the probability value of the actual spatial position of the second time point in the second trajectory segment in the multimodal spatial position probability distribution as the association matching degree, and obtain a multi-target tracking result based on the association matching degree to indicate whether the first trajectory segment and the second trajectory segment belong to the same target.

[0019] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the aforementioned multi-target tracking method based on scene-specific long-term prediction.

[0020] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the described multi-target tracking method based on scene-specific long-term prediction.

[0021] The fifth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the described multi-target tracking method based on scene-specific long-term prediction.

[0022] The multi-target tracking method based on scene-specific long-term prediction provided in this application predetermines the first and second time-term thresholds based on the ratio of the historical spatial location distribution of different targets within the target monitoring space to the system spatiotemporal measurement error being no less than a preset multiple. When selecting the first and second time points, the time interval for association judgment is located on a time scale where the spatial location distribution of different targets is significantly greater than the system spatiotemporal measurement error. This allows the signal-to-noise ratio of association judgment to be controlled above a preset level, and provides robustness across orders of magnitude against spatial positioning errors and time synchronization errors. By inputting historical trajectory segments with a duration of not less than a first duration threshold along with time intervals into a long-term position prediction model pre-trained based on unlabeled historical trajectories accumulated in the target monitoring space, a position mapping from historical trajectory segments to the second time point is directly established. This eliminates the need to predict intermediate time points step by step, thus avoiding error accumulation in recursive prediction. The model outputs a multimodal spatial position probability distribution corresponding to the second time point. This probability distribution carries the long-term motion patterns unique to the target monitoring space, rather than relying solely on short-term inertial extrapolation. Therefore, even with time intervals on the order of tens of seconds, it can still provide a multimodal representation of the possible location of the target at the second time point, improving the reliability of position prediction in scenarios with long-term occlusion or trajectory breaks, and achieving deployment-ready adaptation without the need for manual parameter tuning. Using the probability value of the actual spatial location at the second time point in the multimodal spatial location probability distribution as the association matching degree, and based on this, the tracking result of whether the first trajectory segment and the second trajectory segment belong to the same target can be obtained. This can directly incorporate the uncertainty of long-term prediction into the association confidence degree, so that the association judgment does not rely on a single prediction point or short-term distance constraints. This can improve the continuity and identity stability of long-term multi-target tracking in real deployment scenarios, and further improve the analysis accuracy and reliability of monitoring applications such as pedestrian movement analysis, dwelling area identification and passenger flow statistics in the target monitoring space, providing more reliable trajectory data support for the safety monitoring and abnormal behavior detection of public places.

[0023] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0024] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a schematic diagram of the first process of a multi-target tracking method based on scene-specific long-term prediction in one embodiment of this application.

[0026] Figure 2 This is a schematic diagram of a second process of a multi-target tracking method based on scene-specific long-term prediction in one embodiment of this application.

[0027] Figure 3 This is a schematic diagram of the third process of a multi-target tracking method based on scene-specific long-term prediction in one embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the fourth process of a multi-target tracking method based on scene-specific long-term prediction in one embodiment of this application.

[0029] Figure 5 This is a flowchart illustrating a scene-specific long-term prediction-based multi-target tracking method in an application example of this application.

[0030] Figure 6 This is a schematic diagram of a sub-process for constructing a long-term location prediction model specific to the target monitoring space in an application example of this application.

[0031] Figure 7 This is a schematic diagram of the long-term probability prediction and correlation sub-process based on the target monitoring space-specific long-term location prediction model in an application example of this application.

[0032] Figure 8 This is a schematic diagram of trajectory prediction and spatial probability distribution in an application example of this application.

[0033] Figure 9 This is a schematic diagram of the structure of a multi-target tracking device based on scene-specific long-term prediction in one embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0035] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the scheme according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0036] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0037] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0038] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0039] First, we will describe the existing indoor space multi-target tracking methods. Currently, the mainstream indoor space multi-target tracking methods mainly fall into three categories.

[0040] The first category is similarity matching methods without explicit spatiotemporal constraints. These methods weight the apparent feature distance between the detection box sequences in a trajectory segment and the spatial distance between their short-term predicted positions, then compare this weighted average with a manually set similarity threshold to determine whether two trajectory segments belong to the same target. Because they do not explicitly model the spatiotemporal constraints of pedestrian trajectories, these methods can only rely on continuously increasing the similarity threshold to avoid mismatches when the time interval between trajectory segments is long. This leads to difficulties in correctly merging trajectory segments of the same target due to fluctuations in apparent features, resulting in trajectory fragmentation and missed matches. Furthermore, since the similarity threshold depends on manual setting, its adaptability to different indoor spaces is poor.

[0041] The second category is explicit spatiotemporal constraint matching methods based on manually preset parameter models. These methods attempt to construct a similarity threshold function that varies with spatial distance and time intervals using a manually selected function form and a set of hyperparameters, replacing the fixed threshold directly set in the first category. However, the function form and key parameters of this type of method are all manually preset and not learned from the indoor space being operated on. Therefore, they cannot express the unique pedestrian movement patterns of that space, such as customers in shopping malls tending to retrace along main aisles, or people in office areas frequently stopping short distances and turning around. Thus, they cannot truly establish a spatiotemporal constraint model of the indoor space.

[0042] The third category is trajectory prediction-based association methods. These methods utilize Kalman filtering, Long Short-Term Memory (LSTM) networks, or Transformer models to predict the future location of the target and use the distance between the predicted location and the actual observed location as motion similarity for trajectory association. These methods are typically trained on general datasets, and the prediction models use general motion assumptions such as constant velocity and constant acceleration. They lack adaptability to specific deployment scenarios and cannot effectively match the spatiotemporal constraints specific to scenarios in indoor spaces, such as reaching location B from location A via stairs or elevators, which takes more than 20 seconds.

[0043] None of the three methods mentioned above adequately address the long-term robust tracking problem in real-world deployment scenarios. Specifically, the spatiotemporal constraints or prediction models relied upon by existing methods are typically trained based on short-term inertial extrapolation or general datasets, with prediction timescales mostly on the order of seconds. Beyond 10 seconds, there are no general spatiotemporal constraints that hold true independently of the specific indoor space, thus making it impossible to reliably predict the target's position tens of seconds later. Furthermore, the spatiotemporal constraints or prediction models in existing methods are not learned from the specific indoor space where deployment takes place, but rather from manually set or general scene training, failing to capture the long-term pedestrian movement patterns unique to that space, resulting in poor adaptability when deployed to new scenarios.

[0044] More importantly, under typical commercial deployment conditions, the spatial positioning error of multi-camera systems covering the target monitoring space can reach the meter level, and the time synchronization error can reach the second level. This means that the predicted position difference and measurement error on which short-term association judgment relies are on the same order of magnitude, resulting in insufficient signal-to-noise ratio for association judgment. Any solution relying on short-term, fine-grained spatiotemporal constraints may fail in practical use due to error contamination. However, existing technologies have not recognized the fundamental dilemma of insufficient signal-to-noise ratio in short-term association judgment, nor do they possess the technical means to reliably predict the future position of the target on a timescale of tens of seconds. Consequently, they cannot reconstruct the signal-to-noise ratio required for association judgment, leading to insufficient association reliability in scenarios with long-term occlusion or trajectory breaks, and prominent issues such as trajectory fragmentation and identity switching.

[0045] Based on this, in order to address the problems of insufficient signal-to-noise ratio in short-term association judgment and inability to reliably predict the future position of the target on a timescale of tens of seconds in existing multi-target tracking methods, embodiments of this application provide a multi-target tracking method based on scene-specific long-term prediction, a multi-target tracking device, electronic device, computer-readable storage medium, and computer program product for executing the multi-target tracking method based on scene-specific long-term prediction. It can pre-train a scene-specific long-term position prediction model based on the unlabeled historical trajectory accumulated in the target monitoring space itself, and perform long-term association judgment between trajectory segments under the time threshold determined by the ratio of historical spatial position distribution to system spatiotemporal measurement error. This allows for the reconstruction of the signal-to-noise ratio required for association judgment on a timescale of tens of seconds, thereby improving the reliability of association and the continuity of tracking in real deployment scenarios.

[0046] It should be noted that, in this application, "scene" refers to the target monitoring space, that is, a specific indoor space covered by multiple cameras for pedestrian target tracking. For example, the target monitoring space could be a retail store with an area of ​​approximately 600 square meters, covered by 37 cameras. In this application, the scene-specific long-term position prediction model is a target monitoring space-specific long-term position prediction model. This model is pre-trained based on the unlabeled historical trajectories accumulated within the target monitoring space itself, and is used to capture the unique long-term movement patterns of pedestrians within that target monitoring space.

[0047] The following examples will provide a detailed description.

[0048] Based on this, embodiments of this application provide a multi-target tracking method based on scene-specific long-term prediction, which can be implemented by a multi-target tracking device based on scene-specific long-term prediction. This multi-target tracking device can be a functional module in an electronic device or a stand-alone multi-target tracking device. The multi-target tracking device is communicatively connected to each camera in a multi-camera system covering the target monitoring space. It receives video streams acquired by each camera, or receives a sequence of detection boxes obtained after the multi-camera system performs pedestrian detection on the video stream. Based on the received data, it executes the aforementioned multi-target tracking method based on scene-specific long-term prediction and outputs the multi-target tracking result. See also... Figure 1 The multi-target tracking method based on scene-specific long-term prediction specifically includes the following: Step 100: Based on the condition that the duration of the historical trajectory of the first trajectory segment collected from the target monitoring space before the first time point is not less than the first duration threshold corresponding to the target monitoring space, and the time interval between the first time point and the second time point in the second trajectory segment collected from the target monitoring space is not less than the second duration threshold, select the first time point from the first trajectory segment and select the second time point from the second trajectory segment; wherein, the first duration threshold and the second duration threshold are predetermined based on the ratio of historical spatial location dispersion to system spatiotemporal measurement error being not less than a preset multiple; the system spatiotemporal measurement error is the composite value of the spatial positioning error and time synchronization error of the multi-camera system covering the target monitoring space; the historical spatial location dispersion is the arithmetic mean of the spatial location differences between different targets in the target monitoring space under the candidate time span used to determine the first duration threshold and the second duration threshold.

[0049] In one or more embodiments of this application, the target monitoring space refers to a specific indoor space covered by multiple cameras for pedestrian target tracking. For example, the target monitoring space may be a retail store with an area of ​​approximately 600 square meters, covered by 37 cameras.

[0050] Both the first and second trajectory segments refer to pedestrian trajectory segments collected from within the target monitoring space. Each trajectory segment consists of several pedestrian observation positions arranged in chronological order, with each pedestrian observation position corresponding to a time point. For example, the first trajectory segment could be the trajectory formed by a pedestrian walking inside the store between 10:00:00 and 10:00:30, and the second trajectory segment could be the trajectory formed by a pedestrian walking inside the store between 10:00:50 and 10:01:10.

[0051] The first time point refers to a point in time on the first trajectory segment, and the second time point refers to a point in time on the second trajectory segment, with the second time point being later than the first time point. For example, the first time point is 10:00:30, and the second time point is 10:00:50.

[0052] The time interval refers to the time difference between the first time point and the second time point, i.e. ,in Indicates the first point in time. This indicates the second time point. In the example above, the time interval is 20 seconds.

[0053] The duration of the historical trajectory refers to the length of time that the portion of the trajectory before the first time point in the first trajectory segment is available for model input. For example, if the first time point is 10:00:30 and the first trajectory segment starts at 10:00:00, then the duration of the historical trajectory before the first time point is 30 seconds.

[0054] The first duration threshold refers to the lower limit of the duration of the historical trajectory that the first trajectory segment must have before the first time point. For example, the first duration threshold can be 30 seconds.

[0055] The second duration threshold refers to the lower limit of the time interval between the first time point and the second time point. For example, the second duration threshold can be 20 seconds.

[0056] Historical spatial location dispersion refers to the arithmetic mean of the spatial location differences between different targets within the target monitoring space under the candidate time spans used to determine the first and second duration thresholds. Specifically, the spatial location difference is the Euclidean distance between the spatial locations of two different targets at the end of the candidate time span; the arithmetic mean is the value obtained by taking the arithmetic mean of the Euclidean distances between the spatial locations of multiple pairs of trajectories belonging to different targets at the end of the candidate time span. The candidate time span refers to the time span considered when determining the first and second duration thresholds, such as each candidate time span in a preset candidate time span set. For example, in a retail store with a target monitoring space of approximately 600 square meters, after statistically analyzing the trajectories of 500 pairs of different pedestrians, it was found that when the time span was 20 seconds, the arithmetic mean of the spatial location differences between different pedestrians was approximately 15 meters.

[0057] The system spatiotemporal measurement error refers to the combined value of measurement errors in spatial positioning and time synchronization of a multi-camera system covering the target monitoring space. Spatial positioning error refers to the deviation between the pedestrian's spatial position output by the multi-camera system and the actual position, which can reach 1 to 2 meters under typical commercial deployment conditions, with lens distortion areas reaching 3 to 4 meters. Time synchronization error refers to the time difference in the occurrence of the same physical event in different cameras, which can reach 1 to 3 seconds under typical commercial deployment conditions. The time synchronization error can be multiplied by a preset pedestrian walking speed to convert it into an equivalent spatial error, and then combined with the spatial positioning error to obtain the system spatiotemporal measurement error. For example, if the typical value of spatial positioning error is 1.5 meters and the typical value of time synchronization error is 1.0 second, the equivalent spatial error calculated based on a pedestrian walking speed of 1.5 meters / second is 1.5 meters. Taking the larger of the two values, the system spatiotemporal measurement error is 1.5 meters.

[0058] The preset multiple is a pre-defined signal-to-noise ratio target used to ensure that, within the time span corresponding to the second duration threshold, the ratio of the historical spatial location distribution of different targets to the system's spatiotemporal measurement error is not less than this multiple. For example, the preset multiple can be 10.

[0059] In this embodiment, the first and second duration thresholds are predetermined based on the premise that the ratio of the historical spatial location dispersion of different targets within the target monitoring space to the system's spatiotemporal measurement error is not less than a preset multiple. That is, within the time span corresponding to the second duration threshold, the ratio of the historical spatial location dispersion of different targets to the system's spatiotemporal measurement error should not be less than the preset multiple. This ensures that during association judgment, the spatial location difference between different targets is significantly greater than the measurement error, thus guaranteeing the signal-to-noise ratio of the association judgment. For example, when the second duration threshold is 20 seconds, the historical spatial location dispersion of different customers is approximately 15 meters, and the system's spatiotemporal measurement error is approximately 1.5 meters. The ratio of these two is 10, which is not less than the preset multiple of 10.

[0060] In one example, the target monitoring space is a retail store with an area of ​​approximately 600 square meters, covered by 37 cameras. A first duration threshold of 30 seconds and a second duration threshold of 20 seconds are set, with a preset multiplier of 10. Based on the condition that the historical trajectory duration of the first trajectory segment before the first time point is 30 seconds, and the time interval between the first and second time points is 20 seconds, 10:00:30 is selected as the first time point from the first trajectory segment, and 10:00:50 is selected as the second time point from the second trajectory segment.

[0061] In another example, the retail store later deployed a high-precision time synchronization service and manually calibrated the parameters of lenses with significant distortion. The typical measured spatial positioning error was reduced to 0.8 meters, and the typical time synchronization error was reduced to 0.3 seconds, resulting in a system spatiotemporal measurement error of approximately 0.8 meters. While maintaining the preset multiplier of 10, the second duration threshold can be lowered to approximately 10 seconds, because the typical historical spatial location dispersion of different customers over a 10-second time span is approximately 8 meters, which is 10 times the distance to 0.8 meters. In this case, the first duration threshold remains 30 seconds, as 30 seconds can cover a typical movement of a customer from the entrance to the checkout counter within the store.

[0062] It should be noted that the first duration threshold is not less than the second duration threshold. In the two examples above, the first duration threshold of 30 seconds is greater than the second duration threshold of 20 seconds or 10 seconds.

[0063] Step 200: Extract historical trajectory segments from the first trajectory segment that have a duration of not less than the first duration threshold before the first time point, and input the historical trajectory segments and the time interval into a long-term location prediction model specific to the target monitoring space, so that the model outputs the multimodal spatial location probability distribution of the target monitoring space corresponding to the second time point; wherein, the long-term location prediction model is pre-trained based on the unlabeled historical trajectories accumulated by the target monitoring space itself.

[0064] In step 200, a historical trajectory segment refers to a section of the first trajectory segment that is located before the first time point and has a duration of not less than a first duration threshold. This historical trajectory segment, ending at the first time point, traces back the first duration threshold, encompassing the target's various spatial locations within the target monitoring space during that time period. For example, when the first time point is 10:00:30 and the first duration threshold is 30 seconds, the historical trajectory segment is the 30-second trajectory of the target within the store between 10:00:00 and 10:00:30.

[0065] Unlabeled historical trajectories refer to trajectory data obtained by automatically processing historical video data of a target monitoring space using a multi-target tracking algorithm. This trajectory data does not require manual labeling of pedestrian identities. For example, for a retail store with an area of ​​approximately 600 square meters, existing multi-target tracking algorithms can be used to process the store's surveillance video from the past four consecutive weeks, automatically generating approximately 100,000 customer trajectory segments. These trajectory segments do not require manual correction of identity labels and are therefore unlabeled historical trajectories.

[0066] A target-monitored space-specific long-term location prediction model refers to a prediction model pre-trained using only unlabeled historical trajectories accumulated within the target monitoring space itself, rather than a general prediction model trained on a general dataset. This model can learn the long-term movement patterns of pedestrians unique to the target monitoring space, such as the typical path and time taken by customers in this store from the entrance through the shelving area to the checkout counter.

[0067] Multimodal spatial location probability distribution refers to the probability distribution of a target appearing in multiple different spatial locations within a two-dimensional target monitoring space, rather than simply outputting a single predicted location. This probability distribution can express the uncertainty of a pedestrian potentially appearing in multiple different locations at future moments. For example, for a retail store with an area of ​​approximately 600 square meters, the multimodal spatial location probability distribution could be the set of probability values ​​corresponding to each grid location within the store. A grid refers to multiple spatial units of equal area into which the target monitoring space is divided.

[0068] In this embodiment, the extracted historical trajectory segments and time intervals are input into a long-term position prediction model specific to the target monitoring space. This model directly establishes a position mapping from the historical trajectory segments to the second time point, eliminating the need for progressive prediction of intermediate time points between the first and second time points, thus avoiding the gradual accumulation of errors in frame-by-frame recursive prediction. The model outputs a multimodal spatial position probability distribution of the target monitoring space corresponding to the second time point.

[0069] In one example, the long-term location prediction model is a sequence model built on a Transformer encoder and a probability output module. This model takes historical trajectory segments from the past 30 seconds and future time intervals of 20 seconds as input, and outputs the probability distribution of each grid location within the retail store. For example, if a customer enters through the entrance and reaches the shelving area within 30 seconds, the model predicts that 20 seconds later, the customer is likely to be located in the checkout area (50%), the main aisle (20%), and shelving area D (10%), forming a multimodal spatial location probability distribution.

[0070] In another example, a long-term location prediction model can be built based on a recurrent neural network incorporating an attention mechanism. This model also takes historical trajectory fragments and time intervals as input and outputs a multimodal spatial location probability distribution of the target monitoring space.

[0071] In another example, a long-term location prediction model can be built on a temporal convolutional network, as long as the model can take historical trajectory fragments and future time intervals as inputs and output a multimodal spatial location probability distribution.

[0072] It should be noted that the sum of all probability values ​​in the multimodal spatial location probability distribution output by the model can be 1, meaning the probability distribution is normalized. For example, when the output is the probability of each grid cell, the sum of all grid cell probability values ​​is 1.

[0073] Step 300: Use the probability value of the actual spatial position of the second time point in the second trajectory segment in the multimodal spatial position probability distribution as the association matching degree, and obtain the multi-target tracking result based on the association matching degree to indicate whether the first trajectory segment and the second trajectory segment belong to the same target.

[0074] In step 300, the actual spatial location refers to the pedestrian observation position at the second time point in the second trajectory segment. This position comes from the localization result of the multi-camera system for the pedestrian target. For example, when the second time point is 10:00:50, the actual spatial location is a coordinate point near the cashier area inside the store.

[0075] The correlation matching degree refers to the probability value corresponding to the actual spatial location at the second time point, obtained by substituting the actual spatial location at the second time point into the multimodal spatial location probability distribution. This probability value reflects the likelihood that the target will appear at the actual spatial location at the second time point, given the historical trajectory of the first trajectory segment before the first time point and the time interval. For example, when the actual spatial location at the second time point falls into a grid with a probability value of 0.6, the correlation matching degree is 0.6.

[0076] Multi-target tracking results refer to the determination results used to indicate whether the first trajectory segment and the second trajectory segment belong to the same target. For example, when the correlation matching degree is not less than a preset score threshold, the multi-target tracking result is that the first trajectory segment and the second trajectory segment belong to the same target; when the correlation matching degree is less than the preset score threshold, the multi-target tracking result is that they do not belong to the same target.

[0077] In one example, the multimodal spatial location probability distribution represents the target monitoring space in a grid format, with each grid corresponding to a probability value. If the actual spatial location at the second time point falls into a grid with a probability value of 0.6, then 0.6 is used as the association matching degree. The preset score threshold is a pre-set minimum score used to determine whether the association matching degree reaches the minimum value that can be judged as belonging to the same target, and can be determined from experience or a validation set. For example, the preset score threshold can be 0.5. Since 0.6 is greater than 0.5, the first trajectory segment and the second trajectory segment are determined to belong to the same target, and the two are merged.

[0078] In another example, if the actual spatial location at the second time point does not fall into any grid with a probability value, then 0 is used as the correlation matching degree, and it is determined that the first trajectory segment and the second trajectory segment do not belong to the same target.

[0079] To more clearly illustrate the technical contributions of the embodiments of this application, further explanation is provided herein in conjunction with the prior art closest to this application.

[0080] One of the closest existing technologies to this application is a non-Markovian tracking method that uses behavioral patterns to impose global consistency constraints on trajectories. This method associates behavioral patterns with trajectories through an energy function, imposes consistency constraints on trajectories globally, and proposes an iterative learning scheme that does not require real-world annotations. While this method discloses the non-Markovian concept and the learning approach that does not require real-world annotations, it fails to recognize that in real-world deployment scenarios, the presence of spatial positioning and temporal synchronization errors in multi-camera systems covering the target monitoring space can cause the predicted position difference and measurement error on which short-term association judgments rely to be on the same order of magnitude, resulting in insufficient signal-to-noise ratio for association judgments. Furthermore, it does not provide a technical means to quantitatively correlate the time scale of association judgments with the system's spatiotemporal measurement errors, ensuring that the ratio of the spatial distribution of different target locations to the system's spatiotemporal measurement errors is not less than a preset multiple. In addition, the behavioral patterns used in this method are used for trajectory grouping, rather than an explicit long-term position prediction model that takes historical trajectory fragments and future time intervals as input and outputs a multimodal spatial position probability distribution. It also does not provide an association criterion based on the probability value of the actual spatial location in the probability distribution as the association matching degree.

[0081] Another prior art related to this application is a deep trajectory prediction and tracking method based on Long Short-Term Memory (LSTM) networks. This method constructs an LSTM-based trajectory prediction model, taking the target's historical trajectory and scene information as input, and outputting the position information of multiple consecutive future frames. It calculates motion similarity by the distance between the predicted position and the actual position. The prediction model of this method is trained on a general dataset, and the prediction time scale is several future frames, which is a short-term recursive prediction at the second level. It does not recognize the problem of insufficient signal-to-noise ratio in short-term association judgment, nor does it provide a technical means to train a scene-specific long-term position prediction model based on the unlabeled historical trajectory accumulated by the target monitoring space itself. Moreover, its output is deterministic position coordinates, not a multimodal spatial position probability distribution, and it cannot incorporate the uncertainty of long-term prediction into the association criterion.

[0082] Even combining the aforementioned non-Markov tracking methods with LSTM-based trajectory prediction and tracking methods only yields a tracking scheme that uses trajectory prediction as the basis for motion similarity calculation and imposes behavioral pattern constraints globally. This combination still fails to recognize the fundamental dilemma of insufficient signal-to-noise ratio in short-term association judgment, does not provide a technical means to determine long-term timescales based on the ratio of different target spatial location distributions to system spatiotemporal measurement errors, does not provide a scene-specific long-term position prediction model that uses historical trajectory fragments and future time intervals as inputs and outputs a multimodal spatial position probability distribution, and does not provide an association criterion that uses the probability value of the actual spatial location in the probability distribution as the association matching degree. Therefore, none of the aforementioned prior art and its combinations provide the technical inspiration for obtaining the technical solution of this application.

[0083] Based on the above understanding, this application proposes to determine a long time scale of tens of seconds by error statistics or objective laws, use a scene-specific long-term location prediction model to output a multimodal spatial location probability distribution, and use the probability value of the actual spatial location in the probability distribution as the association matching degree, thereby reconstructing the signal-to-noise ratio required for association judgment in real deployment scenarios.

[0084] As described above, the multi-target tracking method based on scene-specific long-term prediction provided in this application, because the first and second time-duration thresholds are predetermined based on the ratio of the historical spatial position distribution of different targets within the target monitoring space to the system's spatiotemporal measurement error being no less than a preset multiple, can ensure that the time interval for association judgment is on a time scale where the spatial position distribution of different targets is significantly greater than the system's spatiotemporal measurement error. This controls the signal-to-noise ratio of association judgment above a preset level, reducing the interference of spatial positioning error and time synchronization error on association judgment. By inputting historical trajectory segments with a duration no less than the first time-duration threshold along with the time interval into a long-term position prediction model pre-trained based on unlabeled historical trajectories accumulated within the target monitoring space, the position mapping from the historical trajectory segments to the second time point is directly established. Without the need to progressively predict intermediate time points, it can output the multimodal spatial position probability distribution corresponding to the second time point, allowing the prediction result to carry the unique long-term motion patterns of the target monitoring space, rather than relying solely on short-term inertial extrapolation. Therefore, even with time intervals on the order of tens of seconds, it can still perform multimodal representation of the possible location of the target at the second time point, improving the reliability of position prediction in scenarios with long-term occlusion or trajectory breakage. Using the probability value of the actual spatial location at the second time point in the multimodal spatial location probability distribution as the association matching degree, and based on this, the tracking result of whether the first trajectory segment and the second trajectory segment belong to the same target can be obtained. This can directly incorporate the uncertainty of long-term prediction into the association confidence, so that the association judgment does not depend on a single prediction point or short-term distance constraints, thereby improving the continuity and identity stability of long-term multi-target tracking in real deployment scenarios.

[0085] To further address the issue in existing technologies where the duration of association judgment is manually set and cannot be objectively determined based on the actual error level of the deployment scenario, this application provides a multi-target tracking method based on scene-specific long-term prediction. (See also...) Figure 2 The multi-target tracking method based on scene-specific long-term prediction includes the following content before step 100: Step 011: Obtain the statistical values ​​of the spatial positioning error and the time synchronization error of the multi-camera system within the target monitoring space.

[0086] In one or more embodiments of this application, the statistical value of spatial positioning error refers to a typical value obtained by statistically analyzing the spatial positioning errors obtained from several measurements after measuring the spatial positioning accuracy of a multi-camera system covering the target monitoring space. The statistical value of spatial positioning error can be the average of multiple measurement results plus one standard deviation, or it can be the 84th percentile value after sorting the multiple measurement results from smallest to largest.

[0087] Specifically, when obtaining the statistical value of spatial positioning error, several reference points with known precise locations can be selected evenly on the ground. A multi-camera system can perform multiple positioning measurements on each reference point. The Euclidean distance between the system output position and the actual position of the reference point in each measurement is taken as a spatial positioning error sample. Then, all samples are statistically analyzed to obtain the statistical value of spatial positioning error.

[0088] The statistical value of time synchronization error refers to the typical value obtained by statistically analyzing the time synchronization error obtained from several measurements after measuring the time synchronization accuracy of a multi-camera system covering the target monitoring space.

[0089] Specifically, when obtaining the statistical value of time synchronization error, a synchronization timestamp can be injected into each video stream to record the time difference of the same physical event in different cameras. Each recorded time difference is taken as a time synchronization error sample, and then all samples are statistically analyzed to obtain the statistical value of time synchronization error.

[0090] In one example, for a retail store with an area of ​​approximately 600 square meters, 20 reference points with known precise locations were evenly selected on the ground. A multi-camera system consisting of 37 cameras repeatedly measured the location of each reference point 10 times, obtaining a total of 200 spatial positioning error samples. Statistical analysis of these 200 samples yielded a statistical value of 1.5 meters for the spatial positioning error. Simultaneously, synchronization timestamps were injected into each video stream, and the experiment was repeated 30 times, obtaining 30 time synchronization error samples. Statistical analysis of these 30 samples yielded a statistical value of 1.0 second for the time synchronization error.

[0091] Step 012: Multiply the statistical value of the time synchronization error by the preset pedestrian walking speed to obtain the equivalent spatial error.

[0092] In one or more embodiments of this application, the preset pedestrian walking speed is a typical value of a pre-set normal pedestrian walking speed. The preset pedestrian walking speed can be any value from 1.0 m / s to 1.5 m / s. For example, the preset pedestrian walking speed can be 1.5 m / s.

[0093] Equivalent spatial error refers to the error value obtained by converting the time synchronization error into spatial distance. Since the dimension of time synchronization error is time, it cannot be directly combined with spatial positioning error. Therefore, the statistical value of time synchronization error is multiplied by the preset pedestrian walking speed to obtain the equivalent spatial position deviation of the pedestrian caused by the time synchronization error.

[0094] In one example, the statistical value of the time synchronization error is 1.0 second, and the preset pedestrian walking speed is 1.5 meters / second. Then the equivalent spatial error is 1.0 second × 1.5 meters / second = 1.5 meters.

[0095] Step 013: Take the larger value between the statistical value of the spatial positioning error and the equivalent spatial error as the system spatiotemporal measurement error.

[0096] In one or more embodiments of this application, the system spatiotemporal measurement error has been defined in step 100 and will not be repeated here. In this step, the statistical value of the spatial positioning error obtained in step 011 is compared with the equivalent spatial error obtained in step 012, and the larger value is taken as the system spatiotemporal measurement error. This allows for the synthesis of the system spatiotemporal measurement error in a more conservative manner when the contributions of the spatial positioning error and the time synchronization error to the system spatiotemporal measurement error are of the same order of magnitude.

[0097] In one example, the statistical value of the spatial positioning error is 1.5 meters, and the equivalent spatial error is 1.5 meters. Taking the larger of the two values, the spatiotemporal measurement error of the system is 1.5 meters.

[0098] Step 014: Within each candidate time span in the preset candidate time span set, statistically analyze the historical spatial location distribution of different targets within the target monitoring space.

[0099] In one or more embodiments of this application, a preset candidate time span set refers to a pre-defined set containing multiple candidate time spans. A candidate time span refers to the length of time used to examine the spatial positional differences between different targets. The candidate time spans in the preset candidate time span set can be arranged in ascending order. For example, the preset candidate time span set can be {5 seconds, 10 seconds, 15 seconds, 20 seconds, 30 seconds, 60 seconds}.

[0100] The method for statistically analyzing the historical spatial distribution of different targets is as follows: randomly select several pairs of trajectories belonging to different targets from the historical trajectory data of the target monitoring space. For each pair of trajectories, calculate the Euclidean distance between the spatial positions of the two targets at the end of a certain candidate time span. Statistically analyze the Euclidean distances of all trajectory pairs (e.g., take the arithmetic mean) to obtain the typical value of the historical spatial distribution of different targets under that candidate time span.

[0101] In one example, for a retail store with an area of ​​approximately 600 square meters, the trajectories of 500 different pedestrians were randomly selected from historical trajectory data. Typical values ​​of the positional differences between the different pedestrians were calculated at candidate time spans of 5 seconds, 10 seconds, 15 seconds, 20 seconds, 30 seconds, and 60 seconds. The statistical results were as follows: approximately 4 meters at 5 seconds, approximately 8 meters at 10 seconds, approximately 12 meters at 15 seconds, approximately 15 meters at 20 seconds, approximately 20 meters at 30 seconds, and approximately 35 meters at 60 seconds.

[0102] Step 015: From each of the candidate time spans, select the shortest time span in which the ratio of the historical spatial location distribution to the system spatiotemporal measurement error is not less than the preset multiple, and use it as the second duration threshold.

[0103] In one or more embodiments of this application, the shortest time span refers to the candidate time span with the shortest duration among the time spans in a preset set of candidate time spans that satisfy the condition that the ratio of historical spatial location distribution to system spatiotemporal measurement error is not less than a preset multiple. The purpose of selecting the shortest time span is to compress the time interval required for association judgment as much as possible while ensuring the signal-to-noise ratio, so that more trajectory segments with medium-length intervals can also participate in long-term association.

[0104] In one example, the system's spatiotemporal measurement error is 1.5 meters, and the preset multiplier is 10, requiring the historical spatial location dispersion to be no less than 15 meters. According to the statistical results of step 014, the dispersion is approximately 8 meters at 10 seconds, which does not meet the requirement; the dispersion is approximately 15 meters at 20 seconds, which meets the requirement. Therefore, 20 seconds is selected as the second duration threshold.

[0105] Step 016: The first duration threshold is the shortest duration of a typical pedestrian movement process that is not less than the second duration threshold and covers the target monitoring space.

[0106] In one or more embodiments of this application, a pre-statistically estimated typical pedestrian movement process refers to the time typically required for a pedestrian to move from one main area to another within the target monitored space, based on the structural layout of the space and pedestrian walking speed. For example, for a retail store with an area of ​​approximately 600 square meters, a typical movement process from the entrance to the checkout counter takes approximately 30 seconds.

[0107] The shortest duration refers to the minimum of the two conditions that simultaneously satisfy both the requirement of being no less than the second duration threshold and the requirement of covering a typical pedestrian movement process. Since this shortest duration ensures that the historical trajectory segment input to the model is long enough to cover a typical pedestrian movement process, and also ensures that the first duration threshold is no less than the second duration threshold, it is used as the first duration threshold.

[0108] In one example, the second duration threshold is 20 seconds, and the typical movement process of a pedestrian is estimated to be about 30 seconds, so the first duration threshold is 30 seconds.

[0109] The first duration threshold determined in the above manner ensures that the model has a sufficiently long historical trajectory input during the inference phase, enabling the model to capture the non-Markovian long-term behavior patterns of the target in the target monitoring space based on complete historical trajectory fragments, rather than relying solely on short-term inertial extrapolation.

[0110] It should be noted that in existing technologies, the time conditions for association judgment are usually set by human experience, or error measurement is only used to adjust the parameters of a manually preset threshold function, rather than to determine the time scale of the association judgment itself. The embodiments of this application use the time scale itself as a core design quantity, determined jointly by the measured error level or objective laws and the upper bound of the error, so that the second duration threshold is no longer a value based on human experience, and structurally ensures that long-term association judgment has a sufficient signal-to-noise ratio.

[0111] As described above, the multi-target tracking method based on scene-specific long-term prediction provided in this application obtains statistical values ​​of spatial positioning error and time synchronization error through error measurement. The time synchronization error is converted into an equivalent spatial error, and the larger value of this error and the spatial positioning error is used to synthesize the system spatiotemporal measurement error. Then, the shortest time span from a preset candidate time span set, where the ratio of historical spatial location distribution to the system spatiotemporal measurement error is not less than a preset multiple, is selected as the second time span threshold. A first time span threshold is determined based on the second time span threshold and a typical pedestrian movement process. Therefore, both the first and second time span thresholds are objectively determined by measured error levels, rather than by human experience. Furthermore, the second time span threshold is compressed to the shortest possible time scale while maintaining the signal-to-noise ratio, expanding the applicability of long-term correlation and enabling more trajectory segments with medium time intervals to participate in long-term correlation, thus improving trajectory integrity.

[0112] In other embodiments, the statistical values ​​of spatial positioning error and time synchronization error can also be obtained indirectly. For example, indirect estimation can be performed using historical trajectory data of the target monitoring space itself, and the magnitude of the measurement error can be inferred by using the abnormal upper bound of the short-term displacement velocity distribution between statistical trajectory segments; or, error statistics can be obtained by using automated periodic detection of camera calibration boards and ground markers.

[0113] In other embodiments, the preset multiple can be any value from 5 to 20. When the preset multiple is larger, the signal-to-noise ratio requirement for association judgment is more stringent, but the second duration threshold increases accordingly; when the preset multiple is smaller, the second duration threshold can be relatively shortened, but the signal-to-noise ratio margin decreases.

[0114] To further address the problems in existing technologies where the duration of association judgment is manually set and cannot be objectively determined based on the actual error level of the deployment scenario, and where error measurement methods require additional measurement work, have high deployment costs, or lack suitable measurement conditions, this application provides a multi-target tracking method based on scenario-specific long-term prediction. (See also...) Figure 3 The multi-target tracking method based on scene-specific long-term prediction includes the following content before step 100: Step 021: Multiply the preset upper bound of the time synchronization error of the multi-camera system in the target monitoring space by the preset pedestrian walking speed to obtain the equivalent spatial error upper bound.

[0115] In one or more embodiments of this application, the preset upper bound of the time synchronization error refers to the maximum possible value of the time synchronization error of a multi-camera system, which is preset based on engineering experience under conventional commercial deployment conditions. The time synchronization error is affected by a combination of factors such as camera time synchronization, encoding and decoding, and network transmission, and generally does not exceed 3 seconds under conventional commercial deployment conditions. Therefore, the preset upper bound of the time synchronization error can be set to 3 seconds.

[0116] The preset pedestrian walking speed has been explained in step 012 and will not be repeated here.

[0117] The equivalent spatial error upper bound refers to the maximum possible error value obtained by converting the preset upper bound of the time synchronization error into spatial distance. Since the preset upper bound of the time synchronization error is in the dimension of time, it cannot be directly synthesized with the preset upper bound of the spatial positioning error. Therefore, it is multiplied by the preset pedestrian walking speed to obtain the equivalent spatial position deviation upper bound caused by the time synchronization error.

[0118] In one example, the preset upper limit of the time synchronization error is 3 seconds, and the preset pedestrian walking speed is 1.5 meters / second. Then the equivalent spatial error upper limit is 3 seconds × 1.5 meters / second = 4.5 meters.

[0119] Step 022: Take the larger of the preset upper bound of the spatial positioning error of the multi-camera system within the target monitoring space and the upper bound of the equivalent spatial error as the system spatiotemporal measurement error.

[0120] In one or more embodiments of this application, the preset upper bound of the spatial positioning error refers to the maximum possible value of the spatial positioning error of a multi-camera system, pre-set based on engineering experience under conventional commercial deployment conditions. The spatial positioning error is influenced by a combination of factors, including camera intrinsic parameter calibration accuracy, lens distortion correction error, installation pose measurement error, and 2D image human positioning error. Under conventional commercial deployment conditions, the spatial positioning error can reach 1 to 2 meters in most areas and 3 to 4 meters in areas with lens distortion; therefore, the preset upper bound of the spatial positioning error can be set to 4 meters.

[0121] In step 022, the larger value between the preset upper bound of the spatial positioning error and the equivalent spatial error upper bound obtained in step 021 is taken as the system spatiotemporal measurement error. In this way, there is no need to perform on-site error measurement of the target monitoring space; the system spatiotemporal measurement error can be conservatively synthesized based solely on the preset error upper bound.

[0122] In one example, the preset upper limit of the spatial positioning error is 4 meters, and the upper limit of the equivalent spatial error is 4.5 meters. Taking the larger value of the two, the spatiotemporal measurement error of the system is 4.5 meters.

[0123] Step 023: Based on the law that the historical spatial location distribution of different targets grows superlinearly with time intervals, while the spatiotemporal measurement error of the system does not accumulate with time intervals, select the shortest time span from each candidate time span in the preset candidate time span set, where the ratio of the historical spatial location distribution to the spatiotemporal measurement error of the system is not less than the preset multiple, as the second duration threshold.

[0124] The preset candidate time span set can be located within a preset experience interval, such as 20 seconds to 60 seconds.

[0125] In one or more embodiments of this application, superlinear growth refers to the phenomenon where the spatial dispersion of historical locations between different targets exhibits a faster growth trend than linear growth as the time interval increases. Statistically, when the time interval is doubled, the typical value of the spatial dispersion between different targets increases by approximately four times. For example, the dispersion is approximately 15 meters when the time interval is 10 seconds, and approximately 60 meters when the time interval is 20 seconds, the latter being four times the former.

[0126] The phrase "does not accumulate with time interval" means that the magnitude of the system's spatiotemporal measurement error is determined by inherent factors such as the calibration accuracy of the multi-camera system, the clock synchronization status, and the quality of the transmission link, and will not significantly increase as the observation time interval increases from a few seconds to tens of seconds. For example, the system's spatiotemporal measurement error is approximately 4.5 meters when the time interval is 10 seconds, and it remains approximately 4.5 meters when the time interval is 60 seconds.

[0127] Based on the above two objective laws, without conducting on-site error measurement of the target monitoring space, the shortest time span that meets the signal-to-noise ratio requirement can be determined by using the pre-set upper bound of the error and the scatter statistics within the pre-set candidate time span set, and this can be used as the second duration threshold.

[0128] In one example, the preset candidate time span set is {10 seconds, 20 seconds, 30 seconds, 60 seconds}, the system's spatiotemporal measurement error is 4.5 meters, and the preset multiplier is 10, requiring a historical spatial location dispersion of no less than 45 meters. Based on a rough estimate using the retail store's historical trajectory data, the typical location dispersion for different customers is approximately 15 meters at 10 seconds and approximately 60 meters at 20 seconds. Therefore, 20 seconds is selected as the second duration threshold.

[0129] Step 024: The first duration threshold is the shortest duration of a typical pedestrian movement process that is not less than the second duration threshold and covers the target monitoring space.

[0130] In step 024, the meaning of the pre-statistical typical pedestrian movement process and the shortest duration has been explained in step 016 and will not be repeated here. The only difference between this step and step 016 is that the second duration threshold is determined by a preset upper bound, rather than by the measured error statistics.

[0131] In one example, the second duration threshold is 20 seconds. The typical movement process of a pedestrian is estimated to be about 30 seconds. Therefore, the first duration threshold is 30 seconds, which satisfies the condition that it is not less than the second duration threshold.

[0132] Steps 021 to 024 are alternatives to steps 011 to 016 in the aforementioned embodiments, with steps 021 to 024 being the default method; steps 011 to 016 are error measurement methods, where the default method is the preferred method and the error measurement method is an optional refinement method; when deployment conditions allow for error measurement, the error measurement method can be used to further compress the lower limit of the second duration threshold, and the specific method can be selected according to actual application requirements.

[0133] As described above, the multi-target tracking method based on scene-specific long-term prediction provided in this application directly synthesizes the system's spatiotemporal measurement error by using a preset upper bound for spatial positioning error and a preset upper bound for time synchronization error. Based on the objective law that the historical spatial location distribution of different targets grows superlinearly with time intervals, while the system's spatiotemporal measurement error does not accumulate with time intervals, the shortest time span that meets the signal-to-noise ratio requirement is selected from a preset candidate time span set as the second duration threshold. This is then combined with a typical pedestrian movement process to determine the first duration threshold. Therefore, the first and second duration thresholds can be objectively determined without on-site error measurement of the target monitoring space, reducing deployment costs. Furthermore, under conventional engineering conditions, the signal-to-noise ratio of the association judgment can still be guaranteed to be no less than a preset multiple, achieving deployment-ready adaptation.

[0134] To further address the problem that existing predictive models, trained on general datasets, cannot capture the long-term behavioral patterns specific to the target monitoring space, this application provides a multi-target tracking method based on scene-specific long-term prediction. (See also...) Figure 2 or Figure 3 The multi-target tracking method based on scene-specific long-term prediction includes the following content before step 100: Step 030: Obtain the unlabeled historical trajectory of the target monitoring space; wherein, the unlabeled historical trajectory is obtained by processing the historical video data of the target monitoring space based on a multi-target tracking algorithm.

[0135] In one or more embodiments of this application, the unlabeled historical trajectory can be obtained by processing historical video data of the target monitoring space based on a multi-object tracking (MOT) algorithm. This step directly obtains the unlabeled historical trajectory as the data basis for subsequent training of a scene-specific long-term location prediction model.

[0136] Step 040: Filter the trajectories from the unlabeled historical trajectories whose duration is not less than the first duration threshold.

[0137] In one or more embodiments of this application, filtering refers to removing trajectories with insufficient duration from unlabeled historical trajectories, retaining only trajectories with a duration of not less than a first duration threshold as training material. This is because only trajectories with a sufficiently long duration can extract complete historical trajectory segments with a duration of not less than the first duration threshold for model training.

[0138] In one example, the first duration threshold is 30 seconds. From approximately 100,000 unlabeled historical trajectories, after removing trajectories with a duration of less than 30 seconds, approximately 20,000 long trajectories with a duration of not less than 30 seconds are obtained.

[0139] Step 050: Sample each trajectory with a duration not less than the first duration threshold after filtering according to the preset sampling interval to obtain multiple sampling time points.

[0140] In one or more embodiments of this application, the preset sampling interval refers to a pre-set time interval for extracting sampling time points on the trajectory. The sampling time point refers to a time point selected from the trajectory according to the preset sampling interval for extracting historical trajectory segments. The preset sampling interval can be 1 second or other values.

[0141] In one example, the preset sampling interval is 1 second. For each filtered trajectory, samples are taken from the trajectory at 1 time point per second, resulting in multiple sampling time points. For example, for a trajectory lasting 30 seconds, sampling at 1-second intervals will yield 30 sampling time points.

[0142] Step 060: For each sampling time point, extract the historical trajectory segment prior to that sampling time point with a duration not less than the first duration threshold.

[0143] In one or more embodiments of this application, the specific method for extracting historical trajectory segments is as follows: taking the current sampling time point as the end time, tracing back a time length of a first duration threshold, and extracting the continuous trajectory of the target in the target monitoring space within that time period as a historical trajectory segment. The historical trajectory segment has been described in step 200 and will not be repeated here.

[0144] In one example, the first duration threshold is 30 seconds. For a certain sampling time point 10:00:30, the trajectory between 10:00:00 and 10:00:30 is extracted as a historical trajectory segment.

[0145] Step 070: For each of the historical trajectory segments, form a training sample with each of the multiple future time intervals not less than the second duration threshold, and use the actual spatial location corresponding to the future time interval as the label of the training sample to obtain a training sample set composed of each of the training samples with the label.

[0146] In one or more embodiments of this application, the future time interval refers to the length of time between the end of a historical trajectory segment and a future predicted target time. During the training phase, for each historical trajectory segment, multiple future time intervals not less than a second duration threshold can be selected, such as 20 seconds, 30 seconds, 60 seconds, etc.

[0147] A training sample refers to a set of input data consisting of a historical trajectory segment and a future time interval. A label refers to the supervision signal corresponding to the training sample, that is, the actual spatial location corresponding to the future time interval—the spatial location of the target actually appearing in the target monitoring space after the end of the historical trajectory segment and the elapsed time interval. A training sample set refers to a collection of multiple training samples, each with a label.

[0148] In one example, the second duration threshold is 20 seconds. For a given historical trajectory segment, three future time intervals of 20 seconds, 30 seconds, and 60 seconds are selected respectively, forming three training samples together with the historical trajectory segment. Taking 20 seconds as an example, the input of this training sample is the historical trajectory segment and 20 seconds, and the label is the actual spatial location of the target 20 seconds after the end of the historical trajectory segment.

[0149] Step 080: Based on the training sample set, train a preset sequence model using the cross-entropy loss function to obtain a long-term location prediction model specific to the target monitoring space; wherein, the long-term location prediction model is used to output the multimodal spatial location probability distribution of the target monitoring space according to the input historical trajectory fragment and the time interval.

[0150] In one or more embodiments of this application, the preset sequence model refers to a pre-defined model structure suitable for processing sequence data. This model can take historical trajectory fragments and future time intervals as input and output a multimodal spatial location probability distribution of the target monitoring space. The specific structure of the preset sequence model can be a Transformer encoder, a recurrent neural network with an attention mechanism, or a temporal convolutional network, etc., and this embodiment does not specifically limit it.

[0151] Cross-entropy loss is a commonly used classification loss function used to measure the difference between the probability distribution output by the model and the true label. During training, for each training sample, historical trajectory fragments and future time intervals are input into a pre-defined sequence model, which outputs a multimodal spatial location probability distribution. Then, using the actual spatial location corresponding to the label of the training sample as the supervision signal, the negative log-likelihood of the output unit corresponding to that actual spatial location in the multimodal spatial location probability distribution is calculated as the cross-entropy loss for that training sample, without imposing additional penalty terms on other output units. Finally, the model parameters are updated based on the cross-entropy losses of multiple training samples until the convergence condition is met, resulting in a well-trained long-term location prediction model specific to the target monitoring space.

[0152] In one example, a Transformer encoder with a probability output module is used as the pre-defined sequence model. After inputting historical trajectory fragments and future time intervals, the model outputs the probability distribution of each spatial location within the target monitoring space. During training, the negative log-likelihood is calculated only for the output unit corresponding to the actual spatial location of the label; no additional penalty is imposed on other output units. After multiple iterations, the model converges, resulting in a long-term location prediction model specific to the target monitoring space.

[0153] It should be noted that the motion priors in existing technologies either come from manually preset rules and parameterized functions, or from motion models trained on general datasets. These are not learned from the deployed target monitoring space and therefore cannot capture the unique pedestrian motion patterns of that space. This embodiment uses only the unlabeled historical trajectories accumulated within the target monitoring space itself to train the model, without relying on manually set thresholds, functional forms, or general motion assumptions.

[0154] As described above, the multi-target tracking method based on scene-specific long-term prediction provided in this application training method trains a model using only unlabeled historical trajectories accumulated within the target monitoring space itself. It then selects long trajectories with a duration not less than a first duration threshold, samples and extracts historical trajectory segments at preset sampling intervals, and pairs each historical trajectory segment with multiple future time intervals not less than a second duration threshold to form training samples. Using the actual spatial location corresponding to the future time interval as the label, a long-term position prediction model specific to the target monitoring space is trained. Thus, the model can learn the unique long-term movement patterns of pedestrians within the target monitoring space, rather than the general movement patterns on general datasets. This enables reliable multimodal prediction of the target's future position on a timescale of tens of seconds, solving the problem that existing general prediction models cannot capture the unique long-term behavioral patterns of the target monitoring space, achieving deployment-adaptation and zero manual parameter tuning. This model constitutes a long-term spatiotemporal constraint model specific to the target monitoring space, forming the basis for long-term probability prediction and trajectory association in subsequent steps 100 and 200.

[0155] To further address the issues of unclear training processes, difficulty in model convergence, or unstable training results in existing trajectory prediction models, step 080 of the multi-target tracking method based on scene-specific long-term prediction provided in this application embodiment further includes the following: The training steps for the current iteration round are executed, wherein the training steps include: inputting historical trajectory fragments and future time intervals from the training samples into the sequence model to obtain the multimodal spatial location probability distribution of the target monitoring space output by the sequence model; using the actual spatial location corresponding to the label of the training sample as the supervision signal, calculating the negative log-likelihood of the output unit corresponding to the actual spatial location in the multimodal spatial location probability distribution as the cross-entropy loss of the training sample, and not imposing additional penalty terms on other output units in the multimodal spatial location probability distribution; and updating the parameters of the sequence model based on the cross-entropy loss of the training samples in the current iteration round. If the preset convergence condition is not met at present, the training steps of the next iteration round are executed; If the preset convergence condition is met, a long-term location prediction model specific to the target monitoring space is obtained based on the parameters of the latest updated sequence model.

[0156] In one or more embodiments of this application, the current iteration round refers to the training round currently being executed during model training. In each training round, several training samples are selected from the training sample set, and forward propagation, loss calculation, and parameter update are performed based on these training samples.

[0157] An output unit refers to an output node in the multimodal spatial location probability distribution output by the sequence model, which corresponds one-to-one with each spatial location within the target monitoring space. For example, when the multimodal spatial location probability distribution is represented in grid form, the output unit is the probability output node corresponding to each grid.

[0158] The negative log-likelihood is the value obtained by taking the natural logarithm of the output probability value of the output cell corresponding to the actual spatial location of the training sample, and then taking the opposite of that value. The smaller the negative log-likelihood, the higher the predicted probability of the model for that actual spatial location.

[0159] Cross-entropy loss is a loss function used in classification tasks to measure the difference between the probability distribution of the model's output and the true label. In this application, for a single training sample, the cross-entropy loss is equal to the negative log-likelihood of the output cell containing the actual spatial location corresponding to the label of that training sample.

[0160] In one example, the actual spatial location corresponding to the label of a training sample is located in the 120th grid, and the probability value of the 120th grid in the model output is 0.6. Therefore, the cross-entropy loss of this training sample is... If the probability value of the 120th grid in the model output is 0.9, then the cross-entropy loss is... A smaller loss indicates that the model's predictions for that training sample are more accurate.

[0161] In this embodiment, no additional penalty term is applied to other output units in the multimodal spatial location probability distribution. That is, only the negative log-likelihood is calculated for the output unit where the actual spatial location corresponding to the label is located, and the prediction bias of other output units is not directly included in the loss. This allows the model to focus on improving the probability corresponding to the true location while retaining the ability to represent other possible locations in a multimodal manner.

[0162] In one example, the multimodal spatial location probability distribution has 800 output units. If the actual spatial location corresponding to the label is located in the 120th output unit, then only the negative log-likelihood is calculated for the 120th output unit, and no additional penalty is imposed on the remaining 799 output units.

[0163] In one or more embodiments of this application, the preset convergence condition is a pre-set condition used to determine whether the model training has reached a state where it can be terminated. The preset convergence condition may include: the number of iterations reaches a preset maximum number of iterations, or the change in loss between two adjacent iterations is less than a preset threshold, or the metrics on the validation set no longer show significant improvement, etc.

[0164] In one example, the preset maximum number of iterations is 20 rounds. When the current iteration round is less than 20 rounds, the preset convergence condition is not met, and the training steps for the next iteration round are executed.

[0165] In one or more embodiments of this application, when a preset convergence condition is met, the parameters of the sequence model obtained after the latest parameter update are used as the final model parameters to obtain a long-term location prediction model specific to the target monitoring space that has been trained.

[0166] In one example, the sequence model is trained using the Adam optimizer with an initial learning rate of 0.0001. When the preset convergence condition is met after 20 iterations, the model parameters updated in 20th iteration are used as the final parameters to obtain a long-term location prediction model specific to the target monitoring space. This model is used to output a multimodal spatial location probability distribution of the target monitoring space based on the input historical trajectory fragments and time intervals.

[0167] As can be seen from the above description, the multi-target tracking method based on scene-specific long-term prediction provided in this application clarifies the complete process of forward propagation, cross-entropy loss calculation, parameter update and convergence judgment in iterative training, and uses the negative log-likelihood of the output unit where the actual spatial location of the label is located as the cross-entropy loss, without imposing additional penalties on other output units. This enables the sequence model to focus on improving the probability corresponding to the real location while retaining the multimodal expressive ability. The training process is clear and easy to converge, thereby stably obtaining a target monitoring space-specific long-term location prediction model.

[0168] To further address the issues of unclear internal structure and inability to map long-historical trajectories to multimodal probability distributions in existing trajectory prediction models, this application provides a multi-target tracking method based on scene-specific long-term prediction. The long-term position prediction model includes a sequence encoding module and a probability output module; detailed explanation follows: The sequence encoding module includes a Transformer encoder, a recurrent neural network with an attention mechanism, a temporal convolutional network, or a neural ODE network, used to encode the historical trajectory fragment and the time interval as a whole input, and output a contextual representation of the target's long-term motion pattern within the first duration threshold.

[0169] In one or more embodiments of this application, the sequence encoding module refers to the module in the long-term location prediction model responsible for encoding the input historical trajectory segments and time intervals. The sequence encoding module arranges the spatial locations in the historical trajectory segments in chronological order and incorporates the time interval information into the encoding process, outputting a fixed-dimensional context representation.

[0170] Contextual representation refers to the feature vector obtained by the sequence coding module after encoding the input historical trajectory segments and time intervals as a whole. This feature vector is used to represent the long-term motion pattern of the target within a first time threshold, including long-term behavioral information such as the path the target has taken, its stopping position, and changes in direction within the target monitoring space.

[0171] The Transformer encoder is a sequence coding structure based on a self-attention mechanism. It directly establishes the association between any two time steps in a historical trajectory segment through self-attention, without needing to iterate sequentially. Therefore, it can globally model long-term dependencies in a complete historical trajectory segment. For example, a Transformer encoder can consist of six self-attention layers and a feedforward network. The spatial coordinates of each historical time step are converted into a 64-dimensional vector through a learnable embedding layer, and a sinusoidal position code is added to represent the temporal order. The time interval is also converted into a 32-dimensional vector through a learnable embedding layer and concatenated with the vectors of each time step in the historical sequence to form the input sequence. The self-attention mechanism calculates the correlation weights between all time steps, outputting a 256-dimensional context vector.

[0172] Recurrent Neural Networks with Introduced Attention Mechanisms refer to structures that incorporate attention mechanisms into Recurrent Neural Networks. Recurrent Neural Networks can be Long Short-Term Memory (LSTM) networks or Gated Recurrent Units (GRUs). The attention mechanism is used to weighted aggregate the hidden states at different time steps in historical trajectory segments, thereby making more effective use of long-term historical information.

[0173] Temporal convolutional networks refer to sequence modeling structures based on one-dimensional convolution operations. They increase the receptive field through multiple layers of convolution and dilated convolution, thereby modeling variable-length historical sequences.

[0174] Neural ODE networks are continuous dynamic models that combine Ordinary Differential Equations (ODEs) with neural networks. They treat the process of updating hidden states as a process of solving differential equations in continuous time and can perform global modeling of variable-length historical sequences.

[0175] In one example, the sequence encoding module employs a transformer encoder. The input consists of a historical trajectory segment spanning the past 30 seconds and a future time interval of 20 seconds. The historical trajectory segment contains the spatial coordinates of 30 sampling time points. Each coordinate is converted into a 64-dimensional vector through an embedding layer. After incorporating sinusoidal position encoding, this vector is concatenated with a 32-dimensional vector obtained from the time interval embedding to form the input sequence. After processing through six layers of self-attention layers and a feedforward network, a 256-dimensional contextual representation is output. This contextual representation integrates information from all time steps in the historical trajectory segment, rather than being derived recursively from the hidden state of the last time step.

[0176] It should be noted that existing trajectory prediction models are mostly based on the Markov property assumption, and are derived only from the current state or a fixed-dimensional hidden state. Historical trajectory information is compressed and cannot express the long-term dependencies of the target on a scale of tens of seconds. In contrast, the sequence encoding module in this embodiment directly inputs complete historical trajectory segments within a first duration threshold, and captures long-term behavioral features of pedestrians such as turning back, detouring, and taking elevators through global modeling, thus exhibiting non-Markov property.

[0177] The probability output module is used to map the context representation to multiple grids into which the target monitoring space is divided, and output the probability of the target appearing in each grid at the second time point as a discrete multimodal spatial location probability distribution; or it is used to map the context representation to a multimodal spatial location probability distribution in the form of a Gaussian mixture distribution or a heatmap.

[0178] In one or more embodiments of this application, the probability output module refers to the module in the long-term location prediction model that is responsible for mapping the context representation output by the sequence coding module to the multimodal spatial location probability distribution of the target monitoring space.

[0179] A Gaussian mixture distribution (GJD) represents a multimodal spatial location probability distribution using a weighted combination of multiple Gaussian distribution components. Each Gaussian component is described by three parameters: mean, covariance, and weight. The mean represents the center of the high-probability region corresponding to that component, the covariance represents the spatial extent of that region, and the weight represents the proportion of that component in the overall distribution. The probability output module outputs these component parameters, thus obtaining a continuous multimodal spatial location probability distribution.

[0180] A heatmap format involves dividing a two-dimensional plane of the target monitoring space into a high-resolution grid and outputting the probability density value of each grid location, representing the multimodal spatial location probability distribution as a two-dimensional heatmap. The resolution of a heatmap is typically higher than that of a discrete grid format, allowing for a more refined representation of the target's future location distribution.

[0181] In one example, the probability output module divides the target monitoring space into a 0.5m × 0.5m grid, totaling approximately 800 grid cells. The probability output module contains a fully connected layer that receives the 256-dimensional context representation output by the sequence encoding module and outputs 800-dimensional probability values. After normalization using a normalized exponential function (softmax function), the probability value for each grid cell is obtained, representing a discrete multimodal spatial location probability distribution. For example, 20 seconds after the end of a customer's historical trajectory segment, the model outputs a 50% probability for a grid cell near the checkout area, a 20% probability for a grid cell in the main aisle, a 10% probability for a grid cell in shelf area D, and lower probabilities for the remaining grid cells, forming a multimodal spatial location probability distribution.

[0182] In another example, the probability output module outputs parameters for several Gaussian mixture components. For instance, it outputs the mean, covariance, and weights of three Gaussian components, corresponding to the checkout area, main aisle, and shelf area D, with weights of 0.5, 0.2, and 0.1, respectively. The covariance matrix reflects the spatial range of each candidate area, thus forming a continuous multimodal spatial location probability distribution.

[0183] In another example, the probability output module outputs the multimodal spatial location probability distribution in the form of a heatmap, that is, it outputs the probability density value of each location on the high-resolution grid within the target monitoring space. The areas with higher probability density in the heatmap correspond to multiple locations where the target may appear in the future.

[0184] In other embodiments, when the probability output module divides the target monitoring space into a grid, the grid side length can be any value between 0.25 meters and 1 meter. For example, the grid side length can be 0.5 meters. The grid side length should not be less than 0.25 meters because an excessively fine grid will result in too many output units, increasing the computational burden and training difficulty of the model; the grid side length should not be greater than 1 meter because an excessively coarse grid will reduce spatial resolution and make it impossible to effectively distinguish the fine positional differences of the target in space.

[0185] As can be seen from the above description, the multi-target tracking method based on scene-specific long-term prediction provided in this application divides the long-term position prediction model into a sequence encoding module and a probability output module. The sequence encoding module encodes the complete historical trajectory segments and time intervals as a whole to obtain a context representation. Then, the probability output module maps the context representation into a discrete or continuous multimodal spatial position probability distribution. This makes the internal structure of the model clear and the data flow clear. Different structural configurations can stably map historical trajectory segments into multimodal spatial position probability distributions, support the prediction of the probability of multiple future positions of the target, and improve the robustness of long-term prediction.

[0186] In other embodiments, the position coordinates in the historical trajectory segment can be represented in any of the following forms: unified scene coordinates from multiple cameras, planar coordinates from a single camera image, or a triple representation of floor index plus planar coordinates. For multi-floor scenes, the triple representation of floor index plus planar coordinates can distinguish spatial locations within different floors.

[0187] To further address the issues of unreliability of single-point prediction or short-term distance constraints in existing technologies at long-term scales and the inability of probability distributions to be converted into matching degrees, this application provides a multi-target tracking method based on scene-specific long-term prediction. In this method, the multimodal spatial location probability distribution is a discrete probability distribution, which includes multiple spatial locations and probability values ​​corresponding to each spatial location. Each spatial location corresponds to a preset spatial region. Different first time points and different second time points constitute different candidate point pairs; wherein, the first trajectory segment has a continuous historical trajectory before the first time point with a duration not less than the first duration threshold.

[0188] Correspondingly, see Figure 4 Step 300 in the multi-target tracking method based on scene-specific long-term prediction specifically includes the following: Step 310: For each candidate point pair, perform a matching degree determination step; wherein the matching degree determination step includes: arranging the plurality of spatial locations in descending order of probability value, and taking the first K spatial locations after sorting as candidate locations; determining whether the actual spatial location of the second time point in the candidate point pair falls into a preset spatial region corresponding to any of the candidate locations; if it falls into the preset spatial region corresponding to any of the candidate locations, then using the probability value of the candidate location in the multimodal spatial location probability distribution as the association matching degree corresponding to the candidate point pair; if it does not fall into the preset spatial region corresponding to any of the candidate locations, then using zero as the association matching degree corresponding to the candidate point pair; wherein K is a preset positive integer.

[0189] In one or more embodiments of this application, a discrete probability distribution refers to a probability distribution defined over a finite number of spatial locations, where each spatial location corresponds to a probability value, and the sum of the probability values ​​for all spatial locations is 1. Discrete probability distributions have been described in the foregoing embodiments and will not be repeated here.

[0190] A candidate point pair refers to a set of time points consisting of a first time point and a second time point. Different candidate point pairs have different first and second time points. A candidate point pair satisfies the following conditions: the first trajectory segment has a continuous historical trajectory prior to the first time point with a duration not less than a first duration threshold, and the time interval between the first time point and the second time point is not less than a second duration threshold.

[0191] Candidate positions refer to the K spatial positions selected from multiple spatial positions after they are sorted in descending order of probability value. K is a preset positive integer used to limit the number of candidate positions participating in the matching degree determination, so that the association judgment is concentrated on the few spatial positions with the highest probability, and the interference of low probability positions on the matching degree is avoided.

[0192] A preset spatial area refers to the spatial range corresponding to each spatial location. In a discrete probability distribution, each spatial location represents a region within the target monitoring space, and the preset spatial area is the area covered by that spatial location. For example, when the target monitoring space is divided into a 0.5m × 0.5m grid, each spatial location corresponds to one grid, and the preset spatial area is the coverage area of ​​that grid.

[0193] In this embodiment, a matching degree determination step is performed for each candidate point pair. Specifically, the matching degree determination step involves: first, arranging multiple spatial locations in descending order of probability value, and selecting the top K spatial locations as candidate locations; then, determining whether the actual spatial location of the candidate point pair at the second time point falls within a preset spatial region corresponding to any candidate location. If it falls within a preset spatial region corresponding to a candidate location, the probability value of that candidate location in the multimodal spatial location probability distribution is used as the association matching degree corresponding to the candidate point pair; if it does not fall within any preset spatial region corresponding to a candidate location, zero is used as the association matching degree corresponding to the candidate point pair.

[0194] In one example, the multimodal spatial location probability distribution includes 800 spatial locations, each corresponding to a 0.5m × 0.5m grid. K is set to 10. The 800 spatial locations are sorted in descending order of probability value, and the top 10 spatial locations after sorting are selected as candidate locations. If the actual spatial location of the second time point in a candidate point pair falls within the grid corresponding to the third spatial location in the candidate points, and the probability value of this spatial location is 0.6, then the association matching degree corresponding to this candidate point pair is 0.6.

[0195] In another example, if the actual spatial location of the second time point in a candidate point pair does not fall within the preset spatial region corresponding to any of the first 10 candidate locations, then the correlation matching degree of the candidate point pair is zero.

[0196] In another example, among the top 10 candidate locations after sorting, the preset spatial regions of 3 candidate locations overlap, and the actual spatial location at the second time point falls within one of the overlapping regions. Therefore, as long as the actual spatial location falls within the preset spatial region corresponding to any candidate location, the probability value of that candidate location in the multimodal spatial location probability distribution is used as the association matching degree corresponding to that candidate point pair.

[0197] In other embodiments, K can be any positive integer from 5 to 20. For example, K can be 10. K should not be less than 5 because if there are too few candidate positions, the target's actual position may not fall into a few candidate regions, resulting in missed matches; K should not be greater than 20 because if there are too many candidate positions, the participation of low-probability candidate positions in the matching degree determination will dilute the distinguishability of the associated matching degree and increase the computational overhead.

[0198] As described above, the multi-target tracking method based on scene-specific long-term prediction provided in this application transforms the multimodal probability distribution into a single, comparable correlation matching degree by arranging multiple spatial locations in the multimodal spatial location probability distribution in descending order of probability value and selecting the top K as candidate locations. The correlation matching degree is then determined by whether the actual spatial location at the second time point falls within a preset spatial region corresponding to the candidate location. Compared to existing technologies that rely on single-point prediction or short-term distance constraints, this embodiment uses multiple high-probability candidate locations in the probability distribution as the basis for judgment, directly incorporating the uncertainty of long-term prediction into the correlation matching degree, thereby improving the reliability and robustness of correlation judgment on a timescale of tens of seconds.

[0199] It is particularly important to note that this embodiment differs fundamentally from the Joint Probabilistic Data Association (JPDA) method in classic Bayesian data association. The JPDA method calculates association probabilities on a second-level timescale, using a unimodal Gaussian assumption and Markov state recursion, and its effectiveness depends on the measurement error being much smaller than the target distance. In contrast, the association criteria in this embodiment simultaneously satisfy all of the following conditions: (a) the time interval is not less than a second duration threshold determined by error statistics or objective laws, falling within the tens of seconds range; (b) the multimodal spatial position probability distribution upon which the criterion is based is conditional on complete historical trajectory segments, carrying long-term motion information of the target within the target monitoring space; (c) the multimodal spatial position probability distribution is in a discrete multimodal form, and the preset spatial regions corresponding to candidate positions do not need to constitute mutually exclusive spatial divisions; the spatial distance between candidate positions can reach tens of meters, far exceeding the measurement error of a multi-camera system. The combination of these three factors makes the association judgment robust across orders of magnitude to sensor measurement errors, something that short-time Bayesian association frameworks cannot achieve in principle.

[0200] To further address the problems in existing technologies where single-set matching degrees are unreliable due to local observation errors, and where matching degrees cannot be converted into binary judgment results, this application provides a multi-target tracking method based on scene-specific long-term prediction, see [link to relevant documentation]. Figure 4 Step 300 in the multi-target tracking method based on scene-specific long-term prediction further includes the following: Step 320: Statistically aggregate the correlation matching degree corresponding to each candidate point pair, and use the aggregation result as the target correlation matching degree.

[0201] In one or more embodiments of this application, statistical aggregation refers to combining the correlation matching degrees of multiple candidate points to obtain a representative summary value. Statistical aggregation methods may include averaging, weighted averaging, median, or minimum values, etc., and this embodiment does not specifically limit these methods.

[0202] The target correlation matching degree refers to the aggregated value obtained by statistically aggregating the correlation matching degrees of each candidate point pair. This target correlation matching degree is used to represent the overall correlation degree between the first trajectory segment and the second trajectory segment. Compared with the correlation matching degree of a single candidate point pair, it can reduce the fluctuations caused by the influence of local observation errors on individual candidate point pairs and improve the stability of correlation judgment.

[0203] In one example, three candidate point pairs were selected, with corresponding association matching scores of 0.6, 0.5, and 0.4, respectively. Taking the average of the three association matching scores as the statistical aggregation result, the target association matching score is (0.6 + 0.5 + 0.4) / 3 = 0.5.

[0204] In another example, three candidate point pairs were selected, with corresponding association matching degrees of 0.7, 0.2, and 0.3, respectively. The median of the three association matching degrees was taken as the statistical aggregation result, resulting in a target association matching degree of 0.3. This avoids the excessive influence of individual abnormally high or low values ​​on the aggregation result.

[0205] Step 330: Compare the target association matching degree with a preset score threshold. When the target association matching degree is not less than the preset score threshold, obtain a determination result that the first trajectory segment and the second trajectory segment belong to the same target. When the target association matching degree is less than the preset score threshold, obtain a determination result that the first trajectory segment and the second trajectory segment do not belong to the same target.

[0206] In one or more embodiments of this application, the preset score threshold refers to a pre-set minimum score used to determine whether the target association matching degree reaches the level that can be judged as the same target. The preset score threshold can be determined before deployment based on the association effect on the validation set, or it can be adjusted according to the emphasis on purity and completeness based on the business scenario.

[0207] The determination result refers to the binary judgment made on whether the first trajectory segment and the second trajectory segment belong to the same target. The determination result includes two possibilities: they belong to the same target, or they do not belong to the same target.

[0208] In one example, the preset score threshold is 0.5. If the target association matching degree obtained in step 320 is 0.5, which is not less than the preset score threshold of 0.5, then the determination result is that the first trajectory segment and the second trajectory segment belong to the same target.

[0209] In another example, the preset score threshold is 0.5. The target association matching degree obtained in step 320 is 0.3, which is less than the preset score threshold of 0.5. Therefore, the result is that the first trajectory segment and the second trajectory segment do not belong to the same target.

[0210] Step 340: Generate and output a multi-target tracking result containing the determination result, the target association matching degree, and trajectory data composed of the first trajectory segment and the second trajectory segment.

[0211] In one or more embodiments of this application, trajectory data refers to a trajectory record formed by associating a first trajectory segment and a second trajectory segment. When the determination result is that they belong to the same target, the trajectory data is the trajectory formed by merging the first trajectory segment and the second trajectory segment; when the determination result is that they do not belong to the same target, the trajectory data can retain the original trajectory information of the first trajectory segment and the second trajectory segment respectively, and attach a determination identifier that the two do not belong to the same target.

[0212] The multi-target tracking result refers to the final output of step 300, which indicates whether the first trajectory segment and the second trajectory segment belong to the same target. This result includes three parts: the judgment result, the target association matching degree, and the trajectory data, so that downstream applications can not only know the judgment conclusion, but also obtain the judgment basis and the corresponding trajectory data.

[0213] In one example, if the determination result is that they belong to the same target, the target association matching degree is 0.5, and the trajectory data is the complete trajectory formed by merging the first trajectory segment and the second trajectory segment, then the above determination result, the target association matching degree, and the merged trajectory data are output as the multi-target tracking result.

[0214] In another example, if the determination result is that they do not belong to the same target, the target association matching degree is 0.3, and the trajectory data keep the first trajectory segment and the second trajectory segment independent, then the above determination result, the target association matching degree and the two independent trajectory data are output as the multi-target tracking result.

[0215] As can be seen from the above description, the multi-target tracking method based on scene-specific long-term prediction provided in this application obtains the target association matching degree by statistically aggregating the association matching degree of multiple candidate point pairs, and obtains the judgment result by comparing the target association matching degree with a preset score threshold. At the same time, the judgment result, the target association matching degree and trajectory data are output together, which can reduce the impact of fluctuations in multiple sets of association matching degrees, make the association judgment result more stable, and the output result fully carries the judgment basis and trajectory data, which is convenient for downstream applications to trace and utilize the tracking results.

[0216] In other embodiments, the probability output module can also output the multimodal spatial location probability distribution in a non-parametric manner. This non-parametric method includes: retrieving several historical trajectories from a historical trajectory library composed of unlabeled historical trajectories whose similarity to the current historical trajectory segment exceeds a preset similarity threshold; obtaining the actual spatial locations of these historical trajectories at a second time point; and statistically forming an empirical distribution as the multimodal spatial location probability distribution. This method does not involve context representation mapping and is a non-parametric implementation of long-term location prediction models.

[0217] In other embodiments, when the multimodal spatial location probability distribution is a continuous distribution, the method for calculating the association matching degree can be replaced by: taking the actual spatial location at the second time point as a condition, calculating the probability density of the multimodal spatial location probability distribution, and using the obtained probability density value as the association matching degree; when the multimodal spatial location probability distribution is in the form of a discrete grid, the method for calculating the association matching degree can be replaced by: using the sum of the probabilities of the grid where the actual spatial location at the second time point is located and its neighboring grids as the association matching degree.

[0218] In other embodiments, the statistical aggregation of multiple candidate point pairs can be achieved by taking any one of the following: average, weighted average, median, or minimum. The weighted average can be normalized and weighted according to the probability values ​​corresponding to each candidate point pair.

[0219] In other embodiments, the association decision can be replaced by: using the association matching degree corresponding to each candidate point pair as the weight of the corresponding edge in the graph clustering, inputting it into the graph clustering algorithm to complete the global association of multiple trajectory segments, so as to obtain the multi-target tracking result.

[0220] In other embodiments, the association decision can also be made in a cascade manner: first, the candidate associations are coarsely screened based on the target association matching degree, then the apparent feature distance of the candidate associations that have passed the coarse screening is calculated for fine ranking, and the multi-target tracking result is obtained based on the fine ranking result.

[0221] To further address the shortcomings of existing technologies that rely solely on long-term probability matching, which suffers from insufficient reliability in cases of apparent similarity or probabilistic ambiguity, and introduce noise through forced fusion at low probabilities, this application provides a multi-target tracking method based on scene-specific long-term prediction. (See also...) Figure 4 Step 300 in the multi-target tracking method based on scene-specific long-term prediction further includes the following content performed between steps 320 and 330: Step 321: Obtain the appearance features of the first trajectory segment and the second trajectory segment respectively; wherein, the appearance features are appearance features extracted based on the image information of the detection boxes in the first trajectory segment and the second trajectory segment.

[0222] In one or more embodiments of this application, appearance features refer to feature vectors extracted from the image regions corresponding to each detection box in a trajectory segment, used to describe pedestrian appearance information. Appearance features can be extracted based on a Person Re-identification (ReID) model, or based on traditional image features such as color histograms and histograms of oriented gradients. Appearance features are used to measure the degree of visual similarity between targets in two trajectory segments, complementing long-term probability matching.

[0223] In one example, a pre-trained person re-identification model is used to extract features from the image regions corresponding to each detection box in the first and second trajectory segments, resulting in a 256-dimensional apparent feature vector for each detection box. The average of the apparent feature vectors of all detection boxes in the first trajectory segment is taken as the apparent feature of the first trajectory segment, and the average of the apparent feature vectors of all detection boxes in the second trajectory segment is taken as the apparent feature of the second trajectory segment.

[0224] In another example, color histograms are used as appearance features. The color histograms of the image regions of each detection box in the first and second trajectory segments are calculated separately, and then the average of the color histograms of the detection boxes is taken to obtain the appearance features of the first and second trajectory segments respectively.

[0225] Step 322: Compare the current target association matching degree with a preset coarse screening threshold; wherein the preset coarse screening threshold is less than the preset score threshold; if the target association matching degree is not less than the preset coarse screening threshold, then proceed to step 323; In one or more embodiments of this application, the preset coarse screening threshold refers to a pre-set minimum threshold used to perform preliminary screening of the target association matching degree before appearance feature fusion. The preset coarse screening threshold is less than the preset score threshold, and its function is to exclude candidate point pairs with excessively low target association matching, and to avoid forcibly fusing with appearance feature distance when the long-term probability matching degree is already very low, thereby introducing noise into the appearance information.

[0226] In one example, the preset score threshold is 0.5, and the preset coarse screening threshold is 0.2. The current target association matching degree is 0.6, which is not less than the preset coarse screening threshold of 0.2, so proceed to step 323.

[0227] In another example, if the current target association matching degree is 0.1, which is less than the preset coarse screening threshold of 0.2, then step 323 is not executed, but step 331 is executed to directly obtain the judgment result that the first trajectory segment and the second trajectory segment do not belong to the same target.

[0228] Step 323: Calculate the apparent feature distance between the apparent features of the first trajectory segment and the second trajectory segment respectively; In one or more embodiments of this application, the apparent feature distance refers to a measure of the difference between the apparent features of a first trajectory segment and the apparent features of a second trajectory segment. The apparent feature distance can be the cosine distance, Euclidean distance, or Mahalanobis distance between two apparent feature vectors, etc. The smaller the apparent feature distance, the more similar the appearances of the targets in the two trajectory segments, and the higher the probability that they belong to the same target.

[0229] In one example, the apparent feature vector of the first trajectory segment is The apparent feature vector of the second trajectory segment is The apparent feature distance is the cosine distance, i.e., the apparent feature distance. .when and When the directions are completely consistent, the cosine distance is 0; when and As the directional difference increases, the cosine distance approaches 1.

[0230] In another example, the apparent feature distance is taken as the Euclidean distance, i.e., the apparent feature distance. The smaller the Euclidean distance, the closer the two apparent feature vectors are in the feature space.

[0231] In one example, the cosine distance between the apparent features of the first trajectory segment and the apparent features of the second trajectory segment is 0.3.

[0232] Step 324: Perform a weighted fusion of the target association matching degree and the apparent feature distance to obtain the fusion result.

[0233] In one or more embodiments of this application, weighted fusion refers to combining the target association matching degree and the apparent feature distance according to preset weights to obtain a fusion result that integrates the two aspects of information. The specific method of weighted fusion can be: fusion result Target correlation matching degree Apparent feature distance), where and To preset weights, The apparent feature distance is converted into apparent similarity after being subtracted by 1, and the degree of matching with the target remains consistent in both dimensions and direction.

[0234] In one example, the target association matching degree is 0.6, the apparent feature distance is 0.3, and the preset weights α and β are 0.7 and 0.3, respectively. The fusion result is... .

[0235] In another example, the target association matching degree is 0.4, the apparent feature distance is 0.8, and the preset weights α and β are 0.7 and 0.3, respectively. The fusion result is... .

[0236] Step 325: Update the target association matching degree using the fusion result. Then proceed to step 330.

[0237] In one or more embodiments of this application, updating the target association matching degree means using the fusion result obtained in step 324 as the new target association matching degree to replace the original target association matching degree obtained in step 320, for comparison with the preset score threshold in subsequent step 330.

[0238] In one example, the target association matching degree obtained in step 320 is 0.6, and the fusion result obtained after step 324 is 0.63. 0.63 is used as the updated target association matching degree, and then step 330 is executed.

[0239] Correspondingly, see Figure 4 Step 300 of the multi-target tracking method based on scene-specific long-term prediction further includes the following content executed after step 322 and before step 340: If, after step 322, it is determined that the target association matching degree is less than the preset coarse screening threshold, then step 331 is executed: directly obtaining the determination result that the first trajectory segment and the second trajectory segment do not belong to the same target. Then step 340 is executed.

[0240] In one or more embodiments of this application, when the target association matching degree is less than a preset coarse screening threshold, it indicates that, based solely on the long-term probability matching degree, the probability that the first trajectory segment and the second trajectory segment belong to the same target is already very low. At this point, the apparent feature distance is no longer calculated, nor is weighted fusion performed; the determination that the two do not belong to the same target is directly obtained. This saves computational overhead and avoids introducing noise by forcibly fusing excessively low matching degrees with apparent feature distances.

[0241] In one example, the target association matching degree obtained in step 320 is 0.1, and the preset coarse screening threshold is 0.2. Since 0.1 is less than 0.2, the determination result that the first trajectory segment and the second trajectory segment do not belong to the same target is directly obtained.

[0242] As can be seen from the above description, the multi-target tracking method based on scene-specific long-term prediction provided in this application pre-screens the target association matching degree with a preset coarse screening threshold before appearance feature fusion. Candidate point pairs with low long-term probability matching are directly determined to not belong to the same target. Only after the target association matching degree reaches the coarse screening threshold is the appearance feature distance calculated and weighted fusion performed to update the target association matching degree. This can avoid the introduction of noise by forced fusion when the probability is low. At the same time, when the long-term probability matching degree is reliable, the accuracy of the judgment is improved by combining appearance information. This makes the association judgment take into account both long-term motion information and appearance information, and improves the association robustness in real deployment scenarios.

[0243] To further address the issues of performance degradation of fixed models and the need for manual parameter readjustment in existing technologies, this application provides a multi-target tracking method based on scene-specific long-term prediction. (See also...) Figure 4 The multi-target tracking method based on scene-specific long-term prediction further includes the following content after step 300: Step 400: Continuously accumulate the trajectory data in the multi-target tracking results within the current update cycle.

[0244] In one or more embodiments of this application, the current update cycle refers to the time period from the completion of the last model update to the current time. The update cycle can be divided by a fixed time length, such as weekly or monthly; it can also be divided by the cumulative time when the data volume reaches a preset threshold; or it can be triggered by events such as scene layout changes. This embodiment does not limit the specific method of dividing the update cycle.

[0245] Continuous accumulation refers to the process of continuously collecting and storing trajectory data from each output of the multi-target tracking method as it runs, serving as the data basis for subsequent training updates. The trajectory data has already been described in step 340 and will not be repeated here.

[0246] In one example, the update cycle is weekly. The system starts accumulating trajectory data from multi-target tracking results at the beginning of the week and ends on the last Sunday, forming a complete set of trajectory data for the entire update cycle. A new update cycle begins the following Monday, and accumulation starts again.

[0247] In another example, the update cycle is not fixed. When the accumulated trajectory data reaches a preset data volume threshold, it is considered the end of an update cycle, and then the next update cycle begins.

[0248] Step 500: If the update triggering condition is met, then update and train the current long-term location prediction model specific to the target monitoring space to obtain the updated long-term location prediction model specific to the target monitoring space.

[0249] The update triggering conditions include at least one of the following: the trajectory data reaches a preset data volume threshold, the correlation confidence distribution drift detection result determined based on the correlation matching degree of multiple targets generated in the current update cycle exceeds a preset drift threshold, and a scene layout change event is detected in the target monitoring space.

[0250] In one or more embodiments of this application, the update triggering condition refers to a judgment condition used to determine whether to initiate the update training of a long-term location prediction model specific to the target monitoring space. The update triggering condition may include at least one of the following three: The first condition is that the trajectory data reaches a preset data volume threshold. The preset data volume threshold refers to a pre-defined minimum number of accumulated trajectory data points required for update training. For example, the preset data volume threshold could be 5000 trajectories. When the number of accumulated trajectory data points reaches 5000 in the current update cycle, the update trigger condition is met.

[0251] The second item is the detection result of the association confidence distribution drift, determined based on the association matching degrees of multiple targets generated within the current update cycle, exceeding a preset drift threshold. The association confidence distribution drift detection result refers to the detection result obtained by monitoring the distribution of the association matching degrees of multiple targets generated within the current update cycle and determining whether this distribution has changed significantly relative to the historical baseline distribution. The preset drift threshold is a pre-set comparison threshold used to determine whether the association confidence distribution has drifted significantly. When the drift detection result exceeds the preset drift threshold, it indicates that the pedestrian movement pattern or system error level within the target monitoring space may have changed, satisfying the update trigger condition.

[0252] In one example, the distribution of target association matching scores generated within the first month of deployment is used as the historical baseline distribution. During the current update cycle, the distributions of newly generated target association matching scores are compared with the historical baseline distribution, and a difference metric, such as Kullback-Leibler divergence, is calculated between the two distributions. When the difference metric exceeds a preset drift threshold, it is determined that the association confidence distribution has drifted significantly, triggering an update training.

[0253] The third item is the detection of a scene layout change event in the target monitoring space. A scene layout change event refers to an event in which the physical layout within the target monitoring space changes, such as the movement of shelves, closure of entrances, or alteration of passageways. Scene layout change events can be triggered by construction notices, manual reports, or the results of layout change analysis of monitoring video.

[0254] In one example, the target monitoring space is a retail store. When the store rearranges its shelves, staff enter a scene layout change event into the management system. Upon detecting this event, the system meets the update trigger conditions and initiates an update training.

[0255] In this embodiment, when any of the above-mentioned update triggering conditions are met, the current long-term location prediction model specific to the target monitoring space is updated and trained. Specific methods for update training may include incremental training, full retraining, sliding window retraining, or knowledge distillation. The training samples used for update training are constructed based on the trajectory data corresponding to the update training, using the same sample construction method as when training the long-term location prediction model specific to the target monitoring space.

[0256] In one or more embodiments of this application, the current long-term location prediction model refers to the target monitoring space-specific long-term location prediction model that is currently in use during system operation and is used for long-term probability prediction in step 200. The updated target monitoring space-specific long-term location prediction model refers to the model obtained after updated training and used to replace the current long-term location prediction model.

[0257] After training is completed, the updated model is evaluated on the validation set. If the association metric of the updated model on the validation set is not lower than that of the current long-term location prediction model, the updated model replaces the current long-term location prediction model; otherwise, the current model is retained.

[0258] As can be seen from the above description, the multi-target tracking method based on scene-specific long-term prediction provided in this application continuously accumulates trajectory data in the multi-target tracking results within the current update cycle, and updates and trains the target monitoring space-specific long-term position prediction model when the update trigger condition is met. This enables the model to be continuously updated as the pedestrian movement patterns and system error levels change within the target monitoring space, avoiding performance degradation of the fixed model after scene changes and eliminating the need for manual parameter retuning, thereby achieving continuous adaptive multi-target tracking.

[0259] It should be noted that online updates are not a necessary step in the embodiments of this application. Even without executing steps 400 and 500, the initially constructed long-term location prediction model specific to the target monitoring space can still provide tracking performance superior to the prior art after deployment; steps 400 and 500 are only used for further optimization and long-term performance maintenance.

[0260] To further address the issues of existing technologies having a single model update strategy and being unable to flexibly select update methods based on data scale and computing resources, this application provides a multi-target tracking method based on scene-specific long-term prediction. In this method, the update training can be any one of incremental training, full retraining, sliding window retraining, or knowledge distillation using the current long-term position prediction model as the teacher model. The training samples used for the update training are constructed based on the trajectory data corresponding to the update training, using the same sample construction method as when training the target monitoring space-specific long-term position prediction model. In the incremental training and knowledge distillation, the trajectory data corresponding to the updated training is the trajectory data accumulated in the current update cycle; in the full retraining, the trajectory data corresponding to the updated training is all the trajectory data accumulated in the target monitoring space; in the sliding window retraining, the trajectory data corresponding to the updated training is the trajectory data accumulated in multiple update cycles.

[0261] In one or more embodiments of this application, incremental training refers to training the model with a small number of iterations using only the newly added training samples corresponding to the trajectory data accumulated in the current update cycle, with the weights of the current long-term location prediction model as the initial values. The number of iterations in incremental training is usually less than that in the initial training, and the learning rate is also appropriately reduced to avoid catastrophic forgetting of existing knowledge.

[0262] In one example, the learning rate for incremental training is half of the initial training learning rate, i.e., 0.00005, and the number of iterations is 5. Using the weights of the current long-term location prediction model as initial values, a new training sample set is constructed using the trajectory data accumulated in the current update cycle for training, resulting in the updated model.

[0263] Full retraining refers to reconstructing the training sample set using all accumulated trajectory data in the target monitoring space, starting from random initialization or preset initial weights, and then training the model completely. Full retraining is suitable for situations where the accumulated trajectory data volume is large and the scene patterns have changed significantly.

[0264] In one example, the total number of trajectory data accumulated in the target monitoring space is 100,000. After filtering out trajectories with a duration of not less than the first duration threshold, a training sample set is constructed in the same way as the initial training. The preset sequence model is then fully trained for 20 rounds to obtain the model after full retraining.

[0265] Sliding window retraining refers to reconstructing the training sample set using only the trajectory data accumulated over the most recent update cycles to fully train the model. By limiting the time range of the training data, sliding window retraining avoids outdated data interfering with the model's fitting of current scene patterns.

[0266] In one example, the sliding window is the last four update cycles. A training sample set is constructed using the trajectory data accumulated over the last four update cycles, and the model is fully trained to obtain the model after sliding window retraining.

[0267] Knowledge distillation refers to using the current long-term location prediction model as the teacher model and constructing a training sample set using trajectory data accumulated in the current update cycle to train a new student model. The student model acquires the teacher model's knowledge by learning the output distribution of the teacher model on the training samples, while adapting to changes in scene patterns in the new data.

[0268] In one example, the current long-term location prediction model is used as the teacher model. The multimodal spatial location probability distribution output by the teacher model for each training sample is used as the soft label. The distillation loss is constructed by combining the true labels of the training samples to train a student model with the same structure or a lighter one. After training, the student model replaces the current long-term location prediction model.

[0269] As described above, the multi-target tracking method based on scene-specific long-term prediction provided in this application offers four selectable update training methods: incremental training, full retraining, sliding window retraining, and knowledge distillation. Each method specifies a corresponding trajectory data range, allowing for flexible selection of model update strategies based on the data scale of the target monitoring space, the degree of scene change, and computational resources. All four update methods prepare training data using the same sample construction method, ensuring consistency in data format and feature distribution between the updated training and initial training. This maintains model consistency and stability while continuously adapting to scene changes.

[0270] It should be noted that the core of the multi-target tracking method based on scene-specific long-term prediction provided in this application is: determining the long-term judgment scale based on error statistics or objective laws, replacing the manually set short-term threshold judgment with scene-specific long-term probability prediction, and extending the time scale of association judgment from the second level to the tens of seconds level, so as to maintain high robustness in real deployment scenarios where multi-camera systems have meter-level spatial positioning errors and second-level time synchronization errors.

[0271] To further address the problem that the duration threshold and signal-to-noise ratio parameters in existing technologies are arbitrarily set manually and cannot adapt to different deployment scenarios, this application provides a multi-target tracking method based on scene-specific long-term prediction. In this method, the first duration threshold ranges from 30 seconds to 120 seconds, the second duration threshold ranges from 10 seconds to 120 seconds, and the first duration threshold is not less than the second duration threshold; the preset multiple ranges from 5 times to 20 times.

[0272] In one or more embodiments of this application, the first duration threshold ranges from 30 seconds to 120 seconds. The first duration threshold should not be less than 30 seconds because when the historical trajectory segment is shorter than 30 seconds, the historical information input to the model is often insufficient to cover a typical movement process of a pedestrian within the target monitoring space, making it difficult to effectively capture the pedestrian's long-term movement pattern. The first duration threshold should also not be greater than 120 seconds because when the historical trajectory segment exceeds 120 seconds, it requires the first trajectory segment to have a very long continuous history before the first time point, resulting in a large number of trajectory segments being unable to participate in the association judgment due to insufficient historical duration. At the same time, excessively long historical trajectory segments also increase the computational burden on the model.

[0273] In a preferred embodiment, the first duration threshold can be set to 20 to 40 seconds, the second duration threshold can be set to 20 to 60 seconds, and the preset multiplier can be selected as 10 times.

[0274] The second duration threshold ranges from 10 to 120 seconds. The second duration threshold should not be less than 10 seconds because even under favorable deployment conditions, a certain time interval is still needed to ensure that the spatial distribution of different targets grows sufficiently to provide a basic signal-to-noise ratio for long-term association judgment. The second duration threshold should also not exceed 120 seconds because association time intervals exceeding 120 seconds are too long, significantly reducing the number of temporally matching trajectory segments and affecting the association coverage.

[0275] The first duration threshold is not less than the second duration threshold. This ensures that when selecting candidate point pairs, the time interval is not less than the second duration threshold, and historical trajectory segments with a duration not less than the first duration threshold can also be extracted before the first time point, so that subsequent model input and probability prediction steps can be executed normally.

[0276] The preset multiplier ranges from 5 to 20 times. The preset multiplier should not be less than 5 times because if it is too small, the ratio of the spatial distribution of different targets to the system's spatiotemporal measurement error will be too low, resulting in insufficient signal-to-noise ratio for association judgment and an increased false association rate. The preset multiplier should also not be greater than 20 times because if it is too large, the second duration threshold required to meet the signal-to-noise ratio requirements will become too long, thus reducing the coverage and practicality of the association.

[0277] In one example, for a retail store with an area of ​​approximately 600 square meters, a first time duration threshold of 30 seconds and a second time duration threshold of 20 seconds are set, with a preset multiple of 10. Under this parameter configuration, the typical location dispersion of different customers over a 20-second time span is approximately 15 meters, and the system's spatiotemporal measurement error is approximately 1.5 meters. The ratio of these two values ​​is 10, which satisfies the condition of not being less than the preset multiple.

[0278] In another example, for a larger indoor space with more complex passageways, a first duration threshold of 120 seconds and a second duration threshold of 60 seconds are set, with a preset multiplier of 10. Under this parameter configuration, the typical dispersion of different target locations over a 60-second time span is approximately 50 meters, which is far greater than 10 times the system's spatiotemporal measurement error.

[0279] In another example, for a scenario where a high-precision time synchronization device is deployed, the first duration threshold is set to 30 seconds, the second duration threshold is set to 10 seconds, and the preset multiple is 5. Under this parameter configuration, the typical dispersion of the positions of different targets over a 10-second time span is approximately 8 meters, the system's spatiotemporal measurement error is approximately 1.5 meters, and the ratio of the two is approximately 5.3, which meets the condition of not being less than the preset multiple.

[0280] In another example, for scenarios with extremely strict requirements on association misjudgment, the preset multiplier can be increased to 20 to ensure the signal-to-noise ratio in a more conservative way. At this time, the second duration threshold needs to be increased accordingly so that the location dispersion of different targets reaches more than 20 times the spatiotemporal measurement error of the system.

[0281] As can be seen from the above description, the multi-target tracking method based on scene-specific long-term prediction provided in this application embodiment, by limiting the value range of the first duration threshold, the second duration threshold and the preset multiple, and clarifying the constraint relationship that the first duration threshold is not less than the second duration threshold, can ensure that the parameter configuration of the method falls within a reasonable range under different deployment scenarios. This avoids both insufficient signal-to-noise ratio due to excessively small parameter values ​​and reduced associated coverage due to excessively large parameter values, thereby improving the feasibility and scene adaptability of the method.

[0282] To further illustrate the above, this application also provides a specific application example of a multi-target tracking method based on scene-specific long-term prediction. Taking a retail store of approximately 600 square meters as the deployment scenario, the target monitoring space is the approximately 600 square meter retail store, covered by 37 cameras. The goal is to track the complete walking trajectory of customers within the store for purposes such as movement analysis and dwell area identification. See also... Figure 5 This demonstrates the complete process from setting the long-term prediction time interval, constructing a long-term location prediction model specific to the target monitoring space, long-term probability prediction, and correlation to online updates.

[0283] 1. First stage: Determine the duration interval of the long-term forecast. This stage defines two time parameters: the second duration threshold, which is the minimum time interval used for long-term prediction and association; and the first duration threshold, which is the duration of the historical trajectory input to the model.

[0284] In a default approach, no specific error measurement is performed on the store; the duration threshold is directly determined based on objective laws. Based on the fact that the historical spatial location distribution of different targets increases superlinearly with time intervals (i.e., when the time interval is doubled, the typical distribution value increases approximately fourfold), and considering the system's spatiotemporal measurement error does not accumulate with time intervals, as well as the upper limit of error magnitude under typical commercial deployment conditions (spatial positioning error is in the meter range, lens distortion area can reach 3 to 4 meters, and time synchronization error is in the second range, generally not exceeding 2 to 3 seconds), the second duration threshold is directly set to 20 seconds. The typical historical spatial location distribution of different customers within this time span can be roughly estimated at approximately 60 meters from historical trajectories. Precise statistics are unnecessary; even using the most conservative estimate compared to the 4-meter error upper limit, the ratio is still not less than the preset multiple of 10, and the system operates within this multiple. The first duration threshold is set to 30 seconds. Based on the store structure (i.e., the distance between the entrance, shelves, and cashier) and the average customer walking speed, a 30-second trajectory can cover one typical movement process and is greater than the second duration threshold.

[0285] In another alternative approach, the lower limit of the second duration threshold is compressed through actual error measurement. The store later deployed a high-precision time synchronization service and manually calibrated parameters for lenses with significant distortion. The measurement method involved uniformly selecting reference points with known precise locations on the ground, repeatedly measuring and statistically analyzing spatial positioning errors using a multi-camera system, and injecting synchronization timestamps into each video stream to statistically analyze the time differences of the same physical event. The measured typical value of the spatial positioning error was reduced to 0.8 meters, and the typical value of the time synchronization error was reduced to 0.3 seconds. While maintaining a preset signal-to-noise ratio of 10, the lower limit of the second duration threshold was lowered from 20 seconds to approximately 10 seconds, allowing moderate-duration occlusions of 10 to 20 seconds to be included in the long-term probability association range, further improving trajectory integrity.

[0286] 2. Second stage: Constructing a long-term location prediction model specific to the target monitoring space. See Figure 6 This diagram illustrates the sub-processes for building a long-term location prediction model specific to the target monitoring space. It specifically demonstrates the complete data processing chain from acquiring historical videos, generating unlabeled historical trajectories, filtering long trajectories, sampling at preset sampling intervals, extracting historical trajectory fragments, constructing a training sample set, to training and obtaining the long-term location prediction model.

[0287] We acquired historical surveillance video from the store over the past four consecutive weeks. Using multi-object tracking algorithms, such as Simple Online and Realtime Tracking with a Deep AssociationMetric (DeepSORT), we processed the video to obtain approximately 100,000 customer trajectory segments. These trajectory segments may contain noise, breaks, and incorrect associations, but no manual correction is required. Due to the large amount of data, common customer walking patterns within the store, such as from the entrance to the shelves, from the shelves to the checkout, and loitering in specific areas, can be covered by a large number of trajectories. These trajectory segments constitute the unlabeled historical trajectories.

[0288] Trajectories shorter than 30 seconds were removed from all unlabeled historical trajectories, retaining only those longer than or equal to 30 seconds. This resulted in approximately 20,000 long trajectories, which were used as source material for constructing a long-term location prediction model specific to the target monitoring space.

[0289] For each sampling time point t on each long trajectory, the sampling interval is 1 second, and a complete historical trajectory segment from the previous 30 seconds is taken. For each historical trajectory segment Select multiple future time intervals that are not less than the second duration threshold. The corresponding future actual location is obtained at 20 seconds, 30 seconds, and 60 seconds respectively. All samples constitute the training sample set, and each sample can be represented as... .

[0290] For example, if a customer walks a distance between 10:00:00 and 10:00:30, this 30-second trajectory is taken as the input of historical trajectory segments; then, their position at 10:00:50, 10:01:00, and 10:01:30 is predicted. In this way, approximately 5 million training samples are automatically generated from 20,000 long trajectories.

[0291] A long-term position prediction model is constructed using an architecture of a transformer encoder (such as a Transformer encoder) and a probabilistic output module. The transformer encoder consists of six layers of self-attention layers and a feedforward network. The position coordinates at each historical time step, i.e., two-dimensional scene coordinates, are converted into 64-dimensional vectors through a learnable embedding layer, and a sinusoidal position code is added to represent the temporal order. Future time intervals... The input sequence is also transformed into a 32-dimensional vector through a learnable embedding layer and concatenated with the vector of each time step in the historical sequence. The self-attention mechanism calculates the correlation weights between all time steps and outputs a 256-dimensional context vector that integrates information from all historical time steps.

[0292] The probability output module divides the store space into a 0.5m x 0.5m grid, totaling approximately 800 grid cells. The module contains a fully connected layer that receives a 256-dimensional context vector and outputs 800-dimensional unnormalized probability values. These values ​​are then normalized using a normalized exponential function (softmax function) to obtain the probability value for each grid cell.

[0293] During training, the actual future position will be used. The data is mapped to the corresponding grid indices and optimized using the cross-entropy loss function. Training uses the Adam optimizer with a learning rate of 0.0001, a batch size of 64, and converges after 20 iterations.

[0294] Once trained, the model becomes a long-term location prediction model specific to the target monitoring space of the store. Given a historical trajectory segment from the past 30 seconds and future time intervals, this long-term location prediction model specific to the target monitoring space can output a multimodal spatial location probability distribution of where customers may appear in the store at different times in the future.

[0295] 3. Third stage: Long-term probability prediction and correlation based on a target monitoring space-specific long-term location prediction model. See Figure 7This is a schematic diagram of the long-term probability prediction and association sub-process based on the target monitoring space-specific long-term location prediction model. It specifically shows the complete execution logic from selecting candidate point pairs, extracting historical trajectory fragments and inputting them into the long-term location prediction model, obtaining the multimodal spatial location probability distribution, taking the Top-K candidate positions, calculating the association matching degree, to the aggregation decision of multiple candidate point pairs.

[0296] During real-time tracking, when it is necessary to determine whether two trajectory segments belong to the same target, let the first trajectory segment be... The second trajectory segment is Select a first time point 'a' on the first trajectory segment and a second time point 'b' on the second trajectory segment, such that the time interval... The duration is not less than 20 seconds from the second duration threshold, and there is a continuous historical trajectory of at least 30 seconds before the first time point a.

[0297] Extract the complete historical trajectory segment 30 seconds prior to the first time point 'a' from the first trajectory segment, and input it along with Δt into a long-term location prediction model (also known as a spatiotemporal constraint model) specific to the target monitoring space (or simply the target space). The long-term location prediction model specific to the target monitoring space outputs the probability distribution of the customer's possible location in each grid position of the store after Δt seconds. Select the K=10 grid positions with the highest probabilities as candidate positions, thus obtaining the Top-K candidate positions and their probabilities.

[0298] This application example uses mutually exclusive mesh partitioning, where candidate meshes do not overlap. In other implementations, the spatial regions corresponding to candidate positions can overlap, for example, by using a receptive region centered on the mesh center and larger than the mesh itself, or by directly using a continuous probability distribution, such as a Gaussian mixture distribution, in the output format.

[0299] Check whether the actual spatial location of the second time point b in the second trajectory segment falls within the preset spatial region corresponding to these 10 candidate locations. If it falls within the preset spatial region corresponding to one of the candidate locations, the probability corresponding to that candidate location is taken as the association matching degree of the candidate point pair (which can be simply referred to as the matching degree); if it does not fall within the preset spatial region corresponding to any candidate location, the association matching degree is 0.

[0300] Multiple candidate point pairs that meet the conditions are selected on the first and second trajectory segments, for example, three pairs. The above query is repeated to obtain multiple correlation matching scores. In this application example, the average of these correlation matching scores is taken as the target correlation matching score. If the target correlation matching score is greater than or equal to the preset score threshold (which can be simply referred to as the threshold) of 0.5, the two trajectory segments are determined to belong to the same target; otherwise, they are determined to be different targets.

[0301] See Figure 8 This is a schematic diagram of trajectory prediction and spatial probability distribution. Figure 8 There are five shelves: Shelf A, Shelf B, Shelf C, Shelf D, and Shelf E. Solid arrows represent known walking trajectories. The first time point 'a' (hereinafter referred to as point a) on the first trajectory segment is located on the solid trajectory before shelf B. Before point a, there is a complete historical trajectory of 30 consecutive seconds, i.e., the solid arrow portion from the entrance starting point to point a. The second time point on the second trajectory segment is point b, and the time interval between point a and point b is 20 seconds. Dashed arrows and percentage values ​​(such as 50%, 20%, 10%, etc., are illustrative examples) show the multiple candidate locations where the target may appear at time b, as output by the long-term location prediction model, and their corresponding probabilities. This includes candidate locations in the checkout area and their probabilities, reflecting the multimodal uncertainty of long-term prediction and scene-specific behavioral patterns. The long-term location prediction model of this application does not predict the frame-by-frame walking path from point a to each candidate location. Instead, it takes the complete historical trajectory 30 seconds before point a and the future time interval of 20 seconds as input, and directly predicts multiple candidate locations and their probabilities that the target may appear 20 seconds after point a. It outputs a multimodal spatial location probability distribution instead of a single predicted location, while avoiding the accumulation of errors in frame-by-frame recursive prediction. Figure 8 The scenario shown is merely an exemplary illustration for the purpose of facilitating understanding of the technical solution of this application.

[0302] 4. Experimental Results In the comparative experiment in a retail store scenario, purity represents the proportion of correctly matched targets in the tracked output trajectory, while completeness represents the proportion of the actual target trajectory that is correctly tracked. The method provided in this application example is compared with the existing multi-target tracking method DeepSORT on test data from two different time periods.

[0303] Table 1 shows the measured results of all data in December 2025: The purity of the method provided by the application example of this application is 98.94%, which is only 0.18 percentage points lower than DeepSORT's 99.12%, and basically the same; the integrity is 84.96%, which is 10.16 percentage points higher than 74.80%.

[0304] Table 1 Table 2 shows the measured results for some data in June 2026: The completeness of the method provided in the application example of this application is 87.16%, an improvement of 12.12 percentage points from 75.04%; the purity is 91.73%, a decrease of 4.32 percentage points from 96.05%. In the test data during this period, long-term occlusion was more concentrated. While long-term prediction matching significantly improved trajectory completeness, it introduced a small number of error mergings, resulting in a slight decrease in purity. However, the improvement in completeness far outweighed the decrease in purity, resulting in significant overall gains.

[0305] Table 2 5. Cross-time testing To examine the impact of the timing of the training data on the results, a long-term location prediction model specific to the target monitoring space was trained using historical trajectories from December 2025 and June 2026, respectively, and cross-tested on test data from the two time periods. See Table 3 for the experimental results of the method provided in the application example of this application under different training time data for the target store in December 2025, and Table 4 for the experimental results of the method provided in the application example of this application under different training time data for the target store in June 2026.

[0306] Table 3 Table 4 As shown in Tables 3 and 4, the purity of the two training data sets in December 2025 was 98.94% and 98.95%, respectively, and the completeness was 84.96% and 84.86%, respectively. In June 2026, the purity of the two training data sets was 91.73% and 91.38%, respectively, and the completeness was 87.16% and 87.41%, respectively. The differences in indicators across all cross-combinations did not exceed 0.35 percentage points. This indicates that the spatiotemporal constraints of pedestrian traffic at the store remain stable over time, and the model's performance does not depend on a strict temporal correspondence between the training and testing data. The model trained using historical data from any month can consistently achieve better results than existing methods, with a completeness improvement of over 10 percentage points.

[0307] 6. Fourth stage: Online update stage (optional) After the system is officially launched, new trajectory data generated daily will be continuously accumulated. To avoid excessive data volume, only long trajectory segments that meet preset duration conditions will be retained, and training samples will be constructed from these new trajectories periodically, such as once a week.

[0308] Each weekend, a training sample set is constructed for the newly added long-trajectory data from the week, using the same method as the aforementioned training sample construction. The weights of the currently deployed target monitoring space-specific long-term location prediction model are used as initial values, and incremental training is performed using the new samples. During incremental training, only a small number of iterations are performed, such as 5 iterations, and the learning rate can be appropriately reduced, for example, to 0.00005, to avoid catastrophic forgetting of existing knowledge.

[0309] After incremental training is completed, the association accuracy of the updated model is evaluated on the historical validation set. If the purity and completeness of the updated model are not lower than the corresponding metrics of the current model, it is replaced with the online model; otherwise, the current model is retained and awaits the next cycle for further updates. This ensures continuous model evolution while preventing performance regression due to data fluctuations.

[0310] In this application example, referring to Table 5, which compares the effects of different operating modes, the performance of two operating modes was compared: Mode 1 involves keeping the initial model unchanged without any online updates; Mode 2 involves incremental updates once a week. Based on four consecutive weeks of test data, Mode 1 achieved a purity of 99.25% and a completeness of 84.5%; Mode 2 achieved a purity of 99.28% and a completeness of 85.2%. It is evident that even without online updates, the initial model already achieves significantly better results than existing technologies, which achieve a completeness of 74.8%. Online updates, while maintaining a relatively constant purity, further improve the completeness by approximately 0.7 percentage points.

[0311] Table 5 It should be noted that, as shown in Tables 3 and 4 of the cross-time test, the model performance remains stable over time, and online updates are not a necessary condition for achieving beneficial results; online updates are only an optional optimization method used to further maintain and optimize performance when there are significant changes in scene layout or pedestrian flow patterns.

[0312] As can be seen from the above specific application examples, the multi-target tracking method based on scene-specific long-term prediction provided in this application takes a retail store with an area of ​​about 600 square meters and covered by 37 cameras as the target monitoring space. In real deployment scenarios, the spatial positioning error can reach the meter level and the time synchronization error can reach the second level, resulting in insufficient signal-to-noise ratio for short-term association judgment. By determining a long-term time scale of tens of seconds through error statistics or objective laws, the time interval of association judgment is located on a time scale where the spatial locations of different targets are significantly larger than the spatiotemporal measurement error of the system. Structurally, this ensures that the signal-to-noise ratio of association judgment is not less than a preset multiple.

[0313] This method utilizes only the store's accumulated unlabeled historical trajectories to train a long-term location prediction model specific to the target monitoring space. The model takes complete historical trajectory segments and future time intervals as input, directly establishing a location mapping from the historical trajectory segments to the second time point. This eliminates the need for progressively predicting intermediate time points, avoiding error accumulation in frame-by-frame recursive prediction, and outputs a multimodal spatial location probability distribution carrying the store's unique long-term pedestrian movement patterns. During online tracking, the probability value of the actual spatial location at the second time point in the probability distribution is used as the association matching degree. The target association matching degree is obtained by aggregating multiple candidate point pairs, and then a determination is made as to whether they belong to the same target. This allows the uncertainty of long-term prediction to be directly incorporated into the association confidence, without relying on a single prediction point or short-term distance constraints.

[0314] In real-world testing at retail stores, the method described in this application achieved a completeness of 84.96% on data from December 2025, a 10.16 percentage point improvement over the existing multi-object tracking method DeepSORT's 74.80%; and a completeness of 87.16% on data from June 2026, a 12.12 percentage point improvement over DeepSORT's 75.04%. Cross-time testing showed that the differences in various metrics between models trained using historical data from different months did not exceed 0.35 percentage points, indicating that the model's performance did not depend on a strict temporal correspondence between training and testing data. During the online update phase, by continuously accumulating new trajectory data and incrementally training the model, the completeness can be slightly improved while maintaining a relatively constant purity. Furthermore, online updates are not a necessary condition for achieving beneficial results; they are only an optional optimization method used to further maintain and optimize performance.

[0315] In summary, this application example further verifies that the multi-target tracking method based on scene-specific long-term prediction provided in this application embodiment can reconstruct the signal-to-noise ratio required for association judgment through scene-specific long-term probability prediction in real deployment scenarios where multi-camera systems have meter-level spatial positioning errors and second-level time synchronization errors. This improves the association reliability and identity stability in scenarios with long-term occlusion or trajectory breakage, reduces trajectory fragmentation and identity switching, and achieves deployment-based adaptation, zero manual parameter tuning, and continuous adaptive optimization.

[0316] From a software perspective, this application also provides a multi-target tracking apparatus for performing all or part of the scene-specific long-term prediction-based multi-target tracking method described above. See [link to relevant documentation]. Figure 9 The multi-target tracking device based on scene-specific long-term prediction specifically includes the following components: The candidate point selection module 10 is used to select the first time point from the first trajectory segment and the second time point from the second trajectory segment, based on the following conditions: the duration of the historical trajectory of the first trajectory segment collected from the target monitoring space before the first time point is not less than the first duration threshold corresponding to the target monitoring space, and the time interval between the first time point and the second time point in the second trajectory segment collected from the target monitoring space is not less than the second duration threshold. The first duration threshold and the second duration threshold are predetermined based on the ratio of historical spatial location dispersion to system spatiotemporal measurement error being not less than a preset multiple. The system spatiotemporal measurement error is the composite value of the spatial positioning error and time synchronization error of the multi-camera system covering the target monitoring space. The historical spatial location dispersion is the arithmetic mean of the spatial location differences between different targets in the target monitoring space under the candidate time span used to determine the first duration threshold and the second duration threshold.

[0317] The probability prediction module 20 is used to extract historical trajectory segments from the first trajectory segment that have a duration of not less than the first duration threshold before the first time point, and input the historical trajectory segments and the time interval into a long-term position prediction model specific to the target monitoring space, so that the model outputs a multimodal spatial position probability distribution of the target monitoring space corresponding to the second time point; wherein, the long-term position prediction model is pre-trained based on the unlabeled historical trajectories accumulated by the target monitoring space itself.

[0318] The association decision module 30 is used to take the probability value of the actual spatial position of the second time point in the second trajectory segment in the multimodal spatial position probability distribution as the association matching degree, and obtain a multi-target tracking result based on the association matching degree to indicate whether the first trajectory segment and the second trajectory segment belong to the same target.

[0319] The embodiments of the multi-target tracking device based on scene-specific long-term prediction provided in this application can be used to execute the processing flow of the embodiments of the multi-target tracking method based on scene-specific long-term prediction in the above embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the multi-target tracking method based on scene-specific long-term prediction.

[0320] The scene-specific long-term prediction-based multi-target tracking device can perform the scene-specific long-term prediction-based multi-target tracking portion in either a server or a client device. The choice can be made based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application does not impose any limitations in this regard. If all operations are performed in the client device, the client device may further include a processor for the specific processing of scene-specific long-term prediction-based multi-target tracking.

[0321] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0322] The server and the client device can communicate using any suitable network protocol, including those not yet developed as of the date of this application. Such network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Furthermore, such network protocols may also include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer Protocol) protocols used on top of the aforementioned protocols.

[0323] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the scene-specific long-term prediction-based multi-target tracking method mentioned in the above embodiments. The processor and memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and memory via wired or wireless means.

[0324] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0325] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the scene-specific long-term prediction-based multi-target tracking method in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the scene-specific long-term prediction-based multi-target tracking method in the above method embodiments.

[0326] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0327] The one or more modules are stored in the memory, and when executed by the processor, they execute the multi-target tracking method based on scene-specific long-term prediction in the embodiment.

[0328] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0329] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0330] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0331] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned multi-target tracking method based on scene-specific long-term prediction. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0332] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned multi-target tracking method based on scene-specific long-term prediction.

[0333] It should be noted that the multi-target tracking method based on scene-specific long-term prediction described in this application involves the processing of pedestrian trajectory data from public place surveillance videos during its research, development, and deployment. The relevant data processing strictly complies with the requirements of relevant laws and regulations concerning personal information protection and data security. Specific compliance measures are as follows: The method described in this application aims to maintain public safety and is applicable to scenarios involving passenger flow analysis and security monitoring in public places such as shopping malls, transportation hubs, and industrial parks. The trajectory data processed originates from legally installed public place monitoring equipment, and the data processing is strictly limited to the scope necessary for maintaining public safety.

[0334] For deployment scenarios not intended for public safety purposes, such as business operations analysis, implementers must ensure that they have obtained the data subject's consent in accordance with the law before processing personal information, and inform the data subject of the purpose, method, type of information, and retention period in a prominent manner and in clear and understandable language.

[0335] The pedestrian trajectory data used in this method is legally obtained through one of the following methods: Firstly, in public place surveillance scenarios, prominent warning signs have been set up in the monitored areas, and the collected personal images and trajectory information are only used for the purpose of maintaining public safety; Secondly, in non-public safety scenarios, the purpose, method, and scope of the collection of personal information have been clearly communicated to the data subject before collection, and the data subject's consent has been obtained; Third, the data used comes from legally permitted datasets that have been legally made public, and the personal data has been anonymized as necessary during the use of the data, with the anonymized data stored separately from the information that can be used to reconstruct and identify the individual.

[0336] This method only processes the spatial trajectory coordinate sequence of pedestrians and does not involve the identification or analysis of biometric information such as faces. The use of trajectory data is limited to the model training, validation, and inference purposes of the scene-specific long-term prediction-based multi-target tracking method described in this application, and is not used for any other purpose. All trajectory data is encrypted during transmission and stored in encrypted form to ensure the security of data transmission and storage. The trajectory data used for model training comes from monitoring locations that have obtained legal authorization from the data subject or meet the legal processing conditions. Implementers should ensure the legality of data collection based on the specific deployment scenario. After data processing, personal information is deleted or anonymized to ensure that the processed information cannot identify a specific individual and is irreversible.

[0337] Implementers shall take measures including but not limited to the following to protect data security: encrypt the transmission and storage of trajectory data, establish a data access permission management system, authorize only necessary personnel to access the data, record data access logs, and regularly evaluate and strengthen data security measures.

[0338] The above privacy compliance guidelines apply to various deployment scenarios of the scene-specific long-term prediction-based multi-target tracking method described in this application. Implementers should develop and implement corresponding privacy compliance plans based on the local laws and regulations of the actual deployment scenario, the installation nature of the monitoring equipment, and the specific methods of data processing. This guideline does not constitute legal advice, and implementers should consult professional legal counsel before deployment to ensure full compliance.

[0339] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0340] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0341] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0342] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multi-target tracking method based on scene-specific long-term prediction, characterized in that, include: The method is based on the following conditions: the duration of the historical trajectory of the first trajectory segment collected from the target monitoring space before the first time point is not less than the first duration threshold corresponding to the target monitoring space, and the time interval between the first time point and the second time point in the second trajectory segment collected from the target monitoring space is not less than the second duration threshold. The method then selects the first time point from the first trajectory segment and the second time point from the second trajectory segment. The first and second duration thresholds are predetermined based on the ratio of historical spatial location dispersion to system spatiotemporal measurement error being not less than a preset multiple. The system spatiotemporal measurement error is the composite value of the spatial positioning error and time synchronization error of the multi-camera system covering the target monitoring space. The historical spatial location dispersion is the arithmetic mean of the spatial location differences between different targets within the target monitoring space under the candidate time span used to determine the first and second duration thresholds. Historical trajectory segments with a duration of not less than the first duration threshold before the first time point are extracted from the first trajectory segment. The historical trajectory segments and the time interval are input into a long-term location prediction model specific to the target monitoring space, so that the model outputs a multimodal spatial location probability distribution of the target monitoring space corresponding to the second time point. The long-term location prediction model is pre-trained based on unlabeled historical trajectories accumulated by the target monitoring space itself. The probability value of the actual spatial position of the second time point in the second trajectory segment in the multimodal spatial position probability distribution is used as the association matching degree. Based on the association matching degree, a multi-target tracking result is obtained to indicate whether the first trajectory segment and the second trajectory segment belong to the same target.

2. The multi-target tracking method based on scene-specific long-term prediction according to claim 1, characterized in that, Before selecting the first time point from the first trajectory segment and the second time point from the second trajectory segment, the method further includes: Obtain statistical values ​​of spatial positioning error and time synchronization error of the multi-camera system within the target monitoring space; Multiply the statistical value of the time synchronization error by the preset pedestrian walking speed to obtain the equivalent spatial error; The larger value between the statistical value of the spatial positioning error and the equivalent spatial error is taken as the system spatiotemporal measurement error; Within each candidate time span in the preset candidate time span set, the historical spatial location distribution of different targets within the target monitoring space is statistically analyzed. From each of the candidate time spans, the shortest time span in which the ratio of the historical spatial location distribution to the system spatiotemporal measurement error is not less than the preset multiple is selected as the second duration threshold. The first duration threshold is defined as the shortest duration of a typical pedestrian movement process that is not less than the second duration threshold and covers the target monitoring space. The first duration threshold is not less than the second duration threshold.

3. The multi-target tracking method based on scene-specific long-term prediction according to claim 1, characterized in that, Before selecting the first time point from the first trajectory segment and the second time point from the second trajectory segment, the method further includes: The upper bound of the time synchronization error of the multi-camera system in the target monitoring space is multiplied by the preset pedestrian walking speed to obtain the upper bound of the equivalent spatial error. The larger of the preset upper bound of the spatial positioning error of the multi-camera system within the target monitoring space and the upper bound of the equivalent spatial error is taken as the spatiotemporal measurement error of the system. Based on the principle that the historical spatial location distribution of different targets grows superlinearly with time intervals, while the spatiotemporal measurement error of the system does not accumulate with time intervals, the shortest time span with a ratio of the historical spatial location distribution to the spatiotemporal measurement error of the system not less than the preset multiple is selected from each candidate time span in the preset candidate time span set as the second duration threshold. The first duration threshold is defined as the shortest duration of a typical pedestrian movement process that is not less than the second duration threshold and covers the target monitoring space. The first duration threshold is not less than the second duration threshold.

4. The multi-target tracking method based on scene-specific long-term prediction according to claim 1, characterized in that, Before selecting the first time point from the first trajectory segment and the second time point from the second trajectory segment, the method further includes: Obtain the unlabeled historical trajectory of the target monitoring space; wherein, the unlabeled historical trajectory is obtained by processing the historical video data of the target monitoring space based on a multi-target tracking algorithm; Select trajectories from the unlabeled historical trajectories whose duration is not less than the first duration threshold; Each trajectory with a duration not less than the first duration threshold after filtering is sampled at a preset sampling interval to obtain multiple sampling time points; For each sampling time point, extract historical trajectory segments that occurred before that sampling time point and whose duration is not less than the first duration threshold; For each historical trajectory segment, a training sample is formed by combining each of a plurality of future time intervals not less than the second duration threshold with the historical trajectory segment, and the actual spatial location corresponding to the future time interval is used as the label of the training sample, thereby obtaining a training sample set composed of each training sample with the label respectively. Based on the training sample set, a preset sequence model is trained using the cross-entropy loss function to obtain a long-term location prediction model specific to the target monitoring space; wherein, the long-term location prediction model is used to output the multimodal spatial location probability distribution of the target monitoring space according to the input historical trajectory fragment and the time interval.

5. The multi-target tracking method based on scene-specific long-term prediction according to claim 4, characterized in that, The step of training a preset sequence model using the cross-entropy loss function based on the training sample set to obtain a long-term location prediction model specific to the target monitoring space includes: The training steps for the current iteration round are executed, wherein the training steps include: inputting historical trajectory fragments and future time intervals from the training samples into the sequence model to obtain the multimodal spatial location probability distribution of the target monitoring space output by the sequence model; using the actual spatial location corresponding to the label of the training sample as the supervision signal, calculating the negative log-likelihood of the output unit corresponding to the actual spatial location in the multimodal spatial location probability distribution as the cross-entropy loss of the training sample, and not imposing additional penalty terms on other output units in the multimodal spatial location probability distribution; and updating the parameters of the sequence model based on the cross-entropy loss of the training samples in the current iteration round. If the preset convergence condition is not met at present, the training steps of the next iteration round are executed; If the preset convergence condition is met, a long-term location prediction model specific to the target monitoring space is obtained based on the parameters of the latest updated sequence model.

6. The multi-target tracking method based on scene-specific long-term prediction according to claim 1, characterized in that, The long-term location prediction model includes a sequence encoding module and a probability output module; The sequence encoding module includes a Transformer encoder, a recurrent neural network with an attention mechanism, a temporal convolutional network, or a neural ODE network, used to encode the historical trajectory fragment and the time interval as a whole input, and output a contextual representation of the target's long-term motion pattern within the first duration threshold. The probability output module is used to map the context representation to multiple grids into which the target monitoring space is divided, and output the probability of the target appearing in each grid at the second time point as a discrete multimodal spatial location probability distribution; or it is used to map the context representation to a multimodal spatial location probability distribution in the form of a Gaussian mixture distribution or a heatmap.

7. The multi-target tracking method based on scene-specific long-term prediction according to claim 1, characterized in that, The multimodal spatial location probability distribution is a discrete probability distribution, which includes multiple spatial locations and a probability value corresponding to each spatial location. Each spatial location corresponds to a preset spatial region. Different first time points and different second time points constitute different candidate point pairs; wherein, the first trajectory segment has a continuous historical trajectory before the first time point with a duration not less than the first duration threshold; The step of using the probability value of the actual spatial location of the second time point in the second trajectory segment in the multimodal spatial location probability distribution as the association matching degree includes: For each candidate point pair, perform a matching degree determination step separately; The matching degree determination step includes: The multiple spatial locations are sorted in descending order of probability value, and the top K spatial locations after sorting are taken as candidate locations. Determine whether the actual spatial location of the second time point in the candidate point pair falls into a preset spatial region corresponding to any of the candidate locations; If it falls into the range, the probability value of the candidate position in the multimodal spatial position probability distribution is used as the association matching degree of the candidate point pair. If the candidate point does not fall into any of the preset spatial regions corresponding to the candidate positions, then zero is used as the correlation matching degree corresponding to the candidate point pair; where K is a preset positive integer.

8. The multi-target tracking method based on scene-specific long-term prediction according to claim 7, characterized in that, The process of obtaining a multi-target tracking result based on the correlation matching degree to indicate whether the first trajectory segment and the second trajectory segment belong to the same target includes: The correlation matching degree corresponding to each candidate point is statistically aggregated, and the aggregation result is used as the target correlation matching degree. The target association matching degree is compared with a preset score threshold. When the target association matching degree is not less than the preset score threshold, the first trajectory segment and the second trajectory segment are determined to belong to the same target. When the target association matching degree is less than the preset score threshold, the first trajectory segment and the second trajectory segment are determined to not belong to the same target. Generate and output a multi-target tracking result that includes the determination result, the target association matching degree, and trajectory data composed of the first trajectory segment and the second trajectory segment.

9. The multi-target tracking method based on scene-specific long-term prediction according to claim 8, characterized in that, Before comparing the target correlation matching degree with the preset score threshold, the method further includes: Obtain the appearance features of the first trajectory segment and the second trajectory segment respectively; wherein, the appearance features are appearance features extracted based on the image information of the detection boxes in the first trajectory segment and the second trajectory segment; The current target association matching degree is compared with a preset coarse screening threshold; wherein the preset coarse screening threshold is less than the preset score threshold; If the target association matching degree is not less than the preset coarse screening threshold, then calculate the apparent feature distance between the apparent features of the first trajectory segment and the second trajectory segment respectively; The target association matching degree and the apparent feature distance are weighted and fused to obtain the fusion result; The target association matching degree is updated using the fusion result; The method further includes: If the target association matching degree is less than the preset coarse screening threshold, the determination result that the first trajectory segment and the second trajectory segment do not belong to the same target is directly obtained.

10. The multi-target tracking method based on scene-specific long-term prediction according to claim 8, characterized in that, Also includes: The trajectory data in the multi-target tracking results within the current update cycle is continuously accumulated; If the update triggering conditions are met, the current long-term location prediction model specific to the target monitoring space is updated and trained to obtain the updated long-term location prediction model specific to the target monitoring space. The update triggering conditions include at least one of the following: the trajectory data reaches a preset data volume threshold, the drift detection result of the association confidence distribution determined by multiple target association matching degrees generated in the current update cycle exceeds a preset drift threshold, and a scene layout change event is detected in the target monitoring space.

11. The multi-target tracking method based on scene-specific long-term prediction according to claim 10, characterized in that, The update training can be any one of incremental training, full retraining, sliding window retraining, or knowledge distillation using the current long-term location prediction model as the teacher model; wherein, the training samples used in the update training are constructed based on the trajectory data corresponding to the update training, using the same sample construction method as when training the long-term location prediction model specific to the target monitoring space. In the incremental training and knowledge distillation, the trajectory data corresponding to the updated training is the trajectory data accumulated in the current update cycle; in the full retraining, the trajectory data corresponding to the updated training is all the trajectory data accumulated in the target monitoring space; in the sliding window retraining, the trajectory data corresponding to the updated training is the trajectory data accumulated in multiple update cycles.

12. The multi-target tracking method based on scene-specific long-term prediction according to claim 1, characterized in that, The first duration threshold ranges from 30 seconds to 120 seconds, the second duration threshold ranges from 10 seconds to 120 seconds, and the first duration threshold is not less than the second duration threshold; The preset multiple ranges from 5 to 20 times.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-target tracking method based on scene-specific long-term prediction as described in any one of claims 1 to 12.

Images