Multi-object tracking

Abstract

Disclosed in the present description are a multi-object tracking method and apparatus, a storage medium and an electronic device. The multi-object tracking method provided by the present description comprises: on the basis of environment data collected by a sensor, determining candidate positions of each object at a current moment; for each object, determining subordinate distances from a common position to the trajectory prediction positions of a plurality of objects corresponding to the common position; on the basis of the determined subordinate distances, determining weighting of the association degrees between the common position and the plurality of objects; and by means of the weighting of the association degrees between the common position and the objects, re-determining an association probability. When the estimated positions of the objects are determined, on the basis of the subordinate distances from the common position to the trajectory prediction positions of the plurality of objects corresponding to the common position, the association probability is weighted, such that in dense scenarios having a large number of objects, the candidate positions of the plurality of objects at similar distances can be distinguished on the basis of the subordinate distances, thus determining more accurate object trajectories.

Description

Multi-target tracking Technical Field

[0001] This specification relates to computer technology, and in particular to methods, devices, storage media, and electronic devices for multi-target tracking. Background Art

[0002] Target tracking technology can locate a target and generate its trajectory for motion analysis. Currently, target tracking is widely used in scenarios such as autonomous driving, intelligent transportation systems, and intelligent video surveillance.

[0003] When tracking multiple targets, it is necessary to obtain multiple candidate locations of the targets to be tracked based on the data collected by the sensors at the current moment, then associate each candidate location with each target, determine the candidate location of each target respectively, and determine the trajectory of each target at the current moment based on the candidate location corresponding to each target.

[0004] In densely populated scenes with many targets to be tracked, it is very likely that a candidate location will be associated with multiple targets at the same time. This can lead to incorrect correspondence between candidate locations and targets, or multiple targets corresponding to a single candidate location, resulting in inaccurate trajectories for each target.

[0005] Summary of the Invention

[0006] This specification provides a method, device, storage medium, and electronic device for multi-target tracking to at least partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions:

[0008] This specification provides a method for multi-target tracking, including: obtaining environmental data collected by a sensor at the current moment for a monitored scene, and determining, based on the environmental data, candidate positions of each target in the scene at the current moment; for each target, determining, based on the estimated position of the target at the previous moment, a predicted trajectory position of the target at the current moment, and determining an association probability between each candidate position and the target; based on the predicted trajectory position of each target at the current moment, determining one or more common positions from each candidate position, wherein each common position corresponds to multiple targets; and for each common position, determining, respectively, the predicted trajectory positions of multiple subordinate targets corresponding to the common position. The subordinate distance of the common position is used to determine the association weights of the common position with the multiple subordinate targets based on the determined subordinate distances; for each of the multiple subordinate targets corresponding to the common position, the association probability between the common position and the subordinate target is weighted by using the association weight between the common position and the subordinate target, and the association probability between the common position and the subordinate target is re-determined; for each of the targets, the estimated position of the target at the current moment is determined based on the association probability between the target and the candidate positions; for each of the targets, the trajectory of the target at the current moment is determined based on the estimated position of the target at the current moment and the historical trajectory of the target.

[0009] Optionally, based on the predicted trajectory position of each target at the current moment, one or more common positions are determined from the candidate positions, specifically including: determining the candidate distance at the current moment; for each target at the current moment, with the predicted trajectory position of the target at the current moment as the center, determining the candidate area of ​​the target according to the candidate distance; from the candidate positions, determining the candidate position that falls into the candidate area of ​​multiple targets as the common position.

[0010] Optionally, based on the determined subordinate distances, the association weights of the common position and the multiple subordinate targets are determined, specifically including: determining the total distance based on the sum of the subordinate distances; for each subordinate target corresponding to the common position, determining the association weight of the common position with the subordinate target based on the proportion of the subordinate distance between the predicted trajectory position of the subordinate target at the current moment and the common position to the total distance.

[0011] Optionally, the estimated position of the target at the current moment is determined based on the association probability between the target and the candidate positions, specifically including: determining the candidate probability of the target at the candidate positions by normalizing the association probability of the candidate positions with the target; determining the equivalent position of the target at the current moment by weighted summing the candidate positions according to the candidate probability of the target at the candidate positions; and determining the estimated position of the target at the current moment based on the equivalent position of the target at the current moment and the trajectory predicted position of the target at the current moment through a Kalman filtering algorithm.

[0012] Optionally, obtaining environmental data collected by sensors at the current moment specifically includes: obtaining environmental data collected by different types of sensors at the current moment, wherein the types of sensors include at least image sensors and infrared sensors; for each of the targets, determining the predicted trajectory position of the target at the current moment based on the estimated position of the target at the previous moment, and determining the association probability of each candidate position with the target, specifically including: for each target identified by each sensor, determining the predicted trajectory position of the target at the current moment and the association probability of each candidate position with the target based on the fusion position of the target at the previous moment, wherein the fusion position is determined based on the estimated position of the target determined by each sensor at the previous moment; for each of the targets, determining the trajectory of the target at the current moment based on the estimated position of the target at the current moment and the historical trajectory of the target, specifically including: determining the fusion position of the target at the current moment based on the estimated position of the target determined by each sensor at the current moment; determining the trajectory of the target at the current moment based on the fusion position of the target at the current moment and the historical trajectory of the target.

[0013] Optionally, for each of the targets, the estimated position of the target at the current moment is determined based on the association probability between the target and each candidate position, specifically including: for each sensor, determining each target identified based on the environmental data collected by the sensor; for each target identified by the sensor, normalizing the association probability between each candidate position and the target, and determining the candidate probability of the target at each candidate position; based on the candidate probability of the target at each candidate position, performing weighted summation on each candidate position to determine the equivalent position of the target at the current moment corresponding to the sensor; based on the estimated covariance of each estimated position of the target determined based on the environmental data collected by each sensor at the previous moment, performing A first fusion operation is performed to obtain a first fusion covariance of the target at the previous moment; a second fusion operation is performed based on the first fusion covariance of the target at the previous moment and the estimated covariances of the estimated positions to obtain a second fusion covariance of the target at the previous moment; a second fusion operation is performed based on the estimated positions of the target determined based on the environmental data collected by each sensor at the previous moment to obtain the fused position of the target at the previous moment; based on the equivalent position of the target at the current moment corresponding to the sensor, the fused position of the target at the previous moment and the second fusion covariance of the target at the previous moment, the estimated position of the target corresponding to the sensor at the current moment is determined through a Kalman filtering algorithm.

[0014] Optionally, before performing the first fusion operation, the method further includes: for each sensor, determining whether the estimated covariances corresponding to the estimated positions of each target identified by the sensor at the previous moment all meet the covariance threshold corresponding to the sensor; if so, determining the estimated covariance of the estimated position determined based on the environmental data collected by the sensor, and performing the first fusion operation; if not, determining that the sensor is faulty, not using the estimated covariance of the estimated position determined based on the environmental data collected by the sensor, and performing the first fusion operation or the second fusion operation.

[0015] Optionally, the method further includes: for each sensor, for each target identified by the sensor, determining whether there are other sensors that have identified the target in history; if so, treating the target as a trusted target; if not, treating the target as an untrusted target.

[0016] Optionally, the method further includes: for each trusted target, if at the current moment none of the sensors have determined the candidate position of the trusted target, determining whether the time interval during which the trusted target has not been continuously recognized by each sensor is greater than a first preset value, and if so, deleting the historical trajectory of the trusted target; if not, retaining the historical trajectory of the trusted target, and using the predicted trajectory position of the trusted target at the current moment as the estimated position of the trusted target at the current moment; for each untrusted target, if at the current moment none of the sensors have determined the candidate position of the untrusted target, determining whether the time interval during which the untrusted target has not been continuously recognized by each sensor is greater than a second preset value, and if so, deleting the historical trajectory of the untrusted target; if not, retaining the historical trajectory of the untrusted target, and using the predicted trajectory position of the untrusted target at the current moment as the estimated position of the untrusted target at the current moment, wherein the first preset value is greater than the second preset value.

[0017] This specification provides a device for multi-target tracking, which includes: an acquisition module, which acquires environmental data collected by a sensor for a monitored scene at the current moment, and determines the candidate positions of each target in the scene at the current moment based on the environmental data; an association probability determination module, which determines, for each target, the trajectory prediction position of the target at the current moment based on the estimated position of the target at the previous moment, and determines the association probability of each candidate position with the target; a common position determination module, which determines one or more common positions from the candidate positions based on the trajectory prediction position of each target at the current moment, wherein each common position corresponds to multiple targets; and an association weight determination module, which determines, for each common position, the trajectory of multiple subordinate targets corresponding to the common position. The module comprises a trajectory prediction module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module, a trajectory determination module

[0018] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned multi-target tracking method is implemented.

[0019] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multi-target tracking method when executing the program.

[0020] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0021] In the multi-target tracking method provided in this specification, based on environmental data collected by sensors, the candidate positions of each target at the current moment are determined. For each target, the association probability between the target and each candidate position is determined. The subordinate distances from the positions of the multiple targets corresponding to the common position to the common position are predicted based on the trajectories. Based on the determined subordinate distances, the association weights between the common position and the multiple targets are determined. The association probabilities between the common position and the targets are weighted by the association weights between the common position and the targets, and the association probabilities are re-determined. Based on the re-determined association probability, the estimated position of the target at the current moment is determined. Based on the estimated position and historical trajectory of the target at the current moment, the trajectory at the current moment is determined.

[0022] Because when determining the estimated position of the target, the association probability is weighted according to the subordinate distance between the common position and the trajectory prediction positions of multiple targets corresponding to the common position, the estimated position of the target finally obtained takes into account the subordinate distance between the common position and the trajectory prediction positions of each target, thereby increasing the association probability of targets that are relatively close to the common position. In dense scenes with many targets, the candidate positions of multiple targets with similar distances can be distinguished according to the subordinate distance, and it can be further determined to which target the candidate position belongs among the multiple targets that are close to each other, so that the trajectories of each target obtained are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 is a flow chart of a method for multi-target tracking in this specification.

[0024] FIG2 is a schematic diagram of a method for determining a public location provided in this specification.

[0025] FIG3 is a schematic diagram of the structure of a multi-source sensor fusion process provided in this specification.

[0026] FIG4 is a schematic diagram of a multi-target tracking device provided in this specification.

[0027] FIG5 is a schematic diagram of an electronic device provided in this specification. DETAILED DESCRIPTION

[0028] Tracking a target means determining its position at every moment. Since targets are often in motion, it's necessary to estimate their motion state at each moment to determine their position at each moment. Typically, this process involves establishing a motion equation based on the target's motion to predict its trajectory and determining candidate positions based on sensor data.

[0029] Because the equations of motion are too theoretical and fail to account for factors like friction, wind speed, and the effects of heavy rain on different roads, the trajectory predictions based on these equations of motion contain certain errors. Furthermore, the measurement data collected by sensors cannot directly reveal the position of the tracked target; it requires further processing using specific transformation equations or target detection models. The accuracy of the candidate positions derived from the sensor data is affected by the inherent sensor precision errors and the errors introduced at each step in the subsequent data conversion process.

[0030] Therefore, neither the trajectory prediction position obtained based on the motion equation nor the candidate position obtained based on the data collected by the sensor can be considered as reliable data. In other words, the trajectory prediction position or the candidate position cannot be directly used as the target position to determine the target's motion trajectory.

[0031] In practical applications, the Kalman filter algorithm is often used to comprehensively weigh the prediction error generated by the equation of motion and the measurement error generated by the sensor, and then weight the predicted trajectory position and candidate positions. For example, when the prediction error is large, the candidate position is given a higher weight; correspondingly, when the measurement error is large, the predicted trajectory position is given a higher weight. This method can obtain an estimated position that is closest to the target's true position.

[0032] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0033] FIG1 is a flow chart of a multi-target tracking method in this specification, which specifically includes the following steps.

[0034] S110: Obtaining environmental data collected by the sensor for the monitored scene at the current moment, and determining candidate positions of each target in the scene at the current moment based on the environmental data.

[0035] All steps in the multi-target tracking method provided in this specification can be implemented by any electronic device with computing capabilities, such as a terminal, server, etc. For ease of description, the multi-target tracking method provided in this specification is described below using only the server as the execution entity.

[0036] The server obtains the environmental data collected by the sensor at the current moment, and determines the candidate positions of each target in the scene where the environmental data is located at the current moment based on the environmental data.

[0037] The sensor type for collecting environmental data may be, for example, an image sensor, an infrared sensor, a radar sensor, etc. The server may input the environmental data collected by the sensor into a target detection model corresponding to the sensor type to obtain candidate locations of each target.

[0038] The server needs to associate each candidate position with each target to be tracked, determine the candidate position of each target, and thus determine the trajectory position of each target at the current moment based on the candidate position corresponding to each target.

[0039] In the multi-target tracking method provided herein, the number of sensors collecting environmental data may be one or more; furthermore, the sensor types may be one or more. The following description will first use the example of environmental data being collected by a single sensor of any type. Subsequently, embodiments will be provided in which environmental data is collected by multiple different types of sensors.

[0040] S120: For each target, determine the trajectory prediction position of the target at the current moment based on the estimated position of the target at the previous moment, and determine the probability of association between each candidate position and the target.

[0041] First, for each target, the server uses the Kalman filter algorithm to predict its current position based on its estimated position at the previous moment and the equation of motion. This determines the target's predicted trajectory position at the current moment. The equation of motion is derived by modeling the target's motion state for a specific motion scenario. The server then determines the probability of each candidate position being associated with the target.

[0042] The above-mentioned primary association probability can be determined according to any method well known to those skilled in the art, and this specification does not limit the specific method for determining the primary association probability.

[0043] For example, the server can use a neural network to extract features such as the appearance and posture of the target in its historical trajectory to obtain the target's underlying features. The server then performs the same feature extraction operation on the candidate targets at each candidate location to obtain candidate features. The target's underlying features are then similarly matched against the candidate features to determine the probability of a primary association between each candidate location and the target. The higher the similarity match, the greater the probability of a primary association.

[0044] Alternatively, the server can determine the association distance between the target's current predicted trajectory location and each candidate location, and based on the association distance, determine the probability of a primary association between each candidate location and the target. The closer the association distance, the greater the primary association probability. Alternatively, the server can determine the probability of a primary association between each candidate location and the target based on both the degree of similarity between the target's underlying features and the candidate features, as well as the association distance. The higher the similarity and the closer the association distance, the greater the primary association probability.

[0045] Due to the existence of sensor errors, each candidate position may be the position of the target. The probability of association between each candidate position and the target can be used to weight each candidate position to equate each candidate position to one position, which is the more accurate equivalent position of the target at the current moment.

[0046] S130: Determine at least one common position corresponding to the plurality of targets from the candidate positions according to the trajectory predicted positions of the targets at the current moment.

[0047] In the multi-target tracking method provided in this specification, the server uses candidate locations that are associated with multiple targets at the same time as common locations.

[0048] The number of public locations can be one or more, and the operations performed on each public location are the same. The following description will take one public location as an example.

[0049] First, the server determines the candidate distance at the current moment, and for each target at the current moment, determines the candidate area of ​​the target based on the candidate distance, with the target's predicted trajectory position at the current moment as the center.

[0050] In one or more embodiments of this specification, the candidate distance at the current moment can be determined based on the error at the previous moment. Specifically, for each target, the server determines the prediction error for the target at the previous moment based on the difference between the target's estimated position at the previous moment and the target's predicted trajectory position at the previous moment. The server then determines the candidate distance at the current moment based on the mean square error of the prediction errors for each target at the previous moment. Subsequently, a candidate area can be determined with the candidate distance at the current moment as the radius and the target's predicted trajectory position at the current moment as the center.

[0051] The candidate distance at time t can be determined according to the following formula:

[0052] In the above formula, σ t represents the candidate distance at time t, M t-1 represents the number of targets identified by the sensor at time t-1, represents the estimated position of the mth target at time t-1, Represents the predicted trajectory position of the mth target at time t-1.

[0053] In one or more embodiments of the present specification, the candidate distance can be preset based on an empirical value. Specifically, the server can determine a preset candidate distance, and use the preset candidate distance as a radius to determine a candidate area centered on the trajectory prediction position of the target at the current moment. The server can also determine two preset candidate distances, where the directions of the two candidate distances are orthogonal to each other, and respectively specify the variable intervals of the trajectory prediction position in two mutually orthogonal directions, thereby determining a rectangular candidate area centered on the trajectory prediction position. For example, if the trajectory prediction position of the target at the current moment is (x, y), then the candidate area is: (x±Δx, y±Δy).

[0054] Here, Δx and Δy represent the variable intervals in two orthogonal directions respectively.

[0055] Then, the server determines, from the candidate positions of the respective targets, a candidate position that falls within the candidate areas of the plurality of targets as a common position.

[0056] Figure 2 is a schematic diagram of a method for determining a common position provided in this specification. In which, the rectangle represents the predicted trajectory position of the target, the four diamonds represent the candidate positions, and the two circular areas are the candidate areas of the two targets. As shown in Figure 2, there are 2 targets and 4 candidate positions at the current moment. The 1st to 3rd candidate positions fall into the candidate area of ​​the first target, and the 1st and 4th candidate positions fall into the candidate area of ​​the second target. Then the first candidate position is a common position that has an association relationship with both targets, and the remaining candidate positions are non-public positions.

[0057] Specifically, the common position can be determined by a confirmation matrix, which can be expressed as:

[0058] In the confirmation matrix, the rows represent the candidate locations identified by the sensor at time t, with a total of J, and the columns represent the targets at the current time t, with a total of M. Each element in the matrix It is used to indicate whether the j-th candidate position falls within the candidate area of ​​the m-th target. If so, the element value is 1, otherwise it is 0.

[0059] Therefore, whether the jth candidate position is a public position can be determined according to the following formula:

[0060] Sum the M elements of each row of the confirmation matrix. For example, if the above equation holds for row j, it means the jth candidate location falls within the candidate regions of at least two targets, and the jth candidate location is a common location. If the above equation does not hold, the jth candidate location is a non-common location.

[0061] As shown in Figure 2, at time t, four candidate locations are determined by the environmental data of the sensor, and the number of targets is 2. The corresponding confirmation matrix of Figure 2 can be expressed as follows:

[0062] From the above confirmation matrix, after summing the M elements in the first row, we can get It can be determined that the first candidate position represented by the first row is a public position.

[0063] Because a common location is associated with multiple targets, for each target corresponding to the common location, the candidate locations associated with that target are used to determine the target's equivalent location. The equivalent locations are then used to determine the target's estimated location. The estimated location is the location of the target in its trajectory at the current moment. The target corresponding to a common location is also referred to as the dependent target corresponding to that common location or the dependent target of that common location.

[0064] Therefore, a common location will affect the current trajectories of multiple targets corresponding to that common location. However, when determining the primary association probability, the primary association probability for each candidate location is primarily considered. That is, for a target, the sum of the primary association probabilities for that target is 1. However, the influence of the common location is not considered when determining the primary association probability. As a result, when determining the estimated location of the target based on the primary association probability, it is impossible to distinguish which target the common location actually belongs to, resulting in inaccurate trajectories for each target.

[0065] S140: For each of the common positions, determine the subordinate distances from the predicted trajectory positions of the multiple subordinate targets corresponding to the common position to the common position, and determine the association weights between the common position and the multiple subordinate targets based on the determined subordinate distances.

[0066] To determine a more accurate association probability, the different effects of a common location on multiple subordinate targets can be further considered. For example, the server can determine the subordinate distances from the predicted trajectory positions of multiple subordinate targets corresponding to the common location to the common location. The server then determines the association weights between the common location and the multiple subordinate targets based on the determined subordinate distances.

[0067] Specifically, the server may use the sum of the subordinate distances as the total distance. For each subordinate target corresponding to the common location, the server determines the ratio of the subordinate distance between the current predicted trajectory position of the subordinate target and the common location to the total distance, which serves as the association weight between the common location and the subordinate target.

[0068] When the jth candidate position is a common position and has an association relationship with N subordinate targets, the association weight between the candidate position and the nth subordinate target corresponding to the candidate position can be determined according to the following formula:

[0069] In the above formula, represents the subordinate distance between the jth candidate position at time t and the trajectory prediction position of the nth subordinate target corresponding to the candidate position at the current moment, Represents the association weight between the j-th candidate position at time t and the n-th subordinate target corresponding to the candidate position, where N≤M, and M is the total number of targets at the current moment.

[0070] The association weight is the influence of a common location on its corresponding multiple subordinate targets, calculated from the perspective of the common location. For a common location, the sum of the association weights of its corresponding subordinate targets is 1.

[0071] For each subordinate target among the multiple subordinate targets corresponding to the common location, if the distance between the common location and the trajectory prediction position of the subordinate target (i.e., the aforementioned subordinate distance) is relatively close, the server assigns a larger association weight to the subordinate target; otherwise, the server assigns a smaller association weight.

[0072] S150: For each subordinate target corresponding to each of the public positions, the first-order association probability between the public position and the subordinate target is weighted by utilizing the association weight between the public position and the subordinate target to obtain the second-order association probability between the public position and the subordinate target as the re-determined association probability between the public position and the subordinate target.

[0073] The server weights the primary association probability between the public location and the subordinate target by the association weight between the public location and the subordinate target, that is, updates the primary association probability between the public location and the subordinate target to re-determine the association probability between each candidate location and the subordinate target as the secondary association probability.

[0074] In S140, for each candidate location determined as a public location, the server determines the association weight between the public location and each subordinate target corresponding to the public location. The server can then determine the number m of the nth subordinate target corresponding to the public location in the total target, and thus assign the association weight between the jth candidate location at time t and the nth subordinate target corresponding to the candidate location to Converted into the association weight between the jth candidate position at time t and the mth target among all targets

[0075] Therefore, the definition of the re-determined quadratic association probability is as follows:

[0076] like The server can determine that the jth candidate location at time t is a common location, and then calculate the association weight between the candidate location and the nth subordinate target corresponding to the candidate location. Converted into the association weight between the candidate position and the mth target among all targets Then, use the relevance weight The probability of an association between the jth candidate position and the mth target Weighted, that is, the probability of association between the j-th candidate position and the m-th target Update to get the secondary association probability between the re-determined j-th candidate position and the m-th target

[0077] like The server may determine that the jth candidate location is a non-public location, and then does not update the primary association probability between the candidate location and the corresponding target, that is, directly uses the primary association probability between the candidate location and the target as the secondary association probability between the candidate location and the target.

[0078] The secondary association probability is a comprehensive calculation based on the target and the common location associated with multiple targets. Specifically, this probability updates the primary association probability (which considers only the target) based on the association weight. This takes into account the differential impact of the common location on multiple closely spaced subordinate targets. By weighting the primary association probability based on the subordinate distance to further enhance the discriminability, the secondary association probability is increased, increasing the difference between the secondary association probabilities of multiple closely spaced subordinate targets and the common location, making the trajectories of multiple closely spaced targets easier to distinguish at the current moment.

[0079] S160: For each target, determine the estimated position of the target at the current moment according to the association probability between the target and each candidate position.

[0080] First, for each target, the server normalizes the association probability of the target with each candidate location (specifically, it may include the secondary association probability of the public location in each candidate location and the target, and the primary association probability of the non-public location in each candidate location and the target) to determine the tertiary association probability of the target with each candidate location, as well as the candidate probability of the target appearing in each candidate location.

[0081] Therefore, the candidate probability of the mth target at the jth candidate position is The definition is as follows:

[0082] Because the initial candidate probability only considers the target. Therefore, for a target, the sum of the initial association probabilities of each candidate position with the target is 1. The association weight is based on the common position. Accordingly, for a common position, the sum of the association weights of each target corresponding to the common position and the common position is 1. After the association probability between the common position and each target is re-determined, the sum of the association probabilities of each candidate position corresponding to a target with the target will change, that is, it is very likely that it will no longer be 1, so the re-determined association probability must be normalized to determine the candidate probability of the target at each candidate position.

[0083] Then, according to the candidate probability of the target at each candidate position, a weighted sum is performed on each candidate position to determine the equivalent position of the target at the current moment.

[0084] After normalizing the re-determined association probabilities to obtain the candidate probabilities of the target at each candidate location, the sum of the candidate probabilities of the target at each candidate location is 1. Therefore, by weighting and summing each candidate location according to its candidate probability, each valid candidate location of the target (i.e., a candidate location with an association probability or candidate probability not equal to 0) can be equated to a single location, thereby obtaining the equivalent location of the target at the current moment. This equivalent location is the optimal measurement location of the target determined based on the data collected by the sensor at the current moment.

[0085] Finally, the Kalman filter algorithm is used to determine the estimated position of the target at the current moment based on the equivalent position of the target and the trajectory predicted position of the target at the current moment.

[0086] Since the equivalent position of the target at the current moment is obtained by weighted summing the candidate positions of the target based on the association probability re-determined based on the association weight, it takes into account both the association probability of the same target with different candidate positions (i.e., the initial association probability) and the association weight of the same candidate position with different targets (i.e., weighted update of the initial association probability based on the association), therefore, determining the estimated position of the target at the current moment based on the equivalent position of the target at the current moment is a more accurate prediction of the position of the target at the current moment. Accordingly, in the estimated position, by weighting the association probability difference between the common position and different targets based on the subordinate distance between the common position and the corresponding multiple subordinate targets, the difference between the candidate probabilities of different targets at different candidate positions is increased, which can increase the difference between the estimated positions of different targets. In this way, the motion trajectories of different targets are easier to distinguish, and at the same time, the risk of motion trajectory merging is reduced.

[0087] S170: For each target, determine the current trajectory of the target based on the estimated position of the target at the current moment and the historical trajectory of the target.

[0088] For each of the multiple targets, after the server obtains the estimated position of the target at the current moment, it can add the estimated position of the target at the current moment to the historical trajectory composed of the estimated positions of the target at various moments in history based on the estimated position of the target at the current moment and the historical trajectory of the target to obtain the motion trajectory of the target at the current moment.

[0089] In the multi-target tracking method provided in this specification, the candidate positions of each target at the current moment are determined based on the environmental data collected by the sensor; for each target, the association probability between the target and each candidate position is determined; a common position is determined based on the association probability between each target and each candidate position; for each common position, the association weights of the common position and the corresponding multiple subordinate targets are determined based on the subordinate distances between the common position and the trajectory predicted positions of the corresponding multiple subordinate targets, and the association probability of the common position and the subordinate target corresponding to the common position is weighted based on the association weights of the common position and the subordinate target corresponding to the common position, and the association probability between each target and each candidate position is re-determined. Then, for each target, the estimated position of the target at the current moment is determined based on the re-determined association probability, and the trajectory of the target at the current moment is determined based on the estimated position and historical trajectory of the target at the current moment.

[0090] Because when determining the estimated position of the target, the initial association probability is weighted according to the subordinate distance between the common position and the predicted trajectory positions of multiple subordinate targets corresponding to the common position, which increases the association probability of targets that are relatively close to the common position. In dense scenes with many targets, the candidate positions of multiple targets with similar distances can be distinguished according to the subordinate distance, and it can be further determined which target the candidate position belongs to among multiple targets with similar positions, thereby making the motion trajectory of each target obtained more accurate.

[0091] Different types of sensors have their own advantages and limitations. For example, a sensor's sensitivity range is typically limited. In multi-target tracking, the reliability of candidate locations of each target determined based on environmental data collected by only one sensor is low, resulting in insufficient accuracy in the corresponding determined trajectories of each target. Therefore, this specification provides a method for multi-target tracking based on multi-source sensors, which specifically includes the following steps.

[0092] S210: Acquire environmental data collected by multiple sensors of different types at the current moment, wherein the multiple sensors of different types include at least two of image sensors, infrared sensors, radar sensors, etc.

[0093] S220: For each sensor, determine each target identified based on the environmental data collected by the sensor; for each target identified by the sensor, determine the target's current trajectory predicted position and the probability of association between each candidate position and the target based on the target's fused position at the previous moment. The target's fused position at the previous moment is determined based on the target's estimated position determined by each sensor at the previous moment.

[0094] The specific method for determining the association probability can be referred to the description of the corresponding content in the above S120 and will not be described in detail here.

[0095] In multi-source sensor-based multi-target tracking methods, the server fuses the estimated positions of the target determined by each sensor at the previous moment to obtain the fused position of the target at the previous moment. This fused position is an estimated position of the same target determined by integrating environmental data from different types of sensors. It is the result of correcting the estimated positions determined by data collected by each sensor. It can be considered an optimal approximation of the target's true position at the previous moment. Therefore, the fused position is more accurate than the estimated position.

[0096] After obtaining the target's last fused position, the server uses this fused position to update the target's last estimated position, determined based on the data collected by each sensor. For each target identified by each sensor, a Kalman filter algorithm is used to determine the target's current trajectory position based on its last fused position.

[0097] For the processes of S230 to S250 , reference may be made to the description of the corresponding contents of S130 to S150 above, and will not be repeated here.

[0098] S260: For each target identified by each sensor, the server determines the estimated position of the target corresponding to the sensor at the current moment according to the association probability of the target at each candidate position.

[0099] First, the server determines, for each sensor, each target identified based on the environmental data collected by the sensor.

[0100] Next, for each target identified based on the environmental data collected by the sensor, the server normalizes the secondary association probabilities of the target at each candidate location and the common locations, as well as the primary association probabilities of the target at each candidate location. This determines the target's candidate probability at each candidate location. Based on the candidate probabilities, the server performs a weighted sum of the candidate locations associated with the target, and determines the target's equivalent position at the current moment as the estimated position of the target at the current moment, as determined by the sensor.

[0101] Next, for each target, the server needs to fuse the estimated position of the target identified based on the environmental data collected by each sensor to obtain the fused position of the target at the current moment. The specific fusion steps are as follows.

[0102] Step 1: The server performs a first fusion operation based on the estimated covariances of the estimated positions of the target determined at the last moment based on the environmental data collected by each sensor, to obtain the first fused covariance of the target at the last moment.

[0103] Step 2: The server performs a second fusion operation based on the first fusion covariance of the target at the previous moment and the estimated covariances of the estimated positions of the target at the previous moment to obtain the second fusion covariance of the target at the previous moment.

[0104] Step 3: The server performs a second fusion operation based on the estimated positions of the target at the previous moment to obtain the fused position of the target at the previous moment.

[0105] Step 4: The server determines the estimated position of the target corresponding to the sensor at the current moment through the Kalman filter algorithm based on the equivalent position of the target corresponding to the sensor at the current moment, the fused position of the target at the previous moment, and the second fused covariance of the target at the previous moment.

[0106] During the calculation process of the Kalman filter algorithm, for each target identified by each sensor, the predicted covariance of the target at the current moment is determined based on the estimated covariance of the target at the previous moment.

[0107] In the Kalman filter algorithm, the prediction covariance measures the magnitude of the prediction error generated by the equation of motion, while the estimated covariance measures the magnitude of the sensor's measurement error. The Kalman gain determines the weight of the two errors, giving greater weight to the result with the smaller error, resulting in an estimated target position that is closest to the true position.

[0108] In the above fusion steps, both the first and second steps involve fusing the estimated covariances. This two-step process can disperse the server's computational load and improve computational efficiency during multi-target tracking. Of course, if the target tracking task is within the server's computational load, the first and second steps can be combined into a single step. That is, the server directly performs a second fusion operation on the estimated covariances to obtain a second fused covariance.

[0109] In the above fusion steps, the first fusion operation and the second fusion operation can be implemented through two different neural network layers respectively. The server can use the neural network layer that performs the first fusion operation as the first fusion center and the neural network layer that implements the second fusion operation as the second fusion center.

[0110] Of course, the first and second fusion operations can also be implemented through the same neural network layer. That is, the fusion step is implemented through only one neural network layer, and the server can use this neural network layer as the fusion center. Alternatively, the first fusion operation is implemented through a single neural network layer, and the server uses this neural network layer as the first fusion center. The second fusion operation, the operation to obtain the second fusion covariance, is implemented through a single neural network layer, and the server uses this neural network layer as the second fusion center. The operation to obtain the fusion position is implemented through another neural network layer, and the server uses this neural network layer as the third fusion center. In other words, the fusion steps can be implemented through a total of three neural network layers. This specification does not limit the network structure of the multi-source sensor fusion operation.

[0111] S270: For each target identified by each sensor, determine the fused position of the target at the current moment according to the estimated position of the target corresponding to each sensor; determine the trajectory of the target at the current moment according to the fused position of the target at the current moment and the historical trajectory of the target.

[0112] Since different types of sensors have different sensitive ranges, the targets identified by each sensor may also be different. The server needs to obtain the trajectory of each target based on the union of the targets identified by each sensor as the final target recognition result.

[0113] Therefore, the server must first determine the targets identified by the sensors, and for each target, determine the estimated position of the target corresponding to each sensor.

[0114] Since different types of sensors recognize different targets, the server needs to determine which targets are recognized by each sensor. For a target recognized by multiple sensors, the server needs to determine the estimated positions of the target corresponding to each of the sensors at the current moment.

[0115] Then, for each target identified by multiple sensors, the server determines the estimated position of the target determined by each of the multiple sensors that identify the target, and fuses these estimated positions to obtain a fused position of the target.

[0116] Finally, the server determines the current trajectory of the target based on the fused position of the target at the current moment and the historical trajectory of the target.

[0117] The fused position combines the estimated positions of the same target, determined separately based on environmental data collected by multiple sensors of different types. It corrects the estimated positions determined by each sensor, resulting in a more accurate position than the estimated position based on a single sensor. Therefore, the trajectory of the target derived from the fused position is more accurate than that determined by a single sensor.

[0118] In the aforementioned multi-source sensor-based multi-target tracking method, the accuracy of the determined target trajectories is improved by increasing the number of sensor types. However, as the number of sensors increases, the likelihood of failure in the sensor system composed of multiple different sensor types increases. Because the trajectory of each target is derived by fusing the results of each sensor in the sensor system, any failure of any sensor in the sensor system will affect the data from the fusion process, thereby affecting the accuracy of the trajectory prediction results for each target determined by the entire sensor system.

[0119] In order to prevent the accuracy of the trajectory prediction result based on the entire sensor system from being affected by the failure of a certain sensor, a fault detection may be performed on each sensor before performing the first fusion operation.

[0120] Therefore, in the above step S260, before performing the first fusion operation, the server determines, for each sensor, the estimated covariance corresponding to the estimated position of each target identified by the sensor at the current moment, and determines whether all the estimated covariances meet the covariance threshold corresponding to the sensor.

[0121] If so, the estimated covariance of the estimated position determined based on the environmental data collected by the sensor is determined, and a first fusion operation is performed.

[0122] If not, the sensor is determined to be faulty and the environmental data collected by the sensor is isolated, that is, the estimated covariance of the estimated position determined based on the environmental data collected by the sensor is not used to perform the first fusion operation or the second fusion operation.

[0123] By detecting faults in each sensor, the faulty sensor can be identified in a timely manner, and the environmental data collected by the faulty sensor can be isolated before the fusion operation is performed, so that the accurate trajectory of each target can still be obtained through the remaining sensors that have not failed. The entire sensor system will not be unusable due to the failure of some sensors in the entire sensor system, thereby increasing the durability of the sensor system.

[0124] Furthermore, after a confirmed faulty sensor is repaired and restored to normal operation, the estimated covariance of the estimated positions of each target, obtained based on the environmental data collected by the sensor after restoration, will meet the covariance threshold corresponding to the sensor. Based on this, the isolation of the environmental data collected by the sensor can be lifted, allowing the sensor's data to re-enter the fusion process.

[0125] Figure 3 is a schematic diagram of the structure of a multi-source sensor fusion process provided in this specification. The sensor system includes Y different types of sensors. represents the estimated position of the mth target determined by the yth sensor at time t, represents the estimated covariance of the mth target determined by the yth sensor at time t, represents the first fusion covariance of the mth target at time t, represents the second fusion covariance of the mth target at time t, represents the fusion position of the mth target at time t.

[0126] As shown in Figure 3, after determining the estimated position and estimated covariance of each target identified by each sensor, each sensor is fault-checked based on its estimated covariance. The estimated positions and estimated covariances of each target obtained by healthy sensors are then fused through the first and second fusion centers. Ultimately, the fused position of each target at the current moment, as determined by the sensor system, is obtained, thereby determining the trajectory of each target at the current moment. The second fused covariance and fused position of each target determined at the current moment are transmitted to each sensor to determine the estimated position of each target at the next moment.

[0127] In the above-mentioned multi-target tracking method based on multi-source sensors, different types of sensors may identify different targets. For a target, if multiple sensors all identify the target, then the possibility that the target is the true target of the current target tracking task is high. In addition, when determining the trajectory of the target at the current moment, the estimated positions determined by multiple sensors can be combined to obtain a more accurate fused position of the target, which increases the credibility of the trajectory of the target.

[0128] In order to reduce the memory consumption of the server and improve the computing speed of the target tracking task, the tracks of the targets that have left the current field of view can be deleted.

[0129] Specifically, the server determines, for each sensor and each target identified by the sensor, whether there is another sensor that has identified the target in the past.

[0130] If other sensors have identified the target, the target is considered a trusted target. For each trusted target, if no sensor has currently determined its candidate location, the system determines whether the interval during which the trusted target has not been identified by any sensor is greater than a first preset value. Accordingly, if the interval is greater than the first preset value, the historical trajectory of the trusted target is deleted. If the interval is not greater than the first preset value, the historical trajectory of the trusted target is retained, and the predicted position of the trusted target's trajectory at the current moment is used as the estimated position of the trusted target at the current moment.

[0131] If no other sensor can identify the target, the target is considered an untrusted target. For each untrusted target, if no sensor has determined the candidate location of the trusted target at the current moment, a determination is made as to whether the time interval during which the untrusted target has not been identified by any sensor is greater than a second preset value. Accordingly, if the time interval is greater than the second preset value, the historical trajectory of the untrusted target is deleted. If the time interval is not greater than the second preset value, the historical trajectory of the untrusted target is retained, and the predicted position of the untrusted target's trajectory at the current moment is used as the estimated position of the untrusted target at the current moment. The first preset value is greater than the second preset value.

[0132] If none of the sensors have determined the target's candidate position at the current moment, this could be due to a temporary obstruction or the target leaving the current field of view. Without deleting the target's historical trajectory, the target's current trajectory will be determined based on its fused position at the next moment. However, since the target's candidate position is unavailable at the current moment, neither its estimated position nor its fused position can be determined. To maintain trajectory continuity, the target's current trajectory predicted position is used as its estimated position.

[0133] If the target is a trusted target identified by multiple sensors, it is more likely that the target temporarily leaves the field of view due to occlusion. Therefore, when deciding whether to delete the historical trajectory of the trusted target, the trusted target is given more acceptable disappearance time so that the trajectory of the target can be updated in time when the target moves out of the occlusion.

[0134] If the target is an untrusted target identified by only one sensor, it may be due to the sensor being overly sensitive in a specific environment, generating a noise signal. Therefore, the target is more likely to be a noise signal. Therefore, when deciding whether to delete the historical track of the trusted target, the target is given a shorter acceptable disappearance time. In other words, the second preset value is set to be smaller than the first preset value.

[0135] For example, in multi-target tracking tasks for autonomous driving, the vehicle's control system tracks targets within a preset field of view around the vehicle based on environmental data from multiple sensors, allowing for timely adjustments to the vehicle's route. The targets in this task can be other vehicles and people. In this task, only the trajectories of targets within a period of time close to the current moment, or the trajectories of targets within the field of view, will affect the vehicle's route. Furthermore, storing target trajectories with excessively long time intervals will cause data redundancy in the vehicle's storage unit, slowing down the control system's decision-making process and potentially leading to serious consequences.

[0136] Therefore, the targets identified by each sensor are divided into trusted targets and untrusted targets, and the tracks of targets whose candidate positions have not been determined by each sensor within the preset time interval are deleted to ensure the continuous and efficient performance of the target tracking task.

[0137] The above is the multi-target tracking method provided in this specification. Based on the same idea, this specification also provides a corresponding multi-target tracking device, as shown in FIG4 .

[0138] FIG4 is a schematic diagram of a multi-target tracking device provided in this specification, which specifically includes:

[0139] The acquisition module 300 acquires the environmental data collected by the sensor for the monitored scene at the current moment, and determines the candidate positions of each target in the scene at the current moment based on the environmental data; the association probability determination module 302 determines, for each of the targets, the trajectory prediction position of the target at the current moment based on the estimated position of the target at the previous moment, and determines the association probability of each candidate position with the target; the common position determination module 304 determines one or more common positions from the candidate positions based on the trajectory prediction position of each of the targets at the current moment, wherein each common position corresponds to multiple targets; the association weight determination module 306 determines, for each of the common positions, the trajectory prediction positions of multiple subordinate targets corresponding to the common position to the common position. The subordinate distances are set, and the association weights of the common position and the multiple subordinate targets are determined according to the determined subordinate distances; the association probability updating module 308, for each subordinate target among the multiple subordinate targets corresponding to the common position, weights the association probability of the common position and the subordinate target by using the association weight of the common position and the subordinate target, and re-determines the association probability of the common position and the subordinate target; the estimated position determination module 310, for each of the targets, determines the estimated position of the target at the current moment according to the association probability of the target with the candidate positions; the trajectory determination module 312, for each of the targets, determines the trajectory of the target at the current moment according to the estimated position of the target at the current moment and the historical trajectory of the target.

[0140] Optionally, the common position determination module 304 is specifically used to determine the candidate distance at the current moment. For each target at the current moment, the candidate area of ​​the target is determined based on the candidate distance, with the predicted trajectory position of the target at the current moment as the center. From the candidate positions, a candidate position that falls into the candidate area of ​​multiple targets is determined as the common position.

[0141] Optionally, the association weight determination module 306 is specifically used to determine the total distance based on the sum of each of the subordinate distances, and for each subordinate target corresponding to the common position, determine the association weight between the common position and the subordinate target based on the proportion of the subordinate distance between the trajectory prediction position of the subordinate target at the current moment and the common position to the total distance.

[0142] Optionally, the estimated position determination module 310 is specifically used to determine the candidate probability of the target at each candidate position by normalizing the association probability of each candidate position with the target, determine the equivalent position of the target at the current moment by weighted summing the candidate positions according to the candidate probability of the target at the candidate positions, and determine the estimated position of the target at the current moment based on the equivalent position of the target at the current moment and the trajectory predicted position of the target at the current moment through the Kalman filtering algorithm.

[0143] Optionally, the acquisition module 300 is specifically configured to acquire environmental data collected by different types of sensors at the current moment, wherein the types of sensors include at least image sensors and infrared sensors.

[0144] Optionally, the association probability determination module 302 is specifically used to determine, for each target identified by each sensor, the predicted trajectory position of the target at the current moment and the association probability of each candidate position with the target based on the fusion position of the target at the previous moment, wherein the fusion position is determined based on the estimated position of the target determined by each sensor at the previous moment.

[0145] Optionally, the trajectory determination module 312 is specifically used to determine the fused position of the target at the current moment based on the estimated position of the target determined by each sensor at the current moment, and determine the trajectory of the target at the current moment based on the fused position of the target at the current moment and the historical trajectory of the target.

[0146] Optionally, the estimated position determination module 310 is specifically used to determine, for each sensor, each target identified based on the environmental data collected by the sensor, and for each target identified by the sensor, normalize the association probability of each candidate position with the target to determine the candidate probability of the target at each candidate position; perform weighted summation on each candidate position based on the candidate probability of the target at each candidate position to determine the equivalent position of the target at the current moment corresponding to the sensor; perform a first fusion operation based on the estimated covariance of each estimated position of the target determined based on the environmental data collected by each sensor at the previous moment to obtain the equivalent position of the target at the previous moment. a first fusion covariance of the target; performing a second fusion operation based on the first fusion covariance of the target at the previous moment and the estimated covariances of the estimated positions to obtain the second fusion covariance of the target at the previous moment; performing a second fusion operation based on the estimated positions of the target determined based on the environmental data collected by each sensor at the previous moment to obtain the fused position of the target at the previous moment; determining the estimated position of the target at the current moment corresponding to the sensor through a Kalman filtering algorithm based on the equivalent position of the target at the current moment, the fused position of the target at the previous moment and the second fusion covariance of the target at the previous moment.

[0147] Optionally, the estimated position determination module 310 is specifically used to determine, for each sensor, whether the estimated covariances corresponding to the estimated positions of each target identified by the sensor at the previous moment all meet the covariance threshold corresponding to the sensor; if so, determine the estimated covariance of the estimated position determined based on the environmental data collected by the sensor, and perform a first fusion operation; if not, determine that the sensor is faulty, and do not use the estimated covariance of the estimated position determined based on the environmental data collected by the sensor to perform the first fusion operation or the second fusion operation.

[0148] Optionally, the device also includes a track deletion module 314, which is specifically used to determine, for each sensor and for each target identified by the sensor, whether there are other sensors that have identified the target in history. If so, the target is treated as a trusted target; if not, the target is treated as an untrusted target.

[0149] Optionally, the trajectory deletion module 314 is specifically used to, for each trusted target, if at the current moment none of the sensors have determined the candidate position of the trusted target, determine whether the time interval during which the trusted target has not been continuously recognized by each sensor is greater than a first preset value; if so, delete the historical trajectory of the trusted target; if not, retain the historical trajectory of the trusted target, and use the predicted trajectory position of the trusted target at the current moment as the estimated position of the trusted target at the current moment; for each untrusted target, if at the current moment none of the sensors have determined the candidate position of the untrusted target, determine whether the time interval during which the untrusted target has not been continuously recognized by each sensor is greater than a second preset value; if so, delete the historical trajectory of the untrusted target; if not, retain the historical trajectory of the untrusted target, and use the predicted trajectory position of the untrusted target at the current moment as the estimated position of the untrusted target at the current moment, wherein the first preset value is greater than the second preset value.

[0150] This specification also provides a computer-readable storage medium, which stores a computer program. The computer program can be used to execute the multi-target tracking method provided in FIG. 1 above.

[0151] This specification also provides a schematic structural diagram of the electronic device shown in Figure 5. As shown in Figure 5, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the multi-target tracking method described in Figure 1 above. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0152] Improvements to a technology can be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with technological advancements, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using a hardware module. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system onto a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0153] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0154] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0155] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0156] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0157] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0158] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0160] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0161] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0162] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0163] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0164] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0165] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0166] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0167] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of this application.

Claims

1. A method for multi-object tracking, characterized in that Including: Obtain the environmental data collected by the sensor for the monitored scenario at the current moment; Determine the candidate positions of each target in the scenario at the current moment according to the environmental data; For each target, determine the predicted trajectory position of the target at the current moment according to the estimated position of the target at the previous moment, and determine the association probability between each candidate position and the target; Determine one or more common positions from the candidate positions according to the predicted trajectory positions of the targets at the current moment, wherein each common position corresponds to multiple targets; For each of the common positions, Determine the subordinate distances from the predicted trajectory positions of the multiple subordinate targets corresponding to the common position to the common position respectively; Determine the association degree weights between the common position and the multiple subordinate targets respectively according to the determined subordinate distances; For each subordinate target among the multiple subordinate targets corresponding to the common position, re-determine the association probability between the common position and the subordinate target by weighting the association probability between the common position and the subordinate target by using the association degree weight between the common position and the subordinate target; For each target, determine the estimated position of the target at the current moment according to the association probability between the target and the candidate positions; For each target, determine the trajectory of the target at the current moment according to the estimated position of the target at the current moment and the historical trajectory of the target; 2. The method according to claim 1, wherein, Determine one or more common positions from the candidate positions according to the predicted trajectory positions of the targets at the current moment, specifically including: Determine the candidate distance at the current moment; For each target at the current moment, with the predicted trajectory position of the target at the current moment as the center, determine the candidate area of the target according to the candidate distance; Determine the candidate positions that fall into the candidate areas of multiple targets from the candidate positions as the common positions; 3. The method according to claim 1, wherein Determine the association degree weights between the common position and the multiple subordinate targets respectively according to the determined subordinate distances, specifically including: Determine the total distance according to the sum of the subordinate distances; For each subordinate target corresponding to the common position, determine the association degree weight between the common position and the subordinate target according to the proportion of the subordinate distance between the predicted trajectory position of the subordinate target at the current moment and the common position to the total distance; 4. The method according to claim 1, wherein Determine the estimated position of the target at the current moment according to the association probability between the target and the candidate positions, specifically including: Determine the candidate probability of the target at each candidate position by normalizing the association probability between each candidate position and the target; Determine the equivalent position of the target at the current moment by performing weighted summation on the candidate positions according to the candidate probability of the target at each candidate position; Determine the estimated position of the target at the current moment by using the Kalman filtering algorithm according to the equivalent position of the target at the current moment and the predicted trajectory position of the target at the current moment; 5. The method according to any one of claims 1 to 4, characterized in that Obtain the environmental data collected by the sensor at the current moment, specifically including: Obtain the environmental data collected by different types of sensors at the current moment, where the types of the sensors at least include an image sensor and an infrared sensor; For each of the targets, determine the trajectory of the target at the current moment according to the estimated position of the target at the previous moment Predicted position, and determine the association probability between each candidate position and the target, specifically including: For each target identified by each sensor, determine the trajectory prediction position of the target at the current moment and the association probability between each candidate position and the target according to the fusion position of the target at the previous moment, where the fusion position is determined according to the estimated positions of the target determined by each sensor at the previous moment; For each of the targets, determine the trajectory of the target at the current moment according to the estimated position of the target at the current moment and the historical trajectory of the target, specifically including: Determine the fusion position of the target at the current moment according to the estimated positions of the target determined by each sensor at the current moment; Determine the trajectory of the target at the current moment according to the fusion position of the target at the current moment and the historical trajectory of the target.

6. The method according to claim 5, wherein For each of the targets, determine the estimated position of the target at the current moment according to the association probability between the target and each candidate position, specifically including: For each sensor, Determine each target identified based on the environmental data collected by the sensor; For each target identified by the sensor, Normalize the association probability between each candidate position and the target to determine the candidate probability of the target at each candidate position; Perform a weighted sum on each candidate position according to the candidate probability of the target at each candidate position to determine the equivalent position of the target corresponding to the sensor at the current moment; Perform a first fusion operation according to the estimated covariance of each estimated position of the target determined based on the environmental data collected by each sensor at the previous moment to obtain the first fusion covariance of the target at the previous moment; Perform a second fusion operation according to the first fusion covariance of the target at the previous moment and the estimated covariance of each estimated position to obtain the second fusion covariance of the target at the previous moment; Perform a second fusion operation according to the estimated positions of the target determined based on the environmental data collected by each sensor at the previous moment to obtain the fusion position of the target at the previous moment; Determine the estimated position of the target corresponding to the sensor at the current moment through the Kalman filter algorithm according to the equivalent position of the target corresponding to the sensor at the current moment, the fusion position of the target at the previous moment, and the second fusion covariance of the target at the previous moment.

7. The method according to claim 6, characterized in that, Before performing the first fusion operation, the method further includes: For each sensor, determine whether the estimated covariances corresponding to the estimated positions of each target identified by the sensor at the previous moment all meet the covariance threshold corresponding to the sensor; If they meet, determine the estimated covariance of the estimated position determined according to the environmental data collected by the sensor and perform the first fusion operation; If they do not meet, determine that the sensor is faulty and do not use the estimated covariance of the estimated position determined according to the environmental data collected by the sensor to perform the first fusion operation or the second fusion operation.

8. The method according to claim 6, characterized in that, The method further includes: For each sensor, For each target identified by the sensor, determine whether there are other sensors that have identified the target in history; If so, regard the target as a credible target; If not, regard the target as a non-credible target.

9. The method according to claim 8, characterized in that, The method further includes: For each credible target, If the candidate positions of the credible target have not been determined by each sensor at the current moment, determine whether the time interval during which the credible target has not been identified by each sensor continuously is greater than a first preset value, If it is greater than the first preset value, delete the historical trajectory of the credible target, If it is not greater than the first preset value, retain the historical trajectory of the credible target, and use the predicted position of the trajectory of the credible target at the current moment as the estimated position of the credible target at the current moment; For each non-credible target, If the candidate positions of the non-credible target have not been determined by each sensor at the current moment, determine whether the time interval during which the non-credible target has not been identified by each sensor continuously is greater than a second preset value, If it is greater than the second preset value, delete the historical trajectory of the non-credible target, If it is not greater than the second preset value, retain the historical trajectory of the non-credible target, and use the predicted position of the trajectory of the non-credible target at the current moment as the estimated position of the non-credible target at the current moment, wherein, the first preset value is greater than the second preset value.

10. An apparatus for multi-object tracking, characterized in that, It includes: An acquisition module that acquires the environmental data collected by the sensor for the monitored scene at the current moment, and determines the candidate positions of each target in the scene at the current moment according to the environmental data; An association probability determination module that, for each target, determines the predicted position of the trajectory of the target at the current moment according to the estimated position of the target at the previous moment, and determines the association probability between each candidate position and the target; A common position determination module that determines one or more common positions from the candidate positions according to the predicted positions of the trajectories of the targets at the current moment, wherein each common position corresponds to multiple targets; An association degree weight determination module that, for each common position, respectively determines the subordinate distances from the predicted positions of the trajectories of the multiple subordinate targets corresponding to the common position to the common position, and determines the association degree weights between the common position and the multiple subordinate targets according to the determined subordinate distances; An association probability update module that, for each subordinate target among the multiple subordinate targets corresponding to the common position, re-determines the association probability between the common position and the subordinate target by weighting the association probability between the common position and the subordinate target by using the association degree weight between the common position and the subordinate target; An estimated position determination module that, for each target, determines the estimated position of the target at the current moment according to the association probability between the target and each candidate position; A trajectory determination module that, for each target, determines the trajectory of the target at the current moment according to the estimated position of the target at the current moment and the historical trajectory of the target.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 9 above is implemented.

12. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the method described in any one of claims 1 to 9 above is implemented.

Images