A target tracking method and device, electronic equipment and storage medium

Abstract

This invention discloses a target tracking method, apparatus, electronic device, and storage medium. The method includes: in response to a target tracking event being triggered, determining first position information for each target within the radar detection range; for each target within the radar detection range, determining the group structure of the current target based on the first position information, and determining the positional relationship between the group structure and all other targets within the radar detection range (excluding the current target); wherein the group structure is the projection area of ​​the camera's field of view in the radar coordinate system determined based on the first position information; generating a target distribution result based on the positional relationship between the group structure of each target within the radar detection range and other targets; controlling the camera to perform target tracking based on the target distribution result; wherein the camera tracks all targets within the same group structure as a single target. This solution can improve the efficiency and stability of target tracking and reduce camera hardware consumption.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and in particular to a target tracking method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of technology, security systems are playing an increasingly important role in protecting people's lives and property. When detecting intrusion targets, radar-linked PTZ cameras (referred to as "radar-ball linkage") are often used to detect the location and behavior of intrusion targets in real time. However, traditional radar-ball linkage solutions have many problems when handling multi-target tracking, such as frequent target switching and redundant target tracking. This not only leads to poor multi-target tracking performance but also increases the hardware load on the PTZ camera due to frequent rotation, resulting in a higher failure rate. Summary of the Invention

[0003] This invention provides a target tracking method, apparatus, electronic device, and storage medium, which solves the problems of frequent target switching and low tracking efficiency in traditional multi-target tracking methods.

[0004] According to one aspect of the present invention, a target tracking method is provided, comprising:

[0005] In response to a target tracking event being triggered, determine the first position information of each target within the radar detection range;

[0006] For each target within the radar detection range, the group structure of the current target is determined based on the first position information of the current target, and the positional relationship between each other target within the radar detection range (excluding the current target) and the group structure of the current target is determined; wherein, the group structure is the projection area of ​​the camera's field of view in the radar coordinate system determined based on the first position information;

[0007] The target distribution result is generated based on the group structure of each target within the radar detection range and the positional relationship between the other targets;

[0008] The camera is controlled to track targets based on the target distribution results; wherein, the camera is controlled to track all targets within the same group structure as a whole.

[0009] According to another aspect of the present invention, a target tracking device is provided, comprising:

[0010] The first position information determination module is used to determine the first position information of each target within the radar detection range in response to a target tracking event being triggered.

[0011] The positional relationship determination module is used to determine the group structure of the current target based on the first position information of the current target for each target within the radar detection range, and to determine the positional relationship between each other target within the radar detection range (excluding the current target) and the group structure of the current target; wherein, the group structure is the projection area of ​​the camera's field of view in the radar coordinate system determined based on the first position information;

[0012] The target distribution result generation module is used to generate target distribution results based on the group structure of each target within the radar detection range and the positional relationship between the other targets;

[0013] The target tracking module is used to control the camera to track targets based on the target distribution results; wherein, the camera is controlled to track all targets within the same group structure as a whole.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the target tracking method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the target tracking method according to any embodiment of the present invention.

[0019] The target tracking scheme of this invention, in response to a target tracking event being triggered, determines the first position information of each target within the radar detection range; for each target within the radar detection range, it determines the group structure of the current target based on the first position information of the current target, and determines the positional relationship between each other target within the radar detection range (excluding the current target) and the group structure of the current target; wherein, the group structure is the projection area of ​​the camera's field of view in the radar coordinate system determined based on the first position information; it generates a target distribution result based on the positional relationship between the group structure of each target within the radar detection range and the other targets; and it controls the camera to perform target tracking based on the target distribution result; wherein, the camera is controlled to track all targets within the same group structure as a whole target. The technical solution provided by this invention not only solves the problems of frequent target switching and low tracking efficiency in traditional multi-target tracking methods, but also improves the efficiency and stability of target tracking and reduces camera hardware consumption.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a target tracking method provided in Embodiment 1 of the present invention;

[0023] Figure 2 A schematic diagram of the field of view of a camera is provided for an embodiment of the present invention;

[0024] Figure 3 This is a flowchart of a target tracking method provided in Embodiment 2 of the present invention;

[0025] Figure 4 A schematic diagram of a group structure provided in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of a target distribution result provided in an embodiment of the present invention;

[0027] Figure 6 This is a flowchart of a target tracking method provided in Embodiment 3 of the present invention;

[0028] Figure 7 This is a schematic diagram illustrating a method for predicting the position information of a target at the next moment based on IMM, as provided in an embodiment of the present invention.

[0029] Figure 8 This invention provides a schematic diagram illustrating the change in target distribution within a group structure based on IMM prediction, as provided in an embodiment of the invention.

[0030] Figure 9 This is a schematic diagram illustrating the changes in the distribution of stable targets within a group structure, as provided in an embodiment of the present invention.

[0031] Figure 10 This is a schematic diagram illustrating the change in the distribution of unstable targets within a group structure, provided by an embodiment of the present invention.

[0032] Figure 11 This is a schematic diagram of the structure of a target tracking device provided in Embodiment 4 of the present invention;

[0033] Figure 12 A schematic diagram of the structure of an electronic device for implementing the target tracking method of this embodiment of the invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] Example 1

[0037] Figure 1This is a flowchart illustrating a target tracking method according to Embodiment 1 of the present invention. This embodiment is applicable to situations involving target tracking. The method can be executed by a target tracking device, which can be implemented in hardware and / or software. The target tracking device can be configured within a target tracking system, which may include radar and a camera. Figure 1 As shown, the method includes:

[0038] S110, in response to a target tracking event being triggered, determine the first position information of each target within the radar detection range.

[0039] In this embodiment of the invention, when a target tracking command input by a user is received, it can be determined that a target tracking event has been triggered. In response to the triggering of the target tracking event, the first position information of each target detected by the radar within its detection range is acquired. The radar detection range may include one target or multiple targets; this embodiment of the invention does not limit the number of targets included within the radar detection range. As time progresses and targets move, the targets within the radar detection range change dynamically. The target may be a human body or a vehicle; this embodiment of the invention does not limit the type of target.

[0040] S120. For each target within the radar detection range, determine the group structure of the current target based on the first position information of the current target, and determine the positional relationship between each other target within the radar detection range (excluding the current target) and the group structure of the current target; wherein, the group structure is the projection area of ​​the camera's field of view in the radar coordinate system determined based on the first position information.

[0041] In this embodiment of the invention, when tracking targets, a radar-camera linkage is required to detect the position and behavior of intruding targets in real time. For each target within the radar detection range, the camera's field of view is determined based on the target's first position information. Within this field of view, the camera can capture a clear image of the target at the position corresponding to the first position information. The projection area of ​​the camera's field of view determined based on the first position information in the radar coordinate system is determined, and this projection area is used as the group structure of the current targets. It can be understood that the group structure of each target within the radar detection range can be determined in the above manner.

[0042] Optionally, for each target within the radar detection range, determining the group structure of the current targets based on the first position information of the current targets includes: for each target within the radar detection range, determining the camera's field of view based on the first position information of the current targets, and determining each boundary point of the camera's field of view; determining the group structure of the current targets based on the first position information of the current targets and each boundary point of the field of view. For example, for each target within the radar detection range, determining the camera's focal length and field of view angle based on the first position information of the current targets, and determining the camera's field of view based on the camera's focal length and field of view angle, wherein a longer focal length results in a smaller field of view angle (FOV), and a smaller field of view corresponding to the camera's field of view; a shorter focal length results in a larger field of view angle (FOV), and a larger field of view corresponding to the camera's field of view. For example, Figure 2 This is a schematic diagram of the camera's field of view provided in an embodiment of the present invention. For example... Figure 2 As shown, the camera's field of view (FOV) is a conical field of view defined by the camera's focal length and field of view (FOV). The FOV includes a horizontal and a vertical field of view, defining the width of the field of view, while the camera's focal length determines the degree of image magnification. The various FOV boundary points are determined, and the group structure of the current target is determined based on the first position information of the current target and these FOV boundary points.

[0043] For each target within the radar detection range, based on the first position information of all other targets within the radar detection range (excluding the current target), it is determined whether each other target falls within the group structure of the current target, and the positional relationship between each other target and the group structure of the current target is determined based on the determination result. The positional relationship between each other target and the group structure of the current target can include other targets being inside the group structure of the current target and other targets being outside the group structure of the current target. It can be understood that the positional relationship between the group structure of each target within the radar detection range and other targets can be determined using the above method.

[0044] S130. Generate target distribution results based on the group structure of each target within the radar detection range and the positional relationship between the other targets.

[0045] In this embodiment of the invention, based on the positional relationship between the group structure of each target within the radar detection range and other targets, the targets contained within each group structure within the radar detection range can be determined. This achieves grouping of all targets within the radar detection range (i.e., treating targets in the same group structure as the same group), thereby generating the target distribution result for all targets within the radar detection range. A target may fall into the group structure of multiple targets simultaneously. In this case, the target can be randomly assigned to any one of the multiple group structures it falls into, or a preset target partitioning strategy can be used to select one of the multiple group structures to which the target belongs.

[0046] S140. Control the camera to perform target tracking based on the target distribution results; wherein, control the camera to track all targets within the same group structure as a whole target.

[0047] In this embodiment of the invention, the camera linked to the radar tracks targets within the radar's detection range based on target distribution results. For example, the camera tracks all targets within the same group structure as a single target. Since multiple group structures exist, a pre-set tracking strategy can be used to switch between different group structures to cover all targets within the radar's detection range. For example, the camera can be controlled in a polling manner to sequentially track targets within each group structure involved in all targets within the radar's detection range for a preset duration.

[0048] The target tracking method of this invention, in response to a target tracking event being triggered, determines the first position information of each target within the radar detection range; for each target within the radar detection range, it determines the group structure of the current target based on the first position information of the current target, and determines the positional relationship between each other target within the radar detection range (excluding the current target) and the group structure of the current target; wherein, the group structure is the projection area of ​​the camera's field of view in the radar coordinate system determined based on the first position information; it generates a target distribution result based on the positional relationship between the group structure of each target within the radar detection range and the other targets; and it controls the camera to perform target tracking based on the target distribution result; wherein, the camera is controlled to track all targets within the same group structure as a whole target. The technical solution provided by this invention not only solves the problems of frequent target switching and low tracking efficiency in traditional multi-target tracking methods, but also improves the efficiency and stability of target tracking and reduces camera hardware consumption.

[0049] Example 2

[0050] Figure 3 A flowchart of a target tracking method provided in Embodiment 2 of the present invention is shown below. Figure 3 As shown, the method includes:

[0051] S310, in response to a target tracking event being triggered, determines the first position information of each target within the radar detection range.

[0052] S320. For each target within the radar detection range, determine the camera's field of view based on the first position information of the current target, and determine each boundary point of the camera's field of view.

[0053] S330. Construct the camera coordinate system and the radar coordinate system, and determine the transformation matrix between the camera coordinate system and the radar coordinate system.

[0054] In this embodiment of the invention, a camera coordinate system is constructed with the optical center of the camera (i.e., the position of the lens) as the origin, the horizontal rightward direction of the plane where the camera is located as the positive X-axis, the vertical downward direction of the plane where the camera is located as the positive Y-axis, and the camera's shooting direction as the positive Z-axis. A radar coordinate system is constructed with the radar's installation position as the origin, the horizontal forward direction along the ground as the positive X-axis, the horizontal rightward direction as the positive Y-axis, and the vertical upward direction of the radar's installation plane as the positive Z-axis (the definition may vary depending on the installation position). Since the camera and radar may have different installation angles, it is necessary to determine the rotation matrix between the camera coordinate system and the radar coordinate system. For example, the rotation matrix for transforming the camera coordinate system to the radar coordinate system can be composed of an X-axis rotation matrix, a Y-axis rotation matrix, and a Z-axis rotation matrix, wherein:

[0055] The X-axis rotation matrix is:

[0056]

[0057] The Y-axis rotation matrix is:

[0058]

[0059] The Z-axis rotation matrix is:

[0060]

[0061] Where α is the angle of rotation of the camera along its own line of sight axis, β is the angle of tilt of the camera up or down, and γ is the angle of rotation of the camera around the vertical axis (i.e., the Z-axis).

[0062] A rotation matrix P for transforming the camera coordinate system to the radar coordinate system is generated based on the X-axis rotation matrix, Y-axis rotation matrix, and Z-axis rotation matrix, where:

[0063] R = R z (γ)·R y (β)·R x(α) (4)

[0064] Since the camera and radar may be installed in different locations, it is necessary to determine the translation matrix between the camera coordinate system and the radar coordinate system. Assume the camera's position in the radar coordinate system is (x... cam ,y cam ,z cam Then, the translation matrix for transforming the camera coordinate system to the radar coordinate system is:

[0065]

[0066] For ease of calculation, homogeneous coordinates are used. By introducing a fourth coordinate (usually 1), the transformation matrix M for transforming the camera coordinate system to the radar coordinate system is defined:

[0067]

[0068] For any point P in the camera coordinate system cam =(x i ,y i ,z i Using the transformation matrix M, it can be transformed to the radar coordinate system:

[0069] P radar =M·P cam (7)

[0070] S340. Based on the first position information of the current target in the radar coordinate system and the transformation matrix, determine the second position information of the current target in the camera coordinate system.

[0071] In this embodiment of the invention, any target P within the radar detection range is selected. i Let P be the current objective. i The first position information in the radar coordinate system can be represented as P. i =(x i ,y i ,z i ,1), then by transforming the inverse matrix M of matrix M, we can obtain the matrix M. -1 The current target P can be obtained. i Position in the camera coordinate system (i.e., second position information) P cam =(x cam ,y cam ,z cam ,1), where the inverse matrix M -1 The definition is as follows:

[0072]

[0073] S350. In the camera coordinate system, based on the second position information and each of the visible field boundary points, determine the field of view area centered on the position point corresponding to the second position information, and determine each of the first field of view boundary points of the field of view area.

[0074] In this embodiment of the invention, assuming the camera's Z-axis is the line-of-sight direction, the camera's field of view, determined based on the first position information of the current target, can be calculated as follows:

[0075]

[0076]

[0077] Where, x left x right y top y bottom These represent the x-coordinates of the left, right, upper, and lower boundary vertices, respectively, within the visible field of view in the camera coordinate system. z represents the distance from the current target to the camera, determined based on the target's initial position information.

[0078] In actual camera target tracking, since the target's position on the z-axis only affects the calculation of the field of view depth, when calculating whether the target falls within a group structure, the z-axis has no impact on the group structure calculation when the field of view depth is known. Therefore, only the target's position on the x-axis and y-axis is considered. In the camera coordinate system, based on the second position information P... cam =(x cam ,y cam ,z cam The visible boundary points determined by formulas (9)-(10) are used to determine the boundary points of P. cam =(x cam ,y cam ,z cam 1) Define the central visual field region and determine the first visual field boundary points of the visual field region, wherein the first visual field boundary points are:

[0079] Top left corner view boundary point:

[0080]

[0081] Top right corner view boundary point:

[0082]

[0083] Bottom left corner view boundary point:

[0084]

[0085] Bottom right corner view boundary point:

[0086]

[0087] S360. Determine the second field-of-view boundary points of each of the first field-of-view boundary points in the radar coordinate system according to the transformation matrix, and use the boundary regions generated based on each of the second field-of-view boundary points as the group structure of the current target.

[0088] In this embodiment of the invention, the coordinates of each first field-of-view boundary point in the camera coordinate system are transformed to coordinates in the radar coordinate system using the transformation matrix M, thereby generating each second field-of-view boundary point in the radar coordinate system. It can be understood that this can be achieved using formula P. radar =M·P cam Set the top left corner view boundary point P left_top The upper right corner of the field of view boundary point P right_top P, the boundary point of the lower left field of view left_bottom and the lower right corner view boundary point P right_bottom Converted to the corresponding second field-of-view boundary points p' in the radar coordinate system left_top p' right_top p' left_bottom p' right_bottom Based on each second view boundary point p' left_top p' right_top p' left_bottom p' right_bottom The boundary region formed is used as the target P i The group structure. For example, Figure 4 This is a schematic diagram of a group structure provided in an embodiment of the present invention.

[0089] S370. Calculate the gating value between the current target's group structure and each of the other targets within the radar detection range, based on each vertex of the current target's group structure and the first position information of each other target within the radar detection range.

[0090] S380. Determine the positional relationship of the group structure of each of the other targets and the current target based on the gating value.

[0091] In this embodiment, for each target other than the current target within the radar detection range, a threshold value between the current target's group structure and other targets is calculated based on the first position information of the other targets and the vertices of the current target's group structure. The threshold value is then used to determine whether the other targets fall within the current target's group structure. For example, for any target P within the radar detection range... j Determine whether it is located at target P. i Within the group structure, that is, judging the target P jIs it located by p' left_top p' right_top p' left_bottom p' right_bottom Within the polygon formed, such as Figure 4 As shown, by p' left_top p' right_top p' left_bottom p' right_bottom The polygonal bounding box formed constitutes the target P i Gating of the group structure. In this embodiment of the invention, the included angle and discriminant method can be used to determine the target P. j Is it located at target P? i Within a group structure, for example, the target P can be calculated using the following formula. j With target P i Gating values ​​between group structures:

[0092]

[0093] in, Indicates target P i The q-th vertex of the group structure, where n represents the target P. i The number of vertices in the group structure; Representing vectors and dot product, Representing vectors Magnitude and vector The product of the modulus and length.

[0094] In this embodiment of the invention, a tolerance ε can be set, if |f (j) If |≤ε, then the target P can be determined. j Falling into target P i Within the group structure. It should be noted that the tolerance ε can be reasonably set according to different scenarios and the motion characteristics of the targets within the scenario. For example, targets within the radar detection range are generally in a fast-moving state, so the tolerance ε should be set to a small value to keep the number of targets within the group structure within a small range, thereby improving the accuracy of target tracking.

[0095] S390. Generate target distribution results based on the group structure of each target within the radar detection range and the positional relationship between the other targets.

[0096] For example, Figure 5 This is a schematic diagram of a target distribution result provided in an embodiment of the present invention.

[0097] S3100, Control the camera to perform target tracking based on the target distribution result; wherein, control the camera to track all targets within the same group structure as a whole target.

[0098] The target tracking method of this invention determines the group structure of each target within the radar detection range and calculates the gating value between each other target and its own group structure to determine whether other targets fall within their own group structure, thereby generating the target distribution result within the radar detection range. This allows the camera to track all targets within the same group structure as a whole, which not only solves the problems of frequent target switching and low tracking efficiency in traditional multi-target tracking methods, but also improves the efficiency and stability of target tracking and reduces camera hardware consumption.

[0099] Example 3

[0100] Figure 6 A flowchart of a target tracking method provided in Embodiment 3 of the present invention is shown below. Figure 6 As shown, the method includes:

[0101] S610, in response to a target tracking event being triggered, determines the first position information of each target within the radar detection range.

[0102] S620. For each target within the radar detection range, determine the group structure of the current target based on the first position information of the current target, and determine the positional relationship between each other target within the radar detection range (excluding the current target) and the group structure of the current target; wherein, the group structure is the projection area of ​​the camera's field of view in the radar coordinate system determined based on the first position information.

[0103] S630. Generate target distribution results based on the group structure of each target within the radar detection range and the positional relationship between the other targets.

[0104] S640. Control the camera to perform target tracking based on the target distribution result; wherein, control the camera to track all targets within the same group structure as a whole target.

[0105] S650. During the process of controlling the camera to track targets based on the target distribution results, the second position information of each target within the radar detection range at the next moment is predicted based on the interactive multi-model filtering algorithm.

[0106] Because the motion of targets within the same group is free, their motion state may change at any time. This can lead to situations where a target in the group jumps out of the group during tracking, or a new target that meets the group's threshold value is not included in the group. Therefore, dynamic multi-target prediction is needed to continuously update the group structure through the prediction output, thereby updating the target distribution within the radar detection range. The target's motion model is unknown, uncertain, or even changing. Traditional extended Kalman filters (EKFs) cannot provide sufficient prediction accuracy. While multi-model extended Kalman filters (MM-EKFs) can correct the prediction results using multiple models, making the prediction results closer to the actual measurement, they cannot handle situations where the target switches from one motion model to another in discrete time. Therefore, in this embodiment of the invention, the position of each target within the radar detection range at the next moment is predicted based on the Interactive Multiple Model Filtering (IMM) algorithm. IMM is an algorithm for target tracking in a multi-model environment. IMM allows switching between multiple motion models. By providing an initial correlation matrix for each model and continuously updating the correlation matrix based on the actual prediction results, and by weighting and fusing the prediction outputs of different models according to their correlations, the final prediction result is obtained. This provides a more accurate and stable target state estimate, and compared to traditional Extended Kalman Filter (EKF) or Multi-Model Extended Kalman Filter (MMEKF), it is more adaptable to complex tracking scenarios. For example, Figure 7 This is a schematic diagram illustrating a method for predicting the position information of a target at the next moment based on IMM, as provided in an embodiment of the present invention.

[0107] The process of predicting the second position information of each target within the radar detection range at the next moment based on the interactive multi-model filtering algorithm may include the following steps:

[0108] Step 1: Define the system model; given N different motion models, at any time t k The state transition equations and observation equations for each motion model are as follows:

[0109] State transition equation:

[0110]

[0111] Observation equation:

[0112]

[0113] in, This indicates that the i-th motion model is at time t. k The state vector; This represents the state transition matrix of the i-th motion model; This represents the observation matrix of the i-th motion model; The process noise is represented by a Gaussian distribution. The observation noise represents a Gaussian distribution.

[0114] Step 2: Set the initial state; that is, initialize the state vector of each motion model. State covariance matrix and model confidence Wherein, the initial state vector It can be directly based on time t k The state covariance matrix is ​​determined by radar detecting the target's state information (including velocity and position information). This can be determined based on radar measurement accuracy or noise level. For example, based on the radar detecting the target at time t... k The initial position (x0, y0) and initial velocity (v) x0 v y0 ), and the standard deviation of radar measurement error σ x σ v It can be set as follows:

[0115]

[0116] The initial confidence level for each model can be set based on prior knowledge or experience, and must satisfy the following:

[0117]

[0118] In IMM, the initial state vector of each motion model State covariance matrix To ensure comparability between motion models, consistency is crucial; therefore, a mixed initialization of the state vector and covariance matrix is ​​necessary. This can be achieved by mixing all motion models to initialize the state vector and state covariance matrix separately, using the following formula:

[0119]

[0120]

[0121] in: Let represent the initial confidence level of the j-th motion model. This represents the initial state vector of the j-th motion model.

[0122] Step 3: Calculate the state estimate and mixture covariance matrix of the mixture model; at any time t k Based on the model confidence level at the previous time step The transition probability π between models ij Calculate the mixed confidence level of the model interaction

[0123]

[0124] Where, π ij Let j represent the probability of switching from the j-th motion model to the i-th motion model. The switching probability satisfies Based on mixed confidence level Mixed state estimates can be calculated. and mixed covariance matrix in:

[0125]

[0126]

[0127] Step 4: Use a Kalman filter for state prediction; for any time t k For each motion model i, a Kalman filter is used to predict the state of the target, and the state prediction result is obtained. Covariance prediction results

[0128]

[0129]

[0130] Step 5: Use a Kalman filter for state update; for each motion model i, combine the state update at time t. k Radar detected the target's status information. The state prediction results are updated using a Kalman filter update formula, where...

[0131] The Kalman gain is:

[0132]

[0133] The formula for updating the state prediction results is:

[0134]

[0135] The covariance update formula is:

[0136]

[0137] Step 6: Update the confidence of the motion model; based on the time t k Radar detected the target's status information. Calculate the likelihood value for each motion model.

[0138]

[0139]

[0140] in, The innovation covariance matrix can be represented using the likelihood values ​​of the motion model. Update the confidence level of the corresponding motion model:

[0141]

[0142] Step 7: Calculate the weighted fusion state and covariance; based on the confidence level of each motion model. The state prediction results and covariance are weighted and fused to obtain the final state prediction result of the target based on the interactive multi-model filtering algorithm:

[0143]

[0144]

[0145] The final state prediction result of the target includes the target's position information, that is, the target's second position information at the next moment.

[0146] S660. Update the target distribution result based on the second location information, and control the camera to continue target tracking based on the updated target distribution result.

[0147] In this embodiment of the invention, since the second position information of each target within the radar detection range is different from its corresponding first position information, a new group structure can be determined for each target within the radar detection range based on the target's second position information, and the positional relationship between other targets and the newly determined group structure of the target can be re-evaluated, thereby updating the target distribution results.

[0148] Optionally, updating the target distribution result based on the second location information includes: updating the corresponding first location information based on the second location information of each target within the radar detection range, and returning to perform the following steps: for each target within the radar detection range, determining the group structure of the current target based on the first location information of the current target, so as to update the group structure of each target within the radar detection range; re-determining the positional relationship between each of the other targets and the updated group structure of the current target based on the second location information of each of the other targets within the radar detection range; and updating the target distribution result based on the re-determined positional relationship between the updated group structure of each target within the radar detection range and the other targets.

[0149] For example, the predicted target Pi The location point corresponding to the second location information at the next moment is used The representation is based on the location point. The position information (i.e., the second position information) is used to recalculate the camera's horizontal field of view when the target is at this position. Vertical field of view and depth of vision Then, according to formulas (11)-(14) and formula (7), the target P can be calculated. i The vertices of the group structure after predicting the second position information at the next moment. Thus, each vertex The boundary region formed is used as the target P i The group structure at the next moment. The group structure of each target within the radar detection range can be updated in the manner described above.

[0150] In this embodiment of the invention, each target within the radar detection range is traversed. For each other target within the radar detection range (excluding the current target), based on the second position information of the other targets and the vertices of the updated group structure of the current target, the gating values ​​between the other targets and the updated group structure of the current target are recalculated. The gating values ​​are then used to determine whether the other targets fall within the updated group structure of the current target. Based on the determination results, the positional relationship between each other target and the updated group structure of the current target is redefined. The target distribution result is then updated based on the redefined positional relationship between the updated group structure of each target within the radar detection range and other targets.

[0151] Optionally, before re-determining the positional relationship between each of the other targets and the updated group structure of the current target based on the second position information of each other target within the radar detection range, the method further includes: determining the fusion state covariance of at least two motion models involved in the interactive multi-model filtering algorithm for each target during the process of predicting the second position information of each target within the radar detection range at the next moment based on the interactive multi-model filtering algorithm; determining the positional uncertainty of the second position information of the corresponding target based on the fusion state covariance; and adjusting the updated group structure of the target based on the positional uncertainty. The advantage of this configuration is that the updated group structure of each target within the radar detection range can be determined more accurately.

[0152] In this embodiment of the invention, when predicting the second position information of each target within the radar detection range at the next moment based on the interactive multi-model filtering algorithm, N motion models are defined, and the fusion state covariance of the N motion models involved in the interactive multi-model filtering algorithm for each target is determined. For example, the covariance in formula (34) is... The fusion state covariance corresponds to the target. The position uncertainty of the second position information corresponding to the target is determined based on the fusion state covariance. The position uncertainty of each position component in the second position information can be determined using the following formula:

[0153]

[0154] Where, σ x σ represents the positional uncertainty of the x-axis component in the second positional information. y This indicates the positional uncertainty of the y-axis component in the second position information.

[0155] According to σ x σ y It can predict the target location Adjust each vertex of the group structure:

[0156]

[0157] Where α represents the scaling factor, which depends on the magnitude of the uncertainty. Indicates the predicted target location The i-th adjusted vertex of the group structure, Indicates the predicted target location The i-th vertex of the group structure before adjustment.

[0158] based on The adjusted vertices and target P of the group structure j The second location information (i.e., location point) at the next moment The location point can be calculated using the following formula. With the goal Gating values ​​between group structures:

[0159]

[0160] The location point determined by formula (37) With the goal The gating values ​​between group structures are used to determine the location points. Has the target been reached? Within the group structure, the positional relationship between other targets and the current target in the updated group structure can be determined. Figure 8 This is a schematic diagram illustrating the change in target distribution within a group structure based on IMM prediction, as provided in an embodiment of the present invention.

[0161] In this embodiment of the invention, at any time t k Time target P i The target set in the group structure is G.i At the predicted time t k+1 Target P i The target set in the group structure is For G i Any target P in j If its predicted target Still located in the set In the middle, it means at time t k At time t k+1 A target is considered stable within the group structure if it is stable; otherwise, it is considered unstable within the group structure and may escape during tracking. Stable targets cannot be added to the group structure of other targets; unstable targets are removed from the original group structure and allowed to join the group structure of other targets or be referred to as a separate group structure. For example, Figure 9 This is a schematic diagram illustrating the change in the distribution of stable targets within a group structure, as provided in an embodiment of the present invention. Figure 10 This is a schematic diagram illustrating the change in the distribution of unstable targets within a group structure, as provided in an embodiment of the present invention. Figure 10 As shown, at time t k+1 Target P3 escaped the group structure of target P1, but it can still be classified into the group structure of target P4, and its predicted position... Still in the predicted position Within the group structure, it is ultimately classified into the group structure of target P4 for tracking; while for target P5, it is predicted that after escaping from the group structure of target P1, it will not meet the gating requirements of any target's group structure, and therefore will be alone in its own group structure.

[0162] It should be noted that at time t k If an independent target P exists in a group structure j If the predicted time t k+1 Location Satisfying a certain objective P i The gating requirements of the group structure then mark the target P. j The target to be merged. At time t k+1 When, add it to target P i In the group structure.

[0163] In this embodiment of the invention, the camera is controlled to continue target tracking based on the updated target distribution results. During the target tracking process, the radar detects in real time whether a new target appears in the radar detection area. If so, it is determined whether the new target at the current moment meets the gating requirements of a target group structure. If it does, the new target is included in the target group structure; if it does not, the target is included in its own group structure.

[0164] The target tracking method provided in this invention determines the group structure of each target within the radar detection range and predicts the position information of each target at the next moment through an interactive multi-model filtering algorithm. The group structure is updated based on the prediction results, thereby constructing a stable group structure. The camera is then controlled to track all targets within the same group structure as a whole. This not only solves the problems of frequent target switching and low tracking efficiency in traditional multi-target tracking methods, but also improves the efficiency and stability of target tracking and reduces camera hardware consumption.

[0165] Example 4

[0166] Figure 11 This is a schematic diagram of a target tracking device provided in Embodiment 4 of the present invention. Figure 11 As shown, the device includes:

[0167] The first position information determination module 1110 is used to determine the first position information of each target within the radar detection range in response to a target tracking event being triggered.

[0168] The position relationship determination module 1120 is used to determine the group structure of the current target based on the first position information of the current target for each target within the radar detection range, and to determine the position relationship between each other target within the radar detection range (excluding the current target) and the group structure of the current target; wherein, the group structure is the projection area of ​​the camera's field of view in the radar coordinate system determined based on the first position information.

[0169] The target distribution result generation module 1130 is used to generate target distribution results based on the group structure of each target within the radar detection range and the positional relationship between the other targets;

[0170] The target tracking module 1140 is used to control the camera to track targets based on the target distribution results; wherein, the camera is controlled to track all targets in the same group structure as a whole target.

[0171] Optionally, the positional relationship determination module includes:

[0172] The visible field boundary point determination unit is used to determine the camera's visible field of view for each target within the radar detection range based on the first position information of the current target, and to determine each visible field boundary point of the camera's visible field of view.

[0173] A group structure determination unit is used to determine the group structure of the current target based on the first position information of the current target and each of the visible field boundary points.

[0174] Optionally, the group structure determining unit is used for:

[0175] Construct a camera coordinate system and a radar coordinate system, and determine the transformation matrix between the camera coordinate system and the radar coordinate system;

[0176] Based on the first position information of the current target in the radar coordinate system and the transformation matrix, the second position information of the current target in the camera coordinate system is determined;

[0177] In the camera coordinate system, based on the second position information and each of the visible field boundary points, the field of view area centered on the position point corresponding to the second position information is determined, and each of the first field of view boundary points of the field of view area is determined;

[0178] The transformation matrix determines the second field-of-view boundary points of each of the first field-of-view boundary points in the radar coordinate system, and the boundary regions generated based on each of the second field-of-view boundary points are used as the group structure of the current target.

[0179] Optionally, the positional relationship determination module is used for:

[0180] Based on each vertex of the current target's group structure and the first position information of each other target within the radar detection range (excluding the current target), calculate the gating value between the current target's group structure and each of the other targets.

[0181] The positional relationship of each of the other targets and the current target in the group structure is determined based on the gating value.

[0182] Optional, also includes:

[0183] The second position information prediction module is used to predict the second position information of each target within the radar detection range at the next moment based on an interactive multi-model filtering algorithm during the process of controlling the camera to track targets based on the target distribution results.

[0184] The target distribution result update module is used to update the target distribution result based on the second location information, and control the camera to continue target tracking based on the updated target distribution result.

[0185] Optionally, the target distribution result update module is used to:

[0186] The first position information is updated based on the second position information of each target within the radar detection range, and the process returns to perform the following: for each target within the radar detection range, the group structure of the current target is determined based on the first position information of the current target, so as to update the group structure of each target within the radar detection range.

[0187] Based on the second position information of each other target within the radar detection range, the positional relationship between each of the other targets and the updated group structure of the current target is re-determined;

[0188] The target distribution result is updated based on the positional relationship between the updated group structure of each target within the radar detection range and the other targets.

[0189] Optional, also includes:

[0190] The fusion state covariance determination module is used to determine the fusion state covariance of at least two motion models involved in the interactive multi-model filtering algorithm for each target before re-determining the positional relationship between each other target and the current target in the updated group structure based on the second position information of each other target in the radar detection range.

[0191] The position uncertainty determination module is used to determine the position uncertainty of the second position information of the corresponding target based on the fusion state covariance.

[0192] The group structure adjustment module is used to adjust the updated group structure based on the target corresponding to the positional uncertainty.

[0193] The target tracking device provided in the embodiments of the present invention can execute the target tracking method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0194] Example 5

[0195] Figure 12 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0196] like Figure 12As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0197] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0198] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as target tracking methods.

[0199] In some embodiments, the target tracking method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the target tracking method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the target tracking method by any other suitable means (e.g., by means of firmware).

[0200] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0201] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0202] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0204] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0205] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0206] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0207] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A target tracking method, characterized in that, include: In response to a target tracking event being triggered, determine the first position information of each target within the radar detection range; For each target within the radar detection range, the group structure of the current target is determined based on the first position information of the current target, and the positional relationship between each other target within the radar detection range (excluding the current target) and the group structure of the current target is determined; wherein, the group structure is the projection area of ​​the camera's field of view in the radar coordinate system determined based on the first position information; The target distribution result is generated based on the group structure of each target within the radar detection range and the positional relationship between the other targets; The camera is controlled to track targets based on the target distribution results; wherein, the camera is controlled to track all targets within the same group structure as a whole.

2. The method according to claim 1, characterized in that, For each target within the radar detection range, the group structure of the current targets is determined based on the first position information of the current targets, including: For each target within the radar detection range, the camera's field of view is determined based on the first position information of the current target, and each boundary point of the camera's field of view is determined. The group structure of the current target is determined based on the first location information of the current target and each of the visible field boundary points.

3. The method according to claim 2, characterized in that, Determining the group structure of the current target based on the first location information of the current target and each of the visible field boundary points includes: Construct a camera coordinate system and a radar coordinate system, and determine the transformation matrix between the camera coordinate system and the radar coordinate system; Based on the first position information of the current target in the radar coordinate system and the transformation matrix, the second position information of the current target in the camera coordinate system is determined; In the camera coordinate system, based on the second position information and each of the visible field boundary points, the field of view area centered on the position point corresponding to the second position information is determined, and each of the first field of view boundary points of the field of view area is determined; The transformation matrix determines the second field-of-view boundary points of each of the first field-of-view boundary points in the radar coordinate system, and the boundary regions generated based on each of the second field-of-view boundary points are used as the group structure of the current target.

4. The method according to claim 1, characterized in that, Determining the positional relationship between the current target and the group structure of all other targets within the radar detection range (excluding the current target) includes: Based on each vertex of the current target's group structure and the first position information of each other target within the radar detection range (excluding the current target), calculate the gating value between the current target's group structure and each of the other targets. The positional relationship of each of the other targets and the current target in the group structure is determined based on the gating value.

5. The method according to claim 1, characterized in that, The process of controlling the camera to track the target based on the target distribution results also includes: Based on the interactive multi-model filtering algorithm, the second position information of each target within the radar detection range at the next moment is predicted; The target distribution result is updated based on the second location information, and the camera is controlled to continue target tracking based on the updated target distribution result.

6. The method according to claim 5, characterized in that, Updating the target distribution result based on the second location information includes: The first position information is updated based on the second position information of each target within the radar detection range, and the process returns to perform the following: for each target within the radar detection range, the group structure of the current target is determined based on the first position information of the current target, so as to update the group structure of each target within the radar detection range. Based on the second position information of each other target within the radar detection range, the positional relationship between each of the other targets and the updated group structure of the current target is re-determined; The target distribution result is updated based on the positional relationship between the updated group structure of each target within the radar detection range and the other targets.

7. The method according to claim 6, characterized in that, Before re-determining the positional relationship between each of the other targets and the updated group structure of the current target based on the second position information of each other target within the radar detection range, the method further includes: In the process of predicting the second position information of each target within the radar detection range at the next moment based on the interactive multi-model filtering algorithm, the fusion state covariance of at least two motion models involved in the interactive multi-model filtering algorithm corresponding to each target is determined; The positional uncertainty of the second position information of the corresponding target is determined based on the fusion state covariance. The updated group structure is adjusted based on the target's location uncertainty.

8. A target tracking device, characterized in that, include: The first position information determination module is used to determine the first position information of each target within the radar detection range in response to a target tracking event being triggered. The positional relationship determination module is used to determine the group structure of the current target based on the first position information of the current target for each target within the radar detection range, and to determine the positional relationship between each other target within the radar detection range (excluding the current target) and the group structure of the current target; wherein, the group structure is the projection area of ​​the camera's field of view in the radar coordinate system determined based on the first position information; The target distribution result generation module is used to generate target distribution results based on the group structure of each target within the radar detection range and the positional relationship between the other targets; The target tracking module is used to control the camera to track targets based on the target distribution results; wherein, the camera is controlled to track all targets within the same group structure as a whole.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the target tracking method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the target tracking method according to any one of claims 1-7.

Images