Target tracking method and device based on ultra wide band and vision

Abstract

The invention discloses a target tracking method and device based on ultra wide band and vision fusion, and the method comprises the following steps: firstly, collecting ultra wide band signal data, and obtaining an ultra wide band positioning space position through ultra wide band deep learning; visual target tracking is realized by using an improved YOLOv8 model Bot-Sort algorithm, and a visual positioning space coordinate is obtained; a sub-filter of a federated Kalman filter is adopted to filter the ultra wide band and the visual positioning coordinates; calculating an information entropy by using the sub-filter state estimation covariance matrix and carrying out normalization processing to obtain a sub-filter normalization weight; and finally, realizing weighted fusion of sub-filters by using an adaptive federated Kalman filtering algorithm based on information entropy, obtaining global estimation of a target position, and realizing target tracking. The device comprises a data acquisition module, a deep learning module, a data processing module and a data publishing module. The method can effectively deal with the problems of target loss, target confusion, difficult target re-identification, poor dynamic tracking effect and the like in a complex scene, and improves the system stability.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a target tracking method and device based on ultra-wideband and vision. Background Art

[0002] Target tracking technology is a technology that continuously identifies and tracks the position and motion state of a target in spatial and temporal dimensions. It is widely used in many fields, such as smart transportation, unmanned driving, security monitoring, augmented reality, following robots, etc. The main task of target tracking is to determine the spatial position of the target in real time, while dealing with possible interference in the environment, such as target occlusion, posture changes, scene illumination changes, etc. In target tracking technology, visual tracking methods have always been one of the most widely used and studied technologies. It relies on image data obtained by the camera and tracks the target by analyzing the changes in the image sequence. However, when multiple targets overlap or intersect in vision, it is often difficult for the visual algorithm to distinguish these targets, resulting in tracking failure or confusion. When tracking fails, it is difficult to re-identify and continue to track the target. Ultra-wideband signals are often used for target positioning and tracking due to their strong penetration and anti-interference capabilities. Ultra-wideband target tracking and visual target tracking methods have good complementarity. Ultra-wideband target positioning data is used to identify the correct target when the target overlaps or intersects, the target is lost, or the tracking is confused during visual target tracking. However, the ultra-wideband positioning and tracking method is greatly affected by the multipath effect, especially in dynamic tracking. At present, there is no target tracking method and device based on ultra-wideband and vision that can effectively solve the problems of target loss, target confusion, difficulty in re-identifying targets, and poor dynamic tracking effect. Summary of the invention

[0003] In order to overcome the problems of target loss, target confusion, difficulty in re-identifying targets, poor dynamic tracking effect, etc. in existing target tracking methods and devices, the present invention provides a target tracking method and device based on ultra-wideband and vision, which can effectively deal with the problems of target loss, target confusion, difficulty in re-identifying targets, poor dynamic tracking effect, etc. in complex scenarios, and enhance system stability.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A target tracking method based on ultra-wideband and visual fusion, comprising the following steps:

[0006] Step 1) Use the ultra-wideband module to collect ultra-wideband data, pass the data to the trained ultra-wideband model through forward propagation, perform inference and predict the ultra-wideband tracking and positioning coordinates of the target;

[0007] Step 2) Use the camera module to collect image data, use the improved YOLOv8 model to detect the target, and combine the Bot-SORT tracking algorithm to track the target in consecutive frames, and finally calculate and output the visual tracking positioning coordinates of the target;

[0008] Step 3) using a sub-filter of a federated Kalman filter to filter the ultra-wideband tracking positioning coordinates and the visual tracking positioning coordinates respectively;

[0009] Step 4) calculating the sub-filter information entropy, and normalizing the calculated sub-filter information entropy to obtain the normalized weight of the sub-filter;

[0010] Step 5) The information entropy-based federated Kalman filter method is used to weightedly fuse the visual and ultra-wideband positioning coordinates to calculate the global position estimate of the tracked target.

[0011] Furthermore, in the step 1), the ultra-wideband model extracts the spatiotemporal features in the ultra-wideband signal through the convolutional long short-term memory network ConvLSTM, uses the attention U-net network Attention U-Net to receive the spatiotemporal features output from the convolutional long short-term memory network ConvLSTM, further enhances the feature representation through the attention mechanism, extracts multi-scale features through downsampling, and restores the spatial information of the feature map in combination with the upsampling process, and finally generates the target position prediction through the regression layer.

[0012] Furthermore, in the step 1), the ultra-wideband data includes ultra-wideband signal strength, base station ranging value, arrival phase difference, arrival time difference, x-axis coordinate position and y-axis coordinate position of the tracking target relative to the device.

[0013] Furthermore, in the step 2), the improved YOLOv8 model uses the large separable kernel attention mechanism LSKA to improve the spatial pyramid pool SPPF, enhance the multi-scale feature extraction capability and the long-distance dependency capture and adaptability capability; and uses the power-adjusted intersection-over-union (PIOU) loss function instead of the complete intersection-over-union (CIOU) loss function to improve the convergence speed of the model and the accuracy of the regression results.

[0014] Furthermore, in step 3), the sub-filter of the federated Kalman filter is used to filter the ultra-wideband tracking target position and the target position obtained by visual tracking to obtain the target ultra-wideband tracking position estimate and the visual tracking position estimate.

[0015] Furthermore, in step 4), the information entropy is calculated using the state estimation covariance matrix of the ultra-wideband tracking sub-filter and the visual tracking sub-filter:

[0016]

[0017] Among them, H i Respectively represent the information entropy of the i-th sub-filter, P i represents the state estimation covariance matrix of the i-th sub-filter, det(P i ) represent the determinant of the state estimation covariance matrix of the i-th sub-filter respectively.

[0018] Furthermore, in step 4), the sub-filter normalization weight calculation process is as follows:

[0019] The unnormalized weights are calculated by the inverse of the information entropy:

[0020]

[0021] Normalize the unnormalized weights:

[0022]

[0023] in, represents the sum of sub-filter information entropy, H i Respectively represent the information entropy of the i-th sub-filter, w i 、w j Both represent unnormalized weights, α i represents the normalized weight.

[0024] Furthermore, in step 5), the entropy-based federated Kalman filtering method uses the normalized weights obtained from the entropy of each sub-filter to adaptively adjust the weights of the sub-filters in the global estimation:

[0025]

[0026] P g =α uwb P uwb +α vision P vision

[0027] in, is the optimal state estimate after fusion, is the state estimate of the UWB tracking subfilter, is the state estimate of the visual tracking subfilter, α uwb and α vision are the normalized weights of the ultra-wideband tracking sub-filter and the visual tracking sub-filter, respectively, and P g is the fused global covariance matrix, P uwb is the covariance matrix of the ultra-wideband tracking sub-filter, P vision is the covariance matrix of the visual tracking sub-filter.

[0028] A target tracking device based on ultra-wideband and visual fusion, comprising:

[0029] Data acquisition module: including camera module, used to collect image information; ultra-wideband module, used to collect ultra-wideband signal data;

[0030] Deep learning module: including improved YOLOv8 model and ultra-wideband model, used for data processing, feature extraction, and positioning result reasoning of the collected data to calculate the position obtained by visual recognition of the target and the position obtained by ultra-wideband;

[0031] Data processing module, used for filtering visual and ultra-wideband positioning data and fusing positioning data;

[0032] The data publishing module is responsible for publishing the target tracking and positioning results.

[0033] The beneficial effects of the present invention are mainly manifested in:

[0034] A target tracking method and device based on ultra-wideband and vision is provided. The improved YOLOv8 model and Bot-SORT target tracking algorithm are used to reduce the probability of target loss and target confusion in visual target tracking; ultra-wideband target tracking based on deep learning is used to alleviate the positioning drift caused by signal multipath effect; by fusing the results of ultra-wideband and visual positioning technologies, the advantages of target positioning are complementary, and the robustness of target tracking can be maintained in challenging scenarios such as limited visual positioning, target overlap or dynamic tracking. Compared with existing target tracking methods, this method facilitates the fusion of ultra-wideband and visual sensor information, can effectively deal with problems such as target loss, target confusion, difficulty in re-identifying targets, and poor dynamic tracking effects in complex scenarios, and enhances the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the target tracking method based on ultra-wideband and vision of the present invention;

[0036] Figure 2 It is a flow chart of the federal Kalman filtering method based on information entropy of the present invention;

[0037] Figure 3 It is a schematic diagram of the LSKA-SPPF network structure of the present invention;

[0038] Figure 4 It is a module schematic diagram of the target tracking device based on ultra-wideband and vision of the present invention;

[0039] Figure 5 It is a comparison diagram of the positioning error effect of the present invention;

[0040] Figure 6This is a comparison diagram of the positioning standard deviation effect of the present invention;

[0041] Figure 7 This is a comparison diagram of the filtering effects of the federal Kalman filter based on information entropy of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below with reference to the accompanying drawings and specific implementations.

[0043] Reference Figure 1 and Figure 2 , a target tracking method based on ultra-wideband and vision, the method comprising the following steps:

[0044] Step 1) Use the ultra-wideband module to collect ultra-wideband data, pass the data to the trained ultra-wideband model through forward propagation, perform inference and predict the coordinates of the target.

[0045] Step 2) Use the camera module to collect image data, use the improved YOLOv8 model for target detection, and combine the Bot-SORT tracking algorithm to track the target in continuous frames, and finally calculate and output the tracking and positioning coordinates of the target.

[0046] Step 3) Use the sub-filters of the federated Kalman filter to filter the ultra-wideband tracking and positioning data and the visual tracking and positioning data respectively.

[0047] Step 4) Calculate the sub-filter information entropy and perform normalization processing to obtain the normalized weight of the sub-filter.

[0048] Step 5) Use the information entropy-based federated Kalman filter method to calculate the global position estimate of the tracked target.

[0049] Execute steps 1)-5) to obtain the target global position estimation, thereby realizing target tracking based on ultra-wideband and vision.

[0050] Example:

[0051] Taking the intelligent following ball bag car as a specific embodiment, pedestrians holding ultra-wideband signal transceivers are used as tracking targets. The target tracking system of the ball bag car relies on vision and ultra-wideband signals to maintain robust target tracking in target occlusion, target loss, and moving scenarios. The application method and steps of the present invention are further explained below.

[0052] The ultra-wideband module collects ultra-wideband data in real time and inputs the data into the trained ultra-wideband model to achieve target position tracking and prediction.

[0053] The input signal of the ultra-wideband deep learning model is X t :

[0054]

[0055] in, UWB signal strength, Base station ranging value, Arrival phase difference, arrival time difference, Coordinate x, Coordinate y.

[0056] ConvLSTM (convolutional long short-term memory network) is used to extract the spatiotemporal features in ultra-wideband signals. The state transfer equation of ConvLSTM is as follows:

[0057] Input Gate:

[0058] i t =σ(W xi *X t +W hi *H t-1 +W ci ×C t-1 +b i )

[0059] Forget Gate:

[0060] f t =σ(W xf *X t +W hf *H t-1 +W cf ×C t-1 +b f )

[0061] Cell status update:

[0062] C t =f t ×C t-1 +i t ×tanh(W xc *X t +W hc *H t-1 +b c )

[0063] Output Gate:

[0064] o t =σ(W xo *X t +W ho *H t-1 +W co ×C t +b o )

[0065] Hide status updates:

[0066] H t =o t ×tanh(C t )

[0067] Among them, i t is the input gate, f t Forget the Gate, t is the output gate, C t is the cell state, H t is the hidden state, X t is the current input, W xi , W xf , W xo , W xc is the weight matrix input to the gate state, W hi , W hf , W ho , W hc is the weight matrix from the previous hidden state to the gate state, W ci , W cf , W co is the weight matrix from cell state to gate state, b i , b f , b o , b c is the bias term, σ is the sigmoid activation function, and tanh is the hyperbolic tangent function.

[0068] In Attention U-Net, downsampling and upsampling of the U-Net structure are used to extract and reconstruct features, and the attention mechanism is combined to improve model performance. The steps of Attention U-Net to enhance feature representation are as follows:

[0069] Downsampling:

[0070] F down =MaxPoll(ReLu(Conv(X))

[0071] Upsampling:

[0072] F up =ReLu(ConvTranspose(F down )+F skip )

[0073] Attention weight calculation:

[0074]

[0075] Enhanced feature representation:

[0076] F att =α×F up

[0077] Position prediction output:

[0078]

[0079] Among them, F down is the feature generated by downsampling, MaxPoll is the maximum pooling layer, ReLu is the activation function, Conv is the convolution layer, X is the input and output, F up is the feature generated by upsampling, ConvTranspose is the deconvolution layer, and F skip is the encoder feature passed by the jump connection, α is the attention weight, softmax is the normalization function, Q is the query vector, K T is the key vector, d k is the dimension of the key vector, used for scaling, F att It is the feature enhanced by the attention mechanism. The target predicted position coordinates.

[0080] Image data is collected through the camera, and the image is preprocessed, including resizing, normalization, and color channel conversion, to ensure that it meets the input requirements of the improved YOLOv8 model. Then, it is input into the improved YOLOv8 model to achieve pedestrian target recognition. The model will output bounding boxes and category labels to indicate the detected targets and their locations. Next, NMS (non-maximum suppression) is applied to remove overlapping bounding boxes and retain the detection results with the highest confidence.

[0081] The improvements to the YOLOV8 model are as follows:

[0082] The LSKA mechanism is introduced into the SPPF structure to enhance the expressiveness of features and the effect of multi-scale feature fusion through the attention mechanism. The SPPF module is used to extract multi-scale features, but there may be problems of insufficient feature extraction when facing complex scenes or targets of different scales. By introducing LSKA, the SPPF's ability to extract multi-scale features and capture long-distance dependencies can be further enhanced. Figure 3 The schematic diagram of the LSKA-SPPF structure after LSKA is used to improve SPPF. First, SPPF is used for multi-scale pooling, and features of different receptive fields are extracted through multiple pooling layers. Then LSKA is applied to the pooling output, a large separable convolution kernel is used to expand the receptive field, and the attention mechanism is used to strengthen the focus on important feature areas.

[0083] The PIOU loss function is used to replace the loss function. PIOU optimizes the IoU calculation process by introducing a power adjustment mechanism, which improves the convergence speed of the model.

[0084]

[0085] Among them, A and B represent the predicted box and the real box respectively, |A∩B| represents their intersection area, and |A∪B| represents the union area. n is the power adjustment coefficient, which is used to adjust the impact of IoU on the loss.

[0086] With this improvement, PIOU not only takes into account the overlap of bounding boxes, but also further optimizes the gradient return during training through the power adjustment mechanism, thereby improving the regression performance and training efficiency of the model.

[0087] On this basis, the Bot-SORT target tracking algorithm is used to match and update the detection results of each frame. The Kalman filter is used to predict the position of the target in the next frame, and the Hungarian algorithm is used to maintain the consistency of the target identity. Finally, the coordinates of the center point of the bottom edge of the target bounding box in each frame are calculated. as the target location.

[0088] Target position obtained for UWB signal Apply ultra-wideband tracking sub-filter to filter the target position obtained by visual signal Apply the visual tracking sub-filter for filtering.

[0089] Ultra-wideband tracking sub-filter update steps:

[0090] Status prediction:

[0091]

[0092] Kalman gain update:

[0093]

[0094] Status Update:

[0095]

[0096] Covariance matrix update:

[0097]

[0098] in, is the predicted state vector at time k, F k is the state transfer matrix, B k is the control input matrix, U k is the control input, W k is the process noise, is the Kalman gain, is the forecast error covariance, H k is the observation matrix, R k is the observation noise covariance, is the observation vector and I is the identity matrix.

[0099] Visual tracking sub-filter update steps:

[0100] Status prediction:

[0101]

[0102] Kalman gain update:

[0103]

[0104] Status Update:

[0105]

[0106] Covariance matrix update:

[0107]

[0108] in, is the predicted state vector at time k, F k is the state transfer matrix, B k is the control input matrix, U k is the control input, W k is the process noise, is the Kalman gain, is the forecast error covariance, H k is the observation matrix, R k is the observation noise covariance, is the observation vector and I is the identity matrix.

[0109] The information entropy is calculated based on the state estimate covariance matrix of the sub-filters.

[0110]

[0111] Among them, H i Respectively represent the information entropy of the i-th sub-filter, P i represents the state estimation covariance matrix of the i-th sub-filter, det(P i ) represent the determinant of the state estimation covariance matrix of the i-th sub-filter respectively.

[0112] The unnormalized weight w is calculated by the inverse of the information entropy i :

[0113]

[0114] Normalize the unnormalized weights to get the normalized weight α i :

[0115]

[0116] The weighted fusion of state estimation and covariance matrix are completed according to the weights of each sub-filter obtained by information entropy:

[0117]

[0118] P g =α uwb P uwb +α vision P vision

[0119] in, is the fused global state estimate, is the state estimate of the UWB tracking subfilter, is the state estimate of the visual tracking subfilter, α uwb and α vision are the normalized weights of the ultra-wideband tracking sub-filter and the visual tracking sub-filter, respectively, and P g is the fused global covariance matrix, P uwb is the covariance matrix of the ultra-wideband tracking sub-filter, P vision is the covariance matrix of the visual tracking sub-filter.

[0120] Correspondingly, this example also provides an ultra-wideband and vision-based target tracking device for executing the above method.

[0121] Figure 4 The schematic diagram of the device module is shown in the figure. As shown in the figure, the target tracking device based on ultra-wideband and vision includes: a camera module for collecting image information; an ultra-wideband module for collecting ultra-wideband data; a deep learning module for processing the collected data, extracting features, and reasoning about the positioning results to calculate the position obtained by visual recognition of the target and the position obtained by ultra-wideband; a data processing module for filtering the visual and ultra-wideband positioning data and fusing the positioning data; and a data publishing module for publishing the target tracking and positioning results.

[0122] The positioning error variation trend of the present invention is as follows Figure 5 As shown in the figure, more than 95% of the time after data fusion can be controlled below 0.5 meters; the positioning standard deviation change trend of the present invention is as follows Figure 6 As shown in Figure 2, after data fusion, the accuracy can be controlled below 0.15 meters for more than 95% of the time. The effect comparison of the federated Kalman filter based on information entropy is as follows: Figure 7 As shown in Figure 2, the fused tracking trajectory is closer to the true trajectory.

[0123] Through the above method and device, target tracking based on ultra-wideband and vision is realized, and problems such as target loss, target confusion, difficulty in re-identifying targets, and poor dynamic tracking effect are effectively handled.

Claims

1. A target tracking method based on ultra-wideband and visual fusion, characterized in that: The following steps are involved: Step 1) Use the ultra-wideband module to collect ultra-wideband data, pass the data to the trained ultra-wideband model through forward propagation, perform inference and predict the ultra-wideband tracking and positioning coordinates of the target; Step 2) Use the camera module to collect image data, use the improved YOLOv8 model to detect the target, and combine the Bot-SORT tracking algorithm to track the target in consecutive frames, and finally calculate and output the visual tracking positioning coordinates of the target; Step 3) using a sub-filter of a federated Kalman filter to filter the ultra-wideband tracking positioning coordinates and the visual tracking positioning coordinates respectively; Step 4) calculating the sub-filter information entropy, and normalizing the calculated sub-filter information entropy to obtain the normalized weight of the sub-filter; Step 5) The information entropy-based federated Kalman filter method is used to weightedly fuse the visual and ultra-wideband positioning coordinates to calculate the global position estimate of the tracked target.

2. The target tracking method based on ultra-wideband and visual fusion as claimed in claim 1, characterized in that: In the step 1), the ultra-wideband model extracts the spatiotemporal features in the ultra-wideband signal through the convolutional long short-term memory network ConvLSTM, uses the attention U-net network Attention U-Net to receive the spatiotemporal features output from the convolutional long short-term memory network ConvLSTM, further enhances the feature representation through the attention mechanism, extracts multi-scale features through downsampling, and restores the spatial information of the feature map in combination with the upsampling process, and finally generates the target position prediction through the regression layer.

3. The target tracking method based on ultra-wideband and visual fusion as claimed in claim 1, characterized in that: In the step 1), the ultra-wideband data includes ultra-wideband signal strength, base station ranging value, arrival phase difference, arrival time difference, x-axis coordinate position and y-axis coordinate position of the tracking target relative to the device.

4. The target tracking method based on ultra-wideband and visual fusion as claimed in claim 1, characterized in that: In the step 2), the improved YOLOv8 model uses the large separable kernel attention mechanism LSKA to improve the spatial pyramid pool SPPF, enhances the multi-scale feature extraction capability and the long-distance dependency capture and adaptability capability; and uses the power-adjusted intersection-over-union (PIOU) loss function instead of the complete intersection-over-union (CIOU) loss function to improve the convergence speed of the model and the accuracy of the regression results.

5. The target tracking method based on ultra-wideband and visual fusion as claimed in claim 1, characterized in that: In the step 3), the sub-filter of the federated Kalman filter is used to filter the ultra-wideband tracking target position and the target position obtained by visual tracking to obtain the target ultra-wideband tracking position estimate and the visual tracking position estimate.

6. The target tracking method based on ultra-wideband and visual fusion as claimed in claim 1, characterized in that: In step 4), the information entropy is calculated using the state estimation covariance matrix of the ultra-wideband tracking sub-filter and the visual tracking sub-filter: Among them, H i Respectively represent the information entropy of the i-th sub-filter, P i represents the state estimation covariance matrix of the i-th sub-filter, det(P i ) represent the determinant of the state estimation covariance matrix of the i-th sub-filter respectively.

7. The target tracking method based on ultra-wideband and visual fusion as claimed in claim 1, characterized in that: In step 4), the sub-filter normalization weight calculation process is as follows: The unnormalized weights are calculated by the inverse of the information entropy: Normalize the unnormalized weights: in, represents the sum of sub-filter information entropy, H i Respectively represent the information entropy of the i-th sub-filter, w i 、w j Both represent unnormalized weights, α i represents the normalized weight.

8. The target tracking method based on ultra-wideband and visual fusion as claimed in claim 1, characterized in that: In the step 5), the entropy-based federated Kalman filtering method uses the normalized weights obtained from the entropy of each sub-filter to adaptively adjust the weights of the sub-filters in the global estimation: P g =α uwb P uwb +α vision P vision in, is the optimal state estimate after fusion, is the state estimate of the UWB tracking subfilter, is the state estimate of the visual tracking subfilter, α uwb and α vision are the normalized weights of the ultra-wideband tracking sub-filter and the visual tracking sub-filter, respectively, and P g is the fused global covariance matrix, P uwb is the covariance matrix of the ultra-wideband tracking sub-filter, P vision is the covariance matrix of the visual tracking sub-filter.

9. A target tracking device based on ultra-wideband and visual fusion, characterized in that: include: Data acquisition module: including camera module, used to collect image information; Ultra-wideband module, used to collect ultra-wideband signal data; Deep learning module: including improved YOLOv8 model and ultra-wideband model, used for data processing, feature extraction, and positioning result reasoning of the collected data to calculate the position obtained by visual recognition of the target and the position obtained by ultra-wideband; Data processing module, used for filtering visual and ultra-wideband positioning data and fusing positioning data; The data publishing module is responsible for publishing the target tracking and positioning results.

Images