A target positioning and tracking method and system using PTZ camera and radar collaboration
Through the multimodal feature fusion and advanced algorithms that coordinate with PTZ cameras and radar, the stability and accuracy of target positioning and tracking of PTZ cameras in complex environments is solved, and efficient and accurate target recognition and tracking are achieved.
Patent Information
- Application Number
- CN202411027024.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-30
AI Technical Summary
The existing PTZ cameras have limited stability and accuracy of target positioning and tracking under the influence of factors such as excessive distance, target occlusion or changes in the lighting environment.
Using the target positioning and tracking method of coordinated PTZ camera and radar, the target features are extracted and digital fingerprints are generated by obtaining image data and radar echo signal data, and the target features are extracted and digital fingerprints are generated, and the PTZ cameras are guided to track, using multimodal feature fusion and advanced Algorithms such as adaptive region growth, nonlinear Fourier transform, dynamic weighted tensor decomposition, fuzzy logic reasoning and reinforcement learning improve the accuracy and stability of positioning and tracking.
It realizes efficient and accurate target positioning and tracking of PTZ cameras in complex environments, improves the robustness and adaptability of the system, enhances the efficiency and accuracy of target recognition and feature extraction, and is suitable for complex monitoring scenarios.
Smart Images

Figure CN118549924B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of target tracking and discloses a target positioning and tracking method and system in which a PTZ camera and a radar are coordinated. Background Art
[0002] PTZ (Pan-Tilt-Zoom) cameras have become an important device in security monitoring systems because of their all-round mobility, lens control functions and wide application scenarios. PTZ cameras track and locate targets by controlling the pan, tilt and zoom functions of the camera. Currently, the positioning and tracking technology of PTZ cameras has been widely used in security monitoring, intelligent transportation and sports events. However, due to factors such as long distances, obstructed targets and frequent changes in lighting environments, the stability and accuracy of target positioning and tracking functions that rely solely on PTZ cameras are affected.
[0003] Therefore, further research and innovation are needed to solve the above problems existing in the prior art. Summary of the invention
[0004] The purpose of the invention is to provide a target positioning and tracking method and system that cooperates with a PTZ camera and a radar to solve the above-mentioned problems existing in the prior art.
[0005] Technical solution, a target positioning and tracking method using a PTZ camera and a radar, comprising the following steps:
[0006] S1, obtaining PTZ camera image data and radar echo signal data, processing the PTZ camera image data and radar echo signal data, extracting target features and generating a digital fingerprint of the target;
[0007] S2, collect multi-source data, and calculate the precise three-dimensional coordinates of the target based on the digital fingerprint and multi-source data of the target; the multi-source data includes radar distance, azimuth data and current parameters of PTZ camera;
[0008] S3, based on the precise three-dimensional coordinates of the target, calculate the PTZ camera control parameters;
[0009] S4. Based on the PTZ camera control parameters, guide the PTZ camera to track the target.
[0010] According to one aspect of the present application, step S1 further comprises:
[0011] S11, obtaining PTZ camera image data and radar echo signal data;
[0012] S12, using a multi-scale adaptive region growing algorithm on the PTZ camera image data to generate a visual feature vector;
[0013] S13, using an adaptive piecewise nonlinear Fourier transform algorithm on the radar echo signal data to generate a radar feature vector;
[0014] S14, using a dynamic weighted tensor decomposition algorithm to fuse the visual feature vector and the radar feature vector to generate a comprehensive feature vector;
[0015] S15. Based on the comprehensive feature vector, an adaptive local sensitive hashing algorithm is used to construct the digital fingerprint of the target.
[0016] According to one aspect of the present application, step S2 is further:
[0017] S21, collect multi-source data, including radar distance, azimuth data and current parameters of PTZ camera;
[0018] S22, based on the digital fingerprint of the target and the current parameters of the PTZ camera, an adaptive fuzzy logic inference algorithm is used to calculate the initial position of the target in the camera coordinate system;
[0019] S23, based on the initial position, radar distance and azimuth data, a multi-level particle swarm optimization Kalman filter algorithm is used to estimate the precise three-dimensional coordinates of the target;
[0020] S24, using a dynamic quaternion interpolation algorithm to convert the precise three-dimensional coordinates of the target from the global coordinate system to the local coordinate system of the PTZ camera to obtain the local coordinates;
[0021] S25. Based on the local coordinates, the theoretical position of the target on the image plane is calculated through an adaptive perspective n-dimensional projection model.
[0022] According to one aspect of the present application, step S3 is further:
[0023] S31, based on the precise three-dimensional coordinates of the target and the theoretical position of the target on the image plane, the hybrid artificial potential field-neural dynamics algorithm is used to calculate the initial PTZ camera control parameters;
[0024] S32, based on the precise three-dimensional coordinates of the target and the theoretical position of the target on the image plane, use the fractal time series prediction model to predict the short-term future position of the target;
[0025] S33, calculating predicted optimal PTZ camera control parameters based on the predicted short-term future position;
[0026] S34. Based on the initial PTZ camera control parameters and the optimal PTZ camera control parameters, a fuzzy cognitive graph-reinforcement learning hybrid controller is used to generate a smooth PTZ control instruction sequence.
[0027] According to one aspect of the present application, step S4 further includes instructing a predetermined number of PTZ cameras to perform adaptive multi-target tracking, specifically:
[0028] S41, obtaining the digital fingerprint and precise three-dimensional coordinates of the predetermined target, and using the dynamic graph matching-spectral clustering algorithm to build a multi-target tracking queue;
[0029] S42, based on the multi-target tracking queue, using an adaptive multi-target differential evolution algorithm, assigning tracking tasks to predetermined PTZ cameras to obtain a camera task assignment solution;
[0030] S43, generating a comprehensive control instruction set based on the PTZ camera control parameters of each target in the camera task allocation scheme;
[0031] S44, based on the comprehensive control instruction set, instructing a predetermined PTZ camera to perform adaptive multi-target tracking.
[0032] According to one aspect of the present application, step S34 is further:
[0033] S341, constructing a fuzzy cognitive map based on the initial PTZ camera control parameters and the optimal PTZ camera control parameters;
[0034] S342, constructing a Q learning model based on the fuzzy cognitive graph, wherein the state space of the Q learning model includes initial PTZ camera control parameters and optimal PTZ camera control parameters;
[0035] S343, optimizing the action selection strategy of the Q learning model through Monte Carlo tree search to obtain a preliminary control sequence;
[0036] S344: Use a spline smoothing algorithm to adjust the preliminary control sequence to generate a final PTZ control instruction sequence.
[0037] According to one aspect of the present application, the precise three-dimensional coordinates of the target in step S2 are also monitored in real time. When it is detected that the target is occluded or lost, an adaptive attention mechanism-Twin network algorithm is used to resume tracking of the target.
[0038] According to one aspect of the present application, step S13 is further:
[0039] S131, preprocessing the radar echo signal data to remove background noise and fixed clutter to obtain a clean signal;
[0040] S132, segmenting the clean signal to obtain a signal segment set with adaptive length;
[0041] S133, based on the signal segment set, using nonlinear phase function and improved fast Fourier transform to obtain time-frequency representation;
[0042] S134, based on the time-frequency representation, using the peak clustering-tracking algorithm to calculate and obtain the spectrum peak tracking result;
[0043] S135. Based on the spectrum peak tracking result, estimate the distance, speed and direction information of the target and generate a radar feature vector.
[0044] According to one aspect of the present application, step S31 is further:
[0045] S311. Based on the precise three-dimensional coordinates of the target, an artificial potential field centered on the theoretical position of the target on the image plane is constructed, and the center is defined as the global minimum point;
[0046] S312, constructing a neural dynamics model, mapping the artificial potential field to the energy function of the neural dynamics model, and forming a hybrid optimization model;
[0047] S313, using an improved Hopfield network to solve the hybrid optimization model to obtain an estimated value of an initial PTZ camera control parameter;
[0048] S314: fine-tune the initial PTZ camera control parameter estimation value using a dynamic annealing strategy to obtain optimized initial PTZ camera control parameters.
[0049] A target positioning and tracking system using a PTZ camera and a radar, comprising:
[0050] at least one processor; and,
[0051] a memory communicatively connected to at least one of the processors; wherein,
[0052] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the target positioning and tracking method of the PTZ camera and radar collaboration described in any of the above technical solutions.
[0053] Beneficial effect: The present invention captures target features comprehensively and accurately through the extraction and fusion of multimodal features, thereby obtaining accurate PTZ camera control parameter values, and improving the stability and accuracy of PTZ camera positioning and tracking of targets. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is a flow chart of the present invention.
[0055] Figure 2 This is a flow chart of step S1 of the present invention.
[0056] Figure 3 This is a flow chart of step S2 of the present invention.
[0057] Figure 4 This is a flow chart of step S3 of the present invention.
[0058] Figure 5 This is a flow chart of step S4 of the present invention.
[0059] Figure 6 The figure is a flowchart of a program according to an embodiment of the present invention. DETAILED DESCRIPTION
[0060] like Figure 1 As shown, this application proposes a target positioning and tracking method that cooperates with a PTZ camera and a radar, and the specific steps are as follows:
[0061] S1, obtaining PTZ camera image data and radar echo signal data, processing the PTZ camera image data and radar echo signal data, extracting target features and generating a digital fingerprint of the target;
[0062] S2, collect multi-source data, and calculate the precise three-dimensional coordinates of the target based on the digital fingerprint and multi-source data of the target; the multi-source data includes radar distance, azimuth data and current parameters of PTZ camera;
[0063] S3, based on the precise three-dimensional coordinates of the target, calculate the PTZ camera control parameters;
[0064] S4. Based on the PTZ camera control parameters, the PTZ camera is guided to track the target. In other words, the PTZ camera control parameters are sent to the PTZ camera, and the PTZ camera responds to the control parameters to achieve target tracking.
[0065] This embodiment realizes the fusion of multi-source data by combining the image data of the PTZ camera and the echo signal data of the radar. The PTZ camera provides high-resolution image information, while the radar provides accurate distance and azimuth information. The fusion of multi-source data improves the accuracy of target detection and positioning, and reduces the error that may be caused by a single sensor; by processing the image data of the PTZ camera and the echo signal data of the radar, the target features are extracted and the digital fingerprint of the target is generated, so that different targets are effectively identified and distinguished, and the robustness and reliability of the system are enhanced; based on the digital fingerprint and multi-source data of the target, the precise three-dimensional coordinates of the target are calculated, which can provide the accurate position of the target in space, which is conducive to accurate target tracking and positioning; based on the precise three-dimensional coordinates of the target, the control parameters of the PTZ camera are calculated. The control parameters are used to guide the PTZ camera to track the target, ensuring that the camera can track the target in real time and keep the target in the field of view; according to the control parameters of the PTZ camera and the current PTZ value of the PTZ camera, the PTZ camera is controlled to adjust from the current state to the specified state, and when the PTZ camera is adjusted to the specified state, the PTZ camera is controlled to collect image data of the target object. Guide the PTZ camera to track the target, and by dynamically adjusting the camera's position and focal length, track the target in real time, adapt to the movement and change of the target, and improve the effectiveness and flexibility of monitoring.
[0066] This embodiment achieves efficient and reliable target tracking through multi-source data fusion and precise target positioning, which not only improves the accuracy of target detection and positioning, but also enhances the real-time and adaptability of the system, and is suitable for various complex monitoring scenarios.
[0067] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:
[0068] S11, obtaining PTZ camera image data and radar echo signal data;
[0069] S12, using a multi-scale adaptive region growing algorithm on the PTZ camera image data to generate a visual feature vector;
[0070] S13, using an adaptive piecewise nonlinear Fourier transform algorithm on the radar echo signal data to generate a radar feature vector;
[0071] S14, using a dynamic weighted tensor decomposition algorithm to fuse the visual feature vector and the radar feature vector to generate a comprehensive feature vector;
[0072] S15. Based on the comprehensive feature vector, an adaptive local sensitive hashing algorithm is used to construct the digital fingerprint of the target.
[0073] This embodiment obtains high-resolution image data of a PTZ camera, applies a multi-scale adaptive region growing algorithm, extracts target contours and color features, and generates a visual feature vector; receives radar echo signal data, uses an adaptive piecewise nonlinear Fourier transform algorithm, performs spectrum analysis, and generates a radar feature vector; fuses the visual feature vector and the radar feature vector, and uses a dynamic weighted tensor decomposition algorithm to generate a comprehensive feature vector; based on the comprehensive feature vector, an adaptive local sensitive hashing algorithm is applied to create a unique digital fingerprint of the target.
[0074] According to one aspect of the present application, step S13 is further:
[0075] S131, preprocessing the radar echo signal data to remove background noise and fixed clutter to obtain a clean signal;
[0076] S132, segmenting the clean signal to obtain a signal segment set with adaptive length;
[0077] S133, based on the signal segment set, using nonlinear phase function and improved fast Fourier transform to obtain time-frequency representation;
[0078] S134, based on the time-frequency representation, using the peak clustering-tracking algorithm to calculate and obtain the spectrum peak tracking result;
[0079] S135. Based on the spectrum peak tracking result, estimate the distance, speed and direction information of the target and generate a radar feature vector.
[0080] In a specific embodiment, high-resolution image data D1 of a PTZ camera is first received, and a novel multi-scale adaptive region growing algorithm is applied to extract target contours and color features, specifically: image data D1 is converted into L*a*b* color space to obtain image D1 with enhanced color distinction. lab ; For image D1 lab Generate a 5-layer Gaussian pyramid and get a multi-scale image set D1 pyramid ; In the multi-scale image set D1 pyramid The improved region growing algorithm is performed on each layer to obtain the preliminary segmentation result D1 seg ; For the initial segmentation result D1 seg Apply morphological operations to obtain the segmentation map D1 with smooth boundaries. smooth ; From the segmentation map D1 smooth The color histogram and shape descriptor of each segmented region are extracted to generate a visual feature vector V1.
[0081] The radar echo signal data D2 is obtained, and the adaptive piecewise nonlinear Fourier transform (AS-NLFT) algorithm is applied to perform spectrum analysis. Specifically, the echo signal data D2 is preprocessed to remove background noise and fixed clutter to obtain a clean signal D2. clean ; Clean signal D2 clean Segmentation to obtain a signal segment set D2 of adaptive length segments ; For signal segment set D2 segments Applying nonlinear phase function and improved fast Fourier transform, we get the time-frequency representation D2 tf ; For time-frequency representation D2 tf Apply the peak clustering-tracking algorithm to obtain the spectrum peak tracking result D2 peaks ; Based on D2 peaks , estimate the distance, speed and direction information of the target, and generate the radar feature vector R1.
[0082] The visual feature vector V1 and the radar feature vector R1 are fused, and a dynamic weighted tensor decomposition algorithm is used, specifically: a third-order tensor T1 is constructed, and the dimensions are V1, R1 and time series respectively; the improved Tucker decomposition is applied to the third-order tensor T1 to obtain the core tensor C1 and the factor matrix set F1; based on the core tensor C1 and the factor matrix set F1, the main components are extracted to form a low-dimensional comprehensive feature representation F2; a nonlinear mapping function is applied to the comprehensive feature representation F2 to generate the final comprehensive feature vector F3.
[0083] Based on the comprehensive feature vector F3, the adaptive local sensitive hashing (ALSH) algorithm is applied to generate a unique digital fingerprint of the target, specifically: according to the distribution of the comprehensive feature vector F3, a hash function family H1 is dynamically constructed; the comprehensive feature vector F3 is multiple-hashed using the hash function family H1 to obtain an initial hash result H2; error correction coding is applied to the initial hash result H2 to obtain a robust hash value H3; the hash value H3 is combined with the target's position, speed, shape and timestamp information to generate the final digital fingerprint DF1; the digital fingerprint DF1 is stored in the fingerprint index structure IS1 to support subsequent similarity search and target re-identification.
[0084] This embodiment achieves efficient and accurate target recognition and feature extraction by innovatively combining a multi-scale adaptive region growing algorithm, an adaptive piecewise nonlinear Fourier transform algorithm, a dynamic weighted tensor decomposition algorithm, and an adaptive local sensitive hashing algorithm. First, when processing PTZ camera image data, the multi-scale adaptive region growing algorithm effectively copes with the diversity of target sizes and shapes in complex scenes by performing adaptive threshold segmentation at different scales. The algorithm improves the accuracy of target contour extraction by dynamically adjusting the growth threshold and introducing shape prior constraints, especially in cases of complex backgrounds and changing lighting conditions. This adaptability enables the system to maintain stable performance in different environments and enhances the robustness of target recognition.
[0085] Secondly, the adaptive piecewise nonlinear Fourier transform algorithm effectively improves the resolution and anti-interference ability of spectrum analysis by adaptively segmenting the signal and applying nonlinear phase functions when processing radar echo signals. This method enables the system to more accurately extract the distance, speed and azimuth information of the target, and maintain a high degree of accuracy even in complex environments with multiple targets and low signal-to-noise ratio. In particular, the algorithm shows excellent performance when dealing with fast-moving or close-range targets, laying a solid foundation for subsequent target tracking. The dynamic weighted tensor decomposition algorithm achieves adaptive weight allocation for different modal data by constructing a third-order tensor and introducing an attention mechanism when fusing visual and radar features. This method not only effectively integrates the advantages of visual and radar information, but also dynamically adjusts the feature weights according to the complexity of the current scene, so as to obtain the optimal feature representation under different environmental conditions. This dynamic fusion strategy improves the adaptability and accuracy of the system in various complex scenarios.
[0086] Finally, the adaptive local sensitive hashing algorithm achieves efficient and robust feature encoding by dynamically constructing a hash function family and introducing a feature selection mechanism based on information entropy when generating a unique digital fingerprint of a target. This method not only reduces the storage space and comparison time of features, but also maintains the similarity relationship in the original feature space. Especially when processing large-scale target databases, the algorithm exhibits excellent retrieval efficiency and accuracy, enabling the system to complete target identification and matching within milliseconds.
[0087] This embodiment improves the efficiency, accuracy and robustness of target recognition and feature extraction. Through the deep fusion and efficient encoding of multimodal data, the system can quickly and accurately identify and characterize targets in various complex environments, laying a solid foundation for subsequent target positioning and tracking.
[0088] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:
[0089] S21, collect multi-source data, including radar distance, azimuth data and current parameters of PTZ camera;
[0090] S22, based on the digital fingerprint of the target and the current parameters of the PTZ camera, an adaptive fuzzy logic inference algorithm is used to calculate the initial position of the target in the camera coordinate system;
[0091] S23, based on the initial position, radar distance and azimuth data, a multi-level particle swarm optimization Kalman filter algorithm is used to estimate the precise three-dimensional coordinates of the target;
[0092] S24, using a dynamic quaternion interpolation algorithm to convert the precise three-dimensional coordinates of the target from the global coordinate system to the local coordinate system of the PTZ camera to obtain the local coordinates;
[0093] S25. Based on the local coordinates, the theoretical position of the target on the image plane is calculated through an adaptive perspective n-dimensional projection model.
[0094] In a specific embodiment, the target digital fingerprint DF1 and the current azimuth angle PA1 and pitch angle TA1 data of the PTZ camera are received, and the initial position of the target in the camera coordinate system is calculated by applying an adaptive fuzzy logic reasoning algorithm, specifically: extracting the rough position information PL1 of the target from the target digital fingerprint DF1; constructing a fuzzy rule base FR1 by combining the current azimuth angle PA1, pitch angle TA1 and rough position information PL1 of the PTZ camera; using a dynamically adjusted membership function MF1 to fuzzify the input to obtain a fuzzy set FS1; using an improved Sugeno reasoning model to reason on the fuzzy set FS1 to obtain a fuzzy output FO1; and clarifying the fuzzy output FO1 by a novel adaptive centrifugalization method to obtain an initial position estimate CP1 of the target in the camera coordinate system.
[0095] The radar distance RD1 and azimuth data RB1 are obtained, and data fusion is performed with the initial position estimate CP1. The multi-level particle swarm optimization Kalman filter (ML-PSO-KF) algorithm is used to estimate the precise three-dimensional coordinates of the target. Specifically, the multi-level particle swarm PS1 is initialized, and each layer represents a state estimate of different accuracy; the improved Kalman filter KF1 is applied to each layer of particles to predict the state vector SV1; the observation vector OV1 is constructed using the initial position estimate CP1, the radar distance RD1 and the azimuth data RB1; the weighted Mahalanobis distance WMD1 between the state vector SV1 and the observation vector OV1 is calculated as the fitness function; the adaptive inertia weight strategy is applied to update the particle velocity and position to obtain the optimized particle swarm PS2, the optimal particle is selected from the particle swarm PS2, and the precise three-dimensional coordinates TC1 of the target are extracted.
[0096] Based on the precise three-dimensional coordinates TC1, the dynamic quaternion interpolation algorithm is applied to transform the target position from the global coordinate system to the local coordinate system of the PTZ camera, specifically: construct the quaternion transformation matrix QM1 from the global coordinate system to the local coordinate system of the camera; apply the quaternion transformation matrix QM1 to the precise three-dimensional coordinates TC1 to obtain the preliminary transformation result LT1; use the spline interpolation method to generate a smooth quaternion curve QC1 to handle possible nonlinear motion of the camera; refine the preliminary transformation result LT1 along the quaternion curve QC1 to obtain the local coordinate system position LC1 that considers the dynamic characteristics of the camera.
[0097] The local coordinate system position LC1 is used in combination with the internal parameters of the PTZ camera (focal length, principal point offset, etc.) to calculate the theoretical position of the target on the image plane through an adaptive perspective n-dimensional projection model. Specifically, an n-dimensional (n>3) generalized perspective projection model GPM1 is constructed to deal with possible lens distortion and large field of view; the local coordinate system position LC1 is mapped to the generalized perspective projection model GPM1 to obtain the preliminary projection result PR1; the preliminary projection result PR1 is optimized using the iterative least squares method, and the nonlinear characteristics of the camera are considered to obtain the corrected projection point PP1; the projection point PP1 is converted into the image plane coordinate system to obtain the final theoretical image position IP1 of the target.
[0098] This embodiment achieves high-precision, real-time multi-source data fusion and target positioning by combining an adaptive fuzzy logic reasoning algorithm, a multi-level particle swarm optimization Kalman filter algorithm, a dynamic quaternion interpolation algorithm, and an adaptive perspective n-dimensional projection model. First, when processing the target digital fingerprint and the current parameters of the PTZ camera, the adaptive fuzzy logic reasoning algorithm effectively handles the uncertainty and ambiguity in the data by constructing a dynamically adjusted membership function and an improved Sugeno reasoning model. This method can adapt to different scene conditions and improve the accuracy of the initial position estimation while maintaining computational efficiency. In particular, when processing sensor data with noise or partial missing data, the algorithm exhibits excellent robustness and provides a reliable initial estimate for subsequent precise positioning.
[0099] The multi-level particle swarm optimization Kalman filter algorithm effectively solves the limitations of traditional Kalman filtering in nonlinear systems by integrating radar distance, azimuth data and camera coordinate system position through multi-level particle representation and adaptive inertia weight strategy. It can not only handle complex nonlinear motion models, but also greatly improve computational efficiency while maintaining high accuracy. Especially when dealing with targets with high-speed motion or sudden trajectory changes, it shows excellent tracking performance, can quickly converge to the real state, and reduce positioning errors.
[0100] The dynamic quaternion interpolation algorithm effectively handles the possible nonlinear motion of the PTZ camera by constructing a smooth quaternion curve when performing coordinate system conversion. It not only ensures the continuity and smoothness of the coordinate conversion, but also accurately captures the dynamic characteristics of the camera. Compared with the traditional Euler angle representation, quaternion interpolation avoids the gimbal lock problem and can still maintain high precision when rotating at large angles, thereby improving the accuracy and stability of coordinate conversion.
[0101] The adaptive perspective n-dimensional projection model effectively handles complex situations such as lens distortion and large field of view by constructing a high-dimensional perspective projection model and applying iterative least squares when calculating the theoretical position of the target's image plane. It can not only accurately simulate complex imaging processes, but also adaptively adjust model parameters to adapt to different camera configurations. Especially when dealing with wide-angle lenses or severe distortion, the model shows excellent correction capabilities and improves the accuracy of image plane positioning.
[0102] This embodiment improves the accuracy, real-time and robustness of multi-source data fusion and target positioning. Through the comprehensive application of adaptive algorithms and high-dimensional models, the system can achieve high-precision target positioning in complex environments and motion scenes, not only overcoming the limitations of a single sensor, but also giving full play to the complementary advantages of multi-source data, enabling the system to maintain stable high performance in various challenging scenes. Especially when dealing with highly dynamic, nonlinear moving targets, this solution shows obvious advantages, providing an accurate and reliable position information basis for subsequent PTZ control and target tracking.
[0103] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:
[0104] S31, based on the precise three-dimensional coordinates of the target and the theoretical position of the target on the image plane, the hybrid artificial potential field-neural dynamics algorithm is used to calculate the initial PTZ camera control parameters;
[0105] S32, based on the precise three-dimensional coordinates of the target and the theoretical position of the target on the image plane, use the fractal time series prediction model to predict the short-term future position of the target;
[0106] S33, calculating predicted optimal PTZ camera control parameters based on the predicted short-term future position;
[0107] S34. Based on the initial PTZ camera control parameters and the optimal PTZ camera control parameters, a fuzzy cognitive graph-reinforcement learning hybrid controller is used to generate a smooth PTZ control instruction sequence.
[0108] According to one aspect of the present application, step S31 is further:
[0109] S311. Based on the precise three-dimensional coordinates of the target, an artificial potential field centered on the theoretical position of the target on the image plane is constructed, and the center is defined as the global minimum point;
[0110] S312, constructing a neural dynamics model, mapping the artificial potential field to the energy function of the neural dynamics model, and forming a hybrid optimization model;
[0111] S313, using an improved Hopfield network to solve the hybrid optimization model to obtain an estimated value of an initial PTZ camera control parameter;
[0112] S314: fine-tune the initial PTZ camera control parameter estimation value using a dynamic annealing strategy to obtain optimized initial PTZ camera control parameters.
[0113] According to one aspect of the present application, step S34 is further:
[0114] S341, constructing a fuzzy cognitive map based on the initial PTZ camera control parameters and the optimal PTZ camera control parameters;
[0115] S342, constructing a Q learning model based on the fuzzy cognitive graph, wherein the state space of the Q learning model includes initial PTZ camera control parameters and optimal PTZ camera control parameters;
[0116] S343, optimizing the action selection strategy of the Q learning model through Monte Carlo tree search to obtain a preliminary control sequence;
[0117] S344: Use a spline smoothing algorithm to adjust the preliminary control sequence to generate a final PTZ control instruction sequence.
[0118] In a specific embodiment, the final theoretical image position IP1 and the three-dimensional coordinates TC1 are input, and a hybrid artificial potential field-neurodynamic algorithm is applied to calculate the initial PTZ value required to place the target at the center of the image, specifically, an artificial potential field APF1 centered on the final theoretical image position IP1 is constructed, and the image center is defined as the global minimum point; a neurodynamic model NDM1 is designed, in which the neuron state represents the PTZ value; the artificial potential field APF1 is mapped to the energy function of the neurodynamic model NDM1 to form a hybrid optimization model HOP1; an improved Hopfield network is used to solve HOP1 to obtain an initial PTZ value estimate IPV1; a dynamic annealing strategy is applied to fine-tune the initial PTZ value estimate IPV1 to obtain an optimized initial PTZ value OPV1.
[0119] Based on the target feature fingerprint DF1 and position information TC1, a fractal time series prediction model is used to predict the short-term future position of the target. Specifically, the historical trajectory time series TS1 of the target is constructed from the target feature fingerprint DF1 and position information TC1; the Hurst index HI1 of the historical trajectory time series TS1 is calculated using the improved R / S analysis method; the fractal interpolation function FIF1 is constructed based on the Hurst index HI1; the fractal interpolation function FIF1 is used to extrapolate the historical trajectory time series TS1 to obtain the predicted short-term future position FP1 of the target.
[0120] The predicted future position FP1 is input into the hybrid artificial potential field-neurodynamics algorithm to calculate the predicted optimal PTZ value. Specifically, the artificial potential field APF1 is updated using the future position FP1 to obtain a new potential field APF2; APF2 is mapped to the neurodynamics model NDM1 to form a new optimization problem HOP2; the optimization problem HOP2 is solved to obtain the predicted PTZ value PPV1.
[0121] A fuzzy cognitive map-reinforcement learning hybrid controller is applied. Based on the initial PTZ value OPV1 and combined with the predicted PTZ value PPV1, a smooth PTZ control instruction sequence is generated. Specifically, a fuzzy cognitive map FCM1 is constructed, and the nodes represent PTZ control parameters and system states. A Q learning model QLM1 based on the fuzzy cognitive map FCM1 is designed, and the state space includes OPV1 and PPV1. The action selection strategy of the Q learning model QLM1 is optimized through Monte Carlo tree search to obtain a preliminary control sequence CS1. The spline smoothing algorithm is used to adjust the preliminary control sequence CS1 to generate the final PTZ control instruction sequence FCS1.
[0122] This embodiment achieves efficient and accurate calculation and optimization of PTZ control parameters by combining a hybrid artificial potential field-neurodynamics algorithm, a fractal time series prediction model, and a fuzzy cognitive map-reinforcement learning hybrid controller. First, when calculating the initial PTZ value, the hybrid artificial potential field-neurodynamics algorithm constructs an artificial potential field centered on the target image plane position and maps it to the energy function of the neurodynamics model, effectively combining the intuitiveness of the artificial potential field method and the learning ability of the neural network. It can not only quickly generate a smooth control trajectory, but also effectively avoid obstacles during movement. Especially when dealing with multi-target scenes, it can adaptively adjust the potential field distribution to achieve balanced tracking of multiple targets, improving the control stability and flexibility of the system in complex environments.
[0123] When predicting the short-term future position of a target, the fractal time series prediction model effectively captures the long-range correlation and self-similarity of the target's motion by calculating the Hurst exponent of the target's historical trajectory and constructing a fractal interpolation function. It can not only accurately predict the trajectory of a target with regular motion, but also effectively handle complex nonlinear motion with fractal characteristics. Especially when dealing with motion patterns with sudden or periodic changes, the model shows excellent prediction capabilities, improves the system's estimation accuracy of the target's future position, and provides a more reliable prediction basis for PTZ control.
[0124] When generating a smooth PTZ control command sequence, the fuzzy cognitive graph-reinforcement learning hybrid controller constructs a fuzzy cognitive graph to represent the complex relationship between PTZ control parameters and system states, and combines the Q learning model to optimize the action strategy, achieving efficient solutions to complex control problems. It can not only handle the uncertainty and ambiguity in the system, but also continuously optimize the control strategy through continuous learning. In particular, by optimizing the action selection strategy through Monte Carlo tree search, the system can quickly find a near-optimal control sequence in a large-scale state space. In addition, the spline smoothing algorithm is applied to adjust the control sequence, which further ensures the smoothness and continuity of the PTZ movement and effectively reduces mechanical vibration and image jitter.
[0125] This embodiment achieves accurate tracking of the target through accurate calculation of the initial PTZ value and prediction of the future position, and can maintain a stable tracking effect even when the target moves at high speed or suddenly changes direction; the parallel computing characteristics of the hybrid artificial potential field-neurodynamic algorithm and the fuzzy cognitive map-reinforcement learning controller enable the system to complete the calculation and optimization of control parameters within milliseconds to meet the needs of real-time control; through continuous learning and optimization, the system can adapt to different environmental conditions and target motion characteristics, and maintain stable control performance in various complex scenarios; the long-range correlation capture capability of the fractal time series prediction model enables the system to maintain accurate predictions in the face of noise interference and partial observation occlusion, thereby enhancing the robustness of the overall system; through the spline smoothing algorithm, the control instruction sequence generated by the system has a high degree of smoothness, which effectively reduces the mechanical wear of the PTZ camera, extends the service life of the equipment, and improves the quality of image acquisition; the optimized control strategy reduces unnecessary PTZ movement and reduces energy consumption, and is particularly suitable for monitoring scenarios with long-term continuous operation.
[0126] This embodiment improves the accuracy, real-time performance, adaptability and robustness of PTZ control. Through the collaborative work of multiple advanced algorithms, efficient and accurate target tracking is achieved in complex and changing environments, providing reliable technical support for various high-demand application scenarios. It not only improves the overall performance of the system, but also expands its scope of application in practical applications, providing solutions for intelligent video surveillance, autonomous driving, robot vision and other fields.
[0127] like Figure 5 As shown, according to one aspect of the present application, step S4 also includes instructing a predetermined PTZ camera to perform adaptive multi-target tracking, specifically:
[0128] S41, obtaining the digital fingerprint and precise three-dimensional coordinates of the predetermined target, and using the dynamic graph matching-spectral clustering algorithm to build a multi-target tracking queue;
[0129] S42, based on the multi-target tracking queue, using an adaptive multi-target differential evolution algorithm, assigning tracking tasks to predetermined PTZ cameras to obtain a camera task assignment solution;
[0130] S43, generating a comprehensive control instruction set based on the PTZ camera control parameters of each target in the camera task allocation scheme;
[0131] S44, based on the comprehensive control instruction set, instructing a predetermined PTZ camera to perform adaptive multi-target tracking.
[0132] This embodiment continuously receives feature fingerprints and positioning results of multiple targets, applies a dynamic graph matching-spectral clustering algorithm to build a multi-target tracking queue; based on the tracking queue, an adaptive multi-target differential evolution algorithm is used to assign tracking tasks to multiple PTZ cameras; and executes a PTZ control parameter calculation process for each assigned target to generate a comprehensive control instruction set.
[0133] In a specific embodiment, feature fingerprints DF1 and precise three-dimensional coordinates TC1 of multiple targets are continuously received, and a dynamic graph matching-spectral clustering algorithm is applied to construct a multi-target tracking queue, specifically: a target association graph TAG1 is constructed based on the feature fingerprint DF1 and the precise three-dimensional coordinates TC1; an improved spectral clustering algorithm is applied to segment the target association graph TAG1 to obtain a preliminary target grouping result TG1; a dynamic time warping (DTW) algorithm is used to calculate a similarity matrix SM1 of target trajectories in the preliminary target grouping result TG1; hierarchical clustering is applied to the similarity matrix based on SM1 to form a final multi-target tracking queue MTQ1.
[0134] Based on the multi-target tracking queue MTQ1, the adaptive multi-objective differential evolution algorithm (AMODE) is used to assign tracking tasks to multiple PTZ cameras. Specifically, the task assignment problem is encoded into a real vector RV1, where each element represents the target-camera assignment relationship; the population SP1 is initialized, and each individual represents a possible task assignment scheme; and a multi-objective fitness function set MOF1 is designed, which includes the following three sub-goals:
[0135] f1: maximize the overall tracking efficiency;
[0136] f2: minimize camera load imbalance;
[0137] f3: maximize the tracking quality of high priority targets;
[0138] Apply improved differential evolution operations: use adaptive scaling factor SF1, dynamically adjust according to population diversity, adopt Levy flight-based mutation strategy MS1 to enhance global search capabilities, implement non-dominated sorting-based selection mechanism SM1 to maintain elite individuals; introduce adaptive crossover rate CR1, automatically adjust according to the convergence speed of each generation of evolution; apply crowding distance mechanism CM1 to maintain solution diversity; use improved Pareto front approximation method PFA1 to accelerate the identification of non-dominated solutions; iterate until the termination condition or the maximum number of iterations is reached; from the final non-dominated solution set NDS1, use fuzzy decision method FDM1 to select the best compromise solution; decode the selected solution into a specific camera task allocation scheme CTA1;
[0139] For each target assigned in the camera task assignment scheme CTA1, execute the PTZ control parameter calculation process of step S3 to generate the PTZ camera control instruction set CIC1, specifically: for each target-camera pair, apply the algorithm of steps S31-S34, summarize all control instructions, and form a comprehensive control instruction set CIC1.
[0140] This embodiment achieves efficient and stable adaptive multi-target tracking by combining the dynamic graph matching-spectral clustering algorithm and the adaptive multi-target differential evolution algorithm. First, when constructing a multi-target tracking queue, the dynamic graph matching-spectral clustering algorithm constructs a dynamic association graph based on the target feature fingerprint and positioning results, and applies the improved spectral clustering method for graph segmentation, which effectively solves the data association problem in multi-target tracking. It can not only handle scenes with dynamically changing number of targets, but also effectively handle mutual occlusion and intersection between targets. In particular, by introducing the dynamic time warping (DTW) algorithm to calculate the similarity of target trajectories, the system can more accurately distinguish similar targets and improve the tracking accuracy in complex scenes. This method shows excellent performance when processing large-scale multi-target scenes, and can significantly reduce the computational complexity while maintaining high accuracy.
[0141] When assigning tracking tasks to multiple PTZ cameras, the adaptive multi-objective differential evolution algorithm effectively solves the limitations of traditional task allocation methods in dynamic environments by designing a multi-objective fitness function and introducing an adaptive parameter adjustment strategy. It not only considers multiple objectives such as overall tracking efficiency, camera load balance, and target priority, but also dynamically adjusts the optimization strategy according to real-time scene changes. In particular, by introducing the mutation strategy of Levy flight, the algorithm enhances the global search capability while maintaining the convergence speed, effectively avoiding the problem of falling into the local optimum. In addition, the selection mechanism based on non-dominated sorting and crowding distance calculation are adopted to ensure the diversity of the solution, providing the system with a variety of optional task allocation schemes. This multi-objective optimization method not only improves the overall tracking efficiency, but also effectively balances the workload of each PTZ camera and extends the service life of the equipment.
[0142] The dynamic graph matching-spectral clustering algorithm can efficiently manage a large number of targets and maintain stable tracking performance even when the targets frequently enter and leave the field of view. This improves the system's adaptability in complex scenarios and enables it to be applied to scenarios such as monitoring crowded public places or large-scale traffic monitoring. The adaptive multi-target differential evolution algorithm realizes the intelligent allocation of PTZ camera resources, which not only maximizes the overall tracking efficiency, but also ensures the balanced utilization of system resources, which improves the cost-effectiveness of the system and maximizes the effectiveness of limited hardware resources. The entire system can dynamically adjust the tracking strategy according to environmental changes and target behavior through a variety of adaptive mechanisms. This adaptive capability enables the system to maintain efficient and stable performance in various complex and changing scenarios. Despite the use of complex algorithms, the system can still maintain real-time response speed through optimized implementation and parallel processing. This embodiment can be applied to scenarios requiring quick response, such as smart security or automatic driving assistance systems; the modular design of this embodiment makes the system extremely scalable, and new functional modules can be easily added or existing algorithms can be optimized according to specific application requirements without affecting the overall architecture; by comprehensively utilizing a variety of advanced algorithms, the system can maintain accurate tracking of multiple targets for a long time, which is particularly important in applications that require long-term monitoring, such as security monitoring of large-scale events or research on wildlife behavior; multiple verification mechanisms and intelligent feature extraction reduce the false alarm rate of the system, which not only improves the reliability of the system, but also reduces the need for manual intervention and reduces operating costs.
[0143] This embodiment improves the efficiency, accuracy, robustness and adaptability of multi-target tracking. Through the collaborative work of multiple advanced algorithms, the system can achieve stable and reliable multi-target tracking in complex and changing environments, providing strong technical support for various high-demand application scenarios. This embodiment not only greatly improves the system's ability to handle complex scenarios, but also expands its scope of application in practical applications, providing solutions for intelligent video surveillance, smart city management, large-scale event security, traffic flow analysis and other fields. In addition, the flexibility and scalability of this embodiment also provide a solid foundation for future technology upgrades and functional expansions, ensuring that the system can continue to adapt to growing application needs and technological developments.
[0144] According to another aspect of the present application, the precise three-dimensional coordinates of the target in step S2 are also monitored in real time. When it is detected that the target is occluded or lost, an adaptive attention mechanism-Twin network algorithm is used to resume tracking of the target.
[0145] In a specific embodiment, the precise three-dimensional coordinates TC1 are monitored in real time. When it is detected that the target is occluded or lost, the adaptive attention mechanism-twin network algorithm is applied to restore target tracking, specifically: extract the target's visual features VF1 and motion features MF1 from the digital fingerprint DF1; construct an adaptive attention module AAM1 to dynamically adjust the weights of the visual features VF1 and the motion features MF1; design a twin network SN1, one branch processes historical features and the other branch processes current observations; use the features enhanced by the adaptive attention module AAM1 to input the twin network SN1, and calculate the similarity score SS1; perform target matching based on the similarity score SS1, restore the tracking status of the occluded or lost target, and update the multi-target tracking queue MTQ1 to ensure the continuity of tracking.
[0146] The adaptive attention mechanism-twin network algorithm of this embodiment achieves efficient and stable target re-identification by dynamically adjusting the weights of visual features and motion features when dealing with situations where the target is occluded or lost, and using the twin network structure to compare historical features and current observations. This not only effectively copes with complex backgrounds and lighting changes, but also handles situations where the target's appearance changes. Especially when resuming tracking of targets after long-term occlusion, the algorithm exhibits excellent performance and improves the robustness and reliability of the system. In addition, by introducing the adaptive attention mechanism, the system can focus more on the key features of the target, further improving the accuracy and efficiency of re-identification.
[0147] This embodiment enhances the system's ability to recover after a target is temporarily lost, so that the system can maintain continuous and stable tracking in complex real-world environments, such as tracking targets passing through crowded streets or blocked by buildings in urban monitoring.
[0148] In another embodiment of the present application, a target positioning and tracking method in which a PTZ camera and a radar are coordinated comprises the following steps:
[0149] S1. Target recognition and feature extraction.
[0150] S11, receiving high-resolution image data from a PTZ camera, applying an adaptive threshold segmentation algorithm to extract the contour and color features of the target, and obtaining a visual feature vector of the target.
[0151] S12. Acquire the radar echo signal data, perform spectrum analysis using an improved fast Fourier transform (FFT) algorithm, extract the distance, speed and azimuth information of the target, and form a radar feature vector of the target.
[0152] S13, performing feature-level fusion on the visual feature vector obtained in S11 and the radar feature vector obtained in S12, using a principal component analysis (PCA) algorithm to reduce the dimension and generate a comprehensive feature vector.
[0153] S14. Based on the comprehensive feature vector generated in S13, a local sensitive hashing (LSH) algorithm is applied to generate a unique digital fingerprint for each detected target, which contains the location, speed and shape feature information of the target.
[0154] S2. Multi-source data fusion and target positioning.
[0155] S21, receiving the target digital fingerprint generated by S14, and calculating the initial position of the target in the camera coordinate system in combination with the current azimuth and pitch angle data of the PTZ camera.
[0156] S22, obtain the radar distance and azimuth data extracted in S12, perform data fusion with the camera coordinate system position obtained in S21, and use the extended Kalman filter (EKF) algorithm to estimate the precise three-dimensional coordinates of the target.
[0157] S23. Based on the target three-dimensional coordinates obtained in S22, a coordinate transformation algorithm is applied to transform the target position from the global coordinate system to the local coordinate system of the PTZ camera.
[0158] S24, using the local coordinate system position calculated in S23 and the internal parameters of the PTZ camera, a theoretical position of the target on the image plane is calculated through a geometric projection model.
[0159] S3. Calculation and optimization of PTZ control parameters.
[0160] S31, input the target image plane position obtained in S24 and the three-dimensional coordinate information of S22, apply the inverse kinematics algorithm, and calculate the initial PTZ value required to place the target at the center of the image.
[0161] S32, based on the target feature fingerprint of S14 and the location information of S22, uses the long short-term memory network (LSTM) prediction model to predict the short-term future location of the target.
[0162] S33, input the future position predicted by S32 into the inverse kinematics algorithm of S31, and calculate the predicted optimal PTZ value.
[0163] S34, applying the fuzzy PID control algorithm, based on the initial PTZ value of S31 and combined with the predicted PTZ value of S33, generates a smooth PTZ control instruction sequence to compensate for system delay and mechanical inertia.
[0164] S4. Adaptive multi-target tracking.
[0165] S41, continuously receiving feature fingerprints of multiple targets from S1 and positioning results of S2, and building a multi-target tracking queue.
[0166] S42, based on the tracking queue of S41, applies the improved Hungarian algorithm to assign tracking tasks to multiple PTZ cameras to ensure the maximum overall tracking efficiency.
[0167] S43: For each assigned target, execute the PTZ control parameter calculation process of S3 to generate control instructions for each PTZ camera.
[0168] S44 monitors the positioning result of S2 in real time. When it is detected that the target is blocked or lost, a re-identification algorithm based on the appearance and motion model is applied based on the feature fingerprint of S14 to restore target tracking.
[0169] In another embodiment, in order to solve the problem of inaccurate target positioning and tracking by PTZ camera, the positioning and tracking accuracy is improved by interacting with radar data and correcting the calculation result of PTZ value, specifically:
[0170] Step 1. Enter the target ID number;
[0171] Step 2: Match the target ID number in the radar's target list and obtain the target's latitude and longitude information from the radar's echo data;
[0172] Step 3: Calculate the position data of the target relative to the PTZ camera, including the azimuth angle and distance, based on the latitude and longitude information of the target and the position information of the PTZ camera;
[0173] Step 4: Based on the azimuth angle and distance, combined with the basic parameters of the PTZ camera, the camera installation angle, T and Z values corresponding to different P values are obtained by reading the configuration file. The PTZ value corresponding to the position of the target can be calculated and corrected.
[0174] Step 5: Input the PTZ value calculated in step 4 into the camera control function to modify the azimuth and focal length of the PTZ camera, thereby realizing the target positioning function;
[0175] Step 6: After the target position data obtained in step 2 is updated, repeat steps 3 to 5 to complete the real-time position adjustment of the PTZ camera, thereby realizing the target tracking function.
[0176] like Figure 6 As shown, in a specific embodiment, the target ship ID is input; the ship ID is matched in the radar target list; if the ship ID is not matched in the radar target list, the process is terminated; if the ship ID is matched in the radar target list, the real-time position information of the ship can be obtained, including accuracy and latitude; according to the ship's longitude and latitude data, combined with the camera's position data, the relative position of the target ship relative to the camera is calculated, including azimuth angle and distance; according to the obtained relative position, the initial PTZ value can be calculated, and at the same time, combined with the configuration data of the camera, including the installation angle, T and Z value correction values, the initial PTZ value can be corrected to obtain the corrected PTZ value; the calculated corrected PTZ value is input into the camera, and the camera's azimuth and focal length are controlled to achieve the positioning of the target. If you want to continue tracking the target, return to the radar target list to match the ship; if you do not want to track the target, the process is terminated.
[0177] Based on the cooperation and data interaction between the PTZ camera and the radar, this embodiment obtains the position information of the target in the radar target list in real time, obtains a relatively accurate PTZ value through calculation and correction, thereby realizing the positioning and tracking of the target by the PTZ camera; this embodiment can realize the simultaneous monitoring of the target by the PTZ camera and the radar. In the actual monitoring scene, the camera has clear imaging, rich colors, high flexibility, strong radar penetration, and accurate detection information. This embodiment can take into account the advantages of both, and the user can obtain more comprehensive information about the monitored target; this embodiment calibrates the reference focal length corresponding to different angles and distances in advance and saves it to the configuration file. When calibrating the PTZ value, directly matching the relevant data in the configuration file can improve the efficiency and accuracy of the calculation correction; this embodiment can realize the cyclic tracking of multiple targets. When multiple target IDs are entered in the above step 1, the cyclic time interval t can be set. Every t seconds, the camera switches the tracking object to realize the multi-target cyclic tracking function.
[0178] This embodiment realizes the precise positioning of static targets and real-time tracking of dynamic targets based on the coordination of PTZ camera control and radar data.
[0179] A target positioning and tracking system of a PTZ camera and a radar that implements the method described in the above embodiment includes a data acquisition module for acquiring high-resolution image data D1 and echo signal data D2; a feature extraction module for processing the high-resolution image data D1 and the echo signal data D2, extracting target features and generating a digital fingerprint DF1; a target positioning module for calculating precise three-dimensional coordinates TC1 based on multi-source data fusion; a control parameter calculation module for calculating an initial position estimate CP1 of the target in a camera coordinate system based on the precise three-dimensional coordinates TC1; and a target tracking module for using the initial position estimate CP1 of the target in the camera coordinate system to guide the PTZ camera to track the target.
[0180] According to another aspect of the present application, a target positioning and tracking system using a PTZ camera and a radar in coordination includes:
[0181] at least one processor; and,
[0182] a memory communicatively connected to at least one of the processors; wherein,
[0183] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the target positioning and tracking method of the PTZ camera and radar collaboration described in any of the above embodiments.
[0184] Through multimodal feature extraction and fusion, this application can capture target features comprehensively and accurately, thereby improving the accuracy and robustness of target recognition. Multi-source data fusion and high-precision positioning technology provide a reliable location information basis for the subsequent tracking process. Intelligent PTZ control parameter calculation and optimization ensure that the camera can smoothly and accurately follow the target movement. The adaptive multi-target tracking strategy enables the system to efficiently manage and track multiple targets while coping with complex practical scene challenges. The system of this application exhibits excellent environmental adaptability and can maintain stable and efficient performance in various complex scenes; it has powerful real-time processing capabilities and can respond quickly to scene changes; it demonstrates excellent scalability and flexibility and is easy to customize and upgrade according to specific application requirements. In addition, the high degree of automation and intelligence of the system reduces the need for manual intervention, improves operational efficiency and reduces operating costs.
[0185] This embodiment successfully solves many challenges faced by traditional PTZ camera tracking systems, and provides a comprehensive, efficient and reliable technical solution for the fields of intelligent video surveillance, smart city management, large-scale event security, etc., with broad application prospects and important practical value.
[0186] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.
Claims
1. A target positioning and tracking method using a PTZ camera and a radar, characterized in that: The steps include: S1, obtaining PTZ camera image data and radar echo signal data, processing the PTZ camera image data and radar echo signal data, extracting target features and generating a digital fingerprint of the target; S2, collect multi-source data, and calculate the precise three-dimensional coordinates of the target based on the digital fingerprint and multi-source data of the target; the multi-source data includes radar distance, azimuth data and current parameters of PTZ camera; S3, based on the precise three-dimensional coordinates of the target, calculate the PTZ camera control parameters; S4, based on the PTZ camera control parameters, guiding the PTZ camera to track the target; Step S1 is further as follows: S11, obtaining PTZ camera image data and radar echo signal data; S12, using a multi-scale adaptive region growing algorithm on the PTZ camera image data to generate a visual feature vector; S13, using an adaptive piecewise nonlinear Fourier transform algorithm on the radar echo signal data to generate a radar feature vector; S14, using a dynamic weighted tensor decomposition algorithm to fuse the visual feature vector and the radar feature vector to generate a comprehensive feature vector; S15. Based on the comprehensive feature vector, an adaptive local sensitive hashing algorithm is used to construct the digital fingerprint of the target; Step S2 is further as follows: S21, collect multi-source data, including radar distance, azimuth data and current parameters of PTZ camera; S22, based on the digital fingerprint of the target and the current parameters of the PTZ camera, an adaptive fuzzy logic inference algorithm is used to calculate the initial position of the target in the camera coordinate system; S23, based on the initial position, radar distance and azimuth data, a multi-level particle swarm optimization Kalman filter algorithm is used to estimate the precise three-dimensional coordinates of the target; S24, using a dynamic quaternion interpolation algorithm to convert the precise three-dimensional coordinates of the target from the global coordinate system to the local coordinate system of the PTZ camera to obtain the local coordinates; S25, calculating the theoretical position of the target on the image plane based on the local coordinates through an adaptive perspective n-dimensional projection model; Step S3 is further as follows: S31, based on the precise three-dimensional coordinates of the target and the theoretical position of the target on the image plane, the hybrid artificial potential field-neural dynamics algorithm is used to calculate the initial PTZ camera control parameters; S32, based on the precise three-dimensional coordinates of the target and the theoretical position of the target on the image plane, use the fractal time series prediction model to predict the short-term future position of the target; S33, calculating predicted optimal PTZ camera control parameters based on the predicted short-term future position; S34, based on the initial PTZ camera control parameters and the optimal PTZ camera control parameters, a fuzzy cognitive graph-reinforcement learning hybrid controller is used to generate a smooth PTZ control command sequence; Step S4 also includes instructing a predetermined PTZ camera to perform adaptive multi-target tracking, specifically: S41, obtaining the digital fingerprint and precise three-dimensional coordinates of the predetermined target, and using the dynamic graph matching-spectral clustering algorithm to build a multi-target tracking queue; S42, based on the multi-target tracking queue, using an adaptive multi-target differential evolution algorithm, assigning tracking tasks to predetermined PTZ cameras to obtain a camera task assignment solution; S43, generating a comprehensive control instruction set based on the PTZ camera control parameters of each target in the camera task allocation scheme; S44, based on the comprehensive control instruction set, instructing a predetermined PTZ camera to perform adaptive multi-target tracking.
2. The target positioning and tracking method using PTZ camera and radar collaboration according to claim 1, characterized in that: Step S34 is further as follows: S341, constructing a fuzzy cognitive map based on the initial PTZ camera control parameters and the optimal PTZ camera control parameters; S342, constructing a Q learning model based on the fuzzy cognitive graph, wherein the state space of the Q learning model includes initial PTZ camera control parameters and optimal PTZ camera control parameters; S343, optimizing the action selection strategy of the Q learning model through Monte Carlo tree search to obtain a preliminary control sequence; S344: Use a spline smoothing algorithm to adjust the preliminary control sequence to generate a final PTZ control instruction sequence.
3. The target positioning and tracking method using PTZ camera and radar collaboration according to claim 1, characterized in that: The method also includes real-time monitoring of the precise three-dimensional coordinates of the target in step S2. When it is detected that the target is occluded or lost, an adaptive attention mechanism-Siamese network algorithm is used to resume tracking of the target.
4. The target positioning and tracking method using PTZ camera and radar collaboration according to claim 1, characterized in that: Step S13 is further as follows: S131, preprocessing the radar echo signal data to remove background noise and fixed clutter to obtain a clean signal; S132, segmenting the clean signal to obtain a signal segment set with adaptive length; S133, based on the signal segment set, using nonlinear phase function and improved fast Fourier transform to obtain time-frequency representation; S134, based on the time-frequency representation, using the peak clustering-tracking algorithm to calculate and obtain the spectrum peak tracking result; S135. Based on the spectrum peak tracking result, estimate the distance, speed and direction information of the target and generate a radar feature vector.
5. The target positioning and tracking method using PTZ camera and radar collaboration according to claim 1, characterized in that: Step S31 is further as follows: S311. Based on the precise three-dimensional coordinates of the target, an artificial potential field centered on the theoretical position of the target on the image plane is constructed, and the center is defined as the global minimum point; S312, constructing a neural dynamics model, mapping the artificial potential field to the energy function of the neural dynamics model, and forming a hybrid optimization model; S313, using an improved Hopfield network to solve the hybrid optimization model to obtain an estimated value of an initial PTZ camera control parameter; S314: fine-tune the initial PTZ camera control parameter estimation value using a dynamic annealing strategy to obtain optimized initial PTZ camera control parameters.
6. A target positioning and tracking system using a PTZ camera and a radar, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the target positioning and tracking method of PTZ camera and radar collaboration as described in any one of claims 1 to 5.
Citation Information
Cited By
Construction equipment safety system, and method for improving object detection accuracy by fusing multiple pieces of sensor data included in construction equipment safety system
WO2026134504A1
