Blind area relay tracking method and system based on heterogeneous data space-time calibration

Abstract

This invention provides a blind zone relay tracking method and system based on heterogeneous data spatiotemporal calibration. The method includes: acquiring images and radio frequency (RF) signals at the blind zone entrance; calculating the cross-covariance between the image optical flow velocity and the RF Doppler velocity to anchor the visual target to RF hardware features and generate a tracking identifier; within the blind zone, using a joint probabilistic data interconnection algorithm to perform weighted fusion of signal echoes based on spatial distance and RF feature similarity to achieve RF tracking; and at the blind zone exit, using a manifold alignment algorithm to compare the topological distance between RF trajectory fluctuations and the visual motion displacement of candidate objects in low-dimensional space to reconfirm their identity. This invention solves the technical problem of traditional tracking methods relying solely on easily camouflaged and confused visual tracking, leading to the interruption of the tracking chain in the blind zone.

Description

Technical Field

[0001] This invention belongs to the field of data processing and analysis technology, specifically relating to a blind zone relay tracking method and system based on spatiotemporal calibration of heterogeneous data. Background Technology

[0002] In modern security and smart city management systems, large-scale surveillance systems based on networked visual sensors are fundamental technological infrastructure for tracking and analyzing the behavior of targets within a space. The core function is to extract and compare the visual appearance features of targets within the field of view of different camera nodes, enabling cross-camera tracking and continuous association with individual identity. By deploying re-identification algorithms, the movement trajectory of targets can be effectively constructed in areas with good visual continuity, providing crucial data support for public safety management.

[0003] However, due to limitations such as installation conditions and physical obstructions, completely seamless visual coverage is difficult to achieve in real-world scenarios, inevitably creating numerous blind spots. When a continuously tracked target enters such a blind spot, the visual information is interrupted, causing the tracking chain to break. Traditional visual tracking systems rely entirely on the similarity of the target's appearance before and after entering the blind spot for matching. However, appearance features are easily confused or disguised, resulting in often low accuracy in matching results. Therefore, when a candidate object reappears in the field of view of a downstream camera from the blind spot exit, how to restore the previously interrupted identity association in the absence of any observation data within the blind spot is a pressing technical problem that needs to be solved in the field of visual surveillance. Summary of the Invention

[0004] This invention provides a blind zone relay tracking method and system based on heterogeneous data spatiotemporal calibration to solve the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a blind zone relay tracking method based on heterogeneous data spatiotemporal calibration, the method comprising the following steps: Construct a multimodal capture environment at the blind zone entrance and simultaneously acquire image data of the target object entering the multimodal capture environment and radio frequency signals of all mobile devices; The acquired radio frequency signals are analyzed in the frequency domain, and the frequency deviation curve of the mobile device hardware is extracted as radio frequency characteristics. Optical flow analysis is performed on image data to generate displacement velocity sequence of target object. At the same time, the phase change of radio frequency signal is analyzed to generate corresponding Doppler frequency shift velocity sequence. By calculating the cross-covariance between displacement velocity sequence and Doppler frequency shift velocity sequence, the target radio frequency feature with the greatest spatiotemporal correlation is identified with the target object, and a tracking identifier carrying the target radio frequency feature is generated. Within the visual blind zone of the multimodal capture environment, the prior location region of the target object is predicted based on the Kalman filter algorithm and the state of the tracking marker in the previous moment, and the preset distributed antenna array is controlled to search for signal echoes containing radio frequency characteristics. The joint probabilistic data interconnection algorithm is applied to calculate the spatial distance between each signal echo received in the visual blind zone and the prior position area, as well as the feature similarity between each signal echo and the target radio frequency feature in the tracking marker. The spatial distance and feature similarity are combined to generate the association probability weight, and the signal echoes are weighted and fused according to the association probability weight to update the posterior position coordinates of the target object and generate the radio frequency tracking trajectory in the visual blind zone. When the target object leaves the visual blind zone according to the radio frequency tracking trajectory, the radio frequency fluctuation sequence of the radio frequency tracking trajectory and the motion displacement sequence of the candidate object captured by the camera at the exit of the visual blind zone are collected. The radio frequency fluctuation sequence and the motion displacement sequence are mapped to a preset low-dimensional space using a manifold alignment algorithm. In a low-dimensional space, the topological distance between mapped feature vectors is calculated, and candidate objects with topological distances less than a preset threshold are identified as the same entity as the target object.

[0006] Optionally, the step of performing frequency domain analysis on the acquired radio frequency signal and extracting the frequency deviation curve of the mobile device hardware as radio frequency characteristics includes the following steps: The Fast Fourier Transform is used to perform frequency domain analysis on radio frequency signals to identify and extract the preamble sequence of energy burst segments in the radio frequency signals. The preamble sequence is quadrature demodulated to obtain in-phase and quadrature components. The in-phase and quadrature components are then used to calculate the instantaneous phase difference between adjacent symbols in the preamble sequence. The instantaneous offset of the signal source relative to the standard center frequency is calculated based on the instantaneous phase difference and arranged in chronological order to form a frequency deviation curve, which is then used as a radio frequency feature.

[0007] Optionally, the step of performing optical flow analysis on image data to generate a displacement velocity sequence of the target object, and simultaneously analyzing the phase change of the radio frequency signal to generate a corresponding Doppler frequency shift velocity sequence, and then calculating the cross-covariance between the displacement velocity sequence and the Doppler frequency shift velocity sequence to identify the target radio frequency feature with the greatest spatiotemporal correlation and generate a tracking identifier carrying the target radio frequency feature, includes the following steps: The optical flow method is used to track the feature points of pedestrian targets in image data, calculate the displacement pixel difference of feature points on the image plane and convert it into a displacement velocity sequence in physical space; Extract the phase slope change rate of the signal source corresponding to the radio frequency features, and calculate the radial movement speed of the signal source relative to the receiver based on the phase slope change rate to generate a Doppler frequency shift speed sequence; Construct an association matrix with visual targets as rows and radio frequency features as columns, and traverse the association matrix to calculate the Pearson correlation coefficient between the displacement velocity sequence and the Doppler frequency shift velocity sequence corresponding to each element position; The target pair with the highest Pearson correlation coefficient is selected from the correlation matrix, and the radio frequency features in the target pair are marked as the tracking ID of the target object, generating a tracking identifier carrying the target radio frequency features.

[0008] Optionally, the application of the joint probabilistic data interconnection algorithm to calculate the spatial distance between each signal echo received within the visual blind zone and the prior location region, as well as the feature similarity between each signal echo and the target radio frequency feature in the tracking marker, and to generate an association probability weight by combining the spatial distance and feature similarity, and to perform weighted fusion of each signal echo according to the association probability weight to update the posterior location coordinates of the target object, generating the radio frequency tracking trajectory within the visual blind zone includes the following steps: Calculate the position coordinates of each signal echo received within the visual blind zone, and calculate the Euclidean distance between each position coordinate and the center point of the prior position region; Extract the waveform features of each signal echo and calculate the cosine similarity between the waveform features and the target radio frequency features in the tracking identifier; Construct a joint likelihood function that includes Euclidean distance and cosine similarity, and normalize the output of the joint likelihood function to obtain the association probability weight of each signal echo belonging to the target object; The estimated location value is obtained by weighting and summing all location coordinates using the associated probability weights; The state vector of the Kalman filter algorithm is corrected using the position estimate to obtain the updated posterior position coordinates, which are then connected in time series to generate the radio frequency tracking trajectory within the visual blind zone.

[0009] Optionally, after calculating the position estimate by weighted summation of all position coordinates using associated probability weights, the following steps are further included: Determine the current azimuth and pitch angles of the target object within the visual blind spot based on the position estimate; Calculate the phase delay parameters of each element in the distributed antenna array so that the main beam of the distributed antenna array points to the current azimuth and elevation angles; The direction of the source of a signal echo whose correlation probability weight is lower than a preset noise threshold is identified and set as the interference direction. Adjust the array weight vector of the distributed antenna array to create nulls in the interference direction; The distributed antenna array is controlled by applying the adjusted phase delay parameters and array weight vector to enhance the receiving gain of the signal echo containing the target RF characteristics at the next moment.

[0010] Optionally, the construction of a joint likelihood function comprising Euclidean distance and cosine similarity, and the normalization of the output of the joint likelihood function to obtain the association probability weight of each signal echo belonging to the target object, includes the following steps: The joint likelihood function is defined as the product of the spatial Gaussian distribution function and the feature matching probability function; Substitute the Euclidean distance into the spatial Gaussian distribution function to calculate the spatial probability components of the signal echo that conform to the prior prediction in spatial location. The cosine similarity is mapped to a feature matching probability function, and the feature probability components of the signal echo that physically matches the tracking identifier are calculated. The joint likelihood value is obtained by multiplying the spatial probability component and the characteristic probability component, and then the joint likelihood value is corrected by using the clutter density parameter. Summarize the corrected joint likelihood values ​​of all signal echoes, calculate the proportion of each signal echo in the total likelihood value, and obtain the normalized association probability weight of each signal echo belonging to the target object.

[0011] Optionally, the step of acquiring the radio frequency fluctuation sequence of the radio frequency tracking trajectory and the motion displacement sequence of the candidate object captured by the visual blind spot exit camera, and mapping the radio frequency fluctuation sequence and motion displacement sequence to a preset low-dimensional space using a manifold alignment algorithm includes the following steps: Set a time window at the end of the visual blind spot, and extract the signal intensity sequence corresponding to the radio frequency tracking trajectory within the time window as the radio frequency fluctuation sequence; Candidate objects are detected in the monitoring footage of the exit camera in the visual blind spot. The pose estimation model is used to extract the changes in the coordinates of the skeletal key points of the candidate objects in the last time window and generate a motion displacement sequence. Construct the Laplacian matrix of the radio frequency wave sequence and the motion displacement sequence, and calculate the local geometric structure features of the radio frequency wave sequence and the motion displacement sequence respectively; Establish a manifold alignment objective function, which includes constraint terms that preserve the local geometric structure features of each data type and correspondence terms that minimize the correspondence between heterogeneous data types. By solving the generalized eigenvalue problem of the manifold alignment objective function, the radio frequency domain mapping matrix and the visual domain mapping matrix are obtained; The radio frequency fluctuation sequence is projected to a preset low-dimensional space using a radio frequency domain mapping matrix, and the motion displacement sequence is projected to the same low-dimensional space using a visual domain mapping matrix.

[0012] Optionally, the step of calculating the topological distance between mapped feature vectors in the low-dimensional space and determining the candidate object corresponding to the topological distance less than a preset threshold as the same entity as the target object includes the following steps: Obtain the projected RF wave feature vector and the projected motion displacement feature vector in a low-dimensional space; The Euclidean distance between the radio frequency wave feature vector and the motion displacement feature vector is calculated as the topological distance. If the topological distance is less than the preset isomorphism determination threshold, it is confirmed that the fluctuation rhythm of the radio frequency signal and the displacement rhythm of the visual action have manifold isomorphism, and the candidate object and the target object are determined to be the same entity.

[0013] In a second aspect, the present invention also provides a blind zone relay tracking system based on heterogeneous data spatiotemporal calibration, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the blind zone relay tracking method based on heterogeneous data spatiotemporal calibration as described in any one of the first aspects.

[0014] Thirdly, the present invention also provides a computer-readable storage medium storing instructions, characterized in that, when executed by a processor, the instructions cause the processor to be configured to perform the blind zone relay tracking method based on heterogeneous data spatiotemporal calibration according to any one of the first aspects.

[0015] The beneficial effects of this invention are: At the blind zone entrance, by calculating the cross-covariance between the displacement velocity sequence generated by visual optical flow analysis and the Doppler frequency shift velocity sequence of the radio frequency signal, this invention achieves unique identity anchoring between the physical motion of the target object and the radio frequency characteristics of the mobile device it carries at the data source, thereby generating a stable and difficult-to-forge non-visual tracking identifier. During tracking within the visual blind zone, the joint probabilistic data interconnection algorithm not only considers the spatial distance between the signal echo and the prior position predicted by the Kalman filter, but also adds the feature similarity between the echo signal and the anchored radio frequency characteristics, constructing a dual-constraint association probability weight, thus achieving robust updating of the target's posterior position in multi-target, cluttered environments. When the target leaves the blind zone for identity reconfirmation, the manifold alignment algorithm maps the fluctuation sequence of the radio frequency tracking trajectory within the visual blind zone and the visual action displacement sequence of the candidate object at the exit to a unified low-dimensional space for topological distance comparison, providing a re-verification basis for identity relay. This invention solves the technical problem of tracking chain interruption in blind zones caused by traditional tracking methods relying solely on easily spoofed and confused visual tracking. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a blind zone relay tracking method based on heterogeneous data spatiotemporal calibration in one embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0018] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0019] Figure 1 This is a flowchart illustrating a blind zone relay tracking method based on heterogeneous data spatiotemporal calibration in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps. For example Figure 1 As shown, the blind zone relay tracking method based on heterogeneous data spatiotemporal calibration disclosed in this invention specifically includes the following steps: S101. Construct a multimodal capture environment at the blind zone entrance and simultaneously acquire image data of the target object entering the multimodal capture environment and radio frequency signals of all mobile devices.

[0020] The system employs a multi-stage, high-resolution, high-frame-rate industrial-grade camera installed at the entrance to continuously capture video stream data from all target objects entering the area. Simultaneously, a wideband radio frequency (RF) receiver or software-defined radio peripheral is deployed in the same spatial location. These devices must possess omnidirectional antennas or MIMO antenna arrays to indiscriminately monitor and record RF signals emitted by all active mobile devices (such as smartphones and smartwatches) in the surrounding environment, including WiFi probe requests and Bluetooth broadcast packets. To ensure spatiotemporal consistency of the data, the camera and RF receiver must be connected to the same network time protocol server for microsecond-level time synchronization, ensuring that each frame of image and each segment of RF signal has a unified timestamp. This hardware linkage mechanism establishes a data acquisition field with a high degree of overlap between physical and electromagnetic space, avoiding the information loss problem of a single sensor in complex environments and achieving simultaneous capture of the physical and digital characteristics of the target object.

[0021] S102. Perform frequency domain analysis on the acquired radio frequency signals and extract the frequency deviation curve of the mobile device hardware as radio frequency characteristics.

[0022] Because crystal oscillators cannot be manufactured with absolute precision, there is inevitably a slight deviation between the local oscillator frequency and the nominal frequency of each mobile device. This inherent deviation, and its changing trend over time, constitutes a unique fingerprint characteristic. In practice, a Fast Fourier Transform (FFT) is first applied to convert the time-domain RF signal into a frequency-domain signal, and a peak search algorithm is used to locate the center of the carrier frequency. Next, the preamble portion of the signal is extracted, and orthogonal demodulation technology is used to extract the in-phase and quadrature components of the baseband signal. The instantaneous phase difference between adjacent sampling points is calculated, thereby deriving the instantaneous frequency offset. Arranging the continuous frequency offsets over a period of time in a time sequence yields the frequency deviation curve. The RF characteristics extracted in this way have extremely high anti-interference and non-forgeability; even if the device changes its MAC address or network identifier, its underlying hardware frequency deviation characteristics remain stable.

[0023] S103. Perform optical flow analysis on the image data to generate the displacement velocity sequence of the target object, and simultaneously analyze the phase change of the radio frequency signal to generate the corresponding Doppler frequency shift velocity sequence. By calculating the cross-covariance between the displacement velocity sequence and the Doppler frequency shift velocity sequence, the target radio frequency feature with the greatest spatiotemporal correlation is identified and the target object is identified, and a tracking identifier carrying the target radio frequency feature is generated.

[0024] The motion field of a target object is inferred by detecting temporal changes in pixel intensity in an image sequence. Specifically, the Lucas-Cannard method can be used to track the displacement of target feature points between consecutive frames, generating a displacement-velocity sequence describing the target's movement speed in physical space. Simultaneously, for radio frequency signals, the Doppler effect of radio waves is used to analyze the phase change rate of the signal upon arrival at the receiver. As the mobile device moves closer to or away from the receiver, its signal frequency shifts accordingly, and this shift is proportional to the movement speed. By calculating the derivative of the received signal phase, a Doppler frequency shift velocity sequence reflecting the radial movement speed of the signal source can be constructed. The cross-covariance between these two sequences is then calculated to quantify their correlation. When the cross-covariance reaches its maximum value, it indicates that the motion pattern of the visual target is highly consistent with the motion pattern of the radio frequency signal source, thus determining that they belong to the same entity. The extracted hardware frequency deviation features are then bound to the visual target to generate a unique tracking identifier.

[0025] S104. Within the visual blind zone of the multimodal capture environment, the prior location region of the target object is predicted based on the Kalman filter algorithm and the state of the tracking marker in the previous moment, and the preset distributed antenna array is controlled to search for signal echoes containing radio frequency characteristics.

[0026] In this process, when the target object enters a blind spot beyond the camera's coverage, the tracking task smoothly transitions to a pure radio frequency (RF) mode, utilizing the Kalman filter algorithm to maintain continuous estimation of the target's state. Kalman filtering, as a recursive estimation algorithm, can predict the current state using the best estimate from the previous moment and the current input control, making it well-suited for handling noisy dynamic systems. In this step, the target's two-dimensional position coordinates and velocity are used as the state vector. Using the tracking marker and its motion state determined in the previous moment, the system predicts the prior location region where the target might appear at the current moment. To verify this prediction and obtain more accurate position information, the system controls a pre-defined distributed antenna array to focus its beam on this prior region for scanning. The distributed antenna adjusts the transmission phase of each element to form a directional beam, actively searching for and receiving signal echoes containing specific frequency deviation characteristics. This not only narrows the search range and reduces computational complexity but also improves the signal-to-noise ratio of the received signal through beamforming technology. This allows for accurate capture of weak target signals even in the blind spot, despite multipath effects or environmental interference, achieving a seamless transition from visual perception to RF perception.

[0027] S105. The joint probabilistic data interconnection algorithm is applied to calculate the spatial distance between each signal echo received in the visual blind zone and the prior position area, as well as the feature similarity between each signal echo and the target radio frequency feature in the tracking mark. The spatial distance and feature similarity are combined to generate the association probability weight, and the signal echoes are weighted and fused according to the association probability weight to update the posterior position coordinates of the target object and generate the radio frequency tracking trajectory in the visual blind zone.

[0028] In complex tracking environments, received signals often contain direct paths, reflected paths, and noise interference, necessitating the application of a joint probabilistic data interconnection algorithm to accurately determine the target's location. The core idea of ​​this algorithm is not simply to select the strongest signal, but rather to calculate the probability that all candidate echoes belong to the target object. First, the Euclidean distance between the location of the received signal echo within the blind zone and the center of the prior location region predicted by the Kalman filter is calculated; this distance reflects spatial proximity. Simultaneously, the waveform features of the echo are extracted and compared with the target's radio frequency features stored in the tracking identifier to calculate feature similarity. Combining these two dimensions, a Gaussian distribution model is used to generate the associated probability weights for each echo. Finally, the weighted sum of all valid echoes is used to update the target's posterior position coordinates. This weighted fusion mechanism significantly suppresses the influence of false echoes, generating a smooth and accurate radio frequency tracking trajectory.

[0029] S106. When the target object leaves the visual blind zone according to the radio frequency tracking trajectory, the radio frequency fluctuation sequence of the radio frequency tracking trajectory and the motion displacement sequence of the candidate object captured by the camera at the exit of the visual blind zone are collected, and the radio frequency fluctuation sequence and motion displacement sequence are mapped to a preset low-dimensional space using a manifold alignment algorithm.

[0030] When the monitoring system detects that a target is about to leave the blind zone and re-enter the field of view of another surveillance camera, cross-modal identity re-identification is required to complete the closed-loop tracking. At this point, the radio frequency (RF) fluctuation sequence of the final segment of the RF tracking trajectory generated within the blind zone is extracted, and simultaneously, the motion displacement sequences of all candidate objects captured by the camera at the blind zone exit are obtained. Since these two types of data belong to the electromagnetic signal domain and the visual image domain, respectively, and their data dimensions and physical meanings are completely different, direct comparison is extremely difficult. Therefore, a manifold alignment algorithm is used to solve this heterogeneous data matching problem. This algorithm assumes that although the two different modalities of data have different forms in high-dimensional space, their inherent low-dimensional manifold structures are similar when describing the motion process of the same entity. The algorithm finds two specific mapping functions to simultaneously project the high-dimensional RF fluctuation sequence and motion displacement sequence into a shared low-dimensional latent space. In the low-dimensional space, previously unrelated physical quantities are transformed into abstract feature vectors, preserving the geometric structure of data from the same source while maximizing the differences between data from different sources, thus determining the final identity.

[0031] S107. Calculate the topological distance between the mapped feature vectors in the low-dimensional space, and determine the candidate objects and the target objects corresponding to the topological distances that are less than a preset threshold as the same entity.

[0032] After manifold alignment mapping, the fluctuation characteristics of radio frequency signals and the displacement characteristics of visual actions in low-dimensional space are transformed into feature vectors of a unified dimension. At this point, the topological distance between the mapped radio frequency feature vector and the visual feature vector is calculated, typically using Euclidean distance or geodesic distance. If both originate from the same entity, their topological positions on the low-dimensional manifold should be extremely close, even if the original data modes differ. An empirical threshold is set, derived statistically from the distribution of historical training data. If the calculated distance is less than this threshold, the current radio frequency signal source and the candidate object captured by the camera are determined to be the same entity. This manifold topology-based determination method effectively identifies non-rigid correspondences, i.e., the intrinsic connection between the nonlinear fluctuations of radio frequency signal intensity and the non-uniform motion of pedestrians. Once the determination is successful, the previous blind spot trajectory can be stitched together with the current visual trajectory, achieving continuous tracking across blind spots.

[0033] In one embodiment, performing frequency domain analysis on the acquired radio frequency signal and extracting the frequency deviation curve of the mobile device hardware as radio frequency characteristics includes the following steps: The Fast Fourier Transform is used to perform frequency domain analysis on radio frequency signals to identify and extract the preamble sequence of energy burst segments in the radio frequency signals. The preamble sequence is quadrature demodulated to obtain in-phase and quadrature components. The in-phase and quadrature components are then used to calculate the instantaneous phase difference between adjacent symbols in the preamble sequence. The instantaneous offset of the signal source relative to the standard center frequency is calculated based on the instantaneous phase difference and arranged in chronological order to form a frequency deviation curve, which is then used as a radio frequency feature.

[0034] In this embodiment, the original time-domain radio frequency signal acquired by the receiver is first discretized and sampled to form a discrete-time sequence. Due to the presence of environmental noise and unwanted signals, directly detecting the signal start point in the time domain is often inaccurate. Therefore, a Fast Fourier Transform (FFT) is used to convert the time-domain signal into a frequency-domain signal. By calculating the energy distribution on the spectrum, it can be clearly observed that the energy in a specific frequency range surges instantaneously when a data packet arrives. A dynamic energy threshold is set. When the spectral energy within a certain time window exceeds the threshold, it is determined that an energy burst segment of the signal has been detected. In communication protocols, data packets usually begin with a fixed, known sequence, i.e., a preamble, used for synchronization and channel estimation at the receiver. A sliding window algorithm is used to search at the beginning of the energy burst segment, and the intercepted signal fragment is cross-correlated with a locally stored standard preamble template. When the cross-correlation peak exceeds a preset matching degree, the start and end positions of the preamble can be accurately located and extracted for subsequent processing. This process effectively eliminates invalid noise interference, ensuring that subsequent feature extraction is based on the real and valid signal portion.

[0035] Next, the preamble sequence is orthogonally demodulated to obtain in-phase and quadrature components. These components are then used to calculate the instantaneous phase difference between adjacent symbols in the preamble sequence. The instantaneous frequency is defined as the derivative of the instantaneous phase with respect to time; therefore, in the discrete domain, the frequency offset is proportional to the phase difference. Specifically, by using the instantaneous phase difference and the signal's sampling period, the deviation of the current instantaneous frequency relative to the carrier frequency can be derived. Theoretically, if the transmitter's crystal oscillator is perfectly ideal, this frequency offset should be a constant or zero. However, due to differences in hardware manufacturing processes, each mobile device's crystal oscillator will produce a unique frequency drift that changes slowly over time. Arranging the calculated instantaneous frequency offsets corresponding to each sampling point in chronological order creates a curve that fluctuates with time—the frequency offset curve. Because the aging degree, temperature characteristics, and initial errors of crystal oscillators vary among different devices, this frequency offset curve exhibits significant differences in shape, slope, and intercept, demonstrating extremely high discriminative power. Therefore, by using this curve as a radio frequency characteristic, it is possible to accurately identify and track specific mobile devices. Even if the network identifier of the device is changed, the frequency deviation characteristics of its physical layer remain stable.

[0036] In one embodiment, quadrature demodulation of the preamble sequence yields in-phase and quadrature components. Calculating the instantaneous phase difference between adjacent symbols in the preamble sequence using the in-phase and quadrature components includes the following steps: The blind source separation algorithm is used to perform orthogonal demodulation on the preamble sequence to separate the in-phase component and the quadrature component. A nonlinear distortion model based on the complex domain is constructed, and the complex envelope signal of the preamble sequence is reconstructed using in-phase and quadrature components, and the instantaneous phase of the complex envelope signal is calculated. The complex envelope signal is expanded using a Wolterra series to identify and extract the distortion coefficients generated by the power amplifier during signal amplification. Calculate the residual phase error sequence of the instantaneous phase after eliminating the ideal linear phase rotation; The residual phase error sequence after correction using the distortion coefficient is used as the instantaneous phase difference.

[0037] In this embodiment, the received preamble sequence is regarded as the observation signal vector. Assuming the observed signal is from an ideal in-phase component source and orthogonal component sources It is formed through some kind of linear mixing. Independent component analysis is applied as the specific implementation algorithm for blind source separation to construct an unmixing matrix. This makes the output signal The components are made as statistically independent as possible. Since in-phase and quadrature components are statistically independent, the unmixing matrix is ​​iteratively updated by maximizing the non-Gaussianity of the output components. until the two components are separated. and The independence criterion is satisfied. At this point, the two separated independent components correspond to the corrected in-phase components, respectively. and orthogonal components This method does not require prior knowledge of the channel parameters and can adaptively compensate for the IQ mismatch in the quadrature modulator, thereby obtaining high-quality baseband signal components. Then, a nonlinear distortion model based on the complex domain is constructed, and the complex envelope signal of the preamble sequence is reconstructed using in-phase and quadrature components, and the instantaneous phase of the complex envelope signal is calculated. In specific implementation, the in-phase component is used... and orthogonal components Constructing a complex envelope signal using real and imaginary parts. ,in The imaginary unit is used. The constructed nonlinear distortion model... Describes the input complex signal The mapping relationship between the complex envelope signal and the ideal signal. To accurately capture the instantaneous state of the signal, it is necessary to calculate the amplitude of the complex envelope signal at each sampling time. and phase The formula for calculating the amplitude is as follows: The instantaneous phase is calculated using the arctangent function in the four quadrants, i.e. .

[0038] Next, a Wolterra series expansion is performed on the complex envelope signal to identify and extract the distortion coefficients generated by the power amplifier during signal amplification. The Wolterra series, as a powerful tool for describing nonlinear time-invariant systems, can represent the system output as the sum of convolutions of the inputs of various orders, thus accurately fitting the memory effect and nonlinear characteristics of the power amplifier. In specific implementations, for the discretized complex envelope signal... The input-output relationship is modeled as a truncated discrete Wolterra series. Considering computational complexity, odd-order terms are typically retained because even-order terms usually produce out-of-band frequency components. The model expression can be simplified to... ,in The fitted output signal, For the maximum nonlinear order, That is, the first one to be solved The order Vorterra kernel coefficients, also known as distortion coefficients, are used to minimize the fitting error using the least squares algorithm. Solve the above system of equations to estimate the optimal set of coefficients. These extracted distortion coefficients It directly reflects the physical characteristics of the transmitter power amplifier, forming a highly recognizable radio frequency feature.

[0039] In ideal communication, the signal phase either increases linearly with time or changes according to modulation rules. An ideal reference phase trajectory is generated based on the standard symbol sequence of the preamble. Simultaneously, the instantaneous phase obtained from actual measurements was analyzed using linear regression. A fitting process is performed to remove the linear trend term (phase slope caused by carrier frequency offset) and the intercept term (initial phase), resulting in a detrended phase sequence. The detrended phase sequence is then subtracted from the ideal reference phase trajectory to calculate the residual phase error. ,in For the estimated frequency deviation, The sampling interval is... The initial phase is used. Finally, the previously obtained Volterra series distortion coefficients are used. Calculate the phase distortion component caused by nonlinear effects. This component can be calculated from the model output. With input The phase difference between the phases can be obtained, or it can be directly derived from the phase angle of the coefficients. Then, this nonlinear phase distortion component is used to adjust the residual phase error sequence. Perform compensation and calculate the corrected sequence. This correction process effectively cancels out the AM-PM conversion effect of the power amplifier, leaving the remaining phase fluctuations primarily attributable to the phase noise of the local oscillator and the jitter of the frequency synthesizer. Finally, this double-calibrated sequence... Defined as the final instantaneous phase difference feature. This feature not only eliminates first-order frequency deviations but also suppresses higher-order nonlinear interference, greatly improving the stability and consistency of RF fingerprints at different transmit power levels, and achieving accurate identification of mobile devices.

[0040] In one embodiment, optical flow analysis is performed on image data to generate a displacement velocity sequence of the target object. Simultaneously, the phase change of the radio frequency signal is analyzed to generate a corresponding Doppler frequency shift velocity sequence. By calculating the cross-covariance between the displacement velocity sequence and the Doppler frequency shift velocity sequence, the target radio frequency feature with the greatest spatiotemporal correlation is identified and linked to the target object. The generation of a tracking identifier carrying the target radio frequency feature includes the following steps: The optical flow algorithm is used to track the feature points of pedestrian targets in image data, calculate the displacement pixel difference of feature points on the image plane and convert it into a displacement velocity sequence in physical space; Extract the phase slope change rate of the signal source corresponding to the radio frequency features, and calculate the radial movement speed of the signal source relative to the receiver based on the phase slope change rate to generate a Doppler frequency shift speed sequence; Construct an association matrix with visual targets as rows and radio frequency features as columns, and traverse the association matrix to calculate the Pearson correlation coefficient between the displacement velocity sequence and the Doppler frequency shift velocity sequence corresponding to each element position; The target pair with the highest Pearson correlation coefficient is selected from the correlation matrix, and the radio frequency features in the target pair are marked as the tracking ID of the target object, generating a tracking identifier carrying the target radio frequency features.

[0041] In this embodiment, the continuously acquired video frames are preprocessed, including grayscale conversion and Gaussian smoothing, to reduce the impact of illumination changes and noise on feature point extraction. The Shi-Tomasi corner detection algorithm is used to identify several corner points with significant texture features in the first frame as feature points to be tracked. Subsequently, the Lucas-Kanade sparse optical flow algorithm is applied, assuming that the grayscale values ​​of neighboring pixels remain unchanged within a very short time interval, and the displacement vector of the feature point between adjacent frames is solved by constructing a Taylor series equation system. Let... The position of the feature point at time step in the image coordinate system is , The position of the moment is Then the pixel displacement difference is To convert pixel displacement into physical displacement, coordinate transformation is required using the camera's intrinsic and extrinsic parameter matrices. This is achieved based on a pinhole camera model, combined with target depth information obtained from a depth sensor. Using the formula Calculate the displacement velocity in physical space, where The focal length of the camera. This is the inter-frame time interval. The calculated consecutive time intervals... Arranged in chronological order, this generates a sequence of the pedestrian's displacement and velocity.

[0042] When a mobile device moves relative to the receiving antenna, the length of the wireless signal propagation path changes, causing a continuous phase shift in the received signal. The underlying principle is that the rate of phase change directly corresponds to the Doppler shift, which is proportional to the radial velocity. In practice, channel state information is first extracted from the received radio frequency data packets to obtain complex channel responses on different subcarriers. For a specific subcarrier, the phase information of consecutive data packets is extracted and de-wound to eliminate the influence of periodic phase jumps, resulting in a continuous phase curve. The slope of the phase curve within a very short time window, i.e., the rate of phase change, is calculated using linear regression or the finite difference method. Based on electromagnetic wave propagation theory, the phase change rate and radial velocity... The relationship between them can be represented as ,in Let be the carrier wavelength of the radio frequency signal. Using this formula, the extracted phase slope change rate is converted in real time into the radial movement velocity of the signal source relative to the receiver. This calculation process is continuously performed on the time axis, and the calculated velocity values ​​are organized into a time series to generate a Doppler frequency shift velocity sequence reflecting the motion characteristics of the signal source. This sequence strips away the specific content of the signal, retaining only the physical layer characteristics related to motion, and possesses extremely high anti-interference and real-time performance.

[0043] Next, let's assume that the scenario simultaneously contains... Pedestrian targets detected by vision and One radio frequency signal source, to construct a Dimensional association matrix Each element in the matrix Representing the The visual target and the first The correlation probability between each radio frequency signal source. To fill this matrix, the corresponding velocity sequence matching degree needs to be calculated one by one. Take out the... Displacement velocity sequence of a visual target and the Doppler frequency shift velocity sequence of a radio frequency signal source ,in Let be the sequence length. The linear correlation between the two is calculated using the Pearson correlation coefficient formula, which is: ,in and These are the means of the two sequences, respectively. Pearson correlation coefficient. The range of values ​​is The closer the value is to 1, the more consistent the trends of the two changes, i.e., the stronger the positive correlation. By iterating through all possible combinations and calculating the correlation coefficient, the obtained... Fill in the correlation matrix The corresponding position. In the association matrix, the element with the largest value is most likely to represent the true matching relationship; therefore, a greedy algorithm or the Hungarian algorithm can be used to analyze the constructed matrix. Global optimization is performed on the incidence matrix. First, the maximum value in the matrix is ​​searched. If the maximum value exceeds the preset confidence threshold, then the corresponding first... The visual target and the first The radio frequency signal sources are the same entity. After locking this pair, the first one in the matrix will be... row and number Other elements in the column are set to invalid to prevent duplicate matches. After a successful match is confirmed, the hardware layer features of the radio frequency signal source are extracted, and then the visual features are bound to the hardware layer features to generate a composite data structure, namely the tracking identifier.

[0044] In one implementation, a joint probabilistic data interconnection algorithm is applied to calculate the spatial distance between each signal echo received within the visual blind zone and the prior location region, as well as the feature similarity between each signal echo and the target radio frequency feature in the tracking marker. An association probability weight is generated by combining the spatial distance and feature similarity. The signal echoes are then weighted and fused according to the association probability weight to update the posterior location coordinates of the target object. The process of generating a radio frequency tracking trajectory within the visual blind zone includes the following steps: Calculate the position coordinates of each signal echo received within the visual blind zone, and calculate the Euclidean distance between each position coordinate and the center point of the prior position region; Extract the waveform features of each signal echo and calculate the cosine similarity between the waveform features and the target radio frequency features in the tracking identifier; Construct a joint likelihood function that includes Euclidean distance and cosine similarity, and normalize the output of the joint likelihood function to obtain the association probability weight of each signal echo belonging to the target object; The estimated location value is obtained by weighting and summing all location coordinates using the associated probability weights; The state vector of the Kalman filter algorithm is corrected using the position estimate to obtain the updated posterior position coordinates, which are then connected in time series to generate the radio frequency tracking trajectory within the visual blind zone.

[0045] In this embodiment, in the visual blind spot, a distributed antenna array receives multipath signal echoes from the environment. First, a joint localization algorithm using angle of arrival (Angle of Arrival) and time difference of arrival (Time Difference of Arrival) is used to calculate the physical location of each signal echo. For the first... The received signal echo is analyzed, and the incident angle of the signal is calculated by analyzing its phase difference at different antenna elements. A hyperbolic equation system is then established based on the time difference of the signal arriving at each element, and the coordinates of the echo on the two-dimensional plane are obtained by solving it. Meanwhile, the Kalman filter predicts the potential prior location region of the target at the current moment based on the target state at the previous moment. This region is typically defined by the coordinates of a center point. The covariance matrix describes the spatial proximity of each signal echo to the predicted location. This is calculated... and The Euclidean distance between two points. Euclidean distance is a measure of the straight-line distance between two points and effectively reflects the deviation between the signal source's location and the expected location. The calculation formula is as follows: By calculating this distance, echoes that are too far from the prediction center and are clearly environmental noise or irrelevant interference sources can be distinguished. For each received signal echo, a segment of waveform data containing the complete preamble or reference signal is first extracted in the time domain. This waveform segment is then mapped to the frequency domain using a Fast Fourier Transform, and its amplitude and phase spectra are extracted as waveform feature vectors. This feature vector not only contains information about the energy distribution of the signal, but also implicitly contains the inherent frequency deviation and nonlinear distortion characteristics of the transmitter hardware.

[0046] The target object's standard radio frequency feature vector has already been stored in the tracking identifier generated in the multimodal acquisition environment. To quantify the matching degree between the current echo features and standard features, a cosine similarity algorithm is employed. Cosine similarity assesses the directional similarity of two vectors by calculating the cosine of the angle between them. It is insensitive to changes in the absolute value of signal strength and is well-suited for handling signal attenuation issues caused by variations in transmission distance. The calculation formula is as follows: ,in Represents the vector dot product. The magnitude of the vector is used to calculate the similarity. The closer the value is to 1, the more likely the echo is to belong to the target not only in terms of location but also in terms of physical characteristics. However, relying solely on location and distance may lead to misjudgment due to multipath effects, while relying solely on feature similarity may be affected by environmental noise. Therefore, it is necessary to construct a joint likelihood function. To simultaneously evaluate these two metrics, this function is typically designed with a Gaussian distribution, assuming that the target location follows a normal distribution and the feature matching error also follows a normal distribution. Substituting the Euclidean distance and cosine similarity calculated in the previous steps into the likelihood function, all... After calculating the likelihood value of the candidate echo, normalization is required to transform it into a probabilistic weight. The calculation of the... The association probability weight of each echo The formula is , This represents the likelihood value. After normalization, the sum of all weights is 1. Directly reflects the first Each echo originates from the confidence level of the real target.

[0047] Traditional nearest neighbor methods select only the most probable echo as the target measurement, which can lead to tracking divergence if clutter is selected. Therefore, the final target location estimate should not be the coordinates of a single echo, but rather a weighted average of the coordinates of all candidate echoes. In practice, this is achieved by utilizing the correlation probability weights of each echo. As a weighting factor, the position coordinates of all candidate echoes Perform a linear combination. The calculation formula is as follows: Furthermore, if the correlation probability of all echoes is very low in certain extreme cases, then the weights... This will dominate, and the location estimate will rely more on prior predictions than on the current measurement. This weighted fusion mechanism effectively smooths out measurement noise, especially when there are multiple nearby sources of interference or multipath reflection points. It can automatically suppress the influence of outliers through probability weighting, thus obtaining an estimate that is closer to the true target location than a single measurement. .

[0048] Since the Kalman filter already provides a priori prediction of the state based on the state at the previous time step. and the prior covariance matrix At this moment, the new location estimate obtained using weighted fusion... Calculate the Kalman gain using the current observation input. The Kalman gain determines the extent to which current observations should be trusted and the extent to which prior predictions should be retained when updating the state. The calculation formula is as follows: ,in For the observation matrix, To observe the noise covariance, the prior state is corrected using Kalman gain to obtain the posterior state vector. The position component in the posterior state vector is the optimal position estimate of the target within the visual blind zone at the current moment. Simultaneously, the covariance matrix is ​​updated for use in the next iteration. The posterior position coordinates calculated at each moment are stored sequentially according to the timestamp and concatenated to form a continuous, smooth curve, which is the radio frequency tracking trajectory of the target within the visual blind zone, achieving continuous and accurate localization without optical image assistance.

[0049] In one implementation, after calculating the estimated location value by weighted summation of all location coordinates using associated probability weights, the following steps are also included: Determine the current azimuth and pitch angles of the target object within the visual blind spot based on the position estimate; Calculate the phase delay parameters of each element in the distributed antenna array so that the main beam of the distributed antenna array points to the current azimuth and elevation angles; The direction of the source of a signal echo whose correlation probability weight is lower than a preset noise threshold is identified and set as the interference direction. Adjust the array weight vector of the distributed antenna array to create nulls in the interference direction; The distributed antenna array is controlled by applying the adjusted phase delay parameters and array weight vector to enhance the receiving gain of the signal echo containing the target RF characteristics at the next moment.

[0050] In this embodiment, the distributed antenna array in the visual blind spot is typically installed at a fixed reference position, which serves as the origin of the local coordinate system. During implementation, the target position estimate after Kalman filtering smoothing from the previous moment is first obtained. Assume the center of the antenna array is located at the origin. And the array plane is parallel to Plane. Calculate the angular relationship between the target and the array center using inverse trigonometric functions. Azimuth angle. Defined as the projection of the target position vector onto the horizontal plane and The angle between the two axes in the positive direction is calculated using the following formula: Pitch angle Defined as target position vector and The angle between the positive axis and the horizontal plane (or the angle with the horizontal plane, depending on the specific definition; here it is taken as the angle with the horizontal plane). (Including the axis), the calculation formula is as follows: To ensure the continuity and accuracy of angle calculations, it is necessary to perform quadrant determination and correction on the output of the arctangent function, which is done using the atan2 function.

[0051] Next, by changing the relative phases of the feeds to each radiating element in the array, the electromagnetic waves radiated by each element are made to superimpose in phase in a specific direction to form the main lobe, while canceling each other out in other directions to form side lobes or nulls. For the electromagnetic waves radiated by each element... A distributed array consisting of n array elements, assuming the nth element... The coordinates of the positions of each array element in space are To make the main lobe of the beam point in the desired direction. It is necessary to compensate for the path difference caused by the spatial differences of array elements. In specific implementation, the calculation of the first... Phase delay required for each element The formula is ,in The operating wavelength of the radio frequency signal is given. This calculation process needs to be performed in real time, and the phase delay parameter will be dynamically adjusted as the target position is updated. By applying the calculated precise phase delay to each array element, the electromagnetic waves transmitted or received by the array will form coherent enhancement in the target direction, greatly improving the received signal-to-noise ratio and the system's detection range. In actual indoor or multipath environments, the antenna array not only receives the direct signal from the target, but also multipath signals reflected from walls and furniture, as well as co-channel interference signals emitted by other non-target devices. Therefore, it is also necessary to base the calculation on the correlation probability weight. All received signal echoes are classified. An empirical noise threshold is set. This threshold is typically determined based on the ambient noise level and historical false alarm rate. It iterates through all signal echoes; if a certain echo... Association probability weights If the echo originates from a non-target object or is strong reflection clutter, then its source direction can be determined using the spatial spectrum estimation result or angle of arrival measurement value of the invalid echo. .

[0052] By adjusting the complex weighting coefficients of each element in the array, while maintaining the main beam pointing towards the target, extremely deep radiation nulls are formed in the direction of interference, thereby spatially filtering out interference signals. In specific implementation, the covariance matrix of the array's received signal is constructed. This matrix contains statistical information on interference and noise in the environment. The constraint is set as follows: in the target direction... The gain remains constant. Solve for the optimal weight vector. The optimization problem has the analytical solution as follows: ,in The direction guide vector for the array. This indicates the conjugate transpose. This represents matrix inversion. The calculated weight vector... Automatically adjust the antenna pattern to align with the identified interference direction. This produces a minimum response. Finally, the phase delay parameter is... With the optimal weight vector The amplitude and phase information in the data are synthesized. Specifically, for the first... Each array element's final control parameter is a complex weight. ,in It is the amplitude weighting coefficient. This is the phase shift introduced by the weight vector. These parameters are sent to the RF front-end components of the distributed antenna array via a digital control interface, including phase shifters and variable gain amplifiers. The phase shifters adjust the signal delay based on the phase information, and the variable gain amplifiers adjust the signal gain based on the amplitude information. Once the hardware configuration is complete, the antenna array will receive signals from the target direction with extremely high directivity gain in the next sampling period, while deeply suppressing signals from the interference direction. This gives the system strong environmental adaptability during RF tracking, maintaining signal link stability even if the target enters a strong interference area while moving, ensuring continuous high signal-to-noise ratio extraction of RF features, thereby maintaining the continuity of the tracking trajectory.

[0053] In one implementation, constructing a joint likelihood function that includes Euclidean distance and cosine similarity, and normalizing the output of the joint likelihood function to obtain the association probability weight of each signal echo belonging to the target object includes the following steps: The joint likelihood function is defined as the product of the spatial Gaussian distribution function and the feature matching probability function; Substitute the Euclidean distance into the spatial Gaussian distribution function to calculate the spatial probability components of the signal echo that conform to the prior prediction in spatial location. The cosine similarity is mapped to a feature matching probability function, and the feature probability components of the signal echo that physically matches the tracking identifier are calculated. The joint likelihood value is obtained by multiplying the spatial probability component and the characteristic probability component, and then the joint likelihood value is corrected by using the clutter density parameter. Summarize the corrected joint likelihood values ​​of all signal echoes, calculate the proportion of each signal echo in the total likelihood value, and obtain the normalized association probability weight of each signal echo belonging to the target object.

[0054] In this implementation, Bayesian estimation theory posits that the probability of a measurement belonging to a true target is jointly determined by its spatial location bias and feature attribute bias. The spatial location distribution is typically assumed to follow a two-dimensional Gaussian distribution because target location prediction errors and measurement noise often satisfy the central limit theorem. The feature matching probability function is based on the similarity measure of RF fingerprints. Since noise interference also exists during feature extraction, the feature differences can also be approximated as following a Gaussian distribution or a similar bell-shaped distribution. Multiplying these two independent probability density functions constructs the joint likelihood function. The mathematical form of this function can be expressed as: ,in Let Gaussian distribution function be the spatial distribution function. Let be the feature matching probability function. Represents positional variables. Represents the characteristic variable. The first variable, calculated using the previous steps... Each signal echo location The prior location of the region predicted by Kalman filtering Euclidean distance between Assume the target location prediction error follows a mean of 0 and a variance of . The Gaussian distribution, where The standard deviation of the position prediction error is denoted by , and this parameter is determined by the system's positioning accuracy and the noise of the target motion model. The Euclidean distance is then expressed as... Substituting into the standard Gaussian probability density function formula, the spatial probability components are calculated. The calculation formula is: This formula shows that the closer the echo location is to the prediction center (i.e., The smaller the distance, the greater its spatial probability component; conversely, as the distance increases, the probability value decreases exponentially.

[0055] Using the previously calculated first Cosine similarity between the waveform characteristics of a signal echo and the standard features in the target tracking marker Considering similarity values The range is usually in The values ​​between 1 and 1 indicate a higher degree of matching, and therefore need to be mapped to a function that conforms to the characteristics of a probability distribution. This is due to feature matching error (i.e., ...). Features often follow a normal distribution, therefore a feature matching probability function based on a Gaussian kernel is constructed. The standard deviation of the feature matching error is set to... This parameter reflects the stability of the radio frequency characteristics during transmission and the signal-to-noise ratio level of the receiver. Calculate the characteristic probability components. The formula is This function will calculate the similarity. Convert to probability density value, when The function value reaches its peak value when it approaches 1; when When the value decreases, the function value drops rapidly.

[0056] The calculated spatial probability components and feature probability components Multiplying them together yields the uncorrected joint likelihood value. However, in real-world environments, in addition to the target echo, there are also a large amount of clutter signals. Therefore, a clutter density parameter can also be introduced. This parameter represents the average number of false echoes per unit area. To reflect the significance of the target echo relative to clutter, the joint likelihood value needs to be corrected. Specifically, the joint likelihood value is compared with or weighted by the clutter density, or a clutter term is added to the subsequent normalization denominator. In this step, the corrected joint likelihood value is defined. Its calculation logic implicitly incorporates the influence of clutter. Within an observation period, all possible echoes compete mutually exclusively to become the measurement value of the actual target. Therefore, a particular echo... The probability that it is the real target should be equal to its corrected joint likelihood value. The proportion of the total likelihood values ​​of all candidate echoes. The calculation formula is: ,in It is the total number of valid echoes received. It is a likelihood term representing the no-target or pure clutter assumption, typically related to the aforementioned clutter density parameter. and detection probability Related. By introducing The algorithm can handle cases where the target is missed, i.e., when the likelihood values ​​of all echoes are very low. The value will also be small, and location updates will primarily rely on the predicted value. (Calculated...) That is, the first The correlation probability weights of each signal echo, and satisfying This weight value directly reflects the contribution of the echo to the target state update and is used for subsequent weighted position estimation.

[0057] In one embodiment, acquiring the radio frequency fluctuation sequence of the radio frequency tracking trajectory and the motion displacement sequence of the candidate object captured by the visual blind spot exit camera, and mapping the radio frequency fluctuation sequence and the motion displacement sequence to a preset low-dimensional space using a manifold alignment algorithm includes the following steps: Set a time window at the end of the visual blind spot, and extract the signal intensity sequence corresponding to the radio frequency tracking trajectory within the time window as the radio frequency fluctuation sequence; Candidate objects are detected in the monitoring footage of the exit camera in the visual blind spot. The pose estimation model is used to extract the changes in the coordinates of the skeletal key points of the candidate objects in the last time window and generate a motion displacement sequence. Construct the Laplacian matrix of the radio frequency wave sequence and the motion displacement sequence, and calculate the local geometric structure features of the radio frequency wave sequence and the motion displacement sequence respectively; Establish a manifold alignment objective function, which includes constraint terms that preserve the local geometric structure features of each data type and correspondence terms that minimize the correspondence between heterogeneous data types. By solving the generalized eigenvalue problem of the manifold alignment objective function, the radio frequency domain mapping matrix and the visual domain mapping matrix are obtained; The radio frequency fluctuation sequence is projected to a preset low-dimensional space using a radio frequency domain mapping matrix, and the motion displacement sequence is projected to the same low-dimensional space using a visual domain mapping matrix.

[0058] In this implementation, the coordinates of the radio frequency tracking trajectory are first monitored. When the coordinates indicate that the target is approaching the blind zone boundary, a data recording mechanism is triggered. A fixed-length time window is defined. This window typically covers several seconds before and after the target crosses the boundary. During this period, the amplitude values ​​of the received signal strength indication or channel state information recorded by the radio frequency receiver are sampled frequently. Due to multipath effects and human body obstruction, the signal strength fluctuates dramatically but in a specific pattern with even slight movements of the target. All sampled values ​​during this period are filtered and smoothed to remove high-frequency noise and retain the low-frequency fluctuation components reflecting the rhythm of human movement. The processed continuous signal strength values ​​are then arranged in chronological order to form a one-dimensional vector sequence. ,in This represents the number of sampling points. (This is the sequence number.) This refers to the radio frequency fluctuation sequence. When the camera at the exit detects a new pedestrian, the detection process is immediately initiated. This corresponds to the aforementioned final time window. The video frame sequence within that time period is extracted. A deep neural network model is applied to each frame to identify and locate key skeletal nodes of the human body. The two-dimensional coordinates of these key points in the image coordinate system are then extracted. To capture motion information, the displacement vectors of key points between adjacent frames are calculated, or the relative positional changes of key points with respect to the body's center of mass are calculated. Typically, the movement trajectory of the lower limb key points, which are most sensitive to motion, or the overall center of mass is selected as representative. The displacement amplitudes or coordinate changes of consecutive frames are combined in a time series to construct a motion displacement sequence of the same length as the radio frequency fluctuation sequence. .

[0059] Although radio frequency (RF) signals and visual signals reside in different data spaces, if they originate from the motion of the same entity, their local neighborhood relationships in their respective high-dimensional spaces should be similar. In practice, a nearest-neighbor graph is first constructed for RF fluctuation sequences. For each sampling point in the sample, find its Find the nearest neighbors and calculate the edge weights based on the Euclidean distance to construct the adjacency matrix. Similarly, this is a sequence of action displacements. Constructing an adjacency matrix Next, the Laplace matrix is ​​defined based on graph theory. For any mode, its Laplace matrix is... Defined as ,in It is a degree matrix, and its diagonal elements Calculate the Laplace matrix of the radio frequency data respectively. Laplacian matrix of visual data These two matrices encode the local geometric features of their respective data manifolds. Next, a manifold alignment objective function is established, which includes constraint terms that preserve the local geometric features of each data manifold and correspondence terms that minimize the correspondence between heterogeneous data. Specifically, the manifold alignment problem is transformed into an optimization problem.

[0060] objective function It typically consists of two parts: a structure-preserving term that maintains the manifold structure, and a correspondence term. The structure-preserving term ensures that the projected data retains the original local neighborhood relationships, i.e., minimizes... This can be formalized using the Laplace matrix as follows: In the form of, It is the projection matrix to be solved. This is the raw data. The corresponding terms, utilizing a limited number of known correspondences, force the distance between the radio frequency features and visual features at the same moment after projection to be as small as possible, i.e., minimize... By combining these two weighted components, we construct the overall objective function: ,in To adjust the parameters, and These are the mapping matrices for the radio frequency domain and the visual domain, respectively. The objective function aims to find two optimal linear mappings such that the mapped low-dimensional features both preserve the motion patterns of the original data and highly overlap at the same time. The problem of finding the extremum of the quadratic objective function constructed above under constraints can usually be transformed into a generalized eigenvalue problem for solution.

[0061] Specifically, the objective function is mapped to the matrix Taking the derivative and setting it to zero, we can obtain the form: The generalized eigenvalue equations of . Where, matrix It contains correspondence information in manifold alignment, matrix It includes information about the Laplace matrix and regularization terms. The equation is solved using numerical linear algebra algorithms to obtain a series of eigenvalues. and the corresponding feature vector Select the smallest one. The eigenvectors corresponding to the non-zero eigenvalues, arranged column-wise, constitute the required mapping matrix. Specifically, the first half of the solution vector corresponds to the radio frequency domain mapping matrix. The latter half corresponds to the visual field mapping matrix. After obtaining the mapping matrix, for the original radio frequency fluctuation sequence data... , apply formula Calculations are performed to obtain the dimension as follows: Low-dimensional feature representation Similarly, for the original motion displacement sequence data... , apply formula Calculations are performed to obtain the dimension as follows: Low-dimensional feature representation Here It is a preset low-dimensional space dimension. This refers to the sample size. Through this projection operation, radio frequency signal intensity data and visual pixel displacement data, which originally had completely different physical meanings and inconsistent dimensions, are uniformly mapped to the same abstract Euclidean space. In this new low-dimensional space, the coordinates of the data represent implicit motion manifold characteristics. This allows us to directly measure the similarity between radio frequency signal fluctuations and visual motion displacements by calculating the geometric distance between two vectors.

[0062] In one embodiment, calculating the topological distance between mapped feature vectors in a low-dimensional space, and determining candidate objects with topological distances less than a preset threshold as the same entity as the target object includes the following steps: Obtain the projected RF wave feature vector and the projected motion displacement feature vector in a low-dimensional space; The Euclidean distance between the radio frequency wave feature vector and the motion displacement feature vector is calculated as the topological distance. If the topological distance is less than the preset isomorphism determination threshold, it is confirmed that the fluctuation rhythm of the radio frequency signal and the displacement rhythm of the visual action have manifold isomorphism, and the candidate object and the target object are determined to be the same entity.

[0063] In this embodiment, through manifold learning and dimensionality reduction mapping, the original high-dimensional and heterogeneous radio frequency (RF) signals and visual signals have been transformed into points or vectors in the same low-dimensional latent space. Specifically, a pre-trained or real-time solved RF domain mapping matrix is ​​invoked. Visual domain mapping matrix For the raw radio frequency fluctuation sequence acquired at the end of the visual blind spot Multiply it by the transpose of the radio frequency domain mapping matrix on the left, i.e., perform the operation. The radio frequency fluctuation feature vector in the low-dimensional space is obtained. Similarly, for the original motion displacement sequence of the candidate object captured by the exit camera in the blind spot... Multiply it by the transpose of the visual field mapping matrix on the left and perform the operation. This yields the motion displacement feature vector in the same low-dimensional space. These two eigenvectors and The dimensions are completely identical, and their coordinate positions in this space directly encode the inherent manifold structure information of their respective original data. If two different modalities of data originate from the same physical process, then they should be close to each other in the low-dimensional space after manifold alignment projection, exhibiting topological consistency. Therefore, utilizing low-dimensional radio frequency wave feature vectors... and motion displacement feature vector The spatial distance between the two vectors is calculated using the Euclidean distance formula, and this distance is defined as the topological distance. The calculation formula is: ,in For the dimension of low-dimensional space, and The two vectors are respectively in the th... Component values ​​in each dimension. Distance of this topology. The smaller the value, the closer the two feature vectors are in low-dimensional space, meaning the more similar the original radio frequency signal fluctuation pattern and the visual action displacement pattern are in terms of manifold structure, i.e., their intrinsic motion rhythms are highly consistent. Conversely, if the distance is large, it indicates that their motion patterns do not match, and they may originate from different individuals. If the topological distance is less than the preset isomorphism judgment threshold, it is confirmed that the fluctuation rhythm of the radio frequency signal and the displacement rhythm of the visual action have manifold isomorphism, and the candidate object and the target object are determined to be the same entity.

[0064] The present invention also discloses a blind zone relay tracking system based on heterogeneous data spatiotemporal calibration, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the blind zone relay tracking method based on heterogeneous data spatiotemporal calibration as described above.

[0065] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it.

[0066] The memory can be an internal storage unit of a computer device, such as a hard disk or RAM, or an external storage device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) provided on the computer device. Furthermore, the memory can be a combination of internal storage units and external storage devices of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0067] The present invention also discloses a computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to be configured to perform the blind zone relay tracking method based on heterogeneous data spatiotemporal calibration described in any of the above embodiments.

[0068] The computer program can be stored in a machine-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The machine-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the machine-readable medium includes, but is not limited to, the above-mentioned components.

[0069] The blind zone relay tracking method based on heterogeneous data spatiotemporal calibration described in the above embodiments is stored in the computer-readable storage medium and loaded and executed on the processor to facilitate the storage and application of the above method.

[0070] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0071] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A blind zone relay tracking method based on heterogeneous data spatiotemporal calibration, characterized in that, Includes the following steps: Construct a multimodal capture environment at the blind zone entrance and simultaneously acquire image data of the target object entering the multimodal capture environment and radio frequency signals of all mobile devices; The acquired radio frequency signals are analyzed in the frequency domain, and the frequency deviation curve of the mobile device hardware is extracted as radio frequency characteristics. Optical flow analysis is performed on image data to generate displacement velocity sequence of target object. At the same time, the phase change of radio frequency signal is analyzed to generate corresponding Doppler frequency shift velocity sequence. By calculating the cross-covariance between displacement velocity sequence and Doppler frequency shift velocity sequence, the target radio frequency feature with the greatest spatiotemporal correlation is identified with the target object, and a tracking identifier carrying the target radio frequency feature is generated. Within the visual blind zone of the multimodal capture environment, the prior location region of the target object is predicted based on the Kalman filter algorithm and the state of the tracking marker in the previous moment, and the preset distributed antenna array is controlled to search for signal echoes containing radio frequency characteristics. The joint probabilistic data interconnection algorithm is applied to calculate the spatial distance between each signal echo received in the visual blind zone and the prior position area, as well as the feature similarity between each signal echo and the target radio frequency feature in the tracking marker. The spatial distance and feature similarity are combined to generate the association probability weight, and the signal echoes are weighted and fused according to the association probability weight to update the posterior position coordinates of the target object and generate the radio frequency tracking trajectory in the visual blind zone. When the target object leaves the visual blind zone according to the radio frequency tracking trajectory, the radio frequency fluctuation sequence of the radio frequency tracking trajectory and the motion displacement sequence of the candidate object captured by the camera at the exit of the visual blind zone are collected. The radio frequency fluctuation sequence and the motion displacement sequence are mapped to a preset low-dimensional space using a manifold alignment algorithm. In a low-dimensional space, the topological distance between mapped feature vectors is calculated, and candidate objects with topological distances less than a preset threshold are identified as the same entity as the target object.

2. The blind zone relay tracking method based on heterogeneous data spatiotemporal calibration according to claim 1, characterized in that, The step of performing frequency domain analysis on the acquired radio frequency signals and extracting the frequency deviation curve of the mobile device hardware as radio frequency characteristics includes the following steps: The Fast Fourier Transform is used to perform frequency domain analysis on radio frequency signals to identify and extract the preamble sequence of energy burst segments in the radio frequency signals. The preamble sequence is quadrature demodulated to obtain in-phase and quadrature components. The in-phase and quadrature components are then used to calculate the instantaneous phase difference between adjacent symbols in the preamble sequence. The instantaneous offset of the signal source relative to the standard center frequency is calculated based on the instantaneous phase difference and arranged in chronological order to form a frequency deviation curve, which is then used as a radio frequency feature.

3. The blind zone relay tracking method based on heterogeneous data spatiotemporal calibration according to claim 1, characterized in that, The process of performing optical flow analysis on image data to generate a displacement velocity sequence of the target object, and simultaneously analyzing the phase change of the radio frequency signal to generate a corresponding Doppler frequency shift velocity sequence, and calculating the cross-covariance between the displacement velocity sequence and the Doppler frequency shift velocity sequence, anchoring the target radio frequency feature with the greatest spatiotemporal correlation to the target object, and generating a tracking identifier carrying the target radio frequency feature includes the following steps: The optical flow method is used to track the feature points of pedestrian targets in image data, calculate the displacement pixel difference of feature points on the image plane and convert it into a displacement velocity sequence in physical space; Extract the phase slope change rate of the signal source corresponding to the radio frequency features, and calculate the radial movement speed of the signal source relative to the receiver based on the phase slope change rate to generate a Doppler frequency shift speed sequence; Construct an association matrix with visual targets as rows and radio frequency features as columns, and traverse the association matrix to calculate the Pearson correlation coefficient between the displacement velocity sequence and the Doppler frequency shift velocity sequence corresponding to each element position; The target pair with the highest Pearson correlation coefficient is selected from the correlation matrix, and the radio frequency features in the target pair are marked as the tracking ID of the target object, generating a tracking identifier carrying the target radio frequency features.

4. The blind zone relay tracking method based on heterogeneous data spatiotemporal calibration according to claim 1, characterized in that, The application of the joint probabilistic data interconnection algorithm calculates the spatial distance between each signal echo received within the visual blind zone and the prior location region, as well as the feature similarity between each signal echo and the target radio frequency features in the tracking marker. It then combines the spatial distance and feature similarity to generate associated probability weights, and performs weighted fusion of each signal echo based on these associated probability weights to update the posterior location coordinates of the target object. The generation of the radio frequency tracking trajectory within the visual blind zone includes the following steps: Calculate the position coordinates of each signal echo received within the visual blind zone, and calculate the Euclidean distance between each position coordinate and the center point of the prior position region; Extract the waveform features of each signal echo and calculate the cosine similarity between the waveform features and the target radio frequency features in the tracking identifier; Construct a joint likelihood function that includes Euclidean distance and cosine similarity, and normalize the output of the joint likelihood function to obtain the association probability weight of each signal echo belonging to the target object; The estimated location value is obtained by weighting and summing all location coordinates using the associated probability weights; The state vector of the Kalman filter algorithm is corrected using the position estimate to obtain the updated posterior position coordinates, which are then connected in time series to generate the radio frequency tracking trajectory within the visual blind zone.

5. The blind zone relay tracking method based on heterogeneous data spatiotemporal calibration according to claim 4, characterized in that, After calculating the estimated location value by weighted summation of all location coordinates using associated probability weights, the following steps are also included: Determine the current azimuth and pitch angles of the target object within the visual blind spot based on the position estimate; Calculate the phase delay parameters of each element in the distributed antenna array so that the main beam of the distributed antenna array points to the current azimuth and elevation angles; The direction of the source of a signal echo whose correlation probability weight is lower than a preset noise threshold is identified and set as the interference direction. Adjust the array weight vector of the distributed antenna array to create nulls in the interference direction; The distributed antenna array is controlled by applying the adjusted phase delay parameters and array weight vector to enhance the receiving gain of the signal echo containing the target RF characteristics at the next moment.

6. The blind zone relay tracking method based on heterogeneous data spatiotemporal calibration according to claim 4, characterized in that, The steps of constructing a joint likelihood function that includes Euclidean distance and cosine similarity, and normalizing the output of the joint likelihood function to obtain the association probability weight of each signal echo belonging to the target object, include the following: The joint likelihood function is defined as the product of the spatial Gaussian distribution function and the feature matching probability function; Substitute the Euclidean distance into the spatial Gaussian distribution function to calculate the spatial probability components of the signal echo that conform to the prior prediction in spatial location. The cosine similarity is mapped to a feature matching probability function, and the feature probability components of the signal echo that physically matches the tracking identifier are calculated. The joint likelihood value is obtained by multiplying the spatial probability component and the characteristic probability component, and then the joint likelihood value is corrected by using the clutter density parameter. Summarize the corrected joint likelihood values ​​of all signal echoes, calculate the proportion of each signal echo in the total likelihood value, and obtain the normalized association probability weight of each signal echo belonging to the target object.

7. The blind zone relay tracking method based on heterogeneous data spatiotemporal calibration according to claim 1, characterized in that, The process of acquiring the radio frequency fluctuation sequence of the radio frequency tracking trajectory and the motion displacement sequence of the candidate object captured by the visual blind spot exit camera, and mapping the radio frequency fluctuation sequence and motion displacement sequence to a preset low-dimensional space using a manifold alignment algorithm includes the following steps: Set a time window at the end of the visual blind spot, and extract the signal intensity sequence corresponding to the radio frequency tracking trajectory within the time window as the radio frequency fluctuation sequence; Candidate objects are detected in the monitoring footage of the exit camera in the visual blind spot. The pose estimation model is used to extract the changes in the coordinates of the skeletal key points of the candidate objects in the last time window and generate a motion displacement sequence. Construct the Laplacian matrix of the radio frequency wave sequence and the motion displacement sequence, and calculate the local geometric structure features of the radio frequency wave sequence and the motion displacement sequence respectively; Establish a manifold alignment objective function, which includes constraint terms that preserve the local geometric structure features of each data type and correspondence terms that minimize the correspondence between heterogeneous data types. By solving the generalized eigenvalue problem of the manifold alignment objective function, the radio frequency domain mapping matrix and the visual domain mapping matrix are obtained; The radio frequency fluctuation sequence is projected to a preset low-dimensional space using a radio frequency domain mapping matrix, and the motion displacement sequence is projected to the same low-dimensional space using a visual domain mapping matrix.

8. The blind zone relay tracking method based on heterogeneous data spatiotemporal calibration according to claim 7, characterized in that, The step of calculating the topological distance between mapped feature vectors in low-dimensional space and determining the candidate object and the target object corresponding to the topological distance less than a preset threshold as the same entity includes the following steps: Obtain the projected RF wave feature vector and the projected motion displacement feature vector in a low-dimensional space; The Euclidean distance between the radio frequency wave feature vector and the motion displacement feature vector is calculated as the topological distance. If the topological distance is less than the preset isomorphism determination threshold, it is confirmed that the fluctuation rhythm of the radio frequency signal and the displacement rhythm of the visual action have manifold isomorphism, and the candidate object and the target object are determined to be the same entity.

9. A blind zone relay tracking system based on heterogeneous data spatiotemporal calibration, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the blind zone relay tracking method based on heterogeneous data spatiotemporal calibration as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When executed by the processor, the instruction causes the processor to be configured to perform the blind zone relay tracking method based on heterogeneous data spatiotemporal calibration according to any one of claims 1 to 8.

Images