Double reflection non-line-of-sight target tracking method based on time convolution and attention mechanism

Abstract

The application discloses a kind of based on time series convolution and attention mechanism Double-bounce non-line-of-sight target tracking method, step 1, the shadow image of double-bounce non-line-of-sight target is continuously collected using single light source projection system, obtain the time series of non-line-of-sight target shadow image;Step 2, the time series of shadow image obtained in step 1 is input into the time series convolutional neural network of single view input, and the target position is predicted;The time series convolutional neural network of single view input includes multi-stage convolution feature extraction module, local convolution correlation body alignment fusion module and convolution type time series hidden state update module;Step 3, output target position.The double-bounce non-line-of-sight target tracking method based on time series convolution and attention mechanism of the application does not depend on double-view input, and accurate tracking can be realized in low-texture, weak contrast scene only by single-view time image.

Description

Technical Field

[0001] This invention belongs to the field of non-line-of-sight target tracking technology, and relates to a dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanisms. Background Technology

[0002] Traditional target tracking tasks can only capture objects within the direct line of sight. In recent years, however, the development of non-line-of-sight (NLOS) tracking technology has made it possible to track moving objects outside the line of sight. NLOS tracking technology analyzes indirect light signals in the scene, such as diffuse photons from walls and dynamic shadows caused by target movement, to achieve spatial localization and trajectory prediction of dynamic targets. This technology has shown revolutionary potential in fields such as security and rescue, intelligent transportation, medical navigation, and industrial inspection.

[0003] Currently, non-line-of-sight target tracking methods can be divided into transient and steady-state schemes based on their imaging mechanisms. Transient imaging methods are based on ultrafast pulsed lasers and time-resolved detectors, determining the spatial location of hidden objects by reversing photon propagation paths through Time of Flight (ToF) analysis. Zhan et al. used time-of-flight probability analysis to achieve tracking of multiple non-line-of-sight personnel at ultra-long distances exceeding 50 meters. Keyhole Imaging, through single-optical-path time-resolved measurement technology combined with the projection generated by moving objects, uses the expectation-maximization algorithm to achieve three-dimensional trajectory tracking of hidden objects. However, these methods rely on picosecond time-gated devices, resulting in high system costs, long sampling times, and limited environmental adaptability.

[0004] In contrast, steady-state imaging methods utilize low-power continuous-wave lasers or ambient light combined with traditional RGB cameras to achieve low-cost target position inversion and trajectory tracking through the statistical characteristics of spatial light intensity distribution. Existing technologies have utilized the spatiotemporal correlation of dynamic speckle generated by moving target perturbations to achieve real-time trajectory tracking for single-frame exposures. A data-driven NLOS tracking method has been proposed using active flash illumination, and a large dataset has been synthesized using rendering techniques for network training. Others have achieved centimeter-level accuracy for non-line-of-sight target localization under strong ambient light interference by separating ambient light transient noise from target reflection signals. Unlike the aforementioned active illumination methods, some researchers have used RGB cameras to capture continuous changes in ambient light shadows on walls, proposing a Propagation-Calibration Two-Stream Network (PAC-Net) based on differential frame motion information extraction to achieve centimeter-level real-time target tracking. However, existing tracking methods mainly focus on corner non-line-of-sight scenes, and research on target tracking in double-reflection non-line-of-sight scenes such as corridors and tunnels has not been fully explored.

[0005] The core feature of dual-reflection non-line-of-sight scenes lies in their unique geometric layout, which includes two relay walls. This structural advantage facilitates active illumination of one relay wall, thereby illuminating the hidden space and obtaining the outline projection of the hidden object formed on the other relay wall. Compared to corner imaging, where photons need to undergo three reflection paths, in dual-reflection non-line-of-sight scenes, photons reach the hidden space after being reflected by the first relay wall. Photons not blocked by the hidden object are reflected to the second relay wall and finally received by the detector. By moving the laser illumination point to different positions multiple times, multi-view outline projections of the hidden object can be obtained. Based on the spatial relationship between the illumination point and shadow feature points, the feature point matching method can achieve spatial localization of the hidden target. However, the feature point matching method is usually only applicable to static target matching. Since the feature points obtained from the projection of dynamic objects at different times lack spatiotemporal consistency, this method is difficult to achieve accurate spatial localization of dynamic targets.

[0006] Therefore, a new dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanisms is needed to solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to provide a dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanisms to solve the problems of the prior art.

[0008] The technical solution of the present invention is as follows:

[0009] A dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanisms includes the following steps:

[0010] Step 1: Continuously acquire shadow images of dual-reflection non-line-of-sight targets using a single-source projection system to obtain a time series of shadow images of non-line-of-sight targets;

[0011] Step 2: Input the time series of the shadow image obtained in Step 1 into a single-view input temporal convolutional neural network; the single-view input temporal convolutional neural network includes a multi-stage convolutional feature extraction module, a local convolutional correlation volume alignment and fusion module, and a convolutional temporal hidden state update module;

[0012] The multi-stage convolutional feature extraction module performs multi-scale spatial feature extraction on the shadow image to obtain the final feature map F containing spatial details and semantic background. t ∈R h×w×C ;

[0013] The local convolutional correlation volume alignment fusion module estimates the displacement relationship between features of adjacent frames through a local convolutional correlation mechanism, and combines it with a spatial attention mask A. t and channel gating vector g t Achieve feature fusion;

[0014] The convolutional temporal hidden state update module utilizes the temporal convolutional hidden state M t Accumulate target motion information over time to predict target location;

[0015] Step 3: Output the target location.

[0016] Furthermore, the multi-stage convolutional feature extraction module in step 2 includes a shallow representation module, a multi-scale feature mining module, and a context aggregation and refinement module connected in sequence. The shallow representation module includes a 7×7 first convolutional layer and a 3×3 second convolutional layer connected in sequence.

[0017] The multi-scale feature mining module includes a 3×3 first downsampling convolutional layer, a 3×3 second downsampling convolutional layer, a 3×3 first dilated convolutional layer, and a 3×3 second dilated convolutional layer connected in sequence.

[0018] The context aggregation refinement module includes a first residual convolutional layer, a second residual convolutional layer, a third residual convolutional layer, a 1×1 third convolutional layer, a global average pooling layer, and a 3×3 fourth convolutional layer connected in sequence.

[0019] Furthermore, the local convolutional correlation volume alignment fusion module described in step 2 achieves feature fusion through the following formula:

[0020]

[0021] In the formula, H t This is the fused feature map, where ⊙ represents element-wise multiplication;

[0022]

[0023] In the formula, W1 and W2 are learnable weight matrices, and g t Each component Control the retention ratio of channel c. For channel response;

[0024] A t =σ(W A *[ΔF t ,Δ 2 F t ])

[0025] In the formula, W A The kernel is 3×3, [·] represents channel concatenation, σ(·) is the Sigmoid activation function, and ΔF t For first-order time difference features, Δ 2 F t It is a second-order time difference feature, ΔF t =F t -Ft-1 Δ 2 F t =(F t -F t-1 )-(F t-1 -F t-2 );

[0026]

[0027] In the formula, For bilinear resampling operators, Π t (u,v) represents the displacement distribution, and (u,v) represents the candidate displacement. The displacement distribution ∏ is obtained by Softmax normalization over the window Ω. t (u,v), where Ω is the limiting displacement window. Ω={(u,v)∣u,v∈[-r,r], Let r represent the mid-level features at times t-1 and t, and r represent the radius of the confined displacement window.

[0028] Furthermore, in step 2, the convolutional temporal hidden state update module updates the temporal convolutional hidden state M using the following formula. t :

[0029]

[0030] Z t =σ(W z *[M t-1 H t ])

[0031] R t =σ(W r *[M t-1 H t ])

[0032] In the formula, * represents convolution operation, [·] represents channel concatenation, and M t-1 H is the hidden state from the previous step. t For fusion features, σ(·) is the Sigmoid activation function, Z t Indicates the update gate, R t Indicates resetting the door, W h W z and W r These represent the learnable convolutional weight matrices required to generate candidate states, update gates, and reset gates, respectively.

[0033] Furthermore, the loss function of the single-view input temporal convolutional neural network described in step 2 is expressed by the following formula:

[0034]

[0035] p t =GAP(M t )

[0036] In the formula, Let be the loss function for position error. Let λ1 and λ2 be the loss function for speed consistency, and T be the weight hyperparameters. Indicates the predicted target location coordinates, Represents the actual target location coordinates, Indicates the predicted position coordinates at the next moment. p represents the actual position coordinates at the next moment. t M represents the global features of the hidden states of temporal convolutions. t This represents the hidden state of temporal convolution, and GAP() is the global average pooling operation.

[0037] Furthermore, the single-source projection system described in step 1 includes an illumination relay wall, a projection relay wall, an illumination subsystem, and a detection subsystem. The illumination subsystem includes a laser that illuminates the illumination relay wall. The detection subsystem includes an RGB camera facing the projection relay wall. The illumination relay wall and the projection relay wall are positioned opposite each other, with a hidden area separated by an obstruction between them. A hidden target is positioned within this hidden area. The illumination relay wall receives the light from the illumination subsystem and reflects it into the hidden space. The projection relay wall receives the shadow cast by the hidden target. The hidden area between these two walls, separated by an obstruction, is the activity space where a moving target (such as a pedestrian) resides. The core logic of the system is to utilize this unique geometric configuration to infer the dynamic position of the hidden target by capturing and analyzing the shadows projected onto the walls.

[0038] Furthermore, the laser is a galvanometer scanning laser.

[0039] Furthermore, the galvanometer scanning laser operates at a wavelength of 520nm, has an output power of 500mW, and is equipped with a galvanometer with a reflectivity exceeding 99%. The key function of this part is to achieve real-time single-view illumination, i.e., controlling the laser to rapidly and alternately illuminate two specific locations on the illumination relay wall. This mechanism simulates the effect of two virtual light sources working in turn, thereby generating two sets of shadow sequences with different viewpoints on the projection wall. The geometric constraints provided by the dual-viewpoint effectively solve the positioning ambiguity problem existing under single-source illumination.

[0040] The dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanisms as described in claim 6 is characterized in that the laser alternately illuminates two positions on the illumination relay wall.

[0041] Beneficial effects: The dual-reflection non-line-of-sight target tracking method of the present invention based on temporal convolution and attention mechanism does not rely on dual-view input and can achieve accurate tracking in low-texture and low-contrast scenes using only single-view temporal images. Attached Figure Description

[0042] Figure 1 The flowchart shows a dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanisms.

[0043] Figure 2 This is a distributed, non-viewpoint simulation scene built using Blender.

[0044] Figure 3 3D character models for use in walking animation;

[0045] Figure 4 These are real-life scene demonstration images;

[0046] Figure 5 Predict the structure graph for the simulated dataset;

[0047] Figure 6 Track prediction results for real datasets. Detailed Implementation

[0048] Please see Figure 1 As shown, the dual-reflection non-line-of-sight target tracking method of the present invention, based on temporal convolution and attention mechanisms, employs a single-source projection system architecture for dynamic target tracking and constructs a corresponding theoretical error model. This invention can capture the target's motion patterns by utilizing shadow changes from each viewpoint and improve positioning accuracy through spatial correlation of shadows from both views.

[0049] This single-source projection system is primarily designed for Two-bounce NLOS (Two-bounce Non-View-of-Sight) scenarios. Its physical architecture comprises two opposing relay walls. The right wall acts as an illumination relay wall, receiving the laser beam and reflecting it into the hidden space. The left wall acts as a projection relay wall, serving as a "screen" to receive the dynamic shadows projected by the hidden object. Between these two walls lies a hidden area separated by an obstruction—the activity space of a moving target (such as a pedestrian). The core logic of the system utilizes this unique geometric configuration to infer the dynamic position of the hidden target by capturing and analyzing the shadows projected onto the walls.

[0050] The illumination subsystem consists of a galvanometer-scanning laser with a working wavelength of 520nm and an output power of 500mW. It is equipped with a galvanometer with a reflectivity exceeding 99% for precise beam deflection control. The key function of this part is to achieve real-time single-view illumination, i.e., controlling the laser to rapidly and alternately illuminate two specific locations on the illumination relay wall. This mechanism simulates the effect of two virtual light sources working in turn, thereby generating two sets of shadow sequences with different viewpoints on the projection wall. The geometric constraints provided by the dual-viewpoint effectively solve the positioning ambiguity problem inherent in single-source illumination.

[0051] The detection subsystem relies on a high-resolution RGB camera positioned directly opposite the projection relay wall, its field of view completely covering the shadow's imaging area. The camera's task is to acquire shadow images generated by a single light source in real time, converting the optical signals into a digital image sequence. This acquired dual-view shadow data is then input into the core data processing unit—the Attention-Based Dual-Reflection Non-Visual-Field Temporal Network (DAT-NLOSNet). This network uses deep learning algorithms to comprehensively analyze the single-view temporal features and cross-view spatial consistency of the shadow sequence, ultimately calculating and outputting the precise trajectory of the hidden target on a two-dimensional plane.

[0052] Next, this invention constructs a collaborative perception network based on dual-view projection. For each group of continuous single-view shadow images, the projection feature extraction model, projection correlation calibration module and dynamic differential attention module are respectively passed to complete the multi-view fusion of temporal dynamic shadows, and the multilayer perceptron is used to realize the continuous trajectory prediction from the feature space to the real coordinates.

[0053] To overcome the bottleneck of difficulty in obtaining real data, a parameterized simulation dataset was constructed using the Blender physics engine. Poisson-Gaussian mixed noise and dynamic range degradation were introduced to simulate real detector noise.

[0054] Experiments show that the method can achieve millimeter-level positioning accuracy with an error of 0.07cm in simulation scenarios. Furthermore, the model trained in the simulation scenario was transferred to a real scenario for verification, achieving centimeter-level practical accuracy in human motion tracking tasks.

[0055] This invention proposes a temporal convolutional neural network (CNN) target tracking method based on a single-view image sequence, aiming to stably estimate the two-dimensional position coordinates of a target from a single viewpoint, including the following steps:

[0056] First, through a multi-stage convolutional feature extraction module, the network extracts local and global features of the image layer by layer, helping the network understand the shape, boundary and surrounding background information of the target.

[0057] Secondly, the cross-frame feature alignment and fusion module calculates the similarity between adjacent frames through local convolutional correlations to estimate motion displacement. It then uses spatial attention masks and channel gating mechanisms to dynamically adjust the weights of aligned features, ensuring that the network focuses on the target region and suppresses background interference.

[0058] Finally, the temporal information accumulation module uses a convolutional hidden state update mechanism to fuse motion information from historical frames with features from the current frame, ensuring the continuity and robustness of target tracking in dynamic environments.

[0059] Through the synergistic effect of these modules, the algorithm achieves stable and accurate target tracking in challenging scenarios such as low texture, weak contrast, and occlusion. To overcome the difficulty of obtaining real-world data in actual scenarios, a parameterized simulation dataset was constructed using the Blender physics engine, and noise and dynamic range degradation were introduced to simulate the noise conditions in real detectors. Experimental results show that the proposed method can achieve accurate target tracking in simulated scenarios, and the trained model can be transferred to real-world scenarios, achieving high localization accuracy in human motion tracking tasks, demonstrating the feasibility and effectiveness of the method in practical applications.

[0060] The present invention provides a temporal convolutional neural network (CNN) target tracking method based on single-view input, which uses time-series images as input and estimates the two-dimensional normalized position coordinates of the target frame by frame. The core idea of ​​the network is to model temporal continuity as a learnable feature of motion priors and alignment evidence. Through three stages—multi-stage convolutional feature extraction, local convolutional correlation volume alignment fusion, and convolutional temporal hidden state update—robust tracking of non-visual targets is achieved.

[0061] First, the network extracts multi-scale spatial features from the input shadow image to capture edge details and global illumination changes;

[0062] Subsequently, the displacement relationship between features of adjacent frames is estimated through a local convolutional correlation mechanism, and combined with a spatial attention mask A. t and channel gating vector g t Achieve feature fusion;

[0063] Finally, the temporal convolutional hidden state M is utilized. t Accumulate target motion information over time to achieve stable prediction of target position.

[0064] This method does not rely on dual-view input and can achieve accurate tracking in low-texture, low-contrast scenes using only single-view temporal images.

[0065] The method includes the following steps:

[0066] 1. Feature extraction and temporal difference modeling:

[0067] Input image First, a three-stage convolutional feature extraction network is used to obtain a robust spatial feature map F. t The network consists of a shallow representation stage, a multi-scale feature mining stage, and a context aggregation and refinement stage. These three stages work progressively and share parameters in the time dimension.

[0068] In the shallow representation stage, the network is used to capture low-level features such as shadow edges and illumination textures.

[0069] The input image first passes through a 7×7 convolutional layer (64 kernels, stride 2) to extract global shadow contours. Then, it passes through a 3×3 convolutional layer (maintaining 64 channels) with BatchNormalization and ReLU activation to enhance edge response and suppress background noise. This stage preserves high spatial resolution, providing a foundation for subsequent multi-scale convolutions.

[0070] The multi-scale feature mining stage employs four cascaded 3×3 convolutional layers, with downsampling between each pair of layers achieved through convolutions with a stride of 2, expanding the number of channels to 128 and 256 respectively. The third and fourth convolutional layers use dilated convolutions (dilation rates of 2 and 4, respectively) to expand the receptive field and obtain richer contextual information. The mid-level features output from this stage can simultaneously reflect detailed shadow variations and large-scale illumination distribution.

[0071] The context aggregation and refinement stage integrates features through residual connections and global information aggregation. First, the multi-scale features output from the previous stage are input into a residual convolutional module with a unified channel dimension of 256. Then, 1×1 convolution combined with Global Average Pooling (GAP) aggregates the scene context features. Finally, 3×3 convolution refines the feature map F, resulting in the final feature map F containing spatial details and semantic background. t ∈R h×w×C .

[0072] To explicitly capture motion features in the temporal dimension, the network calculates the temporal difference between features in adjacent frames:

[0073] ΔF t =F t -F t-1

[0074] Δ 2 F t =(F t -F t-1 )-(F t-1-F t-2 )

[0075] Among them, the first-order difference ΔF t Describes the motion intensity between frames, second-order difference Δ 2 F t This reflects changes in the direction of motion. Together, these form a temporal prior, which, after 1×1 convolution and normalization, generates a motion feature map used in subsequent stages to compute the spatial attention mask A. t With channel gate vector g t This differential modeling process provides the network with explicit temporal cues of change, enabling the model to perceive the local motion and morphological changes of the target, and providing constraints for subsequent feature fusion and attention guidance.

[0076] 2. Fusion of local convolutional correlation volume and attention guidance:

[0077] In this stage, the network aims to establish pixel-level comparability of features from adjacent time points and, guided by motion priors, selectively enhance target regions and effective channels. Let F... t-1 ,F t ∈R h×w×C The mid-level features are for times t-1 and t. Since the shadow boundary caused by target motion only produces a finite pixel-scale drift, F is directly stitched together. t-1 With F t Convolution often introduces spurious responses at alignment errors. Therefore, a local convolution correlation volume is constructed within a finite displacement window Ω = {(u,v) | u,v ∈ [-r,r]}.

[0078]

[0079] This expression is equivalent to F t-1 As a template in F t Local "convolutional" matching is performed: for each candidate displacement (u,v), the channel inner product of the two frames in the overlapping neighborhood is calculated, thus obtaining a similarity response map for the displacement. To make the response probabilistically interpretable and facilitate gradient propagation, The displacement distribution Π is obtained by Softmax normalization over the window Ω. t (u,v), and then obtain the displacement estimate through expectation operation.

[0080]

[0081] This desired displacement is superior to a hard selection that only takes the maximum value, and it can weight multiple nearest-neighbor peaks in the low signal-to-noise ratio range, mitigating single-peak uncertainty. Subsequently, a differentiable bilinear resampling operator is used. To achieve spatial alignment of features from the previous frame, features from the previous frame are aligned to the coordinates of the current frame, thus achieving consistent spatial alignment between frames at the pixel level.

[0082]

[0083] In boundary handling, resampling uses zero-padding or boundary duplication for coordinates falling outside the image boundary. The difference in training stability between the two is minimal, but the former is more effective at preserving gradient sparsity. Because... The scale is typically no more than a few pixels, and the radius of Ω is usually chosen to be 5. It is important to emphasize that the calculation of the correlator is performed in the feature domain, thus making it more robust to illumination changes. Next, this invention introduces an attention guidance mechanism based on temporal difference, further utilizing the temporal correlation between consecutive frames for feature enhancement. Among these, the spatial attention mask A... t ∈(0,1) h×w With channel gate vector g t ∈(0,1) C Information flow is modulated in both spatial and channel dimensions. The spatial attention mask is generated using differential features.

[0084] A t =σ(W A *[ΔF t ,Δ 2 F t ]),

[0085] Among them, W A The kernel is 3×3, [·] denotes channel concatenation, and σ(·) is the Sigmoid activation function. Mask A t The larger the value, the greater the temporal variation of the pixel, and the more likely it is to carry shadow variations consistent with the target's motion. Therefore, the current observation at that location will be given higher weight during fusion, while reducing the dependence on historical alignment features.

[0086] Channel gate vector g t ∈(0,1) C This reflects the dynamic importance of different convolutional channels. This is achieved by analyzing the first-order temporal difference feature ΔF. t Perform Global Average Pooling (GAP) to obtain channel responses After passing through two layers of perceptron and sigmoid mapping, we obtain:

[0087]

[0088] Where W1 and W2 are learnable weight matrices, g t Each component The retention rate of channel c represents its contribution to the current motion. Since differential operations tend to amplify motion-related high-frequency or boundary-sensitive channels while suppressing static low-frequency channels, this gating adaptively emphasizes convolutional channels that are "most useful for the current motion," such as those related to shadow edge direction, intensity changes, and local contrast enhancement.

[0089] Based on the two types of attention weights mentioned above, cross-frame features are fused in both spatial and channel dimensions. Ultimately, the calculation of the fused features combines spatial and channel weighting mechanisms.

[0090]

[0091] Here, ⊙ denotes element-wise multiplication. Through this fusion process, the network highlights motion regions in the spatial dimension and suppresses redundant information in the channel dimension, resulting in the fused feature map H. t After further feature enhancement through two layers of 3×3 convolutions and one layer of 1×1 convolutions, the feature is sent to the next stage for temporal modeling.

[0092] 3. Hidden state update and coordinate regression:

[0093] To maintain trajectory coherence and robustness to occlusion over longer timescales, the network fuses features H... t Construct the temporal convolutional hidden state M above t Its function is to accumulate historical evidence and filter it for inclusion in current observations, so that the output pose estimate can respond quickly to effective changes without being overly affected by transient noise. Let M be denoted below. t-1 M is the hidden state from the previous step. t Spatial dimensions and number of channels and H t Alignment facilitates convolution fusion.

[0094] M t The update is accomplished using a gated convolutional structure, and its computation process includes three parts: gate generation, candidate state construction, and weighted writing. First, historical states and current fused features are concatenated along the channel dimension, and then the update gate and reset gate are calculated.

[0095] Z t =σ(W z *[M t-1 H t ])

[0096] R t =σ(W r *[M t-1 H t ])

[0097] Where * represents convolution operation, and [·] represents channel concatenation. The update gate controls the proportion of new information written, and the reset gate controls the retention rate of old memories. The initial hidden state is set to a tensor of the same size, all zeros. Z t Large value regions represent areas where the network considers the current fusion features particularly reliable for these locations / channels; Reset gate R t The degree to which historical states are "forgotten" before candidate states are constructed, R t Smaller values ​​provide stronger suppression of older states, thus preventing the accumulation of erroneous memories. Subsequently, candidate hidden states are constructed based on the reset historical states and the current features:

[0098]

[0099] And update the hidden state through interpolation:

[0100]

[0101] This update method is numerically equivalent to maintaining an adaptive forgetting factor for each position / channel, increasing Z when the current evidence is sufficient and consistent. t Write new information; reduce Z when observation noise is high or differences are uncertain. t Preserving the old state allows for immunity to sudden noise and short-term occlusion. Since all gates and convolutions are differentiable, M... t The learning can be optimized together with related bodies, attention branches and feature extraction ends to form an end-to-end temporal representation.

[0102] The spatial distribution of the hidden states encodes the target's trajectory and temporal context information. Finally, global features of the hidden states are extracted using global average pooling.

[0103] p t =GAP(M t )

[0104] The data is then input into two layers of sensors to determine the target location. The prediction is as follows. To enhance prediction stability, the loss function consists of two parts: position error and velocity consistency. The loss function for position error is:

[0105]

[0106] The loss function for speed consistency is:

[0107]

[0108] The overall function expression is:

[0109]

[0110] Through this temporal convolution mechanism, the network can effectively fuse historical information from multiple frames to achieve continuous estimation of the target trajectory. The entire method's mapping from input image to position output is end-to-end learnable: the convolutional extraction module is responsible for spatial perception, the local correlation volume achieves temporal alignment, and the convolutional hidden state module completes motion accumulation and coordinate regression. The three form a closed loop, achieving robust target tracking under single-view conditions.

[0111] Non-line-of-sight data acquisition and experiments:

[0112] 1. Acquisition of non-view data

[0113] Training deep neural networks relies on large datasets with diverse scene types. However, acquiring large-scale, high-quality, and accurately labeled datasets in real-world non-view-of-sight scenes presents significant challenges, particularly in comprehensively covering key variables such as different scene sizes, object materials, and character shapes. Therefore, drawing on previous work, this invention uses Blender software to construct simulated scenes and render simulated data. A physically based rendering approach generates shadow images with characteristics similar to real-world scenes. Furthermore, to verify the generalization and transfer capabilities of the proposed model in real-world physical environments, this invention additionally constructs a practical non-view-of-sight detection system and collects a small-scale real-world dataset in a controlled, real-world non-view-of-sight space for model evaluation.

[0114] 1.1. Simulation Data

[0115] Figure 2 and Figure 3 This work showcases a two-reflection, non-viewpoint simulation scene and character models built using Blender software. The dimensions of the simulation scene were randomly generated within a specific range, with a length between 4 and 8 meters, and the width between the two walls and the scene height between 2 and 4 meters. The camera used to detect the projected wall had a resolution of 652×400 pixels, and its field of view completely covered the expected shadow imaging area. Two virtual point light sources of equal intensity were symmetrically arranged on the illuminated wall, located at 1 / 3 and 2 / 3 of the wall width, respectively, with the diameter of the light sources randomly generated between 2 and 3 cm. Similar to Work 1, all 3D character models used for the walking animation were from Adobe's free animation platform, Mixamo. The movement paths of these character models within the hidden space were created using a random trajectory generation algorithm, including various modes such as uniform motion, variable speed motion, and multi-angle turning.

[0116] Using Blender's built-in Cycles rendering engine, shadow images of a dynamic target from two perspectives are rendered by alternately turning two virtual point lights on and off. For each scene, a temporal image sequence containing 320 consecutive time steps is generated. Simultaneously, the precise 2D coordinates of the target in the latent space at each frame are recorded as ground truth labels for the motion trajectory. The final simulation dataset contains 100 independent off-view scenes, each providing a corresponding dual-view shadow image sequence, ground truth values ​​for the target's motion trajectory, and scene geometric parameters.

[0117] 1.2 Real-world images

[0118] See Figure 4 As shown, this invention constructed a realistic NLOS scene indoors. The two side walls act as relay surfaces, with the right wall serving as the lighting relay surface and the left wall as the projection relay surface, spaced 4.6m apart. A 3.1m × 2.37m × 4.6m non-visual-range hidden space is formed in between by a partition. The physically constructed non-visual-range data acquisition system uses a laser galvanometer on the lighting relay surface to illuminate the hidden space. Its output power is 500mW, wavelength is 520nm, and it is equipped with a ±30° optical deflection angle and a 99% reflectivity lens. The shadows formed by the lighting are captured by a visible light camera (model HIKROBOTMV-CA023-10GC) with an image resolution of 2048 × 1536. To verify the robustness of the algorithm, during the data acquisition phase, this invention had four subjects of different body types walk at different speeds along a preset path. A reference camera of the same model as the one used for shadow acquisition was mounted at the top of the scene, vertically covering the entire hidden space, recording the actual movement of the pedestrians at 30 frames per second. The recorded video is processed through post-processing techniques such as video frame alignment, video clip cropping, and coordinate calibration to obtain the pedestrian movement trajectory as the baseline ground truth.

[0119] 2. Implementation details

[0120] This invention implements the proposed network model within the PyTorch 1.12.1 framework and trains the model on an NVIDIA GeForce GTX 3090 equipped with CUDA 11.3. During the model pre-training phase, the simulation dataset is divided into training and validation sets in a 4:1 ratio, and training is performed on the training set for 200 epochs with a batch size of 32. The Adam optimizer is used to optimize the network weights, with an initial learning rate set to 3 × 10⁻⁶. -4A cosine annealing learning rate update strategy was adopted. Because there are significant differences in imaging characteristics between simulated and real data, gamma transformation and a Poisson-Gaussian mixture noise model were added to each data set during model training to reduce the characteristic differences between simulated and real data. During the fine-tuning phase on the real dataset, the real dataset was also divided into training and validation sets in a 4:1 ratio, and each set was set with a 1×10⁻⁶ resolution. -4 The learning rate was used for 50 training cycles, while other network training parameters remained unchanged.

[0121] To effectively evaluate the tracking results, this invention employs multiple evaluation metrics to quantitatively compare the predicted trajectory curves. Root mean square error (RMSE) is used to evaluate tracking quality by directly calculating the Euclidean distance between the predicted and actual trajectories. Dynamic time warping distance (DTW) is introduced into trajectory morphology analysis to measure temporal similarity, area error is used to quantify spatial trajectory deviation, and process control monitoring (PCM) is employed to assess the temporal matching degree of the motion process. These four metrics together constitute a multi-dimensional performance evaluation system to comprehensively evaluate the performance of the results.

[0122] 3. Evaluation of the effect using simulated datasets

[0123] This invention describes the experimental results of the proposed method on a simulation dataset and compares them with single-light source tracking methods to evaluate the performance of non-line-of-sight tracking methods on simulated data, and demonstrates the tracking effect of the non-line-of-sight tracking system in distributed scenes.

[0124] First, this invention uses a simulated dataset to verify the tracking performance of DAT-NLOSNet and demonstrate the effectiveness of the proposed method. The invention first conducts tracking inference performance tests on the network in a non-view distributed scene with no background and the same human model, obtaining excellent inference results and verifying the feasibility of the model.

[0125] like Figure 5 As shown, to verify the design concept of DAT-NLOSNet, this invention designed a single-source network with the projection correlation calibration module of the input light source 2 image removed, and the trajectory results calculated only based on theoretical errors in the experiment. The preceding and following frames of the single light source were used as inputs for the difference features. The experiment used the controlled variable method, ensuring consistency in scene configuration, target object, motion trajectory, and background parameters between the single-source and multi-source systems.

[0126] The results are shown in Table 1. In the 120-frame dataset test, the localization error (RMSx) of DAT-NLOSNet was 0.070×10⁻², a 91.9% reduction compared to the 0.870×10⁻² of the single-source network, and a 94.2% improvement compared to the theoretical model's 1.203×10⁻². This verifies the effectiveness of multi-source data fusion. When the frame rate increases, leading to a shorter time sampling interval, the prediction error of the dual-source network shows a decreasing trend, consistent with the theoretical model's expectations. The comparative results show that DAT-NLOSNet significantly outperforms the single-source network and the theoretical benchmark in core metrics, maintaining stability, especially in long-distance scenes, without exhibiting the error surge phenomenon seen in traditional methods.

[0127] Table 1 shows the localization errors of DAT-NLOSNet and the single-source network in the 120-frame dataset test.

[0128]

[0129] 3.3.2 Evaluation of effectiveness based on real-world scenario data

[0130] To verify the actual tracking performance of the non-line-of-sight positioning method, this invention collected dynamic human walking data in real-world scenarios. Experimenters walked along a preset path in an occluded area, generating four closed trajectories: "N", "L", "O", and "S". The coordinates of the actual trajectories were recorded by a camera positioned at the top of a hidden space, serving as a baseline for positioning accuracy. This dataset provides an intuitive basis for trajectory shape comparison and error spatial distribution analysis of the tracking algorithm.

[0131] To verify the practical application value of the method, this invention further conducts cross-domain performance evaluation experiments to systematically analyze the generalization ability of different non-line-of-sight positioning methods in real-world scenarios.

[0132] When the inference length of the model exceeds 120 images, the cumulative error of the model increases significantly, posing a critical challenge to long-distance inference tracking. To address this issue, we explored a method to increase the number of images inferenced per batch, gradually increasing the number of input frames per batch from 120 to 240 while maintaining temporal continuity. 120 images were alternately input to the network model for each light source. After 200 epochs of model training, the problem of inference length was effectively mitigated, and the model's sustained inference capability was improved.

[0133] After training on the degraded augmented dataset and extending the model inference length, DAT-NLOSNet achieved sufficient localization accuracy for indoor localization on real-world datasets and demonstrated significant environmental robustness. Figure 6As shown, the overlap between the output trajectory of the augmented model and the real trajectory is significantly improved. Compared with traditional data augmentation methods, the degradation mode generated by this scheme is closer to the real physical process, thus significantly improving the model's robustness to real noise while maintaining high accuracy of the simulation scene. Experimental results show that the domain adaptation strategy based on physical priors can effectively bridge the feature gap between simulation and real data.

Claims

1. A dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanisms, characterized in that, Includes the following steps: Step 1: Continuously acquire shadow images of dual-reflection non-line-of-sight targets using a single-source projection system to obtain a time series of shadow images of non-line-of-sight targets; Step 2: Input the time series of the shadow image obtained in Step 1 into a single-view input temporal convolutional neural network; the single-view input temporal convolutional neural network includes a multi-stage convolutional feature extraction module, a local convolutional correlation volume alignment and fusion module, and a convolutional temporal hidden state update module; The multi-stage convolutional feature extraction module performs multi-scale spatial feature extraction on the shadow image to obtain the final feature map F containing spatial details and semantic background. t ∈R h×w×C ; The local convolutional correlation volume alignment fusion module estimates the displacement relationship between features of adjacent frames through a local convolutional correlation mechanism, and combines it with a spatial attention mask A. t and channel gating vector g t Achieve feature fusion; The convolutional temporal hidden state update module utilizes the temporal convolutional hidden state M t Accumulate target motion information over time to predict target location; Step 3: Output the target location.

2. The dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanism as described in claim 1, characterized in that, The multi-stage convolutional feature extraction module in step 2 includes a shallow representation module, a multi-scale feature mining module, and a context aggregation and refinement module connected in sequence. The shallow representation module includes a 7×7 first convolutional layer and a 3×3 second convolutional layer connected in sequence. The multi-scale feature mining module includes a 3×3 first downsampling convolutional layer, a 3×3 second downsampling convolutional layer, a 3×3 first dilated convolutional layer, and a 3×3 second dilated convolutional layer connected in sequence. The context aggregation refinement module includes a first residual convolutional layer, a second residual convolutional layer, a third residual convolutional layer, a 1×1 third convolutional layer, a global average pooling layer, and a 3×3 fourth convolutional layer connected in sequence.

3. The dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanism as described in claim 1, characterized in that, The local convolutional correlation volume alignment fusion module described in step 2 achieves feature fusion through the following formula: In the formula, H t This is the fused feature map, where ⊙ represents element-wise multiplication; In the formula, W1 and W2 are learnable weight matrices, and g t Each component Control the retention ratio of channel c. For channel response; A t =σ(W A *[ΔF t ,Δ 2 F t ]) In the formula, W A The kernel is 3×3, [·] represents channel concatenation, σ(·) is the Sigmoid activation function, and ΔF t For first-order time difference features, Δ 2 F t It is a second-order time difference feature, ΔF t =F t -F t-1 Δ 2 F t =(F t -F t-1 )-(F t-1 -F t-2 ); In the formula, For bilinear resampling operators, Π t (u,v) represents the displacement distribution, and (u,v) represents the candidate displacement. The displacement distribution ∏ is obtained by Softmax normalization over the window Ω. t (u,v), where Ω is the limiting displacement window. Ω={(u,v)|u,v∈[-r,r],F t-1 ,F t ∈R h×w×C Let r represent the mid-level features at times t-1 and t, and r represent the radius of the confined displacement window.

4. The dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanism as described in claim 1, characterized in that, In step 2, the convolutional temporal hidden state update module updates the temporal convolutional hidden state M using the following formula. t : Z t =σ(W z *[M t-1 ,H t ]) R t =σ(W r *[M t-1 ,H t ]) In the formula, * represents convolution operation, [·] represents channel concatenation, and M t-1 H is the hidden state from the previous step. t For fusion features, σ(·) is the Sigmoid activation function, Z t Indicates the update gate, R t Indicates resetting the door, W h W z and W r These represent the learnable convolutional weight matrices required to generate candidate states, update gates, and reset gates, respectively.

5. The dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanism as described in claim 1, characterized in that, The loss function of the single-view input temporal convolutional neural network described in step 2 is expressed by the following formula: p t =GAP(M t ) In the formula, Let be the loss function for position error. Let λ1 and λ2 be the loss function for speed consistency, and T be the weight hyperparameters. Indicates the predicted target location coordinates, Represents the actual target location coordinates, Indicates the predicted position coordinates at the next moment. p represents the actual position coordinates at the next moment. t M represents the global features of the hidden states of temporal convolutions. t is the hidden state of temporal convolution, and GAP() is the global average pooling operation.

6. The dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanism as described in claim 1, characterized in that, The single-source projection system described in step 1 includes an illumination relay wall, a projection relay wall, an illumination subsystem, and a detection subsystem. The illumination subsystem includes a laser that illuminates the illumination relay wall. The detection subsystem includes an RGB camera that faces the projection relay wall. The illumination relay wall and the projection relay wall are positioned opposite each other. A hidden area separated by an obstruction is provided between the illumination relay wall and the projection relay wall. A hidden target is placed within the hidden area. The illumination relay wall is responsible for receiving the light from the illumination subsystem and reflecting the light into the hidden space. The projection relay wall is responsible for receiving the shadow projected by the hidden target.

7. The dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanism as described in claim 6, characterized in that, The laser is a galvanometer scanning laser.

8. The dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanism as described in claim 7, characterized in that, The galvanometer scanning laser operates at a wavelength of 520nm, has an output power of 500mW, and is equipped with a galvanometer with a reflectivity exceeding 99%.

9. The dual-reflection non-line-of-sight target tracking method based on temporal convolution and attention mechanism as described in claim 6, characterized in that, The laser alternately illuminates two locations on the lighting relay wall.

Images