A deep learning-based passive attitude monitoring method and system

Abstract

The application relates to the technical field of passive sensing and behavior recognition, and discloses a passive posture monitoring method and system based on deep learning, wherein the passive posture monitoring method based on deep learning comprises the following steps: acquiring a target object monitoring image sequence collected by multiple visual angle cameras, and generating a multi-dimensional occlusion label tensor; calculating data validity weights based on the occlusion label tensor, and constructing a sparse multi-view observation tensor; decomposing the observation tensor into a low-rank background tensor, a sparse change tensor and a noise tensor by using a sparse time series tensor decomposition algorithm; inputting the low-rank background tensor and the sparse observation feature obtained through decomposition into a conditional variational autoencoder, and reconstructing a complete time series trajectory; and generating a high-confidence continuous posture monitoring sequence and a gradual abnormality early warning signal through physical constraint optimization and Bayesian inference. The application solves the technical problem of accurately monitoring the slow change of a lighting lamp posture under frequent occlusion conditions.

Description

Technical Field

[0001] This invention relates to the field of passive sensing and behavior recognition technology, and more specifically, to a passive posture monitoring method and system based on deep learning. Background Technology

[0002] In urban road infrastructure management, monitoring the posture of streetlights is a crucial aspect of ensuring traffic safety. Streetlights can experience slow posture changes due to factors such as foundation settlement, material aging, and wind loads, which can lead to tipping accidents in severe cases. Traditional manual inspection methods are inefficient and struggle to detect subtle, gradual changes; therefore, automated monitoring methods based on computer vision are becoming the trend.

[0003] However, monitoring the attitude of streetlights in busy urban areas presents unique challenges: frequent passage of large vehicles causes dynamic obstruction of the monitoring viewpoint, resulting in significant gaps in the collected multi-view time-series data and forming a sparse observation matrix; simultaneously, minute changes in the attitude of the streetlights are easily masked by measurement noise and obstruction. Traditional attitude monitoring methods rely on continuous and complete observation data, which cannot recover continuous attitude trajectories from fragmented data, nor can they detect gradual trends submerged in noise, thus failing to meet the needs of practical applications. Summary of the Invention

[0004] This invention provides a passive attitude monitoring method and system based on deep learning, addressing the severe data loss caused by dynamic occlusion in related technologies. It also solves the technical problem of accurately monitoring the slow changes in the attitude of lighting fixtures under frequent occlusion conditions.

[0005] This invention provides a passive attitude monitoring method and system based on deep learning, comprising:

[0006] Acquire target object monitoring image sequences captured by cameras from multiple perspectives, identify dynamic occlusion regions, and generate a multidimensional occlusion label tensor;

[0007] The step of identifying the dynamically occluded region includes:

[0008] Calculate the optical flow field for each image, and identify dynamic occlusion areas by using optical flow amplitude thresholds;

[0009] Generate a binary occlusion mask, where occluded areas are marked as 1 and unoccluded areas are marked as 0;

[0010] Combine the occlusion masks of all viewpoints and times to form a multidimensional occlusion label tensor;

[0011] Based on the occlusion marker tensor, calculate the data validity weights and construct a sparse multi-view observation tensor;

[0012] The observation tensor is decomposed into three components—low-rank background tensor, sparse variation tensor, and noise tensor—using the sparse temporal tensor decomposition algorithm.

[0013] The objective function of the sparse temporal tensor decomposition algorithm is defined as follows:

[0014] The reconstruction loss term measures the fitting accuracy at the observation location, and only elements at the observation location are retained by the projection operator;

[0015] The low-rank constraint term uses the tensor kernel norm to promote the low-rank structure of the background tensor to capture common patterns from multiple perspectives.

[0016] The sparse constraint term uses the L1 norm to promote the sparsity of the variable tensor to separate anomalous events;

[0017] The noise regularization term uses the Frobenius norm to control the energy of the noise tensor;

[0018] The temporal smoothing constraint term ensures continuity in the temporal dimension by penalizing the differences in the background tensor between adjacent time steps;

[0019] The alternating direction multiplier method is used for iterative solution, updating the three component tensors sequentially until convergence.

[0020] The low-rank background tensor and sparse observation features obtained from the decomposition are input into the conditional variational autoencoder to reconstruct the complete temporal trajectory.

[0021] The conditional variational autoencoder includes:

[0022] The encoder maps observed features to a distribution of latent variables, where conditional information includes time period labels and pose priors;

[0023] The decoder generates reconstructed features by sampling from latent variables;

[0024] The model is trained by maximizing the variational lower bound, and the reconstructed complete pose monitoring sequence is output. The loss function includes the reconstruction loss and the KL divergence regularization term.

[0025] The attitude prior is obtained by processing the historical attitude sequence of effective observation segments through a long short-term memory network, and the attitude prior distribution parameters for each time period are output.

[0026] The local attitude change rate is calculated based on the reconstructed trajectory using sliding window analysis, and the future attitude evolution trend is predicted.

[0027] Cross-validation is performed by fusing multi-view reconstruction results, and high-confidence continuous attitude monitoring sequences and gradual anomaly warning signals are generated through physical constraint optimization and Bayesian inference.

[0028] The sparse temporal tensor decomposition algorithm achieves effective decomposition of incomplete observation data by minimizing an objective function that includes a reconstruction loss term, a low-rank constraint term, a sparse constraint term, a noise regularization term, and a temporal smoothing constraint term.

[0029] Furthermore, the step of constructing the sparse multi-view observation tensor includes:

[0030] For each viewpoint-time pair, the data validity weight is calculated as the proportion of unoccluded viewpoint at that time.

[0031] For images with a validity weight greater than a preset threshold, the pose feature vector of the target object is extracted using a convolutional neural network.

[0032] The attitude feature vector includes key point coordinates, tilt angle and top displacement parameters, and the different physical quantities are standardized to eliminate dimensional differences.

[0033] Construct a sparse multi-view observation tensor, where missing positions are indicated by special markers.

[0034] Furthermore, the convolutional neural network includes:

[0035] Multiple convolutional layers are used to extract image features, and a batch normalization layer is added after each convolutional layer;

[0036] Global average pooling layers pool feature maps into feature vectors;

[0037] Multiple fully connected layers output pose feature vectors;

[0038] During training, a weighted loss function is used, which includes keypoint coordinate errors, tilt angle errors, and top displacement errors. The weight coefficients are determined by the inverse of the variance of each parameter.

[0039] Furthermore, the reconstruction of the complete time-series trajectory also includes:

[0040] The low-rank background tensor is input into a wavelet transform processor to perform multi-scale denoising and extract long-term trend components.

[0041] Based on the change point detection algorithm, trend components and sparse change tensors are analyzed to identify the moment of abrupt attitude change and the starting point of gradual change, and a time period segmentation label sequence is generated.

[0042] The time-segmentation label sequence serves as the conditional information input for the conditional variational autoencoder.

[0043] Furthermore, the predicted future attitude evolution trend includes:

[0044] Extract local segments of the reconstructed trajectory using a sliding window;

[0045] The attitude change rate sequence was calculated using the central difference method;

[0046] The rate of change sequence is input into a recurrent neural network, which outputs a sequence of pose predictions for multiple future time steps.

[0047] The predicted sequence is inversely standardized to obtain the actual attitude parameter prediction values.

[0048] Furthermore, the Bayesian inference includes:

[0049] Construct the likelihood function of the reconstruction results from each perspective, assuming that the observations from each perspective are independent and follow a Gaussian distribution;

[0050] Define a prior distribution based on physical constraints, including tilt angle range constraints and rate of change constraints;

[0051] The optimal pose estimate and its confidence interval after fusion are calculated using maximum a posteriori estimation.

[0052] When the confidence interval exceeds the preset threshold or the attitude change rate exceeds the normal range, a gradual anomaly warning signal is generated.

[0053] This invention provides a passive attitude monitoring system based on deep learning, comprising:

[0054] The image acquisition module is used to acquire monitoring image sequences from cameras with multiple viewing angles;

[0055] The occlusion detection module is used to identify dynamically occluded regions and generate an occlusion marker tensor.

[0056] The tensor construction module is used to extract pose features and construct sparse observation tensors.

[0057] The tensor decomposition module is used to execute the sparse temporal tensor decomposition algorithm.

[0058] The trajectory reconstruction module includes a conditional variational autoencoder for reconstructing complete temporal trajectories;

[0059] The trend prediction module is used to analyze the rate of attitude change and predict future evolution trends.

[0060] The fusion decision module is used to perform Bayesian inference to generate final monitoring results and early warning signals.

[0061] The beneficial effects of this invention are as follows:

[0062] This invention utilizes a sparse temporal tensor decomposition algorithm to organize fragmented multi-view observation data into a high-dimensional tensor structure. By decomposing it into three components—low-rank background, sparse variation, and noise—it effectively separates stable attitude patterns from anomalous change patterns, overcoming the severe data loss caused by dynamic occlusion. Simultaneously, combined with the generative capabilities of a conditional variational autoencoder, it can reconstruct the attitude trajectory for missing time periods based on existing sparse observations and learned attitude evolution patterns, solving the technical challenge of accurately monitoring the slow changes in lighting attitude under frequent occlusion conditions. Attached Figure Description

[0063] Figure 1 This is a flowchart of a passive attitude monitoring method based on deep learning according to the present invention;

[0064] Figure 2 This is a multi-view data integrity comparison bar chart of the present invention;

[0065] Figure 3 This is the dynamic occlusion spatiotemporal distribution heat map of the present invention;

[0066] Figure 4 This is a line graph showing the convergence process of the tensor decomposition algorithm of this invention;

[0067] Figure 5 This is the Sankey diagram of the tensor component energy distribution of the present invention;

[0068] Figure 6 This is a scatter plot of the reconstruction quality assessment of the present invention;

[0069] Figure 7 This is a dual Y-axis plot showing the future attitude prediction trend of the present invention. Detailed Implementation

[0070] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0071] At least one embodiment of the present invention discloses a passive attitude monitoring method and system based on deep learning, such as... Figure 1 As shown, it includes the following steps:

[0072] Step 100: Acquire a sequence of lighting monitoring images captured by cameras from multiple perspectives, identify dynamic occlusion areas, and generate a multidimensional occlusion label tensor.

[0073] Specifically, from deployment in different locations The acquisition time of each camera is... Image sequences ,in Indicates the first A perspective at any moment The image.

[0074] Calculate the optical flow field for each image Dynamic occlusion regions are identified by judging optical flow amplitude thresholds, and a binary occlusion mask is generated. The occluded areas are marked with 1, and the unoccluded areas are marked with 0. The occlusion masks for all views and times are combined to form a multidimensional occlusion label tensor. ,in and These represent the height and width of the image, respectively.

[0075] Furthermore, the optical flow field The Lucas-Kanade method is used to calculate: for two consecutive frames of images and Assuming pixel The motion satisfies the optical flow constraint equation ,in For optical flow vector, These represent the spatial and temporal gradients of the image, respectively. The least squares method is used to... Solving for the optical flow vector within a local window: Optical flow amplitude is defined as .in Indicates that Summation of pixel gradients within a local window. Notation convention: superscript. or Indicates matrix / vector transpose; if it appears This represents an exponential function.

[0076] Furthermore, optical flow amplitude threshold Statistical analysis determined that:

[0077] The statistical distribution of optical flow amplitude was calculated for five consecutive frames of images, and the 90th percentile value was used as the dynamic occlusion judgment threshold. The specific calculation was as follows: ,in The function representing the 90th percentile. For pixels The optical flow amplitude at that location. When the optical flow amplitude... At that time, the pixel is marked as a dynamic occlusion area.

[0078] Figure 3This demonstrates the change in occlusion area at different camera positions over a specific time period, corresponding to the effect of dynamic occlusion detection in the optical flow field detection step.

[0079] Step 200: Calculate data validity weights based on the occlusion marker tensor and construct a sparse multi-view observation tensor. For each view-time pair... Calculate data validity weights This represents the unoccluded proportion of the viewpoint at that moment. For validity weights greater than a threshold... Image, extract the pose feature vector of the lighting lamp. ,in As the feature dimension, construct a sparse multi-view observation tensor. The missing positions are indicated by special markers.

[0080] Furthermore, data validity weight threshold The threshold is set to 0.6, meaning that when the unobstructed proportion of the viewpoint image exceeds 60%, the observation data is considered valid. The data validity weight threshold was determined experimentally: when... At times, observational data containing too much obstruction will introduce significant noise; when At that time, the effective observation data was too scarce to support the convergence of the subsequent tensor decomposition algorithm.

[0081] It should be noted that the extraction of the lighting fixture's pose feature vector refers to extracting pose parameters such as key point coordinates, tilt angle, and top displacement of the lighting fixture from the image using a convolutional neural network, forming... 3D feature representation. Before feature extraction, the input RGB image is first preprocessed by normalization, scaling the pixel values ​​from the range [0,255] to the range [0,1]. The extracted pose parameters include different physical quantities: keypoint coordinates (pixel units), tilt angle (radian units), and top displacement (meter units). To eliminate the influence of dimensional differences on subsequent calculations, each parameter is standardized: coordinate parameters are normalized by dividing by the image size, angle parameters are normalized according to the maximum tilt angle range, and displacement parameters are normalized according to the maximum expected displacement value.

[0082] Furthermore, the physical constraint range for the tilt angle of the lighting fixture is set as follows: radians (i.e.) This corresponds to the maximum safe tilt angle of the lighting fixture from a perfectly vertical position to any direction. The standardized processing of the angle parameter is as follows: ,in This is the actual tilt angle. This refers to the standardized angular features.

[0083] The input layer of the aforementioned convolutional neural network receives a size of [size missing]. RGB image, where and These represent the height and width of the image, respectively. The output layer contains... 1 neuron, corresponding to 3D pose feature vector, where the first Wei said The two-dimensional coordinates of the nth key point, the th Dimension represents the tilt angle, the third to The dimension represents the top displacement vector and other attitude parameters.

[0084] Furthermore, the specific architecture of the convolutional neural network includes: First convolutional layer: 32 layers Convolutional kernel with stride of 2 and ReLU activation function; Second convolutional layer: 64 layers Convolutional kernel with stride of 2 and ReLU activation function; third convolutional layer: 128 layers Convolutional kernel with a stride of 1 and ReLU activation function; global average pooling layer: pools the feature map into... The feature vector; Fully connected layer 1: 128 neurons input to 256 neurons, ReLU activation function; Fully connected layer 2: 256 neurons input to Each convolutional layer contains 10 neurons and uses a linear activation function. A BatchNorm layer is added after each convolutional layer for normalization, reducing the number of keypoints. Set to 8 (top and bottom of the light fixture, key support points, etc.), total feature dimensions dimension.

[0085] The aforementioned convolutional neural network was trained using a supervised learning model, employing a dataset of lighting images with manually labeled pose parameters. The optimization strategy used was the Adam optimizer, with a learning rate set to... The loss function is defined as: ;

[0086] in, and The predicted and the actual numbers are respectively the first two. Coordinates of key points and These are the predicted and actual tilt angles, respectively. and These are the predicted and actual top displacement vectors, respectively. These are weighting coefficients. Since coordinate, angle, and displacement parameters have different dimensions, the weighting coefficients... The variance of each item is set by preprocessing the data to ensure that the error terms of different physical quantities have the same numerical scale in the loss function.

[0087] Furthermore, the specific calculation method for the weight coefficients is as follows: Calculate the standardized error variance of each parameter on the training set. ,in This represents variance calculation. It was obtained through experimental statistics: (The variance of the coordinate error of the corresponding key point is approximately 0.1). (Corresponding angular error variance is approximately 0.01) (Corresponding to a displacement error variance of approximately 0.04), ensuring that the three losses are balanced on the same order of magnitude.

[0088] The output of the aforementioned convolutional neural network The actual pose parameters are obtained by inverse normalization of the 3D pose feature vector: (Previous) Multiplying the 3D coordinate features by the image size recovers the pixel coordinates. Dimensional features multiplied by Restore the actual tilt angle in radians, the first to Multiplying the displacement characteristic by the maximum expected displacement value recovers the displacement in meters.

[0089] Furthermore, the maximum expected displacement is set at 2 meters, corresponding to the horizontal displacement distance of the top of the lighting fixture at the maximum allowable tilt angle, calculated based on a lighting fixture height of 8 meters: Meters. To ensure the robustness of the algorithm, the actual setting is 2 meters to cover displacement changes within the normal monitoring range.

[0090] Figure 2 This demonstrates the data acquisition quality from different camera positions and the impact of dynamic occlusion on the validity of data from each viewpoint during the corresponding steps.

[0091] Step 300: Decompose the observed tensor into three components using the sparse temporal tensor decomposition algorithm.

[0092] This step employs an innovative sparse temporal tensor decomposition algorithm, which includes the following sub-steps:

[0093] Step 301: Initialize the low-rank background tensor sparse change tensor and noise tensor .

[0094] Furthermore, the initialization strategy for tensor decomposition is: low-rank background tensor By observing the tensor The result is obtained by performing High-Order Singular Value Decomposition (HOSVD) at the observation location, retaining the previous value. The main modes, in which the rank constraint parameter ; Sparse variation tensor Initialize to zero tensor; noise tensor Initialize to observation residuals Projection at the observation position; Lagrange multiplier tensors initialized to zero tensors, penalty parameters. The initial value is set to 1.0.

[0095] Step 302: Define the objective function: ;

[0096] Furthermore, the complete objective function consists of four components:

[0097] Reconstruction Loss Items Measuring the fitting accuracy at the observation location; low-rank constraint term Promote the low-rank structure of the background tensor to capture common patterns from multiple perspectives; sparse constraint terms Promoting the sparsity of variable tensors to separate anomalous events; noise regularization term Energy for controlling the noise tensor; temporal smoothing constraint. Ensure the continuity of the background tensor in the time dimension, where Indicates the first A background slice with all perspectives and features at any given moment.

[0098] in, For projection operators, only the observation position is preserved. Element; For tensor kernel norm, promote low-rank property; for Norms promote sparsity; It is the Frobenius norm; This is the regularization parameter.

[0099] Furthermore, tensor kernel norm Nuclear norm and definition using multimodal expansion: ,in Tensor Along the first Matrix expansion of each pattern The matrix nuclear norm (sum of singular values) is set to the weighting coefficients. Ensure the balance constraints of each mode. For a third-order tensor... Mode 1 expansion Corresponding perspective mode, mode 2 unfolds Corresponding time mode, mode 3 unfolds Corresponding feature patterns.

[0100] Furthermore, the set of observation locations Defined as a three-dimensional index set of valid observation data: ,in Indicates the first The first perspective, the first The moment, the first Index of each feature dimension. Projection operator. Its function is: when Keep Unchanged, when Time settings This ensures that the optimization process only considers the reconstruction error of the effective observation locations.

[0101] Furthermore, the range of values ​​for the regularization parameter was determined through cross-validation: Controlling the strength of low-rank constraints, Controlling the sparsity constraint strength The regularization strength of the noise term is controlled. The specific value is optimized on the validation set using a grid search method. .

[0102] Step 303: Iteratively solve using the alternating direction multiplier method, updating sequentially. Until it converges.

[0103] The input to the aforementioned Alternating Direction Multiplier Method (ADMM) is: the observation tensor. Observation location set Regularization parameters And the three initialized component tensors. The output is: the converged low-rank background tensor. sparse change tensor and noise tensor The algorithm decomposes the original optimization problem into three subproblems by introducing Lagrange multipliers and augmented Lagrange functions. Each subproblem has a closed-form solution or can be solved efficiently using a soft thresholding operator.

[0104] Furthermore, the solution steps for the three subproblems of the ADMM algorithm are as follows:

[0105] Subproblems: Fixed and Solving this problem using Tensor-Singular Value Thresholding (TVT) ;

[0106] Subproblem: Solved using the soft thresholding operator. ;

[0107] Subproblem: Closed-form solution using quadratic regularization. ;

[0108] Update the Lagrange multipliers: ;

[0109] Furthermore, the convergence criterion for the ADMM algorithm is: when the relative change in the objective function value between two consecutive iterations is less than 1 / 3... At that time, that is Or the 2-norm of both the original residual and the dual residual is less than 1. The algorithm terminates when the time is reached. The maximum number of iterations is set to 1000 to ensure that the algorithm converges within a finite time.

[0110] Step 304: Output the decomposed low-rank background tensor (Representing stable attitude patterns) and sparse variation tensors (Indicates an abnormal change pattern).

[0111] In this embodiment, the temporal smoothing constraint term is integrated into the complete objective function in step 302. The temporal smoothing constraint term ensures the temporal continuity of attitude evolution by penalizing the difference in background tensors at adjacent time points.

[0112] Furthermore, time-series smoothing constraint parameters The time-series smoothing constraint parameter is set to 0.01, determined by balancing reconstruction accuracy and time-series continuity: when At times, the temporal smoothing constraint is too weak, and the background tensor exhibits unreasonable jumps in the time dimension; when At this time, excessively strong smoothing constraints can suppress the true attitude change signal.

[0113] The main difference between the aforementioned sparse temporal tensor decomposition algorithm and traditional tensor decomposition lies in the introduction of a projection operator. Handling missing data by calculating errors only at observation locations; using low-rank constraints. Capture common pose patterns across multiple viewpoints; utilize sparse constraints Separate sudden and anomalous changes; incorporate temporal smoothing constraints to ensure the temporal continuity of the background tensor. The sparse temporal tensor decomposition algorithm extracts continuous attitude evolution patterns from fragmented data by decomposing incomplete observations into three physically meaningful components: stable background, sparse changes, and noise.

[0114] Figure 4 This demonstrates the optimized convergence process of the sparse temporal tensor decomposition algorithm.

[0115] Figure 5 The diagram shows the energy flow distribution of the three components after tensor decomposition, from the original observation tensor to the low-rank background tensor, the sparse variation tensor, and the noise tensor. The line thickness represents the energy percentage, visually demonstrating the signal capture capability of each component.

[0116] Step 400: Input the decomposition results into the conditional variational autoencoder to reconstruct the complete temporal trajectory.

[0117] low-rank background tensor A conditional variational autoencoder (CVAE) is used to input sparse observation features to generate pose compensation features for missing time periods. The encoder of the CVAE maps the observation features to the distribution of latent variables. ,in The conditional information (including time period labels and pose priors) is used; the decoder samples latent variables to generate reconstructed features. By maximizing the variational lower bound, the model is trained and the reconstructed complete pose monitoring sequence is output.

[0118] Furthermore, the goal of variational inference is to maximize the lower bound of evidence (ELBO).

[0119] However, in actual training, a gradient descent optimizer is used to minimize the negative ELBO loss, i.e.: ;

[0120] This transformation ensures consistency in the optimization direction: maximizing ELBO is equivalent to minimizing negative ELBO, thus allowing training using standard gradient descent-type optimizers.

[0121] The aforementioned conditional variational autoencoder consists of two parts: an encoder and a decoder. The encoder contains three fully connected layers: the first layer receives the spliced ​​observation features. and condition information The first layer outputs a hidden representation with a dimension of 256; the second layer maps the hidden representation to intermediate features with a dimension of 128; the third layer outputs the mean vectors of the latent variables. Sum of logarithmic variance vector ,in For the dimension of latent variables.

[0122] Furthermore, the dimension of latent variables Set to 64, condition information dimension Set to 32. The network weights are initialized using the Xavier method: the weight matrix is ​​initialized according to... Distributed initialization, where These represent the number of input and output neurons, respectively; the bias vector is initialized to zero. The ReLU activation function is used, and a linear activation function is used in the output layer.

[0123] The aforementioned decoder also contains three fully connected layers: the first layer receives data from the distribution... latent variables of sampling and condition information The concatenated vectors output a hidden representation of dimension 128; the second layer maps the hidden representation to intermediate features of dimension 256; the third layer outputs the reconstructed pose features. .

[0124] The aforementioned conditional variational autoencoder is trained using a variational inference framework, with the Adam optimizer used as the optimization strategy and a learning rate set to [value missing]. The loss function is defined as a negative evidence lower bound (ELBO): ;

[0125] The first term is the reconstruction loss, expressed as mean squared error. The second term is the KL divergence regularization term. For conditional prior distribution ; These are the KL weighting coefficients, used to balance reconstruction quality and regularization of latent variables. Since the reconstruction loss has dimensions of attitude characteristics while the KL divergence is a dimensionless term, The coefficients are adaptively adjusted based on the average reconstruction loss on the training set to maintain a balance between the two terms on the numerical scale.

[0126] Furthermore, The adaptive adjustment strategy for the coefficients is as follows: after each training epoch, calculate the moving average of the reconstruction loss. Then follow the formula Update the KL weighting coefficients. Initial values ​​are set to... The scope of adjustment is limited to Between these parameters, ensure that the KL divergence term is neither oversuppressed nor dominates the loss function.

[0127] The reconstructed pose features output by the aforementioned conditional variational autoencoder The same reverse normalization process as the output of the convolutional neural network is required: restore the pixel coordinates, radian angles and meter unit displacements according to the physical meaning of the feature dimensions, so as to obtain a parameter sequence that can be directly used for attitude monitoring.

[0128] It should be noted that step 400 also includes the following sub-steps:

[0129] Step 401: Input the low-rank background tensor into the wavelet transform processor and perform multi-scale denoising to extract the long-term trend component.

[0130] The input to the aforementioned wavelet transform is: the low-rank background tensor. The time series data in the data, i.e., for each view-feature pair Extracting one-dimensional signals in the time dimension The output is the denoised long-term trend component sequence. The algorithm uses Discrete Wavelet Transform (DWT), selects the Daubechies wavelet as the basis function, decomposes the signal into detail coefficients and approximation coefficients at different scales through multi-level decomposition, performs soft thresholding denoising on the high-frequency detail coefficients, retains the low-frequency approximation coefficients, and finally reconstructs the smooth trend signal through inverse wavelet transform.

[0131] Furthermore, the threshold for soft-threshold denoising. An adaptive determination method is adopted: ,in The noise standard deviation estimate of the detail coefficients is calculated according to... calculate, These are the detail coefficients of the first-level wavelet decomposition. The time series length is specified. The number of decomposition layers is set to 4 to ensure effective separation of signal components at different time scales.

[0132] Step 402: Analyze the trend components and sparse change tensors based on the change point detection algorithm, identify the moment of attitude change and the starting point of gradual change, and generate a time period segmentation label sequence.

[0133] The input to the aforementioned change point detection algorithm is: the denoised trend component sequence output from step 401; and the sparse change tensor. The time slice. Output: Set of moments of change. The algorithm divides the label sequence according to the corresponding time period. It employs the PELT (PrunedExactLinearTime) method, which detects moments of significant change in statistical characteristics by minimizing a penalized cost function. The penalty parameter controls the detection sensitivity, and the cost function uses the negative log-likelihood of a normal distribution. The elements are as follows: , This indicates the number of detected change points.

[0134] Furthermore, the penalty parameters of the PELT algorithm Determined through information criteria: ,in The penalty parameter is set based on the BIC (Bayesian Information Criterion) to balance model complexity and fitting accuracy, specifying the time series length. The minimum segment length is set to 5 time steps to ensure that detected changes are statistically significant. When the distance between adjacent changes is less than 10 time steps, they are merged into a single change to avoid over-segmentation.

[0135] Step 403: Use a long short-term memory network to process the historical attitude sequence of the effective observation segments, learn the attitude evolution pattern at different time periods, and output the attitude prior distribution parameters for each time period.

[0136] The input layer reception dimension of the aforementioned Long Short-Term Memory network is... The historical posture sequence, in which For the effective observation segment's time length, The pose feature dimension is used. The LSTM layer contains 128 hidden units and outputs the hidden state at each time step. The output layer consists of a fully connected layer that stores the hidden state at the last moment. Parameter vector mapped to the attitude prior distribution , among which the former Dimension represents the prior mean, and denoted by dimensionality. Dimension represents the prior variance.

[0137] Furthermore, the effective observation segment duration The value constraints are: This means that the sequence should contain at least 15 consecutive time steps (corresponding to 7.5 minutes of observation history) to ensure that the LSTM can learn effective temporal patterns, and no more than 50 time steps (corresponding to 25 minutes) to avoid the gradient vanishing problem caused by excessively long sequences. In practical applications, The value is determined based on the change point detection results and represents the effective observation length between adjacent change points. The hidden state of the LSTM is initialized as a zero vector, and the gating parameters adopt a standard forget gate, input gate, and output gate structure.

[0138] The aforementioned Long Short-Term Memory (LSTM) network was trained using supervised learning, employing historical pose sequences and their corresponding future poses as training data. The optimization strategy used the RMSprop optimizer, with a learning rate set to... The loss function is defined as the negative log-likelihood: ;

[0139] in, For the first The true future posture of each sample These are the prior distribution parameters predicted by LSTM. This represents the number of training samples.

[0140] The aforementioned pose prior distribution parameters output by the Long Short-Term Memory network Conditional information used in conditional variational autoencoders , where the mean vector The expected value and variance vector represent the pose characteristics at each time period. These parameters represent the uncertainties of each feature dimension and provide time-related prior constraints for the generation process of CVAE, thereby improving the rationality of the reconstructed trajectory.

[0141] Figure 6 Demonstrates the reconstruction quality assessment of a conditional variational autoencoder.

[0142] In this embodiment of the application, to further improve monitoring accuracy, the following steps are also included:

[0143] Step 500: Calculate the local attitude change rate based on the reconstructed trajectory using sliding window analysis, input the change rate sequence into a recurrent neural network, and predict the future attitude evolution trend.

[0144] The input layer of the aforementioned recurrent neural network has a receiving dimension of 1. The sequence of attitude change rates, where To adjust the sliding window size, For pose feature dimension.

[0145] The RNN hidden layer contains 64 hidden units at each time step. Based on the current input and the hidden state of the previous moment Calculate the new hidden state: ,in This is the weight matrix. This is the bias vector.

[0146] The output layer consists of a fully connected layer that will ultimately hide the state. Mapping to the future pose prediction sequence at each time step .

[0147] Furthermore, the sliding window size Setting it to 10 time steps, corresponding to a historical observation window of about 5 minutes (assuming a sampling interval of 30 seconds), can fully capture the short-term changing trend of the lighting lamp's attitude.

[0148] Future prediction steps The process is set to 6 steps, corresponding to the attitude evolution prediction over the next 3 minutes. The attitude change rate is calculated using the central difference method. ,in The time interval is specified. The RNN weight matrix uses an orthogonal initialization method, and the bias vector is initialized to zero to ensure the stability of gradient propagation.

[0149] The aforementioned recurrent neural network is trained using a supervised learning model, employing historical pose change rate sequences and their corresponding future pose trajectories as training data. The optimization strategy uses an SGD optimizer combined with a momentum term, and the learning rate is set to... The momentum coefficient is 0.9. The loss function is defined as the mean square error of multi-step prediction: ;

[0150] in, For the first The sample in the future Predicted stance of the step, For the corresponding actual posture, This represents the number of training samples.

[0151] The pose prediction sequence output by the aforementioned recurrent neural network The same reverse normalization process as the aforementioned neural network is used: the predicted normalized pose features are converted into actual pixel coordinates, radian angles and meter unit displacements to obtain the pose evolution trajectory of the lighting lamps in future time periods, which can be used to identify potential pose anomalies in advance.

[0152] Figure 7 The results of the recurrent neural network's prediction of future poses are shown in the demonstration steps.

[0153] Step 600: Perform cross-validation by fusing multi-view reconstruction results, and generate high-confidence continuous attitude monitoring sequences and gradual anomaly warning signals through physical constraint optimization and Bayesian inference.

[0154] The input to the aforementioned Bayesian inference is: the reconstructed pose sequence from each viewpoint. Uncertainty estimation from various perspectives Physical constraints on the pose of the lighting fixtures (such as tilt angle range and rate of change limits). Output: The fused optimal pose estimate. And its confidence interval. The algorithm constructs the posterior probability distribution. By combining prior physical constraints, the optimal solution is obtained using maximum a posteriori estimation (MAP), and the confidence level is calculated by the variance of the posterior distribution.

[0155] Furthermore, the specific calculation steps for MAP estimation in Bayesian inference are as follows:

[0156] Construct the likelihood function: Assume that the observations from each perspective are independent and follow a Gaussian distribution;

[0157] Define the prior distribution: The prior mean and covariance matrix are determined based on physical constraints; the posterior distribution is calculated as follows: ;

[0158] MAP estimation: ;

[0159] Confidence interval: posterior covariance matrix The 95% confidence interval is Symbol convention: Here Representing the covariance matrix (uppercase Greek letters) and the summation symbol. different; It is an identity matrix.

[0160] Furthermore, the specific settings for the prior distribution parameters are: prior mean. (Corresponding to the normal pose after standardization), the prior covariance matrix is ​​a diagonal matrix. The prior standard deviation of coordinate features (Corresponding to 10% uncertainty in standardized coordinates), prior standard deviation of angle-type features (correspond (angular uncertainty), prior standard deviation of displacement features (Corresponding to 15% uncertainty in the standardized displacement). The prior distribution parameters reflect the expectation of the normal attitude and its reasonable range of variation.

[0161] Furthermore, the physical constraints on the lighting fixture's attitude are specifically defined as: tilt angle constraints. Constraints on the rate of change of angle between adjacent time points Radius per minute; Top displacement constraint Meters; Displacement rate constraint Meters per minute. These constraints reflect the physical characteristics of the lighting structure and the behavior patterns under normal monitoring scenarios.

[0162] The aforementioned Bayesian inference output is interpreted as: optimal pose estimation. The lighting lamp at time was obtained after reverse standardization. The actual attitude parameters (key point coordinates, tilt angle, top displacement) and the width of the confidence interval reflect the reliability of the attitude estimation. When the confidence interval exceeds the preset threshold or the attitude change rate exceeds the normal range, the system automatically generates a gradual anomaly warning signal to prompt relevant personnel to conduct further inspection.

[0163] Furthermore, the trigger threshold for the warning signal is specifically set as follows: the width of the tilt angle confidence interval exceeds... Radius; the width of the displacement confidence interval exceeds Meters; the attitude change rate exceeds 80% of the physical constraint limit for three consecutive time steps. When any of these conditions are met, the system automatically marks it as a potential abnormal state and records the abnormality confidence score for subsequent risk assessment.

[0164] Taking the lighting posture monitoring system of the main road Jiefang Road in the city center as an example, the daily traffic flow on this section of road reaches 15,000 vehicles, with large trucks and buses frequently passing through, causing serious dynamic obstruction of the monitoring view of the lighting. Four lighting lamps (numbered L1-L4) with a height of 8 meters were deployed within the monitoring area. Each lamp was equipped with four high-definition cameras at different angles (located in the east, south, west, and north directions) to collect real-time images of the lighting lamps' posture. The monitoring period was from 7:00 to 9:00 AM on weekdays, with an image acquisition frequency of one frame every 30 seconds, totaling data collection at 240 time points.

[0165] Table 1. Statistics of raw multi-view image acquisition data:

[0166]

[0167] Table 2. Dynamic occlusion detection results (typical segment from 7:30 to 7:35):

[0168]

[0169] Table 3. Results of sparse observation tensor construction:

[0170]

[0171] Table 4. Convergence performance and component results of the tensor decomposition algorithm:

[0172]

[0173] Table 5. Energy distribution analysis of the three tensor components:

[0174]

[0175] Table 6. Quality assessment of trajectory reconstructed by conditional variational autoencoder:

[0176]

[0177] Table 7. Future pose prediction results from the recurrent neural network:

[0178]

[0179] Table 8. Optimal pose estimation results of multi-view Bayesian fusion (critical moments):

[0180]

[0181] Through the complete implementation process described above, the system successfully verified the effectiveness of each core step. In step 300, the sparse temporal tensor decomposition algorithm decomposes the observed tensor... Successfully decomposed into three components: low-rank background tensor It captured 78.4% of the signal energy while maintaining a completely dense structure, with sparsely varying tensors. With only 3.7% of non-zero elements, it captured 16.8% of the anomalous energy variation, noise tensor. 4.8% of random measurement noise was effectively separated. In step 400, the conditional variational autoencoder achieved a reconstruction integrity rate of 94.2%-98.1%, with the angle reconstruction error controlled within... Within this range. In step 500, the recurrent neural network predicts the attitude evolution trend over the next 15 minutes based on sliding window analysis, with a prediction confidence level between 0.698 and 0.876, successfully identifying the continuously increasing attitude change pattern. In step 600, Bayesian inference fuses multi-view information to generate the optimal attitude estimate. At 8:00:00 and 8:30:00, the "attention" level abnormality warning signal was accurately triggered.

[0182] The tensor decomposition algorithm converged after 187 iterations. The convergence criterion... The set threshold requirements are met. Regarding reconstruction accuracy, the angular error of the final fusion result is controlled within... Within, the displacement error is controlled within Within a meter, all meet the accuracy requirements for engineering applications. The future attitude prediction function can identify the potential distance of the lighting fixture 15 minutes in advance. The system effectively and proactively supports operational and maintenance decisions by addressing potential high-risk tilting conditions. Its real-time processing capacity reaches 120 frames of image data per minute, meeting the real-time requirements of urban infrastructure monitoring.

[0183] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A passive attitude detection method based on deep learning, characterized in that, Includes the following steps: Acquire target object monitoring image sequences captured by cameras from multiple perspectives, identify dynamic occlusion regions, and generate a multidimensional occlusion label tensor; The step of identifying the dynamically occluded region includes: Calculate the optical flow field for each image, and identify dynamic occlusion areas by using optical flow amplitude thresholds; Generate a binary occlusion mask, where occluded areas are marked as 1 and unoccluded areas are marked as 0; Combine the occlusion masks of all viewpoints and times to form a multidimensional occlusion label tensor; Based on the occlusion marker tensor, calculate the data validity weights and construct a sparse multi-view observation tensor; The observation tensor is decomposed into three components—low-rank background tensor, sparse variation tensor, and noise tensor—using the sparse temporal tensor decomposition algorithm. The objective function of the sparse temporal tensor decomposition algorithm is defined as follows: The reconstruction loss term measures the fitting accuracy at the observation location, and only elements at the observation location are retained by the projection operator; The low-rank constraint term uses the tensor kernel norm to promote the low-rank structure of the background tensor to capture common patterns from multiple perspectives. The sparse constraint term uses the L1 norm to promote the sparsity of the variable tensor to separate anomalous events; The noise regularization term uses the Frobenius norm to control the energy of the noise tensor; The temporal smoothing constraint term ensures continuity in the temporal dimension by penalizing the differences in the background tensor between adjacent time steps; The alternating direction multiplier method is used for iterative solution, updating the three component tensors sequentially until convergence. The low-rank background tensor and sparse observation features obtained from the decomposition are input into the conditional variational autoencoder to reconstruct the complete temporal trajectory. The conditional variational autoencoder includes: The encoder maps observed features to a distribution of latent variables, where conditional information includes time period labels and pose priors; The decoder generates reconstructed features by sampling from latent variables; The model is trained by maximizing the variational lower bound, and the reconstructed complete pose monitoring sequence is output. The loss function includes the reconstruction loss and the KL divergence regularization term. The attitude prior is obtained by processing the historical attitude sequence of effective observation segments through a long short-term memory network, and the attitude prior distribution parameters for each time period are output. The local attitude change rate is calculated based on the reconstructed trajectory using sliding window analysis, and the future attitude evolution trend is predicted. Cross-validation is performed by fusing multi-view reconstruction results, and high-confidence continuous attitude monitoring sequences and gradual anomaly warning signals are generated through physical constraint optimization and Bayesian inference. The sparse temporal tensor decomposition algorithm achieves effective decomposition of incomplete observation data by minimizing an objective function that includes a reconstruction loss term, a low-rank constraint term, a sparse constraint term, a noise regularization term, and a temporal smoothing constraint term.

2. The passive attitude monitoring method based on deep learning according to claim 1, characterized in that, The steps for constructing the sparse multi-view observation tensor include: For each viewpoint-time pair, the data validity weight is calculated as the proportion of unoccluded viewpoint at that time. For images with a validity weight greater than a preset threshold, the pose feature vector of the target object is extracted using a convolutional neural network. The attitude feature vector includes key point coordinates, tilt angle and top displacement parameters, and the different physical quantities are standardized to eliminate dimensional differences. Construct a sparse multi-view observation tensor, where missing positions are indicated by special markers.

3. The passive attitude monitoring method based on deep learning according to claim 2, characterized in that, The convolutional neural network includes: Multiple convolutional layers are used to extract image features, and a batch normalization layer is added after each convolutional layer; Global average pooling layers pool feature maps into feature vectors; Multiple fully connected layers output pose feature vectors; During training, a weighted loss function is used, which includes keypoint coordinate errors, tilt angle errors, and top displacement errors. The weight coefficients are determined by the inverse of the variance of each parameter.

4. The passive attitude monitoring method based on deep learning according to claim 1, characterized in that, The reconstruction of the complete time-series trajectory also includes: The low-rank background tensor is input into a wavelet transform processor to perform multi-scale denoising and extract long-term trend components. Based on the change point detection algorithm, trend components and sparse change tensors are analyzed to identify the moment of abrupt attitude change and the starting point of gradual change, and a time period segmentation label sequence is generated. The time-segmentation label sequence serves as the conditional information input for the conditional variational autoencoder.

5. The passive attitude monitoring method based on deep learning according to claim 1, characterized in that, The predicted future posture evolution trend includes: Extract local segments of the reconstructed trajectory using a sliding window; The attitude change rate sequence was calculated using the central difference method; The rate of change sequence is input into a recurrent neural network, which outputs a sequence of pose predictions for multiple future time steps. The predicted sequence is inversely standardized to obtain the actual attitude parameter prediction values.

6. The passive attitude monitoring method based on deep learning according to claim 1, characterized in that, The Bayesian inference includes: Construct the likelihood function of the reconstruction results from each perspective, assuming that the observations from each perspective are independent and follow a Gaussian distribution; Define a prior distribution based on physical constraints, including tilt angle range constraints and rate of change constraints; The optimal pose estimate and its confidence interval after fusion are calculated using maximum a posteriori estimation. When the confidence interval exceeds the preset threshold or the attitude change rate exceeds the normal range, a gradual anomaly warning signal is generated.

7. A passive attitude monitoring system based on deep learning, used to execute the passive attitude monitoring method based on deep learning as described in any one of claims 1-6, characterized in that, include: The image acquisition module is used to acquire monitoring image sequences from cameras with multiple viewing angles; The occlusion detection module is used to identify dynamically occluded regions and generate an occlusion marker tensor. The tensor construction module is used to extract pose features and construct sparse observation tensors. The tensor decomposition module is used to execute the sparse temporal tensor decomposition algorithm. The trajectory reconstruction module includes a conditional variational autoencoder for reconstructing complete temporal trajectories; The trend prediction module is used to analyze the rate of attitude change and predict future evolution trends. The fusion decision module is used to perform Bayesian inference to generate final monitoring results and early warning signals.

Images