Abnormal motion trail detection method and device based on three-dimensional depth map reconstruction

Abstract

The invention relates to the technical field of three-dimensional reconstruction detection, in particular to an abnormal motion trail detection method and device based on three-dimensional depth map reconstruction, and aims to establish a pointwise corresponding relation based on the same timestamp through depth and infrared, suppress dead pixel and saturation influence, reduce fragmentation noise, reduce projection error accumulation and improve the detection accuracy. The multi-dimensional collaborative judgment capability and dynamic response accuracy of anomaly recognition are enhanced, a trajectory segment set with a direction vector and a distance sequence is obtained through conversion from camera coordinates to world coordinates and superposition of adjacent weight update and spatial association degree quantity, cross-frame continuity and scene scale consistency are improved, misinformation and missing report are reduced, and the accuracy of anomaly recognition is improved. Through the distribution correction of the generative adversarial network, the adaptive ability of the algorithm to the distribution of the complex motion trajectory and the anomaly recognition stability are further enhanced, and the multilayer improvement of trajectory reconstruction and anomaly detection in the aspects of space-time continuity, geometric consistency and distribution judgment is integrally realized.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction and detection technology, and in particular to a method and apparatus for detecting abnormal motion trajectories based on three-dimensional depth map reconstruction. Background Technology

[0002] The field of 3D reconstruction and detection technology mainly studies how to use computer vision and image processing technology to reconstruct, analyze and identify spatial structures by acquiring depth information of objects and scenes. It is widely used in intelligent monitoring, robot navigation, virtual reality, industrial inspection and intelligent transportation. The core is to combine depth image data and algorithm models to recover 3D spatial features from 2D visual information, identify target objects, analyze motion trajectories and detect abnormal behaviors, thereby providing a reliable basis for intelligent analysis and automatic decision-making.

[0003] The abnormal motion trajectory detection method based on 3D depth map reconstruction is an intelligent detection method that uses 3D vision perception technology to model and analyze the motion of target objects in dynamic scenes. This method acquires depth map data of the scene, reconstructs the 3D space, and establishes an accurate motion trajectory model of the monitored object. The purpose is to detect and identify abnormal motion behaviors in real time, such as abnormal movement of people, abnormal paths of objects, and suspicious behavior patterns, thereby improving the security and intelligence level of the system. By combining 3D reconstruction and motion detection, it overcomes the problem of traditional 2D video analysis being greatly affected by lighting, angle, and occlusion, and improves the accuracy, robustness, and environmental adaptability of detection.

[0004] Existing technologies have shortcomings in their operational modes. Depth and intensity information are often loosely synchronized at the frame level, resulting in coarse temporal granularity and difficulty in controlling cross-frame drift, leading to temporal misalignment in trajectory point sequences. Pixel-level processing mainly relies on point-by-point differential measurement, which easily leaves behind bad pixels and saturated pixels. Local noise manifests as pseudo-inflection points and pseudo-abrupt changes on the path. Trajectories are mostly statistically analyzed in the camera coordinate domain, failing to fully isolate the influence of camera pose changes. The same motion can produce directional and scale deviations under different viewpoints, limiting cross-scene comparisons. Path segmentation often relies on a single velocity threshold and equal-interval division, without coupling criteria between directional and distance changes. The true turning points are diluted, and short-term abrupt changes are smoothed out. These issues are particularly noticeable in complex indoor occlusion scenarios and crowd interaction scenarios, with common phenomena including trajectory jitter accumulation, false alarms concentrated in texture-poor areas, and anomaly localization spanning multiple unrelated segments, affecting subsequent decision-making and response efficiency. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and to propose an abnormal motion trajectory detection method and device based on three-dimensional depth map reconstruction.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an abnormal motion trajectory detection method based on three-dimensional depth map reconstruction, comprising the following steps:

[0007] S1: Acquire depth image frames, infrared intensity frames, and timestamp sequences through sensors, correspond pixel depth and infrared values ​​in the same frame point by point and align timestamps, jointly remove bad pixels and saturated pixels and perform adjacent pixel connectivity aggregation to generate a three-dimensional pixel connectivity block data group.

[0008] S2: Based on the three-dimensional pixel connected block data group, call the camera intrinsic parameters focal length and principal point index and convert the pixels to camera coordinates, calculate the weighted sum of the connected block pixels to obtain the block center coordinates and backfill the timestamp, and construct the camera coordinate trajectory node sequence.

[0009] S3: Based on the camera coordinate trajectory node sequence, call the camera extrinsic parameters rotation and translation, use a graph neural network to update the node adjacency weights and spatial associations, convert the camera coordinates into world coordinates, calculate the distance and direction difference between adjacent nodes, perform threshold segmentation and record the labels to obtain the world coordinate trajectory segment structure.

[0010] S4: Based on the camera coordinate trajectory node sequence and the world coordinate trajectory segment structure, match the node time series and synchronously align the path, calculate the velocity difference and direction deviation of the matched nodes, jointly compare the depth layer changes and filter out nodes with direction reversal and velocity change, and output the trajectory turning point candidate set.

[0011] S5: Based on the set of candidate points for trajectory turning points, a generative adversarial network is used to match the candidate points with the time index of the trajectory segment structure in world coordinates and estimate the nearest neighbor curvature. After judging the residual threshold, the segments that exceed the limit are marked, the spatial index of the segments is sorted out, and an abnormal motion trajectory framework is established.

[0012] As a further embodiment of the present invention, the three-dimensional pixel connected component data group includes pixel spatial coordinates, infrared intensity values, and timestamp indexes; the camera coordinate trajectory node sequence includes node spatial coordinates, pixel connected component center coordinates, and timestamp information; the world coordinate trajectory segment structure includes node world coordinates, distances between adjacent nodes, and direction difference labels; the trajectory turning point candidate set includes node spatial coordinates, velocity differences, and direction deviation angles; and the abnormal motion trajectory framework includes an over-limit segment index, a segment spatial coordinate sequence, and curvature residual threshold results.

[0013] As a further aspect of the present invention, the specific steps for generating the three-dimensional pixel connected component data group are as follows:

[0014] The system acquires depth image frames, infrared intensity frames, and timestamp sequences from sensors. It compares the pixel depth and infrared intensity of the same frame, aligns the timestamps with the pixel index, detects the gray point depth difference to remove saturated pixels, rearranges the rows and columns of effective pixels, integrates and maps them, and generates a synchronized pixel mapping set.

[0015] Based on the synchronized pixel mapping set, the grayscale difference and depth difference of adjacent pixels are cross-compared, connected pixels are determined by continuous difference intervals, connected block fusion and isolated points are deleted, and the edge region aggregation structure is reconstructed to generate a three-dimensional pixel connected block data group.

[0016] As a further aspect of the present invention, the specific steps for constructing the camera coordinate trajectory node sequence are as follows:

[0017] Based on the three-dimensional pixel connected block data group, the camera intrinsic parameters focal length and principal point index are called, and coordinate projection is performed by converting the pixel row and column coordinates with the focal length ratio. The values ​​of the offset between the pixel center and the principal point are adjusted, the three-dimensional coordinates of each pixel block are reconstructed and the hierarchy is mapped, and a pixel space conversion set is generated.

[0018] Based on the pixel space conversion set, the weighted summation of connected component pixel coordinates is performed and the center point position is calculated. The timestamp is backfilled by matching the time index table, the node coordinate sequence is sorted and the interval between consecutive nodes is verified, the center coordinates and time information are integrated to form a node sequence, and the camera coordinate trajectory node sequence is constructed.

[0019] As a further aspect of the present invention, the specific steps for obtaining the world coordinate trajectory segment structure are as follows:

[0020] Based on the camera coordinate trajectory node sequence, the external parameters rotation and translation are called. The rotation angle is calculated by converting the node coordinates in the three-dimensional axis direction and the translation vector is superimposed. After correcting the spatial offset of the node position, the coordinate matrix is ​​rearranged to generate the world coordinate node set.

[0021] Based on the world coordinate node set, a graph neural network is used to update the adjacency weights and node association strength, measure the three-dimensional distance between adjacent nodes and calculate the direction vector difference, and extract local mutations and store node identifiers by comparing the intervals of the continuous node distance sequence with the direction change values ​​to generate a node distance-direction difference set.

[0022] Based on the distance-direction difference set of the nodes, the threshold interval of the distance and direction difference is segmented and the node segment labels are divided. The order of the start and end indices of the segments is rearranged and the time series is mapped synchronously. The coordinate sequences of each segment are merged as a whole to obtain the world coordinate trajectory segment structure.

[0023] As a further aspect of the present invention, the execution process of the graph neural network is as follows: based on the input world coordinate node set, an adjacency table between nodes is established and a weight matrix is ​​initialized. The connection strength is calculated by the three-dimensional distance and the angle between directions between nodes, and the weights are mapped to the adjacency matrix. In each round of propagation, the features of adjacent nodes are aggregated according to the weight ratio with the node feature vector as input, and feature normalization and inter-layer transfer are performed to complete the diffusion of local spatial information. Then, the similarity between nodes is recalculated according to the aggregated feature distribution, the adjacency matrix and weight coefficients are updated, and the process is iterated repeatedly until the node features are stable, thus obtaining a structured representation of the spatial dependency between trajectory nodes.

[0024] As a further aspect of the present invention, the specific steps for outputting the trajectory turning candidate point set are as follows:

[0025] Based on the camera coordinate trajectory node sequence and the world coordinate trajectory segment structure, the node time sequence is matched, the node index is compared to complete the path synchronization alignment, the position offset and velocity vector component difference between corresponding nodes are calculated, the directional offset angle is measured and the velocity difference amplitude is statistically analyzed, and a synchronization path matching set is generated.

[0026] Based on the synchronous path matching set, the depth layer changes of continuous nodes are compared, and nodes are screened by jointly identifying direction angle reversal points and velocity change points. The distance intervals between candidate nodes are calculated to aggregate adjacent parts, and after integrating the segment index, the numbers are assigned and the trajectory turning point candidate set is output.

[0027] As a further aspect of the present invention, the specific steps for establishing the abnormal motion trajectory framework are as follows:

[0028] Based on the set of candidate points for trajectory turning points, the candidate points are matched with the time index of the world coordinate trajectory segment structure. The time index is compared with the corresponding correction node position. The distance difference and direction vector change rate between adjacent nodes are calculated. The local curvature is measured and the direction of continuous nodes is smoothly corrected to generate a curvature matching result set.

[0029] Based on the curvature matching result set, a generative adversarial network is used to establish a distribution comparison between the real trajectory and the simulated trajectory. After calculating the curvature residual value of each node, a residual threshold interval is established. The residual is compared with the threshold and the nodes that exceed the limit are screened out. The distance between consecutive nodes that exceed the limit is determined and the segment index is recorded to generate a set of indexes for the segments that exceed the limit.

[0030] Based on the aforementioned over-limit paragraph index set, the spatial coordinates of nodes within the paragraph are aggregated and rearranged, the boundary distances between adjacent paragraphs are determined and the sections are merged, the spatial index table is organized and global trajectory coordinates are mapped, and an abnormal motion trajectory framework is established.

[0031] As a further aspect of the present invention, the execution process of the generative adversarial network is as follows:

[0032] Based on the curvature matching result set of the input, the distribution inputs of real trajectory samples and simulated trajectory samples are constructed. The generator generates simulated trajectory data with the same dimensions as the real samples, using the spatial position, direction vector, and velocity sequence of the trajectory nodes as generation parameters. The differences between the generated samples and the real samples in the distribution space are compared. The discriminator receives the curvature, direction change rate, and residual values ​​of the two sets of samples, calculates the similarity and offset rate in the feature space, and feeds the results back to the generator to update the trajectory generation parameters. The generator adjusts the curvature and spatial coherence of the output samples according to the feedback until the discriminator can no longer distinguish the distribution differences between the two types of samples, thus obtaining a balanced distribution dataset for residual comparison analysis.

[0033] An abnormal motion trajectory detection device based on 3D depth map reconstruction includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above-mentioned abnormal motion trajectory detection method based on 3D depth map reconstruction.

[0034] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0035] 1. In this invention, by establishing a point-to-point correspondence between depth and infrared based on the same timestamp, a three-dimensional connected component set containing pixel spatial coordinates, infrared intensity values, and time indexes is generated, which suppresses bad pixels and saturation effects, reduces fragmented noise, reduces the accumulation of projection errors, and enhances the multi-dimensional collaborative judgment capability and dynamic response accuracy of anomaly identification.

[0036] 2. In this invention, by converting camera coordinates to world coordinates, superimposing adjacency weight updates and spatial correlation metrics, a trajectory segment set with direction vectors and distance sequences is obtained, which improves cross-frame continuity and scene scale consistency, reduces false alarms and false negatives, and improves the accuracy and traceability of anomaly localization.

[0037] 3. In this invention, the distribution correction of generative adversarial networks further enhances the algorithm's adaptability to complex motion trajectory distributions and the stability of anomaly identification, achieving a multi-layered improvement in trajectory reconstruction and anomaly detection in terms of spatiotemporal continuity, geometric consistency, and distribution determination. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the workflow of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] Example 1

[0041] Please see Figure 1 This invention provides a technical solution: an abnormal motion trajectory detection method based on 3D depth map reconstruction, comprising the following steps:

[0042] S1: Acquire depth image frames, infrared intensity frames, and timestamp sequences through sensors, correspond pixel depth and infrared values ​​in the same frame point by point and align timestamps, jointly remove bad pixels and saturated pixels and perform adjacent pixel connectivity aggregation to generate a three-dimensional pixel connectivity block data group.

[0043] S2: Based on the 3D pixel connected component data group, call the camera intrinsic parameters focal length and principal point index and convert the pixels to camera coordinates. The connected component pixels are weighted and summed to obtain the block center coordinates and backfill the timestamp to construct the camera coordinate trajectory node sequence.

[0044] S3: Based on the camera coordinate trajectory node sequence, call the camera extrinsic parameters rotation and translation, use graph neural network to update node adjacency weights and spatial associations, convert camera coordinates to world coordinates, calculate the distance and direction difference between adjacent nodes, perform threshold segmentation and record labels to obtain the world coordinate trajectory segment structure.

[0045] S4: Based on the camera coordinate trajectory node sequence and the world coordinate trajectory segment structure, match the node time series and synchronously align the path, calculate the velocity difference and direction deviation of the matched nodes, jointly compare the depth layer changes and filter out nodes with direction reversal and velocity change, and output the trajectory turning point candidate set.

[0046] S5: Based on the candidate point set of trajectory turning points, a generative adversarial network is used to match the candidate points with the time index of the trajectory segment structure in world coordinates and estimate the nearest neighbor curvature. After judging the residual threshold, the out-of-limit segments are marked, the segment spatial index is sorted out, and an abnormal motion trajectory framework is established.

[0047] The 3D pixel connected component data set includes pixel spatial coordinates, infrared intensity values, and timestamp indexes. The camera coordinate trajectory node sequence includes node spatial coordinates, pixel connected component center coordinates, and timestamp information. The world coordinate trajectory segment structure includes node world coordinates, distances between adjacent nodes, and orientation difference labels. The trajectory turning point candidate set includes node spatial coordinates, velocity differences, and orientation deviation angles. The abnormal motion trajectory framework includes out-of-limit segment indexes, segment spatial coordinate sequences, and curvature residual threshold results.

[0048] The specific steps for generating a 3D pixel connected component data set are as follows:

[0049] The system acquires depth image frames, infrared intensity frames, and timestamp sequences from sensors. It compares the pixel depth and infrared intensity of the same frame, aligns the timestamps with the pixel index, detects the gray point depth difference to remove saturated pixels, rearranges the rows and columns of effective pixels, integrates and maps them, and generates a synchronized pixel mapping set.

[0050] Based on the synchronous pixel mapping set, the gray level difference and depth difference of adjacent pixels are cross-compared. Connected pixels are determined by continuous difference intervals, connected block fusion and isolated points are deleted, and the edge region aggregation structure is reconstructed to generate a three-dimensional pixel connected block data group.

[0051] Based on depth image frames, infrared intensity frames, and timestamp sequences acquired by sensors, the pixel depth and infrared intensity of the same frame are matched. An index table is built for each pixel in row and column order. Linear time interpolation is used to synchronize the timestamp. The time difference between the start frame and the end frame is divided into 200 equally spaced segments. The pixel timestamp is inserted into the intermediate time point according to the sequence position. The depth difference of bad pixels is detected in the aligned data. The depth threshold is set to 0.05 meters. When the depth difference between adjacent pixels exceeds the threshold, it is marked as a bad pixel. Pixels with infrared intensity values ​​exceeding 255 are judged as saturated pixels and removed. The removal process traverses the pixel matrix in row and column order and covers invalid pixels with zero values. The remaining valid pixels are renumbered and rearranged in row and column order. The numbering range is from 000001 to the maximum valid pixel index. After the rearrangement, the row and column index of each pixel and the synchronization timestamp are mapped to the three-dimensional coordinate index table. The unified spatial index number corresponds to the depth value and infrared intensity value, generating a synchronized pixel mapping set.

[0052] Based on the synchronous pixel mapping set, the gray level difference and depth difference of adjacent pixels are compared. The gray level difference threshold of adjacent pixels is set to 12, and the depth difference threshold is set to 0.03 meters. Pixels that meet the conditions are marked as connected pixels in the same region. The marking number starts from 1 and increases sequentially. The numbers of connected regions are summarized. Regions with fewer than 5 connected block pixels are deleted. To prevent edge breakage, the boundary row and column range of the connected region is extended by two pixels. The empty rows and columns are filled with the average gray level and depth difference of the surrounding area to form a continuous boundary structure. When merging adjacent connected regions, if the minimum distance between the boundaries of two regions is less than 3 pixels, the boundaries are merged into a unified region. Isolated images are deleted. The isolation judgment condition is that there are no connected neighboring points in the eight directions of up, down, left, right and diagonal. The merged and corrected connected regions are reordered according to the center coordinate position. The sorting priority is based on the vertical row number and the horizontal column number. The pixel set of the connected region is output in the form of a number table to generate a three-dimensional pixel connected block data group.

[0053] The specific steps for constructing the camera coordinate trajectory node sequence are as follows:

[0054] Based on the 3D pixel connected component data set, the camera intrinsic parameters focal length and principal point index are called. The coordinate projection is performed by converting the pixel row and column coordinates with the focal length ratio. The values ​​of the offset between the pixel center and the principal point are adjusted to reconstruct the 3D coordinates of each pixel block and map the hierarchy, generating a pixel space conversion set.

[0055] Based on the pixel space conversion set, the weighted summation of connected component pixel coordinates is performed and the center point position is calculated. The time index table is matched and backfilled with timestamps. The node coordinate sequence is sorted and the interval between consecutive nodes is verified. The center coordinates and time information are integrated to form a node sequence and a camera coordinate trajectory node sequence is constructed.

[0056] Based on a 3D pixel connected component dataset, the pixel row and column coordinates of each connected component are transformed. The camera intrinsic focal length is set to 1200 pixels, and the principal point index is (640, 480). Using the image center as the reference origin, the principal point index offset is subtracted from the pixel row and column values. The row coordinate offset is multiplied by a focal length scaling factor of 0.0012, and the column coordinate offset is also multiplied by a focal length scaling factor of 0.0012. The transformation results are mapped to the camera coordinate plane, establishing a pixel-to-spatial point mapping table. Within each connected component, the weighted depth average of the pixels is taken, with the weight set to the pixel's infrared intensity and the standard deviation of the block's depth. The inverse ratio is calculated by using the standard deviation of depth values ​​within the block as the denominator and the corresponding pixel infrared intensity as the numerator. The resulting ratio is normalized to the range of 0 to 1. The spatial coordinates of the block center are obtained by summing the weighted coordinates in each connected block in row and column order and dividing by the number of pixels. The obtained spatial coordinates are recorded in the three-dimensional coordinate matrix with index numbers. The row number corresponds to the row coordinates of the original image, and the column number corresponds to the column coordinates of the original image. The depth value is used as the Z-axis value input. The center coordinates of all pixel blocks are reorganized in layers according to the depth distance interval. The depth layer division interval is 0.1 meters, and the numbering is sequentially increased from near to far to generate a pixel space conversion set.

[0057] Based on the pixel spatial conversion set, the coordinates of pixels in the weighted connected components are summed by their indexes in the spatial conversion set. The weighting factor is the ratio of the infrared intensity value of each pixel to the sum of the infrared intensity values ​​within the block. The three-dimensional coordinates of the center point are obtained by dividing the weighted coordinates of the connected components by the number of pixels. Each connected component index is matched with a time index table. The time index table stores timestamps in the acquisition order, with a starting time of 0.000 seconds and a sampling interval of 0.033 seconds. The timestamp values ​​of the corresponding indices are backfilled into the center point record. In the node sequence formed by all center point coordinates, the nodes are sorted in ascending order of timestamps. After sorting, the Euclidean distance between adjacent node coordinates is checked, with an interval threshold of 0.02 meters. Nodes exceeding the threshold are marked as breakpoints and recorded with numbers. The center coordinates and time information are integrated into the node sequence that passes the check to form a three-dimensional coordinate matrix and a time sequence correspondence table, thus constructing the camera coordinate trajectory node sequence.

[0058] The specific steps to obtain the structure of the world coordinate trajectory segment are as follows:

[0059] Based on the camera coordinate trajectory node sequence, the external parameters rotation and translation are called. The rotation angle is calculated by converting the node coordinates in the three-dimensional axis direction and the translation vector is superimposed. After correcting the spatial offset of the node position, the coordinate matrix is ​​rearranged to generate the world coordinate node set.

[0060] Based on the world coordinate node set, a graph neural network is used to update the adjacency weight and node association strength, measure the three-dimensional distance between adjacent nodes and calculate the direction vector difference, and extract local mutation amount and store node identifier by comparing the distance sequence between continuous nodes with the direction change value interval, thus generating a node distance direction difference set.

[0061] Based on the distance-direction difference set of nodes, the threshold interval of the distance and direction difference is segmented and the node segment labels are divided. The order of the start and end indices of the segments is rearranged and the time series is mapped synchronously. The coordinate sequences of each segment are merged as a whole to obtain the world coordinate trajectory segment structure.

[0062] Based on the camera coordinate trajectory node sequence, the external parameters rotation and translation are called to perform rotation conversion on the coordinates of each node in the three-dimensional axis direction. The rotation angle is expressed in Euler angle form, with a rotation angle of 15 degrees around the X-axis, 10 degrees around the Y-axis, and 5 degrees around the Z-axis. The rotation order is set to ZYX sequentially. The matrix product of the node coordinates is calculated for each axis. The rotation matrix is ​​multiplied by the original node coordinate matrix. The rotation matrix has a size of 3×3. After the calculation is completed, the translation vector is superimposed on the rotation result. The translation vector values ​​are (0.12, 0.08, 0.04). After superimposing the translation vector, the columns in the coordinate matrix are rearranged according to the coordinate axis order, with the row number corresponding to the X-axis, the column number corresponding to the Y-axis, and the depth value corresponding to the Z-axis. The overall matrix is ​​normalized, and the coordinate values ​​are restricted to the range of 0 to 1. The nodes are then reordered according to their index numbers to generate the world coordinate node set.

[0063] Based on a world coordinate node set, a graph neural network algorithm is used. The input node feature dimension is set to 3, corresponding to the 3D coordinate values ​​of the node. The adjacency matrix size is N×N, where N is the number of nodes. The initial adjacency weight coefficient is 0.5, the number of iterations is set to 10 rounds, and the learning rate is set to 0.001. The ReLU function is used for feature activation. The feature vectors of each node's neighborhood are weighted and aggregated in each round of propagation. The aggregation coefficient is calculated from the 3D distance and direction vector difference between nodes. The direction vector difference is represented as the angle between 3D unit vectors and recorded in radians. The distance and direction difference between adjacent nodes are normalized after each iteration, with the value range limited to 0 to 1. The weights are updated using a weight smoothing strategy, with the smoothing factor set to 0.8. The node feature matrix after the iteration is subjected to node difference, and the node identifiers with distance changes exceeding 0.03 meters and direction changes greater than 0.2 radians are extracted. The node number and the difference value table are stored to generate a node distance and direction difference set.

[0064] Based on the distance-direction difference set of nodes, the distance difference range is divided, with the threshold set to a range of 0.02 meters to 0.05 meters and the direction difference range set to a range of 0.15 radians to 0.25 radians. Nodes that meet both thresholds are divided into segments according to the continuity of their serial numbers. Each segment is assigned a label, with the label range increasing from 1. The start and end indices of the segments are rearranged according to the node time series order. The time synchronization step is set to 0.033 seconds. The time index of each segment is mapped to the spatial coordinates. After synchronization is completed, the coordinate sequences of all segments are merged. The merging is sorted in ascending order of the time series, and the overall coordinate sequence table is output to obtain the world coordinate trajectory segment structure.

[0065] The execution process of a graph neural network is as follows: based on the input set of world coordinate nodes, an adjacency table between nodes is established and a weight matrix is ​​initialized. The connection strength is calculated by the three-dimensional distance and the angle between directions between nodes, and the weights are mapped to the adjacency matrix. In each round of propagation, the features of adjacent nodes are aggregated according to the weight ratio with node feature vectors as input, and feature normalization and inter-layer propagation are performed to complete the diffusion of local spatial information. Then, the similarity between nodes is recalculated based on the aggregated feature distribution, the adjacency matrix and weight coefficients are updated, and the process is iterated repeatedly until the node features are stable, thus obtaining a structured representation of the spatial dependencies between trajectory nodes.

[0066] Graph neural networks, according to the formula:

[0067] ;

[0068] in: For the first Layer node feature matrix, For the first Layer node feature matrix, For degree matrix, The degree matrix is ​​the negative first power of 2. For the first The node and the first The three-dimensional Euclidean distance between nodes For the first The node and the first The difference in the angle between the direction vectors of each node For the first The node and the first The difference between nodes in the depth direction. For the first The node and the first Time difference between nodes For the first The node and the first The normalized curvature value of each node. This is the distance attenuation coefficient. The direction penalty coefficient, This is the depth penalty coefficient. This is the time penalty coefficient. For curvature modulation coefficients, For the first Layer weight matrix, For residual coefficients, It is a non-linear activation function. Let represent the index number of any two nodes in the graph. Number the target node. Number the adjacent nodes;

[0069] Execution process: First, the 3D coordinates, direction vectors, depth values, and timestamp information of each node are read from the camera coordinate trajectory node sequence, and a node index table is built in the world coordinate system for each pair of nodes. Calculate the three-dimensional Euclidean distance respectively , difference in direction angle Depth layer difference and time difference Then, the four physical quantities are substituted into the exponential weighting term. And introduce curvature normalization parameters Linear modulation is performed to form weighted adjacency values ​​between nodes, based on the degree matrix. Symmetrical normalization of adjacency weights yields the normalized adjacency operator, which is then used to normalize the feature matrix of the current layer nodes. Adjacency operator and weight matrix Multiply to obtain the propagation feature matrix, and then add a residual term to it. Apply the ReLU activation function Perform a nonlinear mapping on the output matrix, and the output... This is the updated set of node features.

[0070] The specific steps for outputting the candidate set of trajectory turning points are as follows:

[0071] Based on the camera coordinate trajectory node sequence and the world coordinate trajectory segment structure, the node time series is matched, the node index is compared to complete the path synchronization alignment, the position offset and velocity vector component difference between corresponding nodes are calculated, the directional offset angle is measured and the velocity difference amplitude is statistically analyzed, and a synchronization path matching set is generated.

[0072] Based on the synchronous path matching set, the depth layer changes of continuous nodes are compared, and nodes are screened by joint identification of direction angle reversal points and velocity change points. The distance interval between candidate nodes is calculated to aggregate adjacent parts, and after integrating the segment index, the number is assigned and the trajectory turning candidate point set is output.

[0073] Based on the camera coordinate trajectory node sequence and the world coordinate trajectory segment structure, the node timestamp sequence and the trajectory segment time index are aligned. The time index interval is set to 0.033 seconds. The two time sequences are sorted in ascending order, and missing items are inserted at the same time nodes using linear interpolation. After alignment, the node index correspondence is calculated, with the index range from 1 to N, where N is the number of nodes. The 3D coordinate difference of each pair of matching nodes is calculated. The coordinate difference calculation is performed on a component basis, with the difference calculated in the X, Y, and Z directions and the absolute value taken. The obtained difference is normalized, with the range limited to 0 to 1. The position offset is calculated. The sum of squares of the three-dimensional differences is expressed in meters. The spatial positions of adjacent nodes are removed, and the velocity components are calculated at time intervals, with the velocity unit being meters per second. The absolute value of the velocity component difference between adjacent nodes is taken and recorded in the velocity difference array. The maximum velocity difference is set to 3.0 meters per second, and any part exceeding the upper limit is truncated at 3.0 meters per second. The angle between the direction vectors of adjacent nodes is calculated after normalizing the three-dimensional coordinate difference and is recorded in radians. The threshold for the direction offset angle is set to 0.2 radians. The calculated velocity difference and direction offset angle of the nodes are stored in the time matching table with the node number as the index, generating a synchronization path matching set.

[0074] Based on the synchronous path matching set, the depth layer changes of consecutive nodes are compared. The depth layer difference threshold is set to 0.05 meters. The depth layer difference sequence is calculated for three consecutive nodes. The depth layer reversal point is detected and judged by the change of positive and negative signs. The reversal node number is recorded. At the same time, the velocity change point is detected. Nodes with a velocity difference greater than 2.0 meters per second in the velocity difference array are marked as change nodes. The intersection of the numbers of the direction angle reversal node and the velocity change node is extracted. The nodes that appear at the same time are taken as candidate nodes. The spatial distance between adjacent candidate nodes is calculated as Euclidean distance. The distance threshold is set to 0.1 meters. The node intervals with a distance less than the threshold are aggregated and merged. The merged segments are assigned consecutive numbers, which start from 1 and increment. The indexes of each segment are rearranged in time order. The segment time index is mapped to the original timestamp sequence. The number of the integrated segment index table is sorted and integrated. A comprehensive record table containing node number, time index and spatial coordinate index is output. The set of trajectory turning candidate points is output.

[0075] The specific steps for establishing an abnormal motion trajectory framework are as follows:

[0076] Based on the candidate point set of trajectory turning points, the candidate points are matched with the time index of the world coordinate trajectory segment structure. The time index is compared with the corresponding correction node position. The distance difference and direction vector change rate between adjacent nodes are calculated. The local curvature is measured and the direction of continuous nodes is smoothly corrected to generate a curvature matching result set.

[0077] Based on the curvature matching result set, a generative adversarial network is used to establish a distribution comparison between the real trajectory and the simulated trajectory. After calculating the curvature residual value of each node, a residual threshold interval is established. The residual is compared with the threshold and the nodes that exceed the limit are screened out. The distance between consecutive nodes that exceed the limit is determined and the segment index is recorded to generate a set of indexes for the segments that exceed the limit.

[0078] Based on the out-of-limit paragraph index set, the spatial coordinates of nodes within the paragraph are aggregated and rearranged, the boundary distance between adjacent paragraphs is determined and the sections are merged, the spatial index table is organized and mapped to the global trajectory coordinates, and an abnormal motion trajectory framework is established.

[0079] Based on the candidate point set for trajectory turning points, the time series of candidate points are compared with the time index of the trajectory segment structure in world coordinates. The time index step size is set to 0.033 seconds. The time index sequence is sorted in ascending order and missing time points are filled in. Missing points are filled in by linear interpolation. The node position is checked for successfully matched time indices. The check process is traversed in the order of node number. For each node, the distance difference between the coordinates of the two adjacent points is calculated. The distance unit is meters and the result is rounded to three decimal places. The unit vector of the node direction vector is obtained by the three-dimensional coordinate difference. The angle between the direction vectors of adjacent nodes is calculated. The angle is expressed in radians and the value range is 0 to π. The rate of change of the direction of continuous nodes is obtained by dividing the angle by the time interval. The unit is radians per second. The curvature of local nodes is measured. The window contains node numbers i-1, i, and i+1. The radius of curvature is obtained by the vector cross product method and the reciprocal is taken as the curvature value. The curvature unit is per meter. The curvature sequence is smoothed and corrected. The corrected three-dimensional unit vector of the node direction is recalculated and the direction matrix is ​​updated to generate a curvature matching result set.

[0080] Based on the curvature matching result set, a generative adversarial network (GAN) algorithm is used to compare the distribution of real trajectory samples and simulated trajectory samples. The input feature dimension is set to 5, including node curvature, rate of change of direction, distance difference, time index, and node number. The generator has 4 layers: 64 nodes in the input layer, 128, 256, and 128 nodes in the hidden layer, and 64 nodes in the output layer. The negative slope parameter is 0.2. The discriminator has a 3-layer structure: 64 nodes in the input layer, 128 and 64 nodes in the hidden layer, and 1 node in the output layer. The activation function is Sigmoid. The loss function is binary cross-entropy, the optimizer is Adam, the learning rate is set to 0.0002, the batch size is 32, and the number of training iterations is 200. After the network training is completed, the curvature residual value of each node is calculated, and the residual threshold range is set to 0.03 to 0.08. The residual value of each node is compared with the upper limit of the threshold. Nodes with residuals exceeding the upper limit of the threshold are marked as out-of-limit nodes. The spatial distance between consecutive out-of-limit nodes is measured in meters. Nodes with a distance of less than 0.2 meters are grouped into the same segment and the start and end index numbers are recorded to generate an out-of-limit segment index set.

[0081] Based on the over-limit paragraph index set, the three-dimensional spatial coordinates of nodes within each paragraph are aggregated and calculated. The aggregation method is to average the coordinate components, and the average precision is retained to three decimal places. The nodes within the paragraph are sorted in chronological order with a time step of 0.033 seconds. The Euclidean distance between the boundary nodes of adjacent paragraphs is determined, and the distance threshold is set to 0.1 meters. If it is less than the threshold, the adjacent paragraphs are merged and renumbered. The paragraph numbers are sequentially incremented, starting from 1. After merging, the spatial index table of each paragraph node is reorganized, and the index number corresponds to the timestamp number and spatial coordinate number. The nodes are mapped in the global coordinate system, and the coordinate value precision is retained to three decimal places. A global record table containing paragraph number, node index, spatial coordinates and timestamp is output to establish an abnormal motion trajectory framework.

[0082] The execution process of a generative adversarial network is as follows:

[0083] Based on the curvature matching result set of the input, the distribution input of real trajectory samples and simulated trajectory samples is constructed. The generator generates simulated trajectory data with the same dimension as the real samples. The spatial position, direction vector and velocity sequence of the trajectory nodes are used as generation parameters. The difference between the generated samples and the real samples in the distribution space is compared. The discriminator receives the curvature, direction change rate and residual value of the two sets of samples, calculates the similarity and offset rate in the feature space, and feeds the results back to the generator to update the trajectory generation parameters. The generator adjusts the curvature and spatial coherence of the output samples according to the feedback until the discriminator can no longer distinguish the distribution difference between the two types of samples, and obtains a balanced distribution dataset for residual comparison analysis.

[0084] Generate adversarial networks according to the formula:

[0085] ;

[0086] in: To generate the total loss value of the adversarial network, For the discriminant function, For generating functions, For the feature vector of the real sample, The noise input vector, For conditional feature set, For linear interpolation sample input, To determine the gradient of the discriminant function with respect to the interpolated samples, To generate the curvature values ​​of the sample, The curvature value is the actual value of the sample. To generate the rate of change of direction of the sample, The rate of change of direction of the real sample. The time index interval between adjacent nodes. To counteract the loss term coefficient, To generate the coefficients of the loss term, The coefficient of the gradient penalty term, The coefficients of the curvature residual term, For time consistency coefficients;

[0087] Execution process: First, extract the curvature of the real trajectory nodes from the curvature matching result set. , rate of change of direction Distance difference With time index Composition condition feature set And generate the corresponding real sample feature vector. Sample noise vectors from a Gaussian distribution ,Will With condition set Common input generation function Output a set of simulated trajectory sample features, which includes the generated curvature , rate of change of direction And the corresponding time index. Next, the generator's output samples and the real samples are input into the discriminant function. Calculate the authenticity score, the first item The second item evaluates the discriminator's ability to identify real samples. To measure the deceptive power of generated samples, interpolation is performed between real and generated samples. Calculate the gradient penalty term Introducing curvature residual term Consistency with time item Constrain the spatiotemporal continuity of the generated samples, and assign weights to each item. to The total loss is obtained by weighted summation. .

[0088] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An abnormal motion trajectory detection method based on 3D depth map reconstruction, characterized in that, Includes the following steps: S1: Acquire depth image frames, infrared intensity frames, and timestamp sequences through sensors, correspond pixel depth and infrared values ​​in the same frame point by point and align timestamps, jointly remove bad pixels and saturated pixels and perform adjacent pixel connectivity aggregation to generate a three-dimensional pixel connectivity block data group. S2: Based on the three-dimensional pixel connected block data group, call the camera intrinsic parameters focal length and principal point index and convert the pixels to camera coordinates, calculate the weighted sum of the connected block pixels to obtain the block center coordinates and backfill the timestamp, and construct the camera coordinate trajectory node sequence. S3: Based on the camera coordinate trajectory node sequence, call the camera extrinsic parameters rotation and translation, use a graph neural network to update the node adjacency weights and spatial associations, convert the camera coordinates into world coordinates, calculate the distance and direction difference between adjacent nodes, perform threshold segmentation and record the labels to obtain the world coordinate trajectory segment structure. S4: Based on the camera coordinate trajectory node sequence and the world coordinate trajectory segment structure, match the node time series and synchronously align the path, calculate the velocity difference and direction deviation of the matched nodes, jointly compare the depth layer changes and filter out nodes with direction reversal and velocity change, and output the trajectory turning point candidate set. S5: Based on the set of candidate points for trajectory turning points, a generative adversarial network is used to match the candidate points with the time index of the trajectory segment structure in world coordinates and estimate the nearest neighbor curvature. After judging the residual threshold, the segments that exceed the limit are marked, the spatial index of the segments is sorted out, and an abnormal motion trajectory framework is established.

2. The abnormal motion trajectory detection method based on three-dimensional depth map reconstruction according to claim 1, characterized in that, The three-dimensional pixel connected component data group includes pixel spatial coordinates, infrared intensity values, and timestamp indexes. The camera coordinate trajectory node sequence includes node spatial coordinates, pixel connected component center coordinates, and timestamp information. The world coordinate trajectory segment structure includes node world coordinates, distances between adjacent nodes, and direction difference labels. The trajectory turning point candidate set includes node spatial coordinates, velocity differences, and direction deviation angles. The abnormal motion trajectory framework includes an over-limit segment index, a segment spatial coordinate sequence, and curvature residual threshold results.

3. The abnormal motion trajectory detection method based on three-dimensional depth map reconstruction according to claim 1, characterized in that, The specific steps for generating the three-dimensional pixel connected component data group are as follows: The system acquires depth image frames, infrared intensity frames, and timestamp sequences from sensors. It compares the pixel depth and infrared intensity of the same frame, aligns the timestamps with the pixel index, detects the gray point depth difference to remove saturated pixels, rearranges the rows and columns of effective pixels, integrates and maps them, and generates a synchronized pixel mapping set. Based on the synchronized pixel mapping set, the grayscale difference and depth difference of adjacent pixels are cross-compared, connected pixels are determined by continuous difference intervals, connected block fusion and isolated points are deleted, and the edge region aggregation structure is reconstructed to generate a three-dimensional pixel connected block data group.

4. The abnormal motion trajectory detection method based on three-dimensional depth map reconstruction according to claim 1, characterized in that, The specific steps for constructing the camera coordinate trajectory node sequence are as follows: Based on the three-dimensional pixel connected block data group, the camera intrinsic parameters focal length and principal point index are called, and coordinate projection is performed by converting the pixel row and column coordinates with the focal length ratio. The values ​​of the offset between the pixel center and the principal point are adjusted, the three-dimensional coordinates of each pixel block are reconstructed and the hierarchy is mapped, and a pixel space conversion set is generated. Based on the pixel space conversion set, the weighted summation of connected component pixel coordinates is performed and the center point position is calculated. The timestamp is backfilled by matching the time index table, the node coordinate sequence is sorted and the interval between consecutive nodes is verified, the center coordinates and time information are integrated to form a node sequence, and the camera coordinate trajectory node sequence is constructed.

5. The abnormal motion trajectory detection method based on three-dimensional depth map reconstruction according to claim 1, characterized in that, The specific steps to obtain the structure of the world coordinate trajectory segment are as follows: Based on the camera coordinate trajectory node sequence, the external parameters rotation and translation are called. The rotation angle is calculated by converting the node coordinates in the three-dimensional axis direction and the translation vector is superimposed. After correcting the spatial offset of the node position, the coordinate matrix is ​​rearranged to generate the world coordinate node set. Based on the world coordinate node set, a graph neural network is used to update the adjacency weights and node association strength, measure the three-dimensional distance between adjacent nodes and calculate the direction vector difference, and extract local mutations and store node identifiers by comparing the intervals of the continuous node distance sequence with the direction change values ​​to generate a node distance-direction difference set. Based on the distance-direction difference set of the nodes, the threshold interval of the distance and direction difference is segmented and the node segment labels are divided. The order of the start and end indices of the segments is rearranged and the time series is mapped synchronously. The coordinate sequences of each segment are merged as a whole to obtain the world coordinate trajectory segment structure.

6. The abnormal motion trajectory detection method based on three-dimensional depth map reconstruction according to claim 1, characterized in that, The execution process of a graph neural network is as follows: based on the input set of world coordinate nodes, an adjacency table between nodes is established and a weight matrix is ​​initialized. The connection strength is calculated by the three-dimensional distance and the angle between directions between nodes, and the weights are mapped to the adjacency matrix. In each round of propagation, the features of adjacent nodes are aggregated according to the weight ratio using the node feature vector as input, and feature normalization and inter-layer propagation are performed to complete the diffusion of local spatial information. Then, the similarity between nodes is recalculated based on the aggregated feature distribution, the adjacency matrix and weight coefficients are updated, and the process is iterated repeatedly until the node features are stable, resulting in a structured representation of the spatial dependencies between trajectory nodes.

7. The abnormal motion trajectory detection method based on three-dimensional depth map reconstruction according to claim 1, characterized in that, The specific steps for outputting the candidate set of trajectory turning points are as follows: Based on the camera coordinate trajectory node sequence and the world coordinate trajectory segment structure, the node time sequence is matched, the node index is compared to complete the path synchronization alignment, the position offset and velocity vector component difference between corresponding nodes are calculated, the directional offset angle is measured and the velocity difference amplitude is statistically analyzed, and a synchronization path matching set is generated. Based on the synchronous path matching set, the depth layer changes of continuous nodes are compared, and nodes are screened by jointly identifying direction angle reversal points and velocity change points. The distance intervals between candidate nodes are calculated to aggregate adjacent parts, and after integrating the segment index, the numbers are assigned and the trajectory turning point candidate set is output.

8. The abnormal motion trajectory detection method based on three-dimensional depth map reconstruction according to claim 1, characterized in that, The specific steps for establishing the abnormal motion trajectory framework are as follows: Based on the set of candidate points for trajectory turning points, the candidate points are matched with the time index of the world coordinate trajectory segment structure. The time index is compared with the corresponding correction node position. The distance difference and direction vector change rate between adjacent nodes are calculated. The local curvature is measured and the direction of continuous nodes is smoothly corrected to generate a curvature matching result set. Based on the curvature matching result set, a generative adversarial network is used to establish a distribution comparison between the real trajectory and the simulated trajectory. After calculating the curvature residual value of each node, a residual threshold interval is established. The residual is compared with the threshold and the nodes that exceed the limit are screened out. The distance between consecutive nodes that exceed the limit is determined and the segment index is recorded to generate a set of indexes for the segments that exceed the limit. Based on the aforementioned over-limit paragraph index set, the spatial coordinates of nodes within the paragraph are aggregated and rearranged, the boundary distances between adjacent paragraphs are determined and the sections are merged, the spatial index table is organized and global trajectory coordinates are mapped, and an abnormal motion trajectory framework is established.

9. The abnormal motion trajectory detection method based on three-dimensional depth map reconstruction according to claim 1, characterized in that, The execution process of a generative adversarial network is as follows: Based on the curvature matching result set of the input, the distribution inputs of real trajectory samples and simulated trajectory samples are constructed. The generator generates simulated trajectory data with the same dimensions as the real samples, using the spatial position, direction vector, and velocity sequence of the trajectory nodes as generation parameters. The differences between the generated samples and the real samples in the distribution space are compared. The discriminator receives the curvature, direction change rate, and residual values ​​of the two sets of samples, calculates the similarity and offset rate in the feature space, and feeds the results back to the generator to update the trajectory generation parameters. The generator adjusts the curvature and spatial coherence of the output samples according to the feedback until the discriminator can no longer distinguish the distribution differences between the two types of samples, thus obtaining a balanced distribution dataset for residual comparison analysis.

10. An abnormal motion trajectory detection device based on 3D depth map reconstruction, comprising a memory and a processor, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the abnormal motion trajectory detection method based on three-dimensional depth map reconstruction as described in any one of claims 1 to 9.

Images