A multi-modal data abnormal behavior three-dimensional visualization method and system

Abstract

The application discloses a kind of multimodal data abnormal behavior three-dimensional visualization method and system, belong to data visualization technical field, method includes: obtaining target multimodal data;Target multimodal data is time-synchronized, noise filtering and spatial coordinate alignment, obtain the target multimodal standard data containing video stream frame standard data and one-dimensional time sequence acceleration standard signal;One-dimensional time sequence acceleration standard signal is converted into two-dimensional velocity spectrum, and video stream frame standard data and two-dimensional velocity spectrum are fused to obtain unified fusion feature vector, and the reconstruction error of unified fusion feature vector is fused and reconstructed;The spatial offset of target node corresponding to target multimodal data relative to cluster center point is calculated, and the abnormal behavior of target multimodal data is judged and output based on reconstruction error and spatial offset;Abnormal behavior three-dimensional visualization.The application can improve the abnormal recognition precision of multimodal data, data correlation and visual expression ability.

Description

Technical Field

[0001] This invention relates to a three-dimensional visualization method and system for multimodal data anomaly behavior, belonging to the field of data visualization technology. Background Technology

[0002] With the development of artificial intelligence and intelligent sensing technologies, multimodal data, such as video images, LiDAR point clouds, inertial sensor data, and environmental monitoring data, are increasingly widely used in the detection and analysis of abnormal behavior. Existing technologies for abnormal behavior recognition and visualization mainly include methods based on single-modal data analysis and methods based on multimodal data fusion analysis.

[0003] In single-modal data processing, existing methods are mostly based on video images or sensor data for abnormal behavior detection. These methods generally include steps such as image acquisition, feature extraction, behavior modeling, and anomaly detection, and have the advantages of simple implementation and relatively low computational cost. However, because they rely on only a single data source, these methods are easily affected by factors such as changes in lighting, occlusion, and viewing angle limitations in complex environments, leading to decreased detection accuracy and poor robustness.

[0004] To improve the accuracy of abnormal behavior identification, existing technologies further incorporate multimodal data fusion methods. For example, by fusing visual data with LiDAR point cloud data, spatial localization and behavior analysis of targets can be achieved; or by combining inertial navigation data with environmental sensor data, a comprehensive judgment of dynamic target behavior can be made. These methods typically include steps such as multi-source data acquisition, temporal and spatial alignment, multimodal feature fusion, and abnormal behavior discrimination, which improves identification accuracy and environmental adaptability to a certain extent.

[0005] Furthermore, in terms of visualization, existing technologies mostly employ two-dimensional planar displays or simple three-dimensional modeling to showcase anomalous behavior. For example, they present the spatial distribution and evolution of anomalous behavior through trajectory overlay, heatmap analysis, or three-dimensional scene reconstruction. While these methods can intuitively display some information, they still have shortcomings in the expression of multimodal data fusion.

[0006] However, the aforementioned existing technologies still have the following drawbacks:

[0007] (1) The degree of multimodal data fusion is limited. Usually, only simple data overlay or low-level feature fusion is performed. There is a lack of a unified data expression model, making it difficult to fully explore the correlation information between different modalities.

[0008] (2) There is a lack of tight coupling between the abnormal behavior identification results and the visualization display. Most methods only display the results after detection, and cannot achieve synchronous linkage between the analysis process and the visualization.

[0009] (3) Existing 3D visualization methods focus on static scene reconstruction, which makes it difficult to dynamically reflect the spatiotemporal evolution of abnormal behavior, and lacks interactivity and expressive ability.

[0010] (4) In complex application scenarios, existing methods have limited real-time processing capabilities for multi-source heterogeneous data, making it difficult to meet the application requirements of high precision and real-time performance. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a three-dimensional visualization method and system for multimodal data anomaly behavior, which solves the problems of low anomaly recognition accuracy, weak data correlation and insufficient visualization expression ability in the prior art.

[0012] To achieve the above objectives, the present invention employs the following technical solution:

[0013] In a first aspect, this invention proposes a three-dimensional visualization method for anomaly behavior in multimodal data, comprising:

[0014] Acquire target multimodal data; the multimodal data includes video stream frame data and one-dimensional time-series acceleration signals;

[0015] The target multimodal data is processed by time synchronization, noise filtering and spatial coordinate alignment to obtain target multimodal standard data containing video stream frame standard data and one-dimensional temporal acceleration standard signal;

[0016] The one-dimensional time-series acceleration standard signal is converted into a two-dimensional velocity spectrum. The video stream frame standard data and the two-dimensional velocity spectrum are fused to obtain a unified fusion feature vector. The unified fusion feature vector is then reconstructed to obtain the reconstruction error.

[0017] Calculate the cluster center point of the target multimodal data and the spatial offset of the target node corresponding to the target multimodal data relative to the cluster center point. Based on the reconstruction error and the spatial offset, determine and output the abnormal behavior of the target multimodal data. The cluster center point is the center node of the space where the target multimodal data is located, and the target node is the corresponding node in the space where the target multimodal data is located.

[0018] The abnormal behavior is mapped to a three-dimensional spatial coordinate system to generate the coordinates and motion trajectory of the abnormal behavior, and the abnormal behavior is visualized in three dimensions based on the coordinates and motion trajectory.

[0019] Furthermore, the multimodal data also includes 3D point cloud data and environmental perception data;

[0020] The acquisition of target multimodal data includes:

[0021] Real-time acquisition of video stream frame data using a visual sensor;

[0022] Acquire 3D point cloud data using LiDAR;

[0023] One-dimensional time-series acceleration signals are acquired using an inertial measurement unit;

[0024] Environmental sensing data is collected through environmental sensing devices.

[0025] Furthermore, the time synchronization includes: using the acquisition frequency of the video stream frame data as a global reference clock, performing timestamp matching on the one-dimensional time-series acceleration signal, and using linear interpolation to fill in the missing data moments, thereby obtaining video stream frame data and one-dimensional time-series acceleration signal that are fully aligned under the same time reference.

[0026] The noise filtering includes: removing image noise by using a combination of Gaussian filtering and median filtering on fully aligned video stream frame data under the same time reference to obtain denoised video stream frame data; and suppressing high-frequency jitter and random errors by using low-pass filtering or Kalman filtering on fully aligned one-dimensional time-series acceleration signals under the same time reference to obtain filtered one-dimensional time-series acceleration signals.

[0027] The spatial coordinate alignment includes: establishing a unified world coordinate system based on the sensor extrinsic calibration results; performing coordinate transformation on the video coordinate system corresponding to the denoised video stream frame data and the inertial measurement coordinate system corresponding to the filtered one-dimensional temporal acceleration signal through rotation matrix R and translation matrix T; and uniformly mapping the denoised video stream frame data and the filtered one-dimensional temporal acceleration signal to the same spatial reference coordinate system to obtain target multimodal standard data containing standard video stream frame data and one-dimensional temporal acceleration standard signal.

[0028] Further, the conversion of the one-dimensional time-series acceleration standard signal into a two-dimensional velocity spectrum includes:

[0029] Using Gram angle and field coding algorithms, the one-dimensional time-series acceleration standard signal is normalized to [-1, 1] and mapped to a polar coordinate system for angle coding. The angle mapping in the polar coordinate system is calculated using the following formula:

[0030] ;

[0031] in, One-dimensional time-series acceleration standard signal Angle mapping in a coordinate system;

[0032] A two-dimensional symmetric matrix P is constructed using trigonometric function angle sums. The correlation characteristics of the time-series signal in the global domain are characterized by the angular combination relationships between different time sampling points, resulting in a two-dimensional velocity spectrum that can completely preserve the temporal correlation. ;

[0033] Two-dimensional velocity spectrum median coordinate pixel value at The formula is:

[0034] ;

[0035] in, For the first Polar coordinate angle mapping corresponding to each sampling point For the first Polar coordinate angle mapping corresponding to each sampling point This represents the angle combination relationship between the i-th sampling point and the j-th sampling point. For the first Normalized amplitude of each sampling point For the first Normalized amplitude of each sampling point.

[0036] Further, the step of fusing the standard data of the video stream frames and the two-dimensional velocity spectrum to obtain a unified fused feature vector, and then performing fused feature reconstruction on the unified fused feature vector to obtain a reconstruction error, includes:

[0037] Video image features are extracted by performing convolutional layers, pooling layers, and nonlinear activation layers on the standard data of the video stream frames. The two-dimensional velocity spectrum is input into a convolutional neural network to extract acceleration spectrum features. ;

[0038] The video image features are then processed using a pre-trained fusion model based on a cross-modal guided attention mechanism. Mapped to query matrix The acceleration spectrum features Mapped to a key matrix Sum matrix The formula is:

[0039] ;

[0040] ;

[0041] ;

[0042] in, For query matrix The corresponding feature mapping weight matrix, Key matrix The corresponding feature mapping weight matrix, Value matrix The corresponding feature mapping weight matrix;

[0043] Based on the query matrix Key matrix Sum matrix Calculate the guiding weight of the video image features on the acceleration spectrum features. The guiding weight formula is as follows:

[0044] ;

[0045] in, The guiding weights for the video image features on the acceleration spectrum features are... The scaling factor for the dimension of the feature vector. Representing the query matrix AND key matrix The correlation calculation results between them Represents the matrix transpose symbol. The normalized activation function;

[0046] Using the guided weights on the value matrix Weighted calculations are performed to obtain cross-modal fusion features, using the following formula:

[0047] ;

[0048] in, For cross-modal fusion features;

[0049] The fused feature vector The graph nodes are input into the graph convolutional network for topological aggregation, resulting in a unified fusion feature vector with deep semantic relationships. The formula is as follows:

[0050] ;

[0051] in, To unify and merge feature vectors, For graph structure adjacency matrix, Adjacency matrix The corresponding degree matrix, For the trainable weight parameter matrix of a graph convolutional network, It is a non-linear activation function. The input graph node feature matrix;

[0052] The unified fused feature vector is reconstructed using an autoencoder, and the reconstruction error of the fused features is calculated using the following formula:

[0053] ;

[0054] in, The reconstruction error of the fused features, For the encoder to unify and fuse feature vectors Latent feature representation after low-dimensional feature encoding The reconstructed fusion feature is obtained by the decoder after reconstructing the latent feature representation. This represents the square norm of the Euclidean distance.

[0055] Furthermore, the training process of the fusion model based on the cross-modal guided attention mechanism includes:

[0056] Construct a training dataset containing the video image features, the acceleration spectrum features, and abnormal behavior labels, and normalize the data in the training dataset;

[0057] Based on the training dataset, a convolutional neural network is used to extract the modal features of the video image, the acceleration spectrum features, and the abnormal behavior labels.

[0058] The parameters of the model are iteratively updated using the backpropagation algorithm, and the loss is calculated using the loss function. Training stops when the loss function converges or the number of iterations reaches a preset number.

[0059] Further, the step of calculating the cluster center point of the target multimodal data and the spatial offset of the target node corresponding to the target multimodal data relative to the cluster center point, and judging and outputting the abnormal behavior of the target multimodal data based on the reconstruction error and the spatial offset, includes:

[0060] Visual sensors, lidar, inertial measurement units, and environmental sensing devices are used as sensing nodes. A graph structure connection relationship is established based on the spatial positional relationship and data association relationship between the sensing nodes to obtain an IoT spatial topology behavior analysis graph in the target scene.

[0061] Calculate the cluster center point of the sensing nodes, and determine the first point distance between the target node corresponding to the target multimodal data and the cluster center point. and relative direction The formula for calculating the cluster center point is:

[0062] ;

[0063] in, The location of the cluster center point. For the first The location coordinates of each sensing node. The total number of sensing nodes;

[0064] When the reconstruction error of the fused feature exceeds a preset abnormal error threshold, and the spatial offset of the target node relative to the cluster center point in a continuous time series of a certain length exhibits a nonlinear abrupt change exceeding a preset range, the target multimodal data is judged to be abnormal, and the abnormal behavior category, spatial location parameters, motion trajectory parameters, and state feature parameters of the target multimodal data are output; the nonlinear abrupt change includes an abnormal increase in the rate of change of spatial displacement, abrupt change in motion direction, abnormal velocity fluctuation, or abnormal change in node topology.

[0065] Furthermore, the step of mapping the abnormal behavior to a three-dimensional coordinate system to generate the coordinates and trajectory of the abnormal behavior, and then performing a three-dimensional visualization of the abnormal behavior based on the coordinates and trajectory, includes:

[0066] The abnormal behavior coordinates of the target multimodal data are mapped to a pre-constructed BIM 3D twin model;

[0067] The spatial folding and compression of the visual edge coordinates of the abnormal behavior coordinates are performed using a nonlinear mapping function to obtain the distance-folded abnormal behavior coordinates.

[0068] The abnormal behavior trajectory is drawn based on the abnormal behavior coordinates after distance folding, and the abnormal behavior trajectory is displayed in three-dimensional visualization.

[0069] Furthermore, the construction process of the BIM 3D twin model includes:

[0070] The architectural structure model of the target scene is built using Revit, and the architectural structure model is then lightweighted and rendered using the Unity3D 3D engine.

[0071] The abnormal behavior is mapped to the three-dimensional scene of the building structure model through spatial coordinate mapping relationship, and a well constructed BIM three-dimensional twin model is obtained.

[0072] Secondly, this invention discloses a three-dimensional visualization system for multimodal data anomaly behavior, comprising:

[0073] The data acquisition module is used to acquire target multimodal data; the multimodal data includes video stream frame data and one-dimensional time-series acceleration signals.

[0074] The data preprocessing module is used to perform time synchronization, noise filtering and spatial coordinate alignment processing on the target multimodal data to obtain target multimodal standard data containing video stream frame standard data and one-dimensional temporal acceleration standard signal.

[0075] The multimodal fusion and reconstruction module is used to convert the one-dimensional time-series acceleration standard signal into a two-dimensional velocity spectrum, fuse the video stream frame standard data and the two-dimensional velocity spectrum to obtain a unified fusion feature vector, and reconstruct the unified fusion feature vector to obtain the reconstruction error.

[0076] An abnormal behavior detection module is used to calculate the cluster center point of the target multimodal data and the spatial offset of the target node corresponding to the target multimodal data relative to the cluster center point, and to judge and output the abnormal behavior of the target multimodal data based on the reconstruction error and the spatial offset; the cluster center point is the center node in the space where the target multimodal data is located, and the target node is the corresponding node in the space where the target multimodal data is located;

[0077] The three-dimensional visualization module is used to map the abnormal behavior to a three-dimensional spatial coordinate system, generate the coordinates and motion trajectory of the abnormal behavior, and perform three-dimensional visualization of the abnormal behavior based on the coordinates and motion trajectory.

[0078] The beneficial effects achieved by this invention are as follows:

[0079] (1) By integrating visual data, sensor data and behavioral trajectory data, a multimodal feature fusion mechanism is introduced to achieve comprehensive discrimination of abnormal behavior, significantly improve the accuracy and stability of abnormal identification, reduce misjudgment and missed judgment, and enable the system to still have high recognition reliability in complex environments.

[0080] (2) By constructing a three-dimensional scene model, the transformation from two-dimensional monitoring to three-dimensional dynamic visualization is realized, which intuitively reflects the spatial distribution characteristics and evolution process of abnormal behavior, improves the integrity and comprehensibility of information expression, and is conducive to operators making quick judgments and decisions;

[0081] (3) By adopting a hierarchical processing and parallel computing mechanism, multimodal data is preprocessed, fused and analyzed, which effectively reduces computational redundancy in the data processing process, improves the overall operating efficiency of the system, and can meet the application scenarios with high real-time requirements.

[0082] (4) The multimodal fusion framework and three-dimensional visualization method proposed in this invention have good modular design characteristics. They can flexibly access new data sources or replace anomaly detection algorithms according to different application scenarios. They have strong versatility and scalability and can be applied to various fields such as smart construction sites, intelligent security, and industrial monitoring. Attached Figure Description

[0083] Figure 1 A flowchart of the three-dimensional visualization method for multimodal data anomaly behavior provided by the present invention;

[0084] Figure 2 The structural diagram of the three-dimensional visualization system for multimodal data anomaly behavior provided by the present invention is shown. Detailed Implementation

[0085] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0086] Example 1, such as Figure 1 and Figure 2 As shown, this invention discloses a three-dimensional visualization method for multimodal data anomaly behavior, comprising:

[0087] Acquire target multimodal data; the multimodal data includes video stream frame data and one-dimensional time-series acceleration signals;

[0088] The target multimodal data is processed by time synchronization, noise filtering and spatial coordinate alignment to obtain target multimodal data containing standard video stream frame data and one-dimensional temporal acceleration standard signal;

[0089] The one-dimensional time-series acceleration standard signal is converted into a two-dimensional velocity spectrum. The video stream frame standard data and the two-dimensional velocity spectrum are fused to obtain a unified fusion feature vector. The unified fusion feature vector is then reconstructed to obtain the reconstruction error.

[0090] Calculate the cluster center point of the target multimodal data and the spatial offset of the target node corresponding to the target multimodal data relative to the cluster center point. Based on the reconstruction error and the spatial offset, determine and output the abnormal behavior of the target multimodal data. The cluster center point is the center node of the target multimodal data in space, and the target node is the node corresponding to the target multimodal data in space.

[0091] The abnormal behavior is mapped to a three-dimensional spatial coordinate system to generate the coordinates and motion trajectory of the abnormal behavior, and the abnormal behavior is visualized in three dimensions based on the coordinates and motion trajectory.

[0092] The multimodal data also includes 3D point cloud data and environmental perception data;

[0093] The acquisition of target multimodal data includes:

[0094] Real-time acquisition of video stream frame data using a visual sensor;

[0095] Acquire 3D point cloud data using LiDAR;

[0096] One-dimensional time-series acceleration signals are acquired using an inertial measurement unit (IMU). An inertial measurement unit generally refers to a space attitude sensor, which is a device that measures the three-axis attitude angles or angular rates and acceleration of an object.

[0097] Environmental perception data is collected through sensor devices; the environmental perception data includes environmental state information such as ambient temperature, ambient humidity, light intensity, smoke concentration, gas concentration and noise intensity, which is used to reflect the environmental changes in the target scene.

[0098] Among them, video stream frame data is used to acquire target appearance features and behavioral dynamic information, 3D point cloud data is used to describe the spatial structural features of the target scene, one-dimensional temporal acceleration signals are used to reflect the changes in the target's motion state, and environmental sensor data is used to assist in analyzing changes in the environmental state under the target scene. Through the collaborative acquisition of multi-source heterogeneous data, the accuracy of abnormal behavior identification, environmental adaptability, and robustness in complex scenes can be effectively improved, providing a data foundation for subsequent abnormal behavior analysis and 3D visualization.

[0099] The time synchronization includes: using the acquisition frequency of the video stream frame data as a global reference clock, performing timestamp matching on the one-dimensional time-series acceleration signal, and using linear interpolation to fill in the missing data moments to obtain fully aligned video stream frame data and one-dimensional time-series acceleration signal under the same time reference.

[0100] The noise filtering includes: removing image noise by using a combination of Gaussian filtering and median filtering on fully aligned video stream frame data under the same time reference to obtain denoised video stream frame data; and suppressing high-frequency jitter and random errors by using low-pass filtering or Kalman filtering on fully aligned one-dimensional time-series acceleration signals under the same time reference to obtain filtered one-dimensional time-series acceleration signals.

[0101] The spatial coordinate alignment includes: establishing a unified world coordinate system based on the sensor extrinsic calibration results; performing coordinate transformation on the video coordinate system corresponding to the denoised video stream frame data and the inertial measurement coordinate system corresponding to the filtered one-dimensional temporal acceleration signal through rotation matrix R and translation matrix T; and uniformly mapping the denoised video stream frame data and the filtered one-dimensional temporal acceleration signal to the same spatial reference coordinate system to obtain target multimodal standard data containing standard video stream frame data and one-dimensional temporal acceleration standard signal.

[0102] The process of converting the one-dimensional time-series acceleration standard signal into a two-dimensional velocity spectrum includes:

[0103] Using Gram angle and field coding algorithms, the one-dimensional time-series acceleration standard signal is normalized to [-1, 1] and mapped to a polar coordinate system for angle coding. The angle mapping in the polar coordinate system is calculated using the following formula:

[0104] ;

[0105] in, One-dimensional time-series acceleration standard signal Angle mapping in a coordinate system;

[0106] A two-dimensional symmetric matrix P is constructed using trigonometric function angle sums. The correlation characteristics of the time-series signal in the global domain are characterized by the angular combination relationships between different time sampling points, resulting in a two-dimensional velocity spectrum that can completely preserve the temporal correlation. ;

[0107] Two-dimensional velocity spectrum median coordinate pixel value at The formula is:

[0108] ;

[0109] in, For the first Polar coordinate angle mapping corresponding to each sampling point For the first Polar coordinate angle mapping corresponding to each sampling point This represents the angle combination relationship between the i-th sampling point and the j-th sampling point. For the first Normalized amplitude of each sampling point For the first Normalized amplitude of each sampling point.

[0110] This step introduces Gramian Angular Field (GAF) technology to convert one-dimensional time series data into two-dimensional images. This can solve the problems of low efficiency and severe information loss in the extraction of one-dimensional motion features in traditional deep learning networks, and realize the image enhancement of non-visual modal features.

[0111] The process of fusing the standard data of the video stream frames and the two-dimensional velocity spectrum to obtain a unified fused feature vector, and then reconstructing the unified fused feature vector to obtain a reconstruction error, includes:

[0112] Video image features are extracted by performing convolutional layers, pooling layers, and nonlinear activation layers on the standard data of the video stream frames. The two-dimensional velocity spectrum is input into a convolutional neural network to extract acceleration spectrum features. ;

[0113] The video image features are then processed using a pre-trained fusion model based on a cross-modal guided attention mechanism. Mapped to query matrix The acceleration spectrum features Mapped to a key matrix Sum matrix The formula is:

[0114] ;

[0115] ;

[0116] ;

[0117] in, For query matrix The corresponding feature mapping weight matrix, Key matrix The corresponding feature mapping weight matrix, Value matrix The corresponding feature mapping weight matrix;

[0118] Based on the query matrix Key matrix Sum matrix The guiding weights of the video image features on the acceleration spectrum features are calculated using the following formula:

[0119] ;

[0120] in, The guiding weights for the video image features on the acceleration spectrum features are... The scaling factor for the dimension of the feature vector. Representing the query matrix AND key matrix The correlation calculation results between them; This is a normalized activation function used to normalize the correlation results between the query matrix Q and the key matrix K, so as to obtain the attention weight distribution among different modal features;

[0121] Using the guided weights on the value matrix Weighted calculations are performed to obtain cross-modal fusion features, using the following formula:

[0122] ;

[0123] in, For cross-modal fusion features;

[0124] The fused feature vector The graph nodes are input into a graph convolutional network (GCN) for topological aggregation, resulting in a unified fused feature vector with deep semantic relationships. The formula is as follows:

[0125] ;

[0126] in, To unify and merge feature vectors, For graph structure adjacency matrix, Adjacency matrix The corresponding degree matrix, For the trainable weight parameter matrix of a graph convolutional network, It is a non-linear activation function. The input graph node feature matrix;

[0127] The unified fused feature vector is reconstructed using an autoencoder, and the reconstruction error of the fused features is calculated using the following formula:

[0128] ;

[0129] in, The reconstruction error of the fused features, For the encoder to unify and fuse feature vectors Latent feature representation after low-dimensional feature encoding The reconstructed fusion feature is obtained by the decoder after reconstructing the latent feature representation. This represents the square norm of the Euclidean distance.

[0130] The reconstruction error characterizes the degree of difference between the current behavioral features and the normal behavioral pattern. When the target behavior is normal, the autoencoder can accurately reconstruct the input features, resulting in a small reconstruction error. When the target behavior exhibits abnormal changes, the autoencoder struggles to accurately reconstruct the abnormal behavior features due to their deviation from the normal behavior feature distribution, leading to a larger reconstruction error. When the reconstruction error exceeds a preset abnormality threshold, it indicates that the current target behavior deviates from the normal behavioral pattern, thus determining a risk of abnormal behavior. By introducing the reconstruction error as a criterion for abnormal behavior determination, the sensitivity and robustness of abnormal behavior recognition in complex environments can be effectively improved, while reducing the false detection rate and false negative rate caused by a single threshold detection method.

[0131] The fusion model based on the cross-modal guided attention mechanism is trained through supervised learning. The training process includes:

[0132] Construct a training dataset containing the video image features, the acceleration spectrum features, and abnormal behavior labels, and normalize the data in the training dataset;

[0133] Based on the training dataset, a convolutional neural network is used to extract the modal features of the video image, the acceleration spectrum features, and the abnormal behavior labels.

[0134] The parameters of the model are iteratively updated using the backpropagation algorithm, and the loss is calculated using the loss function. Training stops when the loss function converges or the number of iterations reaches a preset number.

[0135] The calculation of the cluster center point of the target multimodal data and the spatial offset of the target node corresponding to the target multimodal data relative to the cluster center point, and the determination and output of abnormal behavior of the target multimodal data based on the reconstruction error and the spatial offset, includes:

[0136] Visual sensors, lidar, inertial measurement units, and environmental sensing devices are used as sensing nodes. A graph structure connection relationship is established based on the spatial positional relationship and data association relationship between the sensing nodes to obtain an IoT spatial topology behavior analysis graph in the target scene.

[0137] Calculate the cluster center point of the sensing nodes, and determine the first point distance between the target node corresponding to the target multimodal data and the cluster center point. and relative direction The formula for calculating the cluster center point is:

[0138] ;

[0139] in, The location of the cluster center point. For the first The location coordinates of each sensing node. The total number of sensing nodes;

[0140] When the reconstruction error of the fused feature exceeds a preset abnormal error threshold, and the spatial offset of the target node relative to the cluster center point in a continuous time series of a certain length exhibits a nonlinear abrupt change exceeding a preset range, the target multimodal data is judged to be abnormal, and the abnormal behavior category, spatial location parameters, motion trajectory parameters, and state feature parameters of the target multimodal data are output; the nonlinear abrupt change includes an abnormal increase in the rate of change of spatial displacement, abrupt change in motion direction, abnormal velocity fluctuation, or abnormal change in node topology.

[0141] If no abnormal behavior is detected, data will continue to be collected for the next moment, and abnormal behavior identification and 3D visualization will be performed on the multimodal data of the next moment.

[0142] The step of mapping the abnormal behavior to a three-dimensional coordinate system to generate the coordinates and trajectory of the abnormal behavior, and then performing a three-dimensional visualization of the abnormal behavior based on the coordinates and trajectory, includes:

[0143] The abnormal behavior coordinates of the target multimodal data are mapped to a pre-built BIM (Building Information Modeling) 3D twin model;

[0144] The spatial folding and compression of the field-of-view edge coordinates of the abnormal behavior coordinates are performed using a nonlinear mapping function to obtain the distance-folded abnormal behavior coordinates; this step can solve the problem of unclear display of distant targets in large-scale monitoring scenarios.

[0145] The abnormal behavior trajectory is drawn based on the abnormal behavior coordinates after distance folding, and the abnormal behavior trajectory is displayed in three-dimensional visualization in the form of dynamic highlighting and color block overlay.

[0146] In summary, this invention achieves collaborative analysis of multimodal heterogeneous data by fusing video stream frame data, 3D point cloud data, one-dimensional temporal acceleration signals, and environmental perception data. Compared to traditional single-modal abnormal behavior detection methods, it effectively reduces false detections caused by occlusion, lighting changes, and dynamic interference in complex environments. Furthermore, by introducing a cross-modal guided attention mechanism, self-encoding reconstruction error analysis, and spatial topology correlation analysis, the accuracy and stability of abnormal behavior identification are enhanced. In addition, combining a BIM 3D twin model with dynamic visualization of abnormal behavior trajectories improves the spatial representation and scene perception capabilities of abnormal behavior, thereby enhancing the system's engineering application value in smart construction sites, smart security, and industrial monitoring scenarios.

[0147] Example 2, based on the same inventive concept as Example 1, provides a three-dimensional visualization method for multimodal data anomaly behavior. The following are the application effects of this method in multiple scenarios:

[0148] (1) Testing in a smart construction site scenario.

[0149] Video was captured using a camera with a resolution of 1920×1080, the LiDAR scanning frequency was 10Hz, and the inertial measurement unit sampling frequency was 100Hz.

[0150] In this process, the behavior data of construction workers is first collected and then time-synchronized and filtered. Then, a feature-level fusion method is used to fuse the multimodal data. Finally, the abnormal behavior results are analyzed and output through an anomaly detection model, and the abnormal behavior trajectory is visualized in a three-dimensional scene.

[0151] The results show that, compared with traditional single-modal visual detection methods, the multimodal fusion method of the present invention has a more stable ability to identify abnormal behavior under complex occlusion, low light and dynamic interference conditions, and can realize continuous three-dimensional visualization of abnormal behavior trajectories.

[0152] (2) Test in a highly dynamic and complex environment.

[0153] We employ cameras with higher frame rates and sensors with higher sampling frequencies, and introduce a dynamic weight allocation method based on an attention mechanism during the fusion process to enhance the adaptability of data from different modalities.

[0154] The attention-based dynamic weight allocation method includes:

[0155] Calculate the correlation weights between different modal features;

[0156] Based on changes in the current scene environment, its own modal confidence, and feature response intensity, the weight ratio of each modal feature in the fusion process is dynamically adjusted.

[0157] When a certain modality of data is occluded or interfered with by noise, its corresponding fusion weight is reduced, while the contribution of other effective modal features is enhanced, thereby improving the overall fusion stability and robustness of abnormal behavior recognition.

[0158] In this environment, even in complex situations such as occlusion or rapid movement, the system can still maintain good abnormal behavior recognition and achieve continuous and stable 3D visualization.

[0159] (3) Test in a low-cost deployment environment.

[0160] Data acquisition was performed using only cameras and inertial measurement units, without the use of LiDAR. Data processing was performed using a lightweight fusion model, and the 3D modeling process was simplified.

[0161] The lightweight fusion model reduces computational complexity by decreasing the number of convolutional network layers, reducing feature dimensions, and using depthwise separable convolutions to reduce the number of model parameters. Meanwhile, during the 3D modeling process, sparse point cloud reconstruction and low-precision mesh simplification methods are used to lightweight the 3D scene, thereby reducing the overall system resource consumption.

[0162] Despite the limitations of hardware, the deployment environment still enables basic abnormal behavior detection and 3D visualization, demonstrating the good adaptability and scalability of the method of this invention.

[0163] Example 3, based on the same inventive concept as other examples, discloses a three-dimensional visualization system for multimodal data anomaly behavior, comprising:

[0164] The data acquisition module is used to acquire target multimodal data; the multimodal data includes video stream frame data and one-dimensional time-series acceleration signals.

[0165] The data preprocessing module is used to perform time synchronization, noise filtering and spatial coordinate alignment processing on the target multimodal data to obtain target multimodal standard data containing video stream frame standard data and one-dimensional temporal acceleration standard signal.

[0166] The multimodal fusion and reconstruction module is used to convert the one-dimensional time-series acceleration standard signal into a two-dimensional velocity spectrum, fuse the video stream frame standard data and the two-dimensional velocity spectrum to obtain a unified fusion feature vector, and reconstruct the unified fusion feature vector to obtain the reconstruction error.

[0167] An abnormal behavior detection module is used to calculate the cluster center point of the target multimodal data and the spatial offset of the target node corresponding to the target multimodal data relative to the cluster center point, and to judge and output the abnormal behavior of the target multimodal data based on the reconstruction error and the spatial offset; the cluster center point is the center node of the target multimodal data in space, and the target node is the node corresponding to the target multimodal data in space;

[0168] The three-dimensional visualization module is used to map the abnormal behavior to a three-dimensional spatial coordinate system, generate the coordinates and motion trajectory of the abnormal behavior, and perform three-dimensional visualization of the abnormal behavior based on the coordinates and motion trajectory.

[0169] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0171] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0173] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A three-dimensional visualization method for anomaly behavior in multimodal data, characterized in that, include: Acquire target multimodal data; The target multimodal data includes video stream frame data and one-dimensional temporal acceleration signals; The target multimodal data is processed by time synchronization, noise filtering and spatial coordinate alignment to obtain target multimodal standard data containing video stream frame standard data and one-dimensional temporal acceleration standard signal; The one-dimensional time-series acceleration standard signal is converted into a two-dimensional velocity spectrum. The video stream frame standard data and the two-dimensional velocity spectrum are fused to obtain a unified fusion feature vector. The unified fusion feature vector is then reconstructed to obtain the reconstruction error. Calculate the cluster center point of the target multimodal data and the spatial offset of the target node corresponding to the target multimodal data relative to the cluster center point. Based on the reconstruction error and the spatial offset, determine and output the abnormal behavior of the target multimodal data. The cluster center point is the central node of the space where the target multimodal data is located, and the target node is the corresponding node in the space where the target multimodal data is located; The abnormal behavior is mapped to a three-dimensional spatial coordinate system to generate the coordinates and motion trajectory of the abnormal behavior, and the abnormal behavior is visualized in three dimensions based on the coordinates and motion trajectory.

2. The three-dimensional visualization method for multimodal data anomaly behavior according to claim 1, characterized in that, The target multimodal data also includes 3D point cloud data and environmental perception data; The acquisition of target multimodal data includes: Real-time acquisition of video stream frame data using a visual sensor; Acquire 3D point cloud data using LiDAR; One-dimensional time-series acceleration signals are acquired using an inertial measurement unit; Environmental sensing data is collected through environmental sensing devices.

3. The three-dimensional visualization method for multimodal data anomaly behavior according to claim 2, characterized in that, The time synchronization includes: using the acquisition frequency of the video stream frame data as a global reference clock, performing timestamp matching on the one-dimensional time-series acceleration signal, and using linear interpolation to fill in the missing data moments to obtain fully aligned video stream frame data and one-dimensional time-series acceleration signal under the same time reference. The noise filtering includes: removing image noise by using a combination of Gaussian filtering and median filtering on fully aligned video stream frame data under the same time reference to obtain denoised video stream frame data; and suppressing high-frequency jitter and random errors by using low-pass filtering or Kalman filtering on fully aligned one-dimensional time-series acceleration signals under the same time reference to obtain filtered one-dimensional time-series acceleration signals. The spatial coordinate alignment includes: establishing a unified world coordinate system based on the sensor extrinsic calibration results; performing coordinate transformation on the video coordinate system corresponding to the denoised video stream frame data and the inertial measurement coordinate system corresponding to the filtered one-dimensional temporal acceleration signal through rotation matrix R and translation matrix T; and uniformly mapping the denoised video stream frame data and the filtered one-dimensional temporal acceleration signal to the same spatial reference coordinate system to obtain target multimodal standard data containing standard video stream frame data and one-dimensional temporal acceleration standard signal.

4. The three-dimensional visualization method for multimodal data anomaly behavior according to claim 3, characterized in that, The process of converting the one-dimensional time-series acceleration standard signal into a two-dimensional velocity spectrum includes: Using Gram angle and field coding algorithms, the one-dimensional time-series acceleration standard signal is normalized to [-1, 1] and mapped to a polar coordinate system for angle coding. The angle mapping in the polar coordinate system is calculated using the following formula: ； In the formula, A one-dimensional time-series acceleration standard signal, Angle mapping in polar coordinates; A two-dimensional symmetric matrix P is constructed using trigonometric function angle sums. The correlation characteristics of the time-series signal in the global domain are characterized by the angular combination relationships between different time sampling points, resulting in a two-dimensional velocity spectrum that fully preserves the temporal correlation. ; Two-dimensional velocity spectrum median coordinate pixel value at The formula is: ； In the formula, This represents the polar coordinate angle mapping corresponding to the i-th sampling point. This represents the polar coordinate angle mapping corresponding to the j-th sampling point. This represents the angle combination relationship between the i-th sampling point and the j-th sampling point. The normalized amplitude of the i-th sampling point, It is the normalized amplitude of the j-th sampling point.

5. The three-dimensional visualization method for multimodal data anomaly behavior according to claim 4, characterized in that, The process of fusing the standard data of the video stream frame and the two-dimensional velocity spectrum to obtain a unified fused feature vector, and then reconstructing the unified fused feature vector to obtain a reconstruction error, includes: Video image features are extracted by performing convolutional layers, pooling layers, and nonlinear activation layers on the standard data of the video stream frames. The two-dimensional velocity spectrum is input into a convolutional neural network to extract acceleration spectrum features. ; The video image features are then processed using a pre-trained fusion model based on a cross-modal guided attention mechanism. Mapped to query matrix The acceleration spectrum features Mapped to a key matrix Sum matrix The formula is: ； ； ； in, For query matrix The corresponding feature mapping weight matrix, Key matrix The corresponding feature mapping weight matrix, Value matrix The corresponding feature mapping weight matrix; Based on the query matrix Key matrix Sum matrix Calculate the guided weights of video image features on acceleration spectrum features. The guided weight formula is as follows: ； in, The guiding weights for video image features on acceleration spectrum features are... The scaling factor for the dimension of the feature vector. Representing the query matrix AND key matrix The correlation calculation results between them Represents the matrix transpose symbol. The normalized activation function; Using the guided weights on the value matrix Weighted calculations are performed to obtain cross-modal fusion features, using the following formula: ； in, For cross-modal fusion features; Cross-modal fusion features The graph nodes are input into the graph convolutional network for topological aggregation, resulting in a unified fusion feature vector with deep semantic relationships. The formula is as follows: ； in, To unify and merge feature vectors, For graph structure adjacency matrix, Adjacency matrix The corresponding degree matrix, For the trainable weight parameter matrix of a graph convolutional network, It is a non-linear activation function. The input graph node feature matrix; The unified fused feature vector is reconstructed using an autoencoder, and the reconstruction error of the fused features is calculated using the following formula: ； in, The reconstruction error of the fused features, For the encoder to unify and fuse feature vectors Latent feature representation after low-dimensional feature encoding The reconstructed fusion feature is obtained by the decoder after reconstructing the latent feature representation. This represents the square norm of the Euclidean distance.

6. The three-dimensional visualization method for multimodal data anomaly behavior according to claim 5, characterized in that, The training process of the fusion model based on the cross-modal guided attention mechanism includes: A training dataset containing video image features, acceleration spectrum features, and abnormal behavior labels is constructed, and the data in the training dataset is normalized. Based on the normalized training dataset, a convolutional neural network is used to extract modal features of video images, acceleration spectrum features, and abnormal behavior labels. The parameters of the model are iteratively updated using the backpropagation algorithm, and the loss is calculated using the loss function. Training stops when the loss function converges or the number of iterations reaches a preset number, resulting in a trained fusion model based on a cross-modal guided attention mechanism.

7. The three-dimensional visualization method for multimodal data anomaly behavior according to claim 5, characterized in that, The calculation of the cluster center point of the target multimodal data and the spatial offset of the target node corresponding to the target multimodal data relative to the cluster center point, and the determination and output of abnormal behavior of the target multimodal data based on the reconstruction error and the spatial offset, includes: Visual sensors, lidar, inertial measurement units, and environmental sensing devices are used as sensing nodes. A graph structure connection relationship is established based on the spatial positional relationship and data association relationship between the sensing nodes to obtain an IoT spatial topology behavior analysis graph in the target scene. Based on the IoT spatial topology behavior analysis graph, the cluster center point of the sensing nodes is calculated, and the first point distance between the target node corresponding to the target multimodal data and the cluster center point is determined. and relative direction The formula for calculating the cluster center point is: ； in, The location of the cluster center point. Let be the location coordinates of the i-th sensing node, and n be the total number of sensing nodes; When the reconstruction error of the fused feature exceeds a preset abnormal error threshold, and the spatial offset of the target node relative to the cluster center point in a continuous time series of a certain length exhibits a nonlinear abrupt change exceeding a preset range, the target multimodal data is judged to be abnormal, and the abnormal behavior category, spatial location parameters, motion trajectory parameters, and state feature parameters of the target multimodal data are output; the nonlinear abrupt change includes an abnormal increase in the rate of change of spatial displacement, abrupt change in motion direction, abnormal velocity fluctuation, or abnormal change in node topology.

8. The three-dimensional visualization method for multimodal data anomaly behavior according to claim 7, characterized in that, The step of mapping the abnormal behavior to a three-dimensional coordinate system to generate the coordinates and trajectory of the abnormal behavior, and then performing a three-dimensional visualization of the abnormal behavior based on the coordinates and trajectory, includes: The abnormal behavior coordinates of the target multimodal data are mapped to a pre-constructed BIM 3D twin model; The abnormal behavior coordinates are spatially folded and compressed using a nonlinear mapping function to obtain distance-folded abnormal behavior coordinates. The abnormal behavior trajectory is drawn based on the abnormal behavior coordinates after distance folding, and the abnormal behavior trajectory is displayed in three-dimensional visualization.

9. The three-dimensional visualization method for multimodal data anomaly behavior according to claim 8, characterized in that, The process of constructing the BIM 3D twin model includes: The architectural structure model of the target scene is created using Revit, and the architectural structure model is then lightweighted and rendered using the Unity3D 3D engine. The abnormal behavior is mapped to the three-dimensional scene of the building structure model through spatial coordinate mapping relationship, and a well constructed BIM three-dimensional twin model is obtained.

10. A three-dimensional visualization system for multimodal data anomaly behavior, characterized in that, include: The data acquisition module is used to acquire target multimodal data; The target multimodal data includes video stream frame data and one-dimensional temporal acceleration signals; The data preprocessing module is used to perform time synchronization, noise filtering and spatial coordinate alignment processing on the target multimodal data to obtain target multimodal standard data containing video stream frame standard data and one-dimensional temporal acceleration standard signal. The multimodal fusion and reconstruction module is used to convert the one-dimensional time-series acceleration standard signal into a two-dimensional velocity spectrum, fuse the video stream frame standard data and the two-dimensional velocity spectrum to obtain a unified fusion feature vector, and reconstruct the unified fusion feature vector to obtain the reconstruction error. An abnormal behavior detection module is used to calculate the cluster center point of the target multimodal data and the spatial offset of the target node corresponding to the target multimodal data relative to the cluster center point, and to judge and output the abnormal behavior of the target multimodal data based on the reconstruction error and the spatial offset; the cluster center point is the center node of the space where the target multimodal data is located, and the target node is the corresponding node in the space where the target multimodal data is located; The three-dimensional visualization module is used to map the abnormal behavior to a three-dimensional spatial coordinate system, generate the coordinates and motion trajectory of the abnormal behavior, and perform three-dimensional visualization of the abnormal behavior based on the coordinates and motion trajectory.

Images