Safety management and early warning system based on multi-source data fusion and AI large model analysis

Abstract

The application discloses a kind of based on multi-source data fusion and AI big model analysis's safety management and control early warning system, it is related to video monitoring and artificial intelligence analysis field, system includes video acquisition module, track intersection detection module, behavior segment construction module, feature coding module, fusion modeling module and behavior discrimination module.Wherein, video acquisition module is used to extract the hand trajectory of target individual, shielding state and image of personal belongings;When hand is shielded, its position is estimated by track prediction;Track intersection detection module identifies the event that two people's hands are close in short time and there is shielding;The system constructs interactive segment with the event, extracts multimodal feature and sends into heterogeneous attention fusion model, finally by discrimination subnetwork whether there is hidden exchange behavior is output, realize the intelligent identification and early warning of illegal transaction on street.The application solves the identification problem of illegal transaction behavior on street, improves the early warning efficiency of public safety.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance and artificial intelligence analysis, and more specifically, to a security management and early warning system based on multi-source data fusion and AI large model analysis. Background Art

[0002] In urban street environments, drug trafficking or other illegal street transactions have become a major challenge in the field of public safety. These activities often involve the exchange of objects through brief individual interactions, using dense crowds and dynamic occlusions to conceal intentions.

[0003] With the prevalence of surveillance technology, video data has become the primary source for identifying such behavior. However, the complexity of street scenes, including changing light and shadow, the presence of multiple people, and the rapidity of hand movements, makes it difficult for traditional methods to capture subtle signs of interaction. Especially when transferring small objects like drugs, participants often use clothing, body parts, or surrounding objects to temporarily obscure them, resulting in discontinuous trajectories and ambiguous behavior patterns, leading to missed detections and delayed responses. These issues not only amplify security risks but also hinder law enforcement efficiency. There is an urgent need to enhance the ability to detect and warn of concealed transactions through multi-source data integration and intelligent analysis. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a security management and early warning system based on multi-source data fusion and AI large model analysis to solve the problems mentioned in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The security control and early warning system based on multi-source data fusion and AI large-scale model analysis includes:

[0007] The video acquisition module is used to capture street scene videos and identify multiple target individuals. It then extracts the hand trajectory sequence, hand occlusion state sequence, and the image of the area of ​​the person's belongings. When a hand is occluded, the hand position is estimated using a trajectory prediction algorithm.

[0008] A trajectory intersection detection module is used to identify hand trajectory intersection events between any two target individuals. The hand trajectory intersection event meets the following conditions: the minimum spatial distance between the hand trajectory sequences of the two target individuals within a set time window is less than a preset threshold, and at least one of the two target individuals is in a continuous occlusion state.

[0009] The behavior segment construction module is used to construct the interaction behavior segment of two target individuals centered on the detected hand trajectory intersection event;

[0010] A feature encoding module, configured to process the interaction behavior segments to obtain multimodal feature encoding;

[0011] A fusion modeling module is used to send the multimodal feature encoding to the heterogeneous attention fusion module to obtain a fused interaction behavior representation vector through cross modeling;

[0012] The behavior discrimination module is used to input the interactive behavior representation vector into the discrimination subnetwork for binary classification judgment, and when the prediction result exceeds a preset threshold, output the recognition result of the existence of hidden exchange behavior between the two target individuals.

[0013] In some embodiments, the portable object area image corresponds to an unobstructed area, and the unobstructed area includes any one or more of the following: a pocket, a backpack, and chest clothing.

[0014] In some embodiments, the interaction behavior segment includes: a hand trajectory sequence of two target individuals, a hand occlusion state sequence, image features of the portable object area image before and after the hand trajectory intersection event, and a semantic category code of the interaction area.

[0015] In some embodiments, wherein:

[0016] The hand trajectory sequence is modeled using position embedding and velocity deviation convolution;

[0017] The hand occlusion state sequence is encoded using a mask time series;

[0018] The image of the personal object area is embedded by extracting image differences through a lightweight convolutional neural network.

[0019] In some embodiments, the behavior discrimination module is trained based on a training data set containing positive and negative samples. Each sample in the training data set includes a fused interaction behavior representation vector output by the fusion modeling module and its corresponding behavior label. Supervised learning is adopted during the training process, and the cross-entropy loss function is used to optimize the parameters of the discriminant sub-network.

[0020] In some embodiments, the hand occlusion state sequence is generated by an occlusion determination module, which determines whether the hand area of ​​the target individual in each frame of the image is occluded by its own body parts, clothing or fixed obstructions in the scene based on the posture estimation result and the image segmentation mask. If it is occluded, it is recorded as 1, and if it is not occluded, it is recorded as 0. The hand occlusion state sequence is a binary sequence arranged in chronological order, which is used to represent the continuous occlusion of the hand during the interactive behavior segment.

[0021] In some embodiments, the fusion modeling module includes a heterogeneous attention fusion module. The heterogeneous attention fusion module uses a cross-attention mechanism to perform interaction modeling on multimodal feature encoding of image difference embedding from hand trajectory sequences, hand occlusion state sequences, and images of the portable object region, as well as semantic category encoding of the interaction region. Specifically, the module includes:

[0022] The feature codes of each modality are input into the corresponding linear transformation layer to generate query vector, key vector and value vector. Through the cross-attention operation with one modality as the dominant modality, its attention weight distribution to other modalities is calculated and the fused feature representation is obtained. Finally, the fusion results under each dominant modality are spliced ​​or weighted fused to obtain the fused interaction behavior representation vector.

[0023] In some embodiments, the fusion modeling module performs a difference score on the degree of change of the image of the body object area before and after the hand trajectory intersection event during the multimodal cross-modeling process, and dynamically adjusts the attention weight of the image difference embedding in the fusion process based on the difference score, wherein the difference score is calculated by the image difference neural network, reflecting the intensity of the change in the shape, texture or edge structure of the image of the body object area before and after the hand trajectory intersection event. The more significant the change intensity, the higher the contribution weight of the image difference embedding in the interaction behavior representation vector after fusion.

[0024] In some embodiments, the trajectory prediction algorithm is Kalman filtering or motion model interpolation.

[0025] In some embodiments, the semantic category encoding of the interaction area is based on the video frame image corresponding to the intersection event, and an image semantic segmentation model is used to identify the scene semantic category of the interaction area, and the semantic category is converted into a vector representation of fixed dimension.

[0026] The present invention's advantage over existing technologies lies in its ability to capture and analyze features of hidden methods commonly used in illegal transactions, such as hand occlusion and trajectory intersection, through a security control and early warning system based on multi-source data fusion and AI large-scale model analysis, providing a precise identification solution. Starting with video acquisition, the system identifies the target individual's hand trajectory sequence, occlusion state sequence, and accompanying object area image. It then detects trajectory intersection events, constructs interaction behavior segments, performs multimodal feature encoding and heterogeneous attention fusion, and ultimately outputs the hidden exchange behavior results through a discriminant subnetwork. This architecture effectively addresses the high concealment of illegal transactions and the susceptibility of traditional surveillance to occlusion interference. It improves detection accuracy and real-time early warning capabilities in street scenarios, reduces false alarms, and supports security automation. Furthermore, the present invention refines the solution in areas such as accompanying object area definition, hand occlusion determination, interaction segment expansion, feature encoding optimization, cross-attention mechanism in the fusion module, dynamic weight adjustment of image differences, trajectory prediction, and supervised training. These designs target the persistent occlusion and hidden object changes in illegal transactions, and strengthen multimodal interaction modeling. For example, difference scores are used to highlight the deformation of objects before and after intersection, and the fusion weight is dynamically improved; binary occlusion sequences are used to capture continuous hidden actions, improving the model's adaptability to complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is an overall schematic diagram of the system of the present invention;

[0028] Figure 2 It is a flow chart of the video acquisition and preprocessing module of the present invention;

[0029] Figure 3 It is a flow chart of the fusion modeling module of the present invention;

[0030] Figure 4 This is a flow chart of behavior identification and early warning output of the present invention. DETAILED DESCRIPTION

[0031] The specific embodiments of the present invention will be described below with reference to the accompanying drawings.

[0032] like Figure 1 As shown, the present invention includes:

[0033] The video acquisition module is used to capture street scene videos and identify multiple target individuals. It then extracts the hand trajectory sequence, hand occlusion state sequence, and the image of the area of ​​the person's belongings. When a hand is occluded, the hand position is estimated using a trajectory prediction algorithm.

[0034] A trajectory intersection detection module is used to identify hand trajectory intersection events between any two target individuals. The hand trajectory intersection event meets the following conditions: the minimum spatial distance between the hand trajectory sequences of the two target individuals within a set time window is less than a preset threshold, and at least one of the two target individuals is in a continuous occlusion state.

[0035] The behavior segment construction module is used to construct the interaction behavior segment of two target individuals centered on the detected hand trajectory intersection event;

[0036] A feature encoding module, configured to process the interaction behavior segments to obtain multimodal feature encoding;

[0037] A fusion modeling module is used to send the multimodal feature encoding to the heterogeneous attention fusion module to obtain a fused interaction behavior representation vector through cross modeling;

[0038] The behavior discrimination module is used to input the interactive behavior representation vector into the discrimination subnetwork for binary classification judgment, and when the prediction result exceeds a preset threshold, output the recognition result of the existence of hidden exchange behavior between the two target individuals.

[0039] like Figure 2 As shown, more specifically, the present invention collects street scene videos through a video acquisition module, first identifying multiple target individuals, such as pedestrians or passengers in a vehicle. These target individuals can be framed out in the video frame by a target detection algorithm such as YOLO or Faster R-CNN to ensure that each individual is uniquely identified.

[0040] Next, the system extracts the hand trajectory sequence for each target individual. This involves using a pose estimation model such as OpenPose or MediaPipe to track the position changes of the key points of the hand in real time to form a time series, where each point includes x, y coordinates and a timestamp. When the hand is occluded, the system uses a trajectory prediction algorithm to estimate the hand position. Specifically, the Kalman filter algorithm can be used, which predicts the current position based on the previous motion state and noise model. For example, it assumes that the hand movement is uniform linear motion, combines observation noise and process noise for state update, or uses a motion model interpolation method to perform linear or spline interpolation through the hand positions of the previous and next frames to ensure the continuity of the trajectory sequence and avoid trajectory interruption due to short-term occlusion.

[0041] At the same time, the system extracts the hand occlusion state sequence generated by the occlusion judgment module. This module judges whether the hand area of ​​the target individual in each frame image is occluded by its own body parts such as arms, clothing such as sleeves, or fixed obstructions in the scene such as lamp posts based on the posture estimation results and image segmentation masks, such as the hand area mask generated by the Mask R-CNN or U-Net model.

[0042] If the hand is occluded, it is recorded as 1, and if it is not occluded, it is recorded as 0, thus forming a binary sequence arranged in time order, which is used to represent the continuous occlusion of the hand throughout the video sequence. For example, in a crowded street scene, if two pedestrians approach and one of them has his hand occluded by a backpack for more than three frames, the sequence will contain continuous 1 values, helping subsequent modules to identify potential concealment behavior.

[0043] The system also extracts images of areas of belongings. These images correspond to unobstructed areas, including pockets, backpacks, and clothing across the chest. Using object detection combined with semantic segmentation, the system locates these areas and crops the corresponding image blocks within the video frame, ensuring they are clear and not covered by hands or other objects. For example, for a pedestrian carrying a backpack, the system extracts an RGB image of the backpack area for subsequent analysis of object changes.

[0044] The trajectory intersection detection module identifies hand trajectory intersection events between any two target individuals. This event must meet two conditions: the minimum spatial distance between the hand trajectory sequences of the two target individuals within a set time window, such as 5 seconds, is less than a preset threshold, such as 0.5 meters. This can be achieved by calculating the Euclidean distance and taking the minimum value; at the same time, at least one of the parties is in a continuous occlusion state, such as more than 2 consecutive frames of 1 value in the occlusion sequence. For efficient detection, the system can first pair up and traverse the trajectories of all target individuals, use the KD tree to accelerate the distance calculation, and verify the occlusion condition when a potential intersection is detected. For example, in a surveillance video, if two pedestrians pass by each other, their hand trajectories are close, and one of the hands is blocked by clothing, the module will mark it as an intersection event.

[0045] The behavior segment construction module, centered around the detected hand trajectory intersection event, constructs a segment of the interaction between two target individuals. Specifically, this segment includes the hand trajectory sequence of the two target individuals, the hand occlusion state sequence, the image features of the accompanying object area before and after the hand trajectory intersection event, and the semantic category encoding of the interaction area.

[0046] Segments can be constructed with the intersection moment as the zero point, extending forward by, for example, 10 frames, and backward by 10 frames, forming a fixed-length subsequence. For example, a hand trajectory sequence can be a coordinate vector of 20 time points, a hand occlusion state sequence can be the corresponding 20 binary points, and an object area image can be constructed by extracting feature vectors from the preceding and following images and calculating them using a pre-trained ResNet model.

[0047] The semantic category encoding of the interaction area is based on the video frame image corresponding to the intersection event. An image semantic segmentation model such as DeepLabv3 is used to identify the scene semantic category, such as "sidewalk", "road edge" or "park bench". The category is then converted into a fixed-dimensional vector representation, such as using one-hot encoding or a pre-trained word embedding model such as a visual variant of BERT to map the semantic label into a 128-dimensional vector, thereby capturing the potential impact of the environment on behavior.

[0048] The feature encoding module processes the interaction behavior segments to generate multimodal feature encodings. The hand trajectory sequence is modeled using position embedding and velocity deviation convolution. Specifically, the trajectory coordinates are first converted into high-dimensional vectors using position embeddings, such as the sinusoidal function. A one-dimensional convolutional layer is then used to capture velocity variations. For example, a convolution kernel size of 3 is used to capture velocity deviations between adjacent points, helping to encode motion dynamics. The hand occlusion state sequence is encoded using masked time series encoding. Using the Transformer's masking mechanism, the binary sequence is embedded into the sequence model, ignoring non-occluded portions or emphasizing consecutive occluded segments. Images of the hand object region are embedded using a lightweight convolutional neural network, such as the MobileNetV2 model. The image difference embedding is extracted from the front and back image pairs and difference features are calculated, such as embedding vectors representing shape and texture changes through subtraction and subsequent convolution. These encoding methods ensure that multimodal data is unified into a similar vector space, facilitating subsequent fusion.

[0049] like Figure 3 As shown in the figure, the fusion modeling module feeds the multimodal feature encoding into the heterogeneous attention fusion module, and obtains the fused interaction behavior representation vector through cross-modeling. This module uses a cross-attention mechanism to interactively model the image difference embeddings from hand trajectory sequences, hand occlusion state sequences, and images of the body object region, as well as the semantic category encoding of the interaction area. The specific process involves inputting the feature encoding of each modality into the corresponding linear transformation layer to generate a query vector, a key vector, and a value vector. Then, through a cross-attention operation with one modality as the dominant modality, the attention weight distribution of the other modalities is calculated to obtain a fused feature representation.

[0050] For example, taking the hand trajectory sequence as the primary focus, the module calculates the dot product attention score between its query vector and the key vectors of other modalities, then performs a softmax on the weighted value vector. Finally, the fusion results from each dominant modality are concatenated or weighted, such as using a gating mechanism to weight the modalities based on their importance, resulting in a comprehensive 512-dimensional representation of the interaction behavior. During multimodal cross-modeling, this module differentially scores the degree of change in the image of the hand object area before and after the hand trajectory intersection event, and dynamically adjusts the attention weight of the image difference embedding during the fusion process based on the differential score.

[0051] The difference score is calculated by an image difference neural network, which can be a variant of a Siamese network. It inputs a pair of before and after images, extracts feature differences through a convolutional layer, and outputs a score between 0 and 1 that reflects the intensity of changes in shape, texture, or edge structure. For example, if the edge of the backpack area is significantly deformed and the score is close to 1, the weight of the corresponding embedding will be increased to 1.5 times through multiplication gating, thereby highlighting the key changes.

[0052] like Figure 4 As shown, the behavior discrimination module inputs the interaction behavior representation vector into the discriminant subnetwork for binary classification. When the prediction result exceeds a preset threshold, such as 0.7, it outputs the identification result of a covert exchange between the two target individuals. This discriminant subnetwork can be a multi-layer perceptron architecture, consisting of three fully connected layers, each followed by ReLU activation and Dropout regularization. The final output layer uses a sigmoid function to generate a probability score. The entire behavior discrimination module is trained based on a training dataset containing positive and negative samples. Each sample includes the fused interaction behavior representation vector output by the fusion modeling module and its corresponding behavior label. Positive samples can be simulated videos of covert exchanges, such as two people approaching and exchanging a small object, while negative samples are normal interactions, such as a handshake.

[0053] During training, supervised learning is employed, using a cross-entropy loss function to optimize the parameters of the discriminative subnetwork. For example, the Adam optimizer is used with a learning rate of 0.001, a batch size of 32, and 50 training epochs. Data augmentation, such as adding noise to trajectory sequences and flipping images, can be incorporated during training to improve generalization. For example, in a training example, if the fused vector captures hand occlusion and high image variance, the model will learn to classify it as positive, enabling accurate identification of potential risks in real-world street surveillance.

[0054] In further embodiments, multi-camera fusion can be integrated into the video acquisition module, for example, using stereo vision to calculate more accurate hand depth information and avoid distance errors caused by two-dimensional projection. In trajectory intersection detection, the preset threshold can be dynamically adjusted based on the scenario, such as a slightly higher threshold in a crowded market and a slightly lower threshold in an open street, learned from historical data. The lightweight network of the feature encoding module can be optimized for edge device deployment to ensure real-time performance, for example, by keeping model parameters within 1M. The attention mechanism of the fusion module can be expanded to multi-head attention, such as eight heads, to improve the ability to capture complex modal interactions. In a practical application, for example, monitoring a street trading scene, the system detects two hands approaching each other, one hand occludes three consecutive frames, and the backpack area image has significant texture changes before and after, with the semantic category being "dark alley." The fused vector is input into the discriminant network, outputting a probability of 0.85, triggering a warning alarm and notifying security personnel to intervene.

[0055] The overall architecture of the system of the present invention is based on AI large-scale model analysis, in which the fusion modeling module and the behavior discrimination module can be regarded as part of an end-to-end neural network. The architecture of the large model includes a coding layer, a fusion layer, and a classification layer. The total number of parameters is about 10M, and it supports GPU acceleration. The training process first collects labeled data sets, such as extracting and manually labeling exchange behavior samples from a public video library. The total sample size is 10,000, divided into 80% training set and 20% verification set. The pre-trained encoding module uses the ImageNet dataset to fine-tune the image part and uses synthetic trajectory data to pre-train the sequence part. Then, the entire model is fine-tuned end-to-end, and the accuracy of the monitoring verification set is more than 95%. During the test, it can be verified in a simulated environment, such as generating virtual pedestrian interaction videos. Through these refined implementations, the system can effectively identify hidden exchange behaviors on the streets and provide real-time safety management and control warnings.

[0056] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A security control and early warning system based on multi-source data fusion and AI large model analysis, characterized by: include: The video acquisition module is used to capture street scene videos and identify multiple target individuals. It then extracts the hand trajectory sequence, hand occlusion state sequence, and the image of the area of ​​the person's belongings. When a hand is occluded, the hand position is estimated using a trajectory prediction algorithm. A trajectory intersection detection module is used to identify hand trajectory intersection events between any two target individuals. The hand trajectory intersection event meets the following conditions: the minimum spatial distance between the hand trajectory sequences of the two target individuals within a set time window is less than a preset threshold, and at least one of the two target individuals is in a continuous occlusion state. The behavior segment construction module is used to construct the interaction behavior segment of two target individuals centered on the detected hand trajectory intersection event; A feature encoding module, configured to process the interaction behavior segments to obtain multimodal feature encoding; A fusion modeling module is used to send the multimodal feature encoding to the heterogeneous attention fusion module to obtain a fused interaction behavior representation vector through cross modeling; The behavior discrimination module is used to input the interactive behavior representation vector into the discrimination subnetwork for binary classification judgment, and when the prediction result exceeds a preset threshold, output the recognition result of the existence of hidden exchange behavior between the two target individuals.

2. The security control and early warning system based on multi-source data fusion and AI large model analysis according to claim 1 is characterized in that: The portable object area image corresponds to an unobstructed area, and the unobstructed area includes any one or more of the following: a pocket, a backpack, and chest clothing.

3. The security control and early warning system based on multi-source data fusion and AI large model analysis according to claim 1 is characterized in that: The interaction behavior fragment includes: a hand trajectory sequence of two target individuals, a hand occlusion state sequence, image features of the portable object area image before and after the hand trajectory intersection event, and a semantic category code of the interaction area.

4. The security control and early warning system based on multi-source data fusion and AI large model analysis according to claim 1 is characterized in that: in: The hand trajectory sequence is modeled using position embedding and velocity deviation convolution; The hand occlusion state sequence is encoded using a mask time series; The image of the personal object area is embedded by extracting image differences through a lightweight convolutional neural network.

5. The security control and early warning system based on multi-source data fusion and AI large model analysis according to claim 1 is characterized in that: The behavior discrimination module is trained based on a training data set containing positive and negative samples. Each sample in the training data set includes a fused interaction behavior representation vector output by the fusion modeling module and its corresponding behavior label. Supervised learning is adopted during the training process, and the cross-entropy loss function is used to optimize the parameters of the discriminant sub-network.

6. The security control and early warning system based on multi-source data fusion and AI large model analysis according to claim 1 is characterized in that: The hand occlusion state sequence is generated by the occlusion judgment module. The occlusion judgment module determines whether the hand area of ​​the target individual in each frame image is occluded by its own body parts, clothing or fixed obstructions in the scene based on the posture estimation result and the image segmentation mask. If it is occluded, it is recorded as 1, and if it is not occluded, it is recorded as 0. The hand occlusion state sequence is a binary sequence arranged in chronological order, which is used to represent the continuous occlusion of the hand during the interactive behavior segment.

7. The security control and early warning system based on multi-source data fusion and AI large model analysis according to claim 1 is characterized in that: The fusion modeling module includes a heterogeneous attention fusion module, which uses a cross-attention mechanism to interactively model the multimodal feature encoding of image difference embedding from hand trajectory sequences, hand occlusion state sequences, and portable object area images, as well as semantic category encoding of the interaction area. Specifically, the module includes: The feature codes of each modality are input into the corresponding linear transformation layer to generate query vector, key vector and value vector. Through the cross-attention operation with one modality as the dominant modality, its attention weight distribution to other modalities is calculated and the fused feature representation is obtained. Finally, the fusion results under each dominant modality are spliced ​​or weighted fused to obtain the fused interaction behavior representation vector.

8. The safety management and early warning system based on multi-source data fusion and AI large model analysis according to claim 1 or 7 is characterized in that: During the multimodal cross-modeling process, the fusion modeling module performs a difference score on the degree of change in the image of the portable object area before and after the hand trajectory intersection event, and dynamically adjusts the attention weight of the image difference embedding in the fusion process based on the difference score. The difference score is calculated by an image difference neural network, reflecting the intensity of the change in the shape, texture or edge structure of the image of the portable object area before and after the hand trajectory intersection event. The more significant the change intensity, the higher the contribution weight of the image difference embedding in the interaction behavior representation vector after fusion.

9. The security control and early warning system based on multi-source data fusion and AI large model analysis according to claim 1 is characterized in that: The trajectory prediction algorithm is Kalman filtering or motion model interpolation.

10. The security control and early warning system based on multi-source data fusion and AI large model analysis according to claim 3 is characterized in that: The semantic category encoding of the interaction area is based on the video frame image corresponding to the intersection event, and an image semantic segmentation model is used to identify the scene semantic category of the interaction area, and the semantic category is converted into a vector representation of fixed dimension.

Images