A system and method for understanding and identifying abnormal behavior of port customs officers

Abstract

This invention discloses a system and method for understanding and identifying abnormal behavior of port clearance personnel, relating to the field of port clearance personnel monitoring technology. The invention acquires visible light, infrared, and millimeter-wave monitoring data of the port clearance area, and forms multimodal behavioral data through spatiotemporal alignment and personnel target association; extracts behavioral features of each modality, determines fusion weights based on data quality, and completes feature fusion; arranges the fused behavioral features chronologically to form a target behavior sequence, and obtains behavioral recognition results through a behavior recognition model; determines abnormal behavior based on abnormal judgment conditions, calculates risk scores, and generates early warning instructions; when identified as clustering behavior, it distinguishes between normal queuing and abnormal clustering based on the personnel arrangement order and movement transmission. This invention can improve the accuracy of abnormal behavior identification and risk warning in complex clearance environments.

Description

Technical Field

[0001] This invention relates to the field of port clearance personnel monitoring technology, specifically a system and method for understanding and identifying abnormal behavior of port clearance personnel. Background Technology

[0002] Port clearance personnel behavior monitoring refers to the collection and analysis of data on the location, actions, and movement status of personnel at port entrances and exits, inspection channels, and waiting areas to identify behaviors such as normal passage, climbing over barriers, tailgating, attempting to break through checkpoints, and gathering. Existing monitoring systems typically collect personnel monitoring data through camera equipment and combine it with target detection, target tracking, and behavior recognition technologies to identify and issue warnings for abnormal behaviors of clearance personnel.

[0003] As the number of people passing through ports increases, personnel behavior identification is gradually shifting from single-modal monitoring data analysis to multi-modal monitoring data fusion analysis. However, different modal monitoring data differ in collection time and spatial location, making it difficult to uniformly collect monitoring data for the same person. Furthermore, different modal behavioral characteristics have varying descriptive abilities for personnel behavior, resulting in the formation of personnel behavior characteristics that cannot accurately reflect the person's behavioral state, thus affecting the accuracy of abnormal behavior identification results. Summary of the Invention

[0004] The purpose of this invention is to provide a system and method for understanding and identifying abnormal behavior of customs clearance personnel at ports, in order to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for understanding and identifying the behavior of port clearance personnel, wherein the method specifically includes:

[0006] Step S1: Obtain multimodal monitoring data of the port clearance area, and perform spatiotemporal alignment on the multimodal monitoring data to form synchronous monitoring data;

[0007] Step S2: Detect personnel targets from the synchronous monitoring data, and associate and aggregate the monitoring data corresponding to the same personnel target in different modalities and at different collection times to form multimodal behavioral data corresponding to each personnel target;

[0008] Step S3: Extract the personnel behavior features corresponding to each modality from the multimodal behavior data, determine the fusion weight of each personnel behavior feature, and fuse the personnel behavior features according to the fusion weight to form the fused behavior features corresponding to each personnel target;

[0009] Step S4: Arrange the fused behavioral features corresponding to each person's target in chronological order to form a target behavior sequence, and input the target behavior sequence into the behavior recognition model to form a behavior recognition result;

[0010] Step S5: Determine the abnormal behavior based on the behavior recognition result, determine the risk level according to the risk classification conditions corresponding to the abnormal behavior, and generate the corresponding early warning instruction.

[0011] Further, step S1 specifically includes:

[0012] Visible light acquisition equipment, infrared acquisition equipment, and millimeter-wave radar are used to collect visible light monitoring data, infrared monitoring data, and millimeter-wave monitoring data in the port clearance area, respectively, to form multimodal monitoring data;

[0013] Multiple calibration points are set up on the ground in the port clearance area. Based on the ground coordinates of each calibration point and its image coordinates in visible light and infrared images, spatial transformation relationships between visible light monitoring data and infrared monitoring data and the same ground coordinate system are established respectively. Spatial transformation relationships between the millimeter-wave radar coordinate system and the ground coordinate system are established through the external parameter matrix.

[0014] Based on the acquisition time of visible light monitoring data, infrared monitoring data and millimeter wave monitoring data that are closest to its acquisition time are searched within the same time window. When the difference between the acquisition times is within the time matching range, they are matched to form synchronous monitoring data.

[0015] Furthermore, the visible light monitoring data includes visible light images continuously acquired by the visible light acquisition device, the infrared monitoring data includes infrared thermal images continuously acquired by the infrared acquisition device, and the millimeter wave monitoring data includes point cloud data or range Doppler data continuously acquired by the millimeter wave radar. Each monitoring data has a corresponding acquisition time.

[0016] The calibration points are fixed points set on the ground in the port clearance area. The ground coordinates of each calibration point in the ground coordinate system are predetermined, and the corresponding image coordinates can be determined in visible light and infrared images. A correspondence is established between the ground coordinates of the same calibration point and its image coordinates in visible light and infrared images, so as to form a spatial transformation relationship from visible light image coordinates and infrared image coordinates to ground coordinates.

[0017] The location of reflection points in millimeter-wave monitoring data is referenced to the location of the millimeter-wave radar itself. Through the spatial transformation relationship between the millimeter-wave radar coordinate system and the ground coordinate system, the location of each reflection point is converted into the corresponding ground coordinates, so that the location of personnel targets in millimeter-wave monitoring data can be compared with the location of personnel targets in visible light monitoring data and infrared monitoring data in the same ground coordinate system.

[0018] The time matching range refers to the allowable time difference that data from different acquisition devices can be considered as data from the same acquisition time. Using the acquisition time of visible light monitoring data as a benchmark, infrared monitoring data and millimeter-wave monitoring data with the closest acquisition times are selected respectively. When the corresponding acquisition time difference does not exceed the time matching range, the three types of monitoring data are grouped into the same set of synchronous monitoring data. When the time matching range is exceeded, the corresponding data is interpolated or discarded.

[0019] Further, step S2 specifically includes:

[0020] Personnel targets are detected from the visible light monitoring data and infrared monitoring data in the synchronous monitoring data respectively, and the reflection points in the millimeter wave monitoring data are clustered to form personnel detection results corresponding to each mode; the clustering is to group the reflection points corresponding to the same personnel target into the same target point set according to the position and movement speed of each reflection point in the millimeter wave monitoring data.

[0021] Extract the personnel target positions from the personnel detection results corresponding to each modality, transform each personnel target position to the ground coordinate system according to the spatial transformation relationship, compare the target positions of the personnel detection results of each modality at the same acquisition time, associate the personnel detection results with the target positions with each other to the same personnel target, and set the corresponding target identifiers;

[0022] By comparing the position and movement status of each person target at adjacent acquisition times, the corresponding person targets are temporally correlated and their corresponding target identifiers are continued.

[0023] Based on the target identifier, monitoring data corresponding to the same person target at different modalities and different collection times are collected to form multimodal behavioral data corresponding to each person target.

[0024] Furthermore, the target identifier is a unified number assigned to the same personnel target. By continuing the target identifiers of corresponding personnel targets at adjacent acquisition times, a continuous trajectory of the personnel target is formed. Data corresponding to the same personnel target in visible light monitoring data, infrared monitoring data, and millimeter wave monitoring data are collected according to the target identifier to form multimodal behavioral data.

[0025] Furthermore, step S3 specifically includes:

[0026] Human appearance features, human key point features, and head posture features are extracted from the multimodal behavioral data to form visible light human behavior features; human heat source features are extracted to form infrared human behavior features; and the position, movement speed, movement direction, and reflection intensity of the human target are extracted to form millimeter wave human behavior features.

[0027] Illuminance, motion ambiguity, occlusion rate, and detection confidence are extracted from visible light monitoring data; heat source integrity, thermal noise, and thermal target confidence are extracted from infrared monitoring data; and signal-to-noise ratio, data continuity, and velocity consistency are extracted from millimeter-wave monitoring data. The indicators corresponding to each mode are weighted and summed according to set coefficients to form visible light quality score, infrared quality score, and millimeter-wave quality score, respectively.

[0028] The visible light quality score, infrared quality score, and millimeter wave quality score are normalized to form fusion weights corresponding to each person's behavioral characteristics. Each person's behavioral characteristics are multiplied by their corresponding fusion weights and then concatenated to form initial fusion features.

[0029] Cross-modal attention processing is performed on the visible light personnel behavior features, infrared personnel behavior features, and millimeter wave personnel behavior features to form cross-modal correlation features. The cross-modal correlation features are then residually fused with the initial fusion features to form fused behavior features corresponding to each personnel target.

[0030] Furthermore, the quality score represents the current reliability of the corresponding modal monitoring data. The higher the quality score, the greater the fusion weight of the corresponding personnel behavior features. The cross-modal attention processing analyzes the correspondence between human key point features and infrared heat source features and millimeter wave channel activation features in visible light personnel behavior features, highlights the interrelated behavioral information in different modalities, and combines the formed cross-modal correlation features with the initial fusion features to obtain fused behavioral features.

[0031] Furthermore, step S4 specifically includes:

[0032] Arrange the fused behavioral features of the same person at different collection times in chronological order to form a target behavior sequence;

[0033] The target behavior sequence is divided into multiple behavior sequence segments according to the set time length and sliding step size;

[0034] Each behavior sequence segment is input into the behavior recognition model. The fused behavior features at continuously collected moments in the behavior sequence segment are analyzed to form short-term action change features. The continuous change relationship between each short-term action change feature is analyzed in chronological order to form the recognition probability corresponding to each behavior category.

[0035] The behavior category with the highest recognition probability and its recognition probability are determined as the behavior recognition result.

[0036] Furthermore, the set time length represents the continuous monitoring duration covered by each behavior sequence segment, and the sliding step size represents the time interval between the start times of adjacent behavior sequence segments. The behavior recognition model includes a short-term action feature extraction part and a temporal relationship analysis part. The short-term action feature extraction part uses a three-dimensional convolutional network to jointly process the fused behavior features collected at continuous moments in the behavior sequence segment, extracting changes in human key points, head posture, heat source distribution, target position, movement speed, movement direction, and millimeter wave reflection response within a short period of time to form short-term action change features. The temporal relationship analysis part uses a Transformer encoder to analyze the continuous changes and sequential relationships between each short-term action change feature in chronological order, forming the recognition probability corresponding to each behavior category: climbing over, tailing, breaking through, gathering, looking left and right, and normal passage.

[0037] Further, step S5 specifically includes:

[0038] The behavior category and its corresponding recognition probability are extracted from the behavior recognition result. When the behavior category is not a normal passage behavior, the behavior category is determined as a candidate behavior, and the abnormal judgment condition corresponding to the candidate behavior is retrieved.

[0039] Extract the target position, motion state, human posture, and behavior duration corresponding to each anomaly determination condition from the multimodal behavior data and target behavior sequence, and determine the satisfaction of each anomaly determination condition to form a rule satisfaction degree;

[0040] The recognition probability and rule satisfaction are weighted and summed according to a set coefficient to form the behavior confidence score; when the behavior confidence score reaches the anomaly judgment threshold corresponding to the candidate behavior, the candidate behavior is determined to be an abnormal behavior.

[0041] The location and duration of the abnormal behavior are determined. Historical abnormal situations corresponding to the same person or location are extracted from historical abnormal events. The confidence level of the behavior, the rule satisfaction level, and the scores corresponding to the location, duration, and historical abnormal situations are weighted and summed to form a risk score. The risk score is compared with the risk classification conditions to determine the corresponding risk level. When the risk level is normal, the candidate behavior is determined as normal behavior. When the risk level is abnormal, the candidate behavior is determined as abnormal behavior, and a corresponding warning instruction is generated.

[0042] The historical abnormal events are abnormal behavior events identified and stored by the behavior understanding and abnormal identification system during previous monitoring, including target identifier, behavior category, occurrence area, occurrence time, duration and risk level; when no historical abnormal event corresponding to the same person target or the same occurrence area is retrieved, the score corresponding to the historical abnormal situation is determined to be zero.

[0043] Furthermore, when the behavior category in the behavior recognition result is a clustering behavior, the determination of the abnormal behavior specifically includes:

[0044] Extract the target positions of each person target corresponding to the aggregation behavior at different collection times from the target behavior sequence, and determine the passage direction according to the entrance and exit positions of the port clearance area;

[0045] Project the target positions of each person onto the direction of travel, and arrange the person targets according to the projection positions to form a personnel arrangement sequence corresponding to each collection time.

[0046] Compare the personnel arrangement sequence at adjacent acquisition times, and determine the order preservation degree by the ratio of the number of adjacent personnel target pairs whose arrangement order has not changed to the total number of adjacent personnel target pairs; when the personnel target in front starts to move, determine the movement transfer degree by the ratio of the number of adjacent personnel target pairs that move with the personnel target in front within a set response time to the total number of adjacent personnel target pairs.

[0047] When the order preservation degree reaches the order preservation threshold and the movement transfer degree reaches the movement transfer threshold, the aggregation behavior is determined as normal queuing behavior; otherwise, the aggregation behavior is determined as abnormal aggregation behavior and is treated as the abnormal behavior.

[0048] Furthermore, the behavior understanding and anomaly recognition system includes a multimodal synchronization module, a personnel data collection module, a behavior feature fusion module, a behavior recognition module, and an anomaly judgment and early warning module;

[0049] The multimodal synchronization module acquires multimodal monitoring data of the port clearance area, performs spatiotemporal alignment on the multimodal monitoring data, and forms synchronized monitoring data.

[0050] The personnel data collection module detects personnel targets from the synchronous monitoring data, and associates and collects monitoring data corresponding to the same personnel target in different modalities and at different collection times to form multimodal behavioral data corresponding to each personnel target;

[0051] The behavior feature fusion module extracts the personnel behavior features corresponding to each modality from the multimodal behavior data, determines the fusion weight of each personnel behavior feature, and fuses the personnel behavior features according to the fusion weight to form the fused behavior features corresponding to each personnel's target.

[0052] The behavior recognition module arranges the fused behavior features corresponding to each person's target in chronological order to form a target behavior sequence, and inputs the target behavior sequence into the behavior recognition model to form a behavior recognition result;

[0053] The anomaly detection and early warning module determines the abnormal behavior based on the behavior recognition result, determines the risk level according to the risk classification conditions corresponding to the abnormal behavior, and generates a corresponding early warning instruction.

[0054] Furthermore, the behavioral feature fusion module includes a feature extraction and evaluation unit and a feature association and fusion unit;

[0055] The feature extraction and evaluation unit extracts visible light human behavior features, infrared human behavior features, and millimeter wave human behavior features from the multimodal behavior data, and extracts the quality indicators corresponding to each mode. The quality indicators are weighted and summed according to a set coefficient to form the corresponding quality score.

[0056] The feature association fusion unit normalizes the quality scores corresponding to each modality to form fusion weights for each person's behavioral features. It then multiplies each person's behavioral features with its corresponding fusion weights and concatenates them to form initial fusion features. Finally, it performs residual fusion with the cross-modal association features formed by cross-modal attention processing and the initial fusion features to form fusion behavioral features.

[0057] Furthermore, the anomaly detection and early warning module includes an anomaly verification unit and a risk classification unit;

[0058] The anomaly verification unit extracts the behavior category and its corresponding recognition probability from the behavior recognition result. When the behavior category is not a normal passage behavior, it retrieves the anomaly judgment condition corresponding to the behavior category, extracts the data corresponding to each anomaly judgment condition from the multimodal behavior data and the target behavior sequence, judges the satisfaction of each anomaly judgment condition, forms the rule satisfaction degree, and weights and sums the recognition probability and the rule satisfaction degree according to the set coefficient to form the behavior confidence degree, and determines the abnormal behavior based on the behavior confidence degree.

[0059] The risk grading unit combines the behavioral confidence level, rule satisfaction level, occurrence area, duration, and historical anomalies of the abnormal behavior to form a risk score. The risk score is compared with the risk grading conditions to determine the corresponding risk level. When the risk level is normal, the candidate behavior is determined as normal behavior. When the risk level is abnormal, the candidate behavior is determined as abnormal behavior, and a corresponding warning instruction is generated.

[0060] Compared with the prior art, the beneficial effects of the present invention are:

[0061] 1. This invention continuously acquires multimodal behavioral data of the same person by performing spatiotemporal alignment and target association on visible light, infrared and millimeter wave monitoring data, and identifies abnormal behavior by combining the temporal changes of behavioral characteristics, thereby improving the accuracy of abnormal behavior identification in complex customs clearance environments.

[0062] 2. This invention determines the fusion weights based on the quality indicators of each modality monitoring data, and strengthens the corresponding behavioral information between different modalities through cross-modal attention processing, thereby reducing the impact of illumination changes, target occlusion and data noise on the behavior recognition results and improving recognition stability.

[0063] 3. After identifying gathering behavior, this invention determines whether people are maintaining a normal queuing state by observing the maintenance of the order of personnel and the transmission of movement behavior, thereby distinguishing between normal queuing behavior and abnormal gathering behavior and reducing the situation where normal clearance queues are mistakenly judged as abnormal gathering. Attached Figure Description

[0064] Figure 1 This is a flowchart illustrating a method for understanding and identifying abnormal behavior of customs clearance personnel at ports, as proposed by the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Example: Figure 1 As shown, the present invention provides a technical solution: a method for understanding and identifying abnormal behavior of port clearance personnel. The method includes the following steps:

[0067] Step S1: Obtain multimodal monitoring data of the port clearance area, and perform spatiotemporal alignment of the multimodal monitoring data to form synchronous monitoring data.

[0068] Visible light acquisition equipment, infrared acquisition equipment, and millimeter-wave radar are deployed in the port clearance area. The visible light acquisition equipment continuously acquires visible light images of the port clearance area, the infrared acquisition equipment continuously acquires infrared thermal images of the corresponding area, and the millimeter-wave radar acquires millimeter-wave point cloud or range Doppler data of the corresponding area. All types of monitoring data are recorded at the corresponding acquisition time.

[0069] At least four calibration points are set up on the ground in the port clearance area. The ground coordinates of each calibration point in the ground coordinate system are predetermined, and the corresponding image coordinates can be determined in both visible light and infrared images. Based on the correspondence between the ground coordinates and visible light image coordinates of the same calibration point, the homography matrix H for transforming the visible light image plane to the ground coordinate system is determined. v Based on the correspondence between the ground coordinates and infrared image coordinates of the same calibration point, determine the homography matrix H for transforming the infrared image plane to the ground coordinate system. i ;

[0070] For image coordinates (u) in visible light images v ,v v The transformation relationship between the coordinates of the ground and the coordinates of the ground is expressed as follows:

[0071]

[0072] For image coordinates (u) in infrared images i ,v i The transformation relationship between its coordinates and ground coordinates is expressed as follows:

[0073]

[0074] In the formula, p v p represents the ground coordinates obtained after visible light image coordinate transformation. i This represents the ground coordinates obtained after the infrared image coordinate transformation.

[0075] The location of the reflection point in the millimeter-wave monitoring data is represented in the coordinate system of the millimeter-wave radar itself, through the extrinsic parameter matrix T. r Mapping the millimeter-wave radar coordinate system to the same ground coordinate system, the spatial transformation relationship is expressed as follows:

[0076]

[0077] In the formula, (x r ,y r ,z r ) represents the coordinates of the reflection point in the millimeter-wave radar coordinate system, p rThis represents the coordinates after transformation to the ground coordinate system. Through the above spatial transformation, the target location in visible light monitoring data, infrared monitoring data, and millimeter wave monitoring data can all be represented using the same ground coordinates.

[0078] After spatial calibration, using the acquisition time of the visible light monitoring data as a benchmark, the infrared monitoring data and millimeter-wave monitoring data with the closest acquisition time are searched within the corresponding time window. When the acquisition time difference between the visible light monitoring data and the other two types of monitoring data does not exceed 80ms, the corresponding visible light image, infrared thermal image, and millimeter-wave monitoring data are grouped into the same set of synchronous monitoring data; when the acquisition time difference of one type of monitoring data exceeds 80ms, the corresponding data is interpolated or the set of data is discarded.

[0079] The synchronous monitoring data generated at the t-th acquisition time is represented as:

[0080] Xt={I v,t ,I i,t ,R t}

[0081] In the formula, I v,t I represents the visible light image corresponding to the t-th acquisition time. i,t Indicates the infrared thermal image matched with its completion time, R t This indicates millimeter-wave monitoring data that matches its completion time.

[0082] Step S2: Detect personnel targets from synchronous monitoring data, and associate and aggregate monitoring data corresponding to the same personnel target in different modalities and at different collection times to form multimodal behavioral data corresponding to each personnel target.

[0083] For the visible light images and infrared thermal images in the synchronous monitoring data, human detection, human key point detection, and personnel target bounding box extraction are performed respectively to obtain visible light personnel detection results and infrared personnel detection results. For millimeter wave monitoring data, clustering is performed according to the position and movement speed of each reflection point, and reflection points corresponding to the same personnel target are grouped into the same target point set to obtain millimeter wave personnel detection results.

[0084] Extract the image location of the person target from the visible light person detection results and the infrared person detection results, and then calculate the location according to the homography matrix H. v and homography matrix H i Convert to ground coordinates; according to the extrinsic parameter matrix T r The reflection points of the millimeter-wave target set are converted into ground coordinates. The positions of the personnel detection results of each modality in the ground coordinate system are compared at the same acquisition time. Personnel detection results with corresponding positions are associated with the same personnel target, and a unified target identifier is set for that personnel target.

[0085] During continuous data acquisition, Kalman filtering is used to predict the position and motion state of each person target at the next acquisition time. Then, Hungarian matching is used to correlate the prediction results with the personnel detection results at the next acquisition time. For corresponding personnel targets, their existing target labels are retained; for newly appearing detection results that cannot be matched with existing personnel targets, new target labels are set, thereby forming the continuous motion trajectory of each personnel target.

[0086] For the j-th personnel target, its ground coordinates at the t-th data acquisition time are represented as:

[0087] p j,t =(x j,t ,y j,t )

[0088] Calculate the velocity vector of the person target based on the ground coordinates at adjacent acquisition times:

[0089]

[0090] And calculate its acceleration vector:

[0091]

[0092] In the formula, Δt represents the time interval between adjacent acquisition times, specifically the time interval between the (t-1)th acquisition time and the tth acquisition time. The direction of movement of the personnel target is determined based on the direction of positional change between adjacent ground coordinates.

[0093] According to the target identifier, the images of people and key points of the human body in the visible light monitoring data, the thermal source information of people in the infrared monitoring data, and the target point set, location and motion information in the millimeter wave monitoring data are collected and arranged according to the collection time to form multimodal behavior data corresponding to each person target.

[0094] Step S3: Extract the personnel behavior features corresponding to each modality from the multimodal behavior data, determine the fusion weight of each personnel behavior feature, and form the fused behavior features corresponding to each personnel's target.

[0095] For the same person target, human appearance features, key human features, and head posture features are extracted from visible light monitoring data to form visible light person behavior characteristics F. v Among them, human appearance features represent the contour and texture information of the human target, human key point features represent the position and changes of various body parts of the human, and head posture features include yaw angle, pitch angle and roll angle.

[0096] The distribution, intensity, and continuous changes of heat sources corresponding to personnel targets are extracted from infrared monitoring data to form infrared personnel behavior characteristics F. i The location, speed, direction of movement, and reflection intensity of personnel targets are extracted from millimeter-wave monitoring data to form millimeter-wave personnel behavior characteristics F. r .

[0097] In this embodiment, the millimeter-wave monitoring data is also converted into a millimeter-wave channel activation map that can participate in cross-modal fusion. The j-th reflection point output by the millimeter-wave radar is represented as:

[0098] P j =(x j ,y j ,z j ,v j ,s j )

[0099] In the formula, (x j ,y j ,z j ) indicates the location of the reflection point, v j The velocity s represents the velocity corresponding to the reflection point. j This represents the reflection intensity. Static clutter filtering is performed on millimeter-wave point clouds to remove background reflection points with a velocity less than 0.05 m / s and a reflection intensity change less than a set value for three consecutive frames. Generally, P is set to... j =(x j ,y j ) represents the two-dimensional position of the j-th reflection point after it is projected onto the ground coordinate system.

[0100] The retained reflection points are projected onto a ground grid in the clearance area, with a grid resolution set from 0.10m to 0.25m. Reflection points falling into the same grid are accumulated according to reflection intensity, absolute velocity, and distance attenuation. The grid image is then diffused to form a millimeter-wave channel activation map.

[0101]

[0102] In the formula, g(x,y) represents the corresponding position in the ground grid, and s j Indicates the intensity of reflection, v j Let σ represent the velocity, and σ represent the diffusion range parameter, typically taken as 0.2m to 0.4m. The millimeter-wave channel activation map is normalized to the 0-1 interval, and the activation change between adjacent acquisition times is calculated.

[0103]

[0104] This enables millimeter-wave human behavior characteristics to characterize the position displacement, velocity changes, and direction of motion of human targets.

[0105] For each modal monitoring data within the current time window, the visible light quality score, infrared quality score, and millimeter-wave quality score are calculated respectively. The visible light quality score is expressed as:

[0106] q v =0.35l+0.25(1−b)+0.25(1−o)+0.15c v

[0107] In the formula, l represents illuminance, b represents motion blur, o represents occlusion rate, and c v This indicates the confidence level of the visible light personnel detection results.

[0108] Infrared mass fraction is expressed as:

[0109] q i =0.45h+0.25(1−n i +0.30c i

[0110] In the formula, h represents the integrity of the heat source, and n i Represents thermal noise, c i This indicates the confidence level of the thermal target in the infrared personnel detection results.

[0111] Millimeter wave mass fraction is expressed as:

[0112] q r =0.40snr+0.35k+0.25e

[0113] In the formula, snr represents the signal-to-noise ratio of millimeter-wave monitoring data, k represents data continuity, and e represents velocity consistency.

[0114] Softmax normalization is applied to the three quality scores to obtain the corresponding fusion weights:

[0115]

[0116] In the formula, m represents the mode currently being calculated for weights, n represents the modes traversed during the summation process, m, n ∈ {v, i, r}, the sum of the three fusion weights is 1, representing the visible light, infrared, and millimeter wave modes respectively, and τ is a temperature parameter, which is taken as 0.5 in this embodiment. The higher the quality score, the greater the fusion weight of the corresponding mode.

[0117] The behavioral characteristics of each person are multiplied by their corresponding fusion weights and then concatenated to form the initial fusion features:

[0118] F cat =[w v F v ,w i F i ,w rF r ]

[0119] The formula indicates that splicing is performed along the direction of the feature channel.

[0120] Based on the initial fusion features, cross-modal attention processing is performed on visible light human body keypoint features, infrared heat source features, and millimeter-wave channel activation features. The cross-modal attention matrix is ​​represented as follows:

[0121]

[0122] In the formula, Q comes from visible light human body key point features, K and V come from infrared heat source features and millimeter wave channel activation features, and d represents the dimension of the corresponding feature.

[0123] Residual fusion is performed between the associated features obtained from cross-modal attention processing and the initial fused features:

[0124] F fuse =Conv 1×1 (F cat )+A*V+F v

[0125] In the formula, Conv 1×1 This indicates channel compression of the initial fused features, A*V represents cross-modal correlation features, and F... v This is used as a residual term. This forms the fusion behavior feature F corresponding to the target person at the current acquisition moment. fuse Before adding the residuals, the initial fusion features, cross-modal correlation features, and visible light human behavior features are mapped to the same feature dimension.

[0126] Step S4: Arrange the fused behavioral features of the same person at different collection times in chronological order to form a target behavior sequence, and generate a behavior recognition result through a behavior recognition model.

[0127] The fused behavioral features with the same target identifier are arranged according to the acquisition time to form a target behavior sequence for the corresponding personnel target. The target behavior sequence is divided into segments with a continuous time length of 2s to 10s, and the sliding step size of adjacent behavior sequence segments is set to 0.5s to 1s, thereby forming multiple behavior sequence segments with partial overlap.

[0128] The behavior recognition model comprises a 3D convolutional network and a Transformer encoder. The 3D convolutional network receives visible light features, infrared thermal features, and millimeter-wave channel activation features from consecutive snapshots within a behavior sequence segment, processing the continuous changes of each feature over a short period to form short-term motion change features. The Transformer encoder analyzes the target trajectory, human key points, head posture angles, and the continuous relationship between various short-term motion change features in chronological order to form the recognition probability corresponding to each behavior category.

[0129] The recognition probability vector output by the behavior recognition model is represented as follows:

[0130] P model =[P 翻越 ,P 尾随 ,P 闯关 ,P 聚集 ,P 左顾右盼 ,P 正常 ]

[0131] Each probability represents the likelihood that the current behavior sequence segment belongs to the behaviors of climbing over, tailing, breaking through obstacles, gathering, looking left and right, and normal passage. The behavior category with the highest recognition probability and its recognition probability are determined as the behavior recognition result of that behavior sequence segment.

[0132] The training samples for the behavior recognition model are derived from manually labeled synchronous multimodal data. Training samples are paired using visible light images, infrared thermal images, and millimeter-wave point clouds or millimeter-wave channel activation maps, all within the same target identifier and time window. The pairs are labeled with the behavior category, start and end times, channel region, and occlusion status. During training, a behavior category cross-entropy loss is employed, supplemented with temporal smoothing loss and category weights to reduce jitter in recognition results between adjacent behavior sequence segments and mitigate the impact of differences in the number of samples from different categories.

[0133] In one training method, samples are divided into training and test sets in an 8:2 ratio, with 10% of the training set used as a validation set. The training set is used for training model parameters and determining class weights, the validation set is used for calibrating behavior recognition thresholds and risk classification thresholds, and the test set is used to verify the recognition results for six types of behaviors.

[0134] Step S5: Determine the abnormal behavior based on the behavior recognition results, determine the risk level according to the risk classification conditions corresponding to the abnormal behavior, and generate the corresponding early warning instruction.

[0135] Extract the behavior category b and its recognition probability P from the behavior recognition results. model,b The behavior category is identified as a candidate behavior, and the corresponding anomaly judgment conditions are retrieved. Each anomaly judgment condition consists of a corresponding behavior rule.

[0136] The abnormal judgment conditions for climbing behavior include the relationship between the height change trend of key points of the human body and the boundary of the obstacle in the port clearance area, which is used to judge whether the human target has a continuous action of jumping, raising his leg or crossing the boundary of the obstacle.

[0137] The abnormal judgment conditions for tailing behavior include the spatiotemporal distance between adjacent targets, the overlap of movement paths, continuous following relationships within a short period of time, and the correlation of historical trajectories.

[0138] The abnormal judgment conditions for the attempt to break through the checkpoint include whether the person or target has crossed the gate line, inspection line or no-entry line, as well as the changes in speed, acceleration and direction of movement during the crossing process.

[0139] The criteria for identifying abnormal gathering behavior include the number of people in the area, population density, average dwell time, activity radius, and their continuous changes. For a candidate area of ​​area A, there exists n at the t-th data collection time. t When there are individual targets, the instantaneous density is expressed as:

[0140]

[0141] The center of gravity of personnel objectives is represented as follows:

[0142]

[0143] The activity radius of the personnel target is expressed as:

[0144]

[0145] The average length of stay for people is expressed as follows:

[0146]

[0147] In the formula, t j,in This indicates the moment when the j-th person enters the candidate region.

[0148] Using L consecutive frames as the review window, the population density growth rate is expressed as:

[0149]

[0150] The convergence of the radius of personnel activity is expressed as:

[0151]

[0152] In the formula, ε is a very small positive number to prevent the denominator from being zero. The above-mentioned personnel density, average stay time, activity radius, density growth rate and activity radius convergence amount serve as rule evidence corresponding to the aggregation behavior.

[0153] Abnormal judgment criteria for looking left and right include changes in head yaw angle, number of times the observation direction is reversed, duration of gazing in the non-traveling direction, duration of dwell time, and walking speed. Head posture is determined by combining two-dimensional feature points of the nose tip, corners of the eyes, corners of the mouth, and ears with standard three-dimensional facial key points.

[0154]

[0155] In the formula, p k P represents the two-dimensional feature points of the head. k Let R represent the standard 3D facial keypoints, K represent the camera intrinsic parameters, R represent the head rotation matrix, t represent the translation vector, and π represent the projection function. The yaw, pitch, and roll angles are determined based on the head rotation matrix R.

[0156] Within a 10-second behavioral sequence segment, if there are at least 3 lateral observations with an absolute yaw angle greater than 25°, at least 2 reversals of the positive and negative yaw angles, a single gaze duration in a non-traveling direction exceeding 1.0s, and the person's dwell time exceeding 3s or the travel speed is less than 0.2m / s, the corresponding conditions will be used as rule evidence for the left-right looking behavior.

[0157] Determine whether each anomaly judgment condition corresponding to the candidate behavior is met, and generate the rule satisfaction degree S in the interval of 0 to 1 from the corresponding behavior template. template,b The recognition probability output by the behavior recognition model is weighted by the rule satisfaction level to form the behavior confidence score:

[0158] P b =αP model,b +(1−α)S template,b

[0159] In the formula, P b P represents the confidence level of behavior corresponding to behavior category b. model,b Let α be the recognition probability output by the behavior recognition model for behavior of type b, ranging from 0.6 to 0.8. When the confidence level of a behavior reaches the anomaly determination threshold corresponding to the candidate behavior, the candidate behavior is determined as an anomalous behavior. The anomaly determination thresholds corresponding to each behavior category are determined through validation set calibration.

[0160] After identifying abnormal behavior, the region and duration of the abnormal behavior are extracted, and historical abnormal situations corresponding to the same personnel target or the same region are extracted from historical abnormal events. A risk score is formed by weighting the behavior confidence level, rule satisfaction level, security sensitivity weight corresponding to the region of occurrence, normalized duration, and correction factors corresponding to historical abnormal situations.

[0161] R raw =0.45P b +0.20S rule+0.15W z +0.10T b +0.10H abn

[0162] In the formula, S rule S represents the normalized satisfaction of the rule evidence corresponding to the abnormal behavior. rule For the S template,b The standardized notation used in the risk scoring process is W. z T represents the security sensitivity weight of the area where abnormal behavior occurs. b H represents the result of normalizing the duration of abnormal behavior according to the corresponding duration range. abn This represents the normalization correction factor for the same personnel target or the same area's historical abnormal events; each value is between 0 and 1.

[0163] Risk score R raw As a basis for determining the risk level. When R raw When R < 0.35, the risk level is L0 (normal); when 0.35 ≤ R raw When R < 0.55, the risk level is L1 (Level of Concern); when R ≤ 0.55, the risk level is L1 (Level of Concern). raw When R < 0.75, the risk level is L2 warning; when 0.75 ≤ R raw When R < 0.90, the risk level is L3 alarm; raw When the value is ≥0.90, the risk level is L4 and mandatory measures are required.

[0164] The warning instruction corresponding to L1 is to record the abnormal behavior to the background and track the trend; the warning instruction corresponding to L2 is to push the abnormal behavior, its target identifier, the area where it occurred, and the confidence level of the behavior to the duty terminal for verification; the warning instruction corresponding to L3 is to trigger an audible and visual prompt and notify on-site personnel to handle the situation; the warning instruction corresponding to L4 is to send a high-priority blocking instruction to the channel control equipment and solidify the corresponding abnormal behavior event.

[0165] As a further method for determining aggregation behavior, when the behavior category in the behavior identification result is aggregation behavior, the ground coordinates of each person participating in the aggregation behavior are extracted from the target behavior sequence at different collection times, and the passage direction is determined according to the entrance and exit positions of the port clearance area.

[0166] The ground coordinates of each personnel target are projected onto the direction of travel, and the personnel targets are arranged according to the projected positions to form a personnel arrangement sequence corresponding to each acquisition time. The personnel arrangement sequences at adjacent acquisition times are compared, and the number of adjacent personnel target pairs whose arrangement order has not changed is counted. The ratio of this number to the total number of adjacent personnel target pairs is determined as the order preservation degree.

[0167] When the target person in front begins to move, the number of adjacent target persons moving along the direction of passage with the target person in front within the set response time is counted, and the ratio of this number to the total number of adjacent target persons is determined as the movement transitivity. When both the order maintenance rate and the movement transitivity reach the movement transitivity threshold, it indicates that the target persons maintain a relatively fixed queuing order and move sequentially with the target person in front. This aggregation behavior is determined as normal queuing behavior and is not considered abnormal behavior. Otherwise, this aggregation behavior is determined as abnormal aggregation behavior, and the risk level is determined and an early warning instruction is generated according to the aforementioned risk scoring method.

[0168] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for understanding and identifying anomalies in the behavior of port clearance personnel, characterized in that: The behavior understanding and anomaly identification method specifically includes: Step S1: Obtain multimodal monitoring data of the port clearance area, and perform spatiotemporal alignment on the multimodal monitoring data to form synchronous monitoring data; Step S2: Detect personnel targets from the synchronous monitoring data, and associate and aggregate the monitoring data corresponding to the same personnel target in different modalities and at different collection times to form multimodal behavioral data corresponding to each personnel target; Step S3: Extract the personnel behavior features corresponding to each modality from the multimodal behavior data, determine the fusion weight of each personnel behavior feature, and fuse the personnel behavior features according to the fusion weight to form the fused behavior features corresponding to each personnel target; Step S4: Arrange the fused behavioral features corresponding to each person's target in chronological order to form a target behavior sequence, and input the target behavior sequence into the behavior recognition model to form a behavior recognition result; Step S5: Determine the abnormal behavior based on the behavior recognition result, determine the risk level according to the risk classification conditions corresponding to the abnormal behavior, and generate the corresponding early warning instruction.

2. The method for understanding and identifying abnormal behavior of port clearance personnel according to claim 1, characterized in that: Step S1 specifically includes: Visible light acquisition equipment, infrared acquisition equipment, and millimeter-wave radar are used to collect visible light monitoring data, infrared monitoring data, and millimeter-wave monitoring data in the port clearance area, respectively, to form multimodal monitoring data; Multiple calibration points are set up on the ground in the port clearance area. Based on the ground coordinates of each calibration point and its image coordinates in visible light and infrared images, spatial transformation relationships between visible light monitoring data and infrared monitoring data and the same ground coordinate system are established respectively. Spatial transformation relationships between the millimeter-wave radar coordinate system and the ground coordinate system are established through the external parameter matrix. Based on the acquisition time of visible light monitoring data, infrared monitoring data and millimeter wave monitoring data that are closest to its acquisition time are searched within the same time window. When the difference between the acquisition times is within the time matching range, they are matched to form synchronous monitoring data.

3. The method for understanding and identifying abnormal behavior of port clearance personnel according to claim 2, characterized in that: Step S2 specifically includes: Personnel targets are detected from the visible light monitoring data and infrared monitoring data in the synchronous monitoring data respectively, and the reflection points in the millimeter wave monitoring data are clustered to form personnel detection results corresponding to each mode; the clustering is to group the reflection points corresponding to the same personnel target into the same target point set according to the position and movement speed of each reflection point in the millimeter wave monitoring data. Extract the personnel target positions from the personnel detection results corresponding to each modality, transform each personnel target position to the ground coordinate system according to the spatial transformation relationship, compare the target positions of the personnel detection results of each modality at the same acquisition time, associate the personnel detection results with the target positions with each other to the same personnel target, and set the corresponding target identifiers; By comparing the position and movement status of each person target at adjacent acquisition times, the corresponding person targets are temporally correlated and their corresponding target identifiers are continued. Based on the target identifier, monitoring data corresponding to the same person target at different modalities and different collection times are collected to form multimodal behavioral data corresponding to each person target.

4. The method for understanding and identifying abnormal behavior of port clearance personnel according to claim 2, characterized in that: Step S3 specifically includes: Human appearance features, human key point features, and head posture features are extracted from the multimodal behavioral data to form visible light human behavior features; human heat source features are extracted to form infrared human behavior features; and the position, movement speed, movement direction, and reflection intensity of the human target are extracted to form millimeter wave human behavior features. Illuminance, motion ambiguity, occlusion rate, and detection confidence are extracted from visible light monitoring data; heat source integrity, thermal noise, and thermal target confidence are extracted from infrared monitoring data; and signal-to-noise ratio, data continuity, and velocity consistency are extracted from millimeter-wave monitoring data. The indicators corresponding to each mode are weighted and summed according to set coefficients to form visible light quality score, infrared quality score, and millimeter-wave quality score, respectively. The visible light quality score, infrared quality score, and millimeter wave quality score are normalized to form fusion weights corresponding to each person's behavioral characteristics. Each person's behavioral characteristics are multiplied by their corresponding fusion weights and then concatenated to form initial fusion features. Cross-modal attention processing is performed on the visible light personnel behavior features, infrared personnel behavior features, and millimeter wave personnel behavior features to form cross-modal correlation features. The cross-modal correlation features are then residually fused with the initial fusion features to form fused behavior features corresponding to each personnel target.

5. The method for understanding and identifying abnormal behavior of port clearance personnel according to claim 1, characterized in that: Step S4 specifically includes: Arrange the fused behavioral features of the same person at different collection times in chronological order to form a target behavior sequence; The target behavior sequence is divided into multiple behavior sequence segments according to the set time length and sliding step size; Each behavior sequence segment is input into the behavior recognition model. The fused behavior features at continuously collected moments in the behavior sequence segment are analyzed to form short-term action change features. The continuous change relationship between each short-term action change feature is analyzed in chronological order to form the recognition probability corresponding to each behavior category. The behavior category with the highest recognition probability and its recognition probability are determined as the behavior recognition result.

6. The method for understanding and identifying abnormal behavior of port clearance personnel according to claim 1, characterized in that: Step S5 specifically includes: The behavior category and its corresponding recognition probability are extracted from the behavior recognition result. When the behavior category is not a normal passage behavior, the behavior category is determined as a candidate behavior, and the abnormal judgment condition corresponding to the candidate behavior is retrieved. Extract the target position, motion state, human posture, and behavior duration corresponding to each anomaly determination condition from the multimodal behavior data and target behavior sequence, and determine the satisfaction of each anomaly determination condition to form a rule satisfaction degree; The recognition probability and rule satisfaction are weighted and summed according to a set coefficient to form the behavior confidence score; when the behavior confidence score reaches the anomaly judgment threshold corresponding to the candidate behavior, the candidate behavior is determined to be an abnormal behavior. The location and duration of the abnormal behavior are determined. Historical abnormal situations corresponding to the same person or location are extracted from historical abnormal events. The confidence level of the behavior, the rule satisfaction level, and the scores corresponding to the location, duration, and historical abnormal situations are weighted and summed to form a risk score. The risk score is compared with the risk grading conditions to determine the corresponding risk level; when the risk level is normal, the candidate behavior is determined as normal behavior; when the risk level is abnormal, the candidate behavior is determined as abnormal behavior, and a corresponding warning instruction is generated.

7. The method for understanding and identifying anomalies in the behavior of port clearance personnel according to claim 1, characterized in that, When the behavior category in the behavior recognition result is clustering behavior, the determination of the abnormal behavior specifically includes: Extract the target positions of each person target corresponding to the aggregation behavior at different collection times from the target behavior sequence, and determine the passage direction according to the entrance and exit positions of the port clearance area; Project the target positions of each person onto the direction of travel, and arrange the person targets according to the projection positions to form a personnel arrangement sequence corresponding to each collection time. Compare the personnel arrangement sequence at adjacent acquisition times, and determine the order preservation degree by the ratio of the number of adjacent personnel target pairs whose arrangement order has not changed to the total number of adjacent personnel target pairs; when the personnel target in front starts to move, determine the movement transfer degree by the ratio of the number of adjacent personnel target pairs that move with the personnel target in front within a set response time to the total number of adjacent personnel target pairs. When the order preservation degree reaches the order preservation threshold and the movement transfer degree reaches the movement transfer threshold, the aggregation behavior is determined as normal queuing behavior; otherwise, the aggregation behavior is determined as abnormal aggregation behavior and is treated as the abnormal behavior.

8. A port clearance personnel behavior understanding and anomaly identification system, applied to the port clearance personnel behavior understanding and anomaly identification method described in any one of claims 1-7, characterized in that: The behavior understanding and anomaly recognition system includes a multimodal synchronization module, a personnel data collection module, a behavior feature fusion module, a behavior recognition module, and an anomaly judgment and early warning module; The multimodal synchronization module acquires multimodal monitoring data of the port clearance area, performs spatiotemporal alignment on the multimodal monitoring data, and forms synchronized monitoring data. The personnel data collection module detects personnel targets from the synchronous monitoring data, and associates and collects monitoring data corresponding to the same personnel target in different modalities and at different collection times to form multimodal behavioral data corresponding to each personnel target; The behavior feature fusion module extracts the personnel behavior features corresponding to each modality from the multimodal behavior data, determines the fusion weight of each personnel behavior feature, and fuses the personnel behavior features according to the fusion weight to form the fused behavior features corresponding to each personnel's target. The behavior recognition module arranges the fused behavior features corresponding to each person's target in chronological order to form a target behavior sequence, and inputs the target behavior sequence into the behavior recognition model to form a behavior recognition result; The anomaly detection and early warning module determines the abnormal behavior based on the behavior recognition result, determines the risk level according to the risk classification conditions corresponding to the abnormal behavior, and generates a corresponding early warning instruction.

9. A port clearance personnel behavior understanding and anomaly identification system according to claim 8, characterized in that: The behavioral feature fusion module includes a feature extraction and evaluation unit and a feature association and fusion unit; The feature extraction and evaluation unit extracts visible light human behavior features, infrared human behavior features, and millimeter wave human behavior features from the multimodal behavior data, and extracts the quality indicators corresponding to each mode. The quality indicators are weighted and summed according to a set coefficient to form the corresponding quality score. The feature association fusion unit normalizes the quality scores corresponding to each modality to form fusion weights for each person's behavioral features. It then multiplies each person's behavioral features with its corresponding fusion weights and concatenates them to form initial fusion features. Finally, it performs residual fusion with the cross-modal association features formed by cross-modal attention processing and the initial fusion features to form fusion behavioral features.

10. A port clearance personnel behavior understanding and anomaly identification system according to claim 8, characterized in that: The anomaly detection and early warning module includes an anomaly verification unit and a risk classification unit; The anomaly verification unit extracts the behavior category and its corresponding recognition probability from the behavior recognition result. When the behavior category is not a normal passage behavior, it retrieves the anomaly judgment condition corresponding to the behavior category, extracts the data corresponding to each anomaly judgment condition from the multimodal behavior data and the target behavior sequence, judges the satisfaction of each anomaly judgment condition, forms the rule satisfaction degree, and weights and sums the recognition probability and the rule satisfaction degree according to the set coefficient to form the behavior confidence degree, and determines the abnormal behavior based on the behavior confidence degree. The risk grading unit combines the behavioral confidence, rule satisfaction, occurrence area, duration, and historical anomalies of the abnormal behavior to form a risk score. The risk score is then compared with the risk grading conditions to determine the corresponding risk level. When the risk level is normal, the candidate behavior is determined as normal behavior; When the risk level is an abnormal level, the candidate behavior is identified as an abnormal behavior, and a corresponding warning instruction is generated.

Images