Secret room abnormal behavior detection method and system based on multi-channel feature fusion

Abstract

The present application relates to the technical field of computer vision, in particular to a secret room abnormal behavior detection method and system based on multi-channel feature fusion. The method comprises: acquiring secret room video frame data, locking target personnel through target detection and tracking algorithm; extracting multi-channel features, including global appearance features, key point features, time sequence dynamic features and local area abnormal features; based on a metric learning framework, training a feature mapping network using a triplet loss function to construct a discrimination boundary between normal behavior and abnormal behavior; and performing consistency measurement on the behavior features of the same person at different time points to realize abnormality determination by calculating the Euclidean distance in the feature space. The present application models behavior features in all directions through a four-channel parallel feature extraction architecture, effectively detects unknown types of abnormalities by introducing an open set recognition mechanism, and breaks through the limitations of single-frame detection by combining the consistency modeling of the same person, thereby significantly improving the accuracy and robustness of abnormal behavior detection in the secret room scene, reducing the false positive rate to below 1%, and having the generalization detection ability for new types of abnormal behavior.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for detecting abnormal behavior in secure rooms based on multi-channel feature fusion. Background Technology

[0002] Anomaly detection technology in secure examination rooms has become a core component of the educational examination security system and a key technological means to ensure the fairness and impartiality of examinations. The anomaly detection system for secure examination rooms integrates multiple advanced technologies, including intelligent video analysis, behavioral pattern recognition, identity verification, and environmental monitoring, constructing a 24 / 7, comprehensive security network for examination papers. This system utilizes specialized equipment such as high-definition intelligent cameras, personnel identification terminals, access control systems, and environmental sensors deployed both inside and outside the secure room to form an intelligent monitoring system covering the entire process of examination paper storage, transportation, and distribution. Based on deep learning algorithms, the system can monitor the activities of personnel within the secure room in real time, automatically identifying and issuing warnings for various violations and anomalies, including but not limited to unauthorized entry, abnormal gatherings of personnel, abnormal contact with examination paper bags, prolonged solo work, unauthorized carrying of electronic devices, and abnormal filming behavior. Simultaneously, the system uses behavioral sequence analysis technology to intelligently learn from historical monitoring data, establishing anomaly recognition models and achieving a significant shift from simple post-event evidence collection to real-time early warning and prevention. Compared to the traditional manual monitoring and post-event review model, the abnormal behavior detection technology in the exam paper security room greatly improves the continuity and accuracy of supervision, effectively makes up for the time blind spots and subjective limitations of manual supervision, and significantly reduces the risk of exam paper leakage.

[0003] However, existing technologies still face the following pressing technical problems in practical applications:

[0004] 1. Insufficient feature representation capability: Existing systems mostly rely on a single modality (such as RGB images or 2D poses) for behavior recognition, which makes it difficult to comprehensively characterize human behavior features in complex scenes. A single feature source cannot simultaneously capture multi-dimensional information such as global appearance, local structure, temporal dynamics, and fine-grained regional anomalies, resulting in weak recognition capabilities for subtle abnormal behaviors such as "abnormally carrying suspicious items" and "concealed operation of electronic devices," leading to a high false negative rate.

[0005] 2. Lack of Open-Set Recognition Capability: Traditional behavior recognition models are mostly closed-set classification models, capable of recognizing only pre-defined abnormal categories in the training set (such as "using a mobile phone" or "smoking"). When faced with novel abnormal behaviors (such as unseen ways of carrying items, disguise behaviors, or the use of new electronic devices), the model, lacking generalization ability, is prone to misclassifying them as normal behavior, resulting in serious false negatives. In secure room scenarios, abnormal behavior types are diverse and constantly evolving, making the closed-set recognition paradigm unsuitable for practical needs.

[0006] 3. High false alarm rate and poor robustness: In complex scenarios such as changes in lighting, occlusion, multi-person interaction, and camera viewpoint shift, the model is prone to misclassifying normal behavior as abnormal, especially in the detection of key areas (such as hands, waist, and shoulders), where the false alarm rate remains high. Traditional methods lack temporal modeling of the consistency of behavior of the same person, and cannot effectively distinguish between "temporary changes in normal behavior" and "real abnormal behavior," which seriously affects the availability of the system and the trust of supervisory personnel. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a method and system for detecting abnormal behavior in secure rooms based on multi-channel feature fusion. By enhancing expressive power through multi-dimensional feature fusion, introducing metric learning to achieve open set recognition, and constructing a consistent model of the same person to overcome temporal limitations, the method significantly improves the accuracy and generalization ability of abnormal behavior detection in complex environments.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] On one hand, embodiments of the present invention provide a method for detecting abnormal behavior in a secure room based on multi-channel feature fusion, the method comprising the following steps:

[0010] S100: Acquire a video sequence of the secure room scene, detect and track personnel in the video sequence, and obtain a sequence of images of personnel with the same tracking ID;

[0011] S200, extract multi-channel features from the personnel image sequence, and merge and splice the multi-channel features to form fusion features of personnel behavior; wherein, each channel corresponds to a feature type and outputs a feature vector of the same dimension;

[0012] S300: Obtain the fusion features of personnel with the same tracking ID at different time points, calculate the Euclidean distance between the fusion features in the constructed feature discrimination space, and output the abnormal behavior detection result based on the comparison result of the Euclidean distance and the preset threshold.

[0013] Optionally, in S200, the step of extracting multi-channel features from the personnel image sequence and merging and splicing the multi-channel features to form fusion features of personnel behavior includes:

[0014] S210, preprocess the single-frame images in the personnel image sequence, obtain the high-level semantic features of the images through a pre-trained feature extraction network, and map the high-level semantic features into a global appearance feature vector of a preset dimension through a multilayer perceptron.

[0015] S220, perform human key point detection on a single frame image in the personnel image sequence to obtain the coordinates and confidence of each joint of the human body, draw a human skeleton diagram based on the coordinates and confidence, input the human skeleton diagram into a pre-trained classification model for feature extraction, and map the extracted features into a key point feature vector of a preset dimension through a multilayer perceptron.

[0016] S230, the personnel image sequence is preprocessed to obtain a fixed-length video segment, and the video segment is used to extract features through a pre-trained three-dimensional convolutional network. The extracted features are then mapped into a temporal interaction feature vector of a preset dimension through a multilayer perceptron.

[0017] S240 determines the key detection areas of the hand, waist and shoulder based on the human body key point detection results, extracts features from each key detection area, fuses the extracted features of each area, and maps the fused features into a local area anomaly feature vector of preset dimension through a multilayer perceptron.

[0018] S250, the global appearance feature vector, key point feature vector, temporal interaction feature vector and local area anomaly feature vector are concatenated to form personnel behavior fusion features.

[0019] Optionally, in S210, the preprocessing of single-frame images in the personnel image sequence, obtaining high-level semantic features of the images through a pre-trained feature extraction network, and mapping the high-level semantic features into a global appearance feature vector of a preset dimension through a multilayer perceptron, includes:

[0020] S211, perform size normalization and pixel value standardization processing on the single frame images in the personnel image sequence;

[0021] S212, input the preprocessed single-frame image into the pre-trained ResNet50 network to extract the high-level semantic features output by the fully connected layer;

[0022] S213, the high-level semantic features are input into a multilayer perceptron, processed by linear mapping and ReLU activation function, and output a global appearance feature vector of a preset dimension.

[0023] Optionally, in S220, the step of detecting human key points in a single frame of the personnel image sequence to obtain the coordinates and confidence scores of each joint of the human body includes:

[0024] S221, The HRNet model is used to detect human key points in a single frame of the human image sequence, and the normalized coordinates of each joint of the human body and the corresponding confidence score are output.

[0025] S222, Draw a human skeleton diagram based on the coordinates and confidence level, input the human skeleton diagram into a ResNet-18 model trained on the human behavior skeleton diagram dataset of the confidential room, and extract the initial features of key points of a preset dimension.

[0026] S223, the initial features of the key points are mapped to key point feature vectors of a preset dimension using a multilayer perceptron.

[0027] Optionally, in S230, the preprocessing of the personnel image sequence to obtain a fixed-length video segment, the feature extraction of the video segment through a pre-trained 3D convolutional network, and the mapping of the extracted features into a temporal interaction feature vector of a preset dimension through a multilayer perceptron include:

[0028] S231, extract multiple consecutive frames of images from the personnel image sequence as video segments, and normalize the size of each frame of the video segment to a preset resolution;

[0029] S232, The video clip is input into a pre-trained C3D network, and spatiotemporal features are extracted through three-dimensional convolutional layers and pooling layers to output temporal initial features of a preset dimension;

[0030] S233, the initial temporal features are input into a multilayer perceptron, and through linear mapping and batch normalization processing, a temporal interaction feature vector of a preset dimension is output.

[0031] Optionally, in S240, the step of determining key detection areas for the hand, waist, and shoulder based on the human body key point detection results, extracting features from each key detection area, fusing the extracted features from each area, and mapping the fused features to a local region anomaly feature vector of a preset dimension through a multilayer perceptron includes:

[0032] S241, Perform human key point detection on a single frame image in the personnel image sequence to obtain the coordinates of 17 key points defined in the COCO dataset;

[0033] S242, determine the target key detection area based on the coordinates of the corresponding key points. The left wrist area is the bounding box centered on the left wrist, the right wrist area is the bounding box centered on the right wrist, the waist area is the bounding box centered on the average coordinates of the left hip and the right hip, and the shoulder area is the bounding box centered on the average coordinates of the left shoulder and the right shoulder.

[0034] S243, crop the target key detection regions from the personnel image, input the target key detection regions into the pre-trained ResNet18 network, and extract the local features of each target key detection region; wherein, the ResNet18 network is a model trained with manually labeled data from the confidential room, and the categories of items carried by the manually labeled personnel include no items, handbag, folder, cardboard box and test paper bag;

[0035] S244: Max pooling fusion is performed on the local features of the key detection areas of each target to obtain regional anomaly features;

[0036] S245, input the regional anomaly features into the multilayer perceptron and output a local regional anomaly feature vector of a preset dimension.

[0037] Optionally, in S300, the constructed feature discrimination space is constructed through the following steps:

[0038] S310, Collect personnel behavior sample data in the secure room scenario. The sample data includes normal behavior samples and abnormal behavior samples of the same person, wherein abnormal behavior samples include behavior samples of carrying a mobile phone and using a computer.

[0039] S320, Select anchor samples, positive samples, and negative samples from the sample data to construct a triplet training dataset. The triplet training dataset includes two construction modes. In mode one, the anchor samples are the normal behavior features of a person at a certain moment, the positive samples are the normal behavior features of the same person in another frame, and the negative samples are the features of the abnormal behavior category of the same person. In mode two, the anchor samples are a certain type of abnormal behavior of a person, the positive samples are the same type of abnormal behavior of the same person, and the negative samples are another type of abnormal behavior or normal behavior of the same person.

[0040] S330, the samples in the triplet training dataset are input into the multilayer perceptron encoder, and the encoder parameters are updated by minimizing the triplet loss function until the model converges, thus obtaining the feature discrimination space; wherein, the triplet loss function is... f(·) represents the feature mapping function, a represents the anchor sample, p represents the positive sample, and n represents the negative sample. This represents the Euclidean distance, and margin is the boundary parameter.

[0041] Optionally, in S300, the step of acquiring the fused features of individuals with the same tracking ID at different time points, calculating the Euclidean distance between the fused features in the constructed feature discrimination space, and outputting the abnormal behavior detection result based on the comparison result of the Euclidean distance and a preset threshold includes:

[0042] S310, obtain the fusion features of people with the same tracking ID at the time of entry and exit;

[0043] S320, calculate the Euclidean distance between the fused features at the time of entry and the fused features at the time of exit;

[0044] S330, compare the Euclidean distance with a preset threshold. If the Euclidean distance is greater than the preset threshold, output the abnormal behavior detection result; if the Euclidean distance is less than or equal to the preset threshold, output the normal behavior detection result.

[0045] On the other hand, embodiments of the present invention provide a system for detecting abnormal behavior in a secure room based on multi-channel feature fusion, comprising:

[0046] At least one processor;

[0047] At least one memory for storing at least one program;

[0048] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0049] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method.

[0050] The beneficial effects of this invention are as follows: This invention discloses a method and system for detecting abnormal behavior in secure rooms based on multi-channel feature fusion. By innovatively constructing a four-channel parallel feature extraction architecture, it comprehensively models human behavior features from four dimensions: global appearance, key point structure, temporal dynamics, and local regional anomalies, significantly enhancing feature representation capabilities. It introduces a triplet loss training framework based on metric learning, constructing an open-set recognition mechanism that does not require predefining all anomaly categories, effectively improving the generalization detection capability for unknown abnormal behaviors. It proposes a method for modeling consistency of human behavior, breaking through the limitations of traditional single-frame detection through temporal feature alignment and Euclidean distance calculation, and accurately identifying "behavioral mutation" anomalies. It designs a ROI local feature extraction and max-pooling fusion mechanism for high-risk areas in secure rooms, enhancing the perception accuracy of subtle abnormal behaviors. Compared with traditional closed-set classification methods, this invention achieves deep fusion of multimodal features, clear open-set discrimination boundaries, and explicit modeling of temporal consistency, providing a high-precision, low-false-alarm, and scalable intelligent monitoring solution for high-risk locations such as secure rooms, and has significant application value in fields such as intelligent examination supervision and archive security. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is an overall flowchart of a method for detecting abnormal behavior in a secure room based on multi-channel feature fusion, provided in an embodiment of the present invention.

[0053] Figure 2 This is an overall framework diagram of a confidential room abnormal behavior detection system based on multi-channel feature fusion, provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0055] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0056] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0058] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0059] The core idea of ​​this invention is to comprehensively extract behavioral features without relying on single-modal features or closed-set classification paradigms. Instead, it introduces metric learning to construct an open-set discrimination space and combines this with temporal consistency modeling of the same person to achieve accurate anomaly detection. Specifically, this invention extracts four-dimensional features from video sequences, uses triplet loss training to cluster features of similar samples and separate dissimilar samples, and identifies abrupt behavioral anomalies by calculating feature distances between the same person at different times during the inference phase.

[0060] refer to Figure 1 ,like Figure 1 The figure shown is an embodiment of the present invention providing a method for detecting abnormal behavior in a secure room based on multi-channel feature fusion. The method includes the following steps:

[0061] S100: Acquire a video sequence of the secure room scene, detect and track personnel in the video sequence, and obtain a sequence of images of personnel with the same tracking ID;

[0062] In one specific embodiment, YOLOv8 is used as the target detector. A 1920×1080 resolution surveillance image of a secure room is input, and pedestrian bounding boxes are detected, with a detection result having a confidence level greater than 0.5 output. Subsequently, the ByteTrack tracking algorithm is used, based on IoU matching and feature similarity of the bounding boxes, to associate the same person in different frames and assign a unique tracking ID. By extracting image regions of people with the same tracking ID in each frame, a temporally continuous image sequence is formed, providing basic data for subsequent time-series analysis.

[0063] S200, extract multi-channel features from the personnel image sequence, and merge and splice the multi-channel features to form fusion features of personnel behavior; wherein, each channel corresponds to a feature type and outputs a feature vector of the same dimension;

[0064] S300: Obtain the fusion features of personnel with the same tracking ID at different time points, calculate the Euclidean distance between the fusion features in the constructed feature discrimination space, and output the abnormal behavior detection result based on the comparison result of the Euclidean distance and the preset threshold.

[0065] In the embodiments provided by this invention, a multi-channel feature extraction and fusion architecture is used to achieve multi-dimensional characterization of human behavior, specifically including a global appearance feature channel, a human body key point feature channel, a temporal interaction feature channel, and a local region anomaly feature channel. Each channel is processed in parallel through independent feature extraction networks, and the final output feature vectors with uniform dimensions are concatenated and fused to form a comprehensive feature representation that can fully reflect the state of human behavior.

[0066] In some embodiments, S200, the step of extracting multi-channel features from the personnel image sequence and merging and splicing the multi-channel features to form fusion features of personnel behavior includes:

[0067] S210, preprocess the single-frame images in the personnel image sequence, obtain the high-level semantic features of the images through a pre-trained feature extraction network, and map the high-level semantic features into a global appearance feature vector of a preset dimension through a multilayer perceptron.

[0068] S220, perform human key point detection on a single frame image in the personnel image sequence to obtain the coordinates and confidence of each joint of the human body, draw a human skeleton diagram based on the coordinates and confidence, input the human skeleton diagram into a pre-trained classification model for feature extraction, and map the extracted features into a key point feature vector of a preset dimension through a multilayer perceptron.

[0069] S230, the personnel image sequence is preprocessed to obtain a fixed-length video segment, and the video segment is used to extract features through a pre-trained three-dimensional convolutional network. The extracted features are then mapped into a temporal interaction feature vector of a preset dimension through a multilayer perceptron.

[0070] S240 determines the key detection areas of the hand, waist and shoulder based on the human body key point detection results, extracts features from each key detection area, fuses the extracted features of each area, and maps the fused features into a local area anomaly feature vector of preset dimension through a multilayer perceptron.

[0071] S250, the global appearance feature vector, key point feature vector, temporal interaction feature vector and local area anomaly feature vector are concatenated to form personnel behavior fusion features.

[0072] This embodiment achieves a comprehensive description of human behavior from static to dynamic and from overall to local dimensions through the synergistic effect of four-channel features. Global appearance features capture macroscopic information such as overall clothing and body shape, providing basic visual cues for behavior recognition; human keypoint features focus on joint positional relationships, reflecting subtle differences in posture through changes in skeletal structure; temporal interaction features utilize 3D convolutional networks to model dynamic changes in video clips, effectively capturing the continuity of human behavior over time; and local anomaly features, through ROI cropping and pre-trained models, specifically enhance the detection of item carrying status in high-risk areas (hands, waist, shoulders) in secure room scenarios. This multimodal feature fusion strategy avoids the limitations of single-feature representation, enabling the fused feature vector to simultaneously encompass multidimensional information such as appearance, structure, dynamics, and local anomalies, laying a solid foundation for the accurate identification of subsequent open-set abnormal behaviors.

[0073] In some embodiments, S210, the preprocessing of single-frame images in the personnel image sequence, obtaining high-level semantic features of the images through a pre-trained feature extraction network, and mapping the high-level semantic features into a global appearance feature vector of a preset dimension through a multilayer perceptron, includes:

[0074] S211, perform size normalization and pixel value standardization processing on the single frame images in the personnel image sequence;

[0075] S212, input the preprocessed single-frame image into the pre-trained ResNet50 network to extract the high-level semantic features output by the fully connected layer;

[0076] S213, the high-level semantic features are input into a multilayer perceptron, processed by linear mapping and ReLU activation function, and output a global appearance feature vector of a preset dimension.

[0077] This embodiment utilizes the deep convolutional structure of the ResNet50 network to effectively capture global visual information such as texture, color, and contours in images. Specifically, size normalization adjusts the image to a uniform 224×224 pixels, ensuring consistent image size input to the network; pixel value standardization uses the mean and standard deviation of the ImageNet dataset (mean=[0.485,0.456,0.406], standard deviation=[0.229,0.224,0.225]), accelerating network convergence and improving feature extraction stability. The ResNet50 network, through the alternating action of 50 convolutional and pooling layers, gradually transforms the original pixel information into feature vectors with high-level semantics. The 2048-dimensional feature vector output by its last fully connected layer contains rich information about the overall appearance of the person. Subsequently, the multilayer perceptron (MLP) consists of two fully connected layers. The first layer maps 2048-dimensional features to 1024-dimensional features and introduces a non-linear transformation through the ReLU activation function. The second layer further maps to a preset 256-dimensional global appearance feature vector, reducing the feature dimension while retaining key information, thus preparing for subsequent multi-channel feature stitching.

[0078] In some embodiments, S220, the step of performing human keypoint detection on a single frame image in the personnel image sequence to obtain the coordinates and confidence scores of each joint of the human body includes:

[0079] S221, The HRNet model is used to detect human key points in a single frame of the human image sequence, and the normalized coordinates of each joint of the human body and the corresponding confidence score are output.

[0080] S222, Draw a human skeleton diagram based on the coordinates and confidence level, input the human skeleton diagram into a ResNet-18 model trained on the human behavior skeleton diagram dataset of the confidential room, and extract the initial features of key points of a preset dimension.

[0081] S223, the initial features of the key points are mapped to key point feature vectors of a preset dimension using a multilayer perceptron.

[0082] This embodiment utilizes the high-resolution feature representation capabilities of the HRNet model to achieve precise localization of 17 key joints in the human body (including the head, neck, shoulder, elbow, wrist, hip, knee, and ankle). Specifically, the HRNet model maintains high-resolution feature representation throughout the network process by parallelly connecting feature maps of multiple resolutions, thereby outputting the normalized (x,y) coordinates (range [0,1]) and confidence score (range [0,1]) of each joint. For joints with a confidence score below 0.3, interpolation based on neighboring frames is used for completion to ensure the integrity of the skeleton structure. Subsequently, based on the detected joint coordinates, a human skeleton image is drawn on a blank canvas: circles with a radius of 3 pixels are drawn with the joint as the center, and skeleton lines with a width of 2 pixels are drawn by connecting adjacent joints (such as shoulder-elbow, elbow-wrist), generating a grayscale skeleton image with a size of 224×224 pixels. This skeleton image only retains the structural information of human movement, effectively filtering out irrelevant interference factors such as clothing and background. To enhance the model's adaptability to human behavior in specific scenarios within a secure room, the drawn skeleton diagrams are input into a ResNet-18 model fine-tuned using a human behavior skeleton diagram dataset for secure rooms. This fine-tuning process uses a skeleton diagram dataset containing normal behaviors (such as standing, walking, and reviewing documents) and typical abnormal behaviors (such as climbing, bending over to hide, and passing items). Transfer learning enables the ResNet-18 model to better capture the structural features of behavior in secure room scenarios. After the ResNet-18 model outputs initial 512-dimensional keypoint features, these are then mapped to 256-dimensional keypoint feature vectors using a multilayer perceptron with one hidden layer (256-dimensional, ReLU activation), maintaining consistency with the global appearance feature vector dimension for subsequent fusion.

[0083] In some embodiments, S230, the preprocessing of the personnel image sequence to obtain a fixed-length video segment, the feature extraction of the video segment through a pre-trained 3D convolutional network, and the mapping of the extracted features into a temporal interaction feature vector of a preset dimension through a multilayer perceptron, includes:

[0084] S231, extract multiple consecutive frames of images from the personnel image sequence as video segments, and normalize the size of each frame of the video segment to a preset resolution;

[0085] S232, The video clip is input into a pre-trained C3D network, and spatiotemporal features are extracted through three-dimensional convolutional layers and pooling layers to output temporal initial features of a preset dimension;

[0086] S233, the initial temporal features are input into a multilayer perceptron, and through linear mapping and batch normalization processing, a temporal interaction feature vector of a preset dimension is output.

[0087] This embodiment effectively captures the dynamic changes in human behavior in video clips through the 3D convolution operation of the C3D network (convolving both spatial and temporal dimensions simultaneously). Specifically, a video clip is composed of 16 consecutive frames centered on the current frame, extracted from the image sequence of the person corresponding to the tracking ID. If the sequence has fewer than 16 frames at the beginning or end, it is padded by repeating the first or last frame to ensure a fixed clip length. Each frame is normalized to 112×112 pixels, and the spatiotemporal dimensions of the video clip are uniformly 16×112×112×3 (number of frames × height × width × number of channels). The pre-trained C3D network is initialized on the Sports-1M dataset. Its network structure includes 8 3D convolutional layers, 5 3D pooling layers, and 2 fully connected layers, extracting hierarchical dynamic features by progressively reducing the spatiotemporal resolution. The 3D convolutional kernels (e.g., 3×3×3) can simultaneously perceive the continuity of actions between adjacent frames, such as the changing trends of actions like reaching out or turning over on the time axis. The final fully connected layer of the C3D network outputs a 4096-dimensional temporal initial feature, which contains dynamic interaction information about human behavior in the video clip. To unify the dimensionality with other channel features, the 4096-dimensional temporal initial feature is input into a multilayer perceptron: first, it is mapped to 256 dimensions through a linear layer; then, batch normalization is performed to accelerate training convergence and prevent overfitting; finally, it is activated by a ReLU function to output a 256-dimensional temporal interaction feature vector. This process not only achieves feature dimensionality reduction but also enhances the discriminative power of the features through nonlinear transformation, enabling the temporal interaction features to effectively complement other static features.

[0088] In some embodiments, S240, the process of determining key detection regions for the hand, waist, and shoulder based on human key point detection results, extracting features from each key detection region, fusing the extracted features, and mapping the fused features to a local region anomaly feature vector of a preset dimension using a multilayer perceptron includes:

[0089] S241, Perform human key point detection on a single frame image in the personnel image sequence to obtain the coordinates of 17 key points defined in the COCO dataset;

[0090] S242, determine the target key detection area based on the coordinates of the corresponding key points. The left wrist area is the bounding box centered on the left wrist, the right wrist area is the bounding box centered on the right wrist, the waist area is the bounding box centered on the average coordinates of the left hip and the right hip, and the shoulder area is the bounding box centered on the average coordinates of the left shoulder and the right shoulder.

[0091] S243, crop the target key detection regions from the personnel image, input the target key detection regions into the pre-trained ResNet18 network, and extract the local features of each target key detection region; wherein, the ResNet18 network is a model trained with manually labeled data from the confidential room, and the categories of items carried by the manually labeled personnel include no items, handbag, folder, cardboard box and test paper bag;

[0092] S244: Max pooling fusion is performed on the local features of the key detection areas of each target to obtain regional anomaly features;

[0093] S245, input the regional anomaly features into the multilayer perceptron and output a local regional anomaly feature vector of a preset dimension.

[0094] In the implementation process, real-time video streams of personnel entering the secure room are first captured. Background modeling and foreground extraction techniques are used to initially locate the areas where personnel are moving. Subsequently, a deep learning-based target detection algorithm (such as YOLOv5) is employed to detect personnel in the captured video frames, obtaining the bounding box coordinates of each person. This is then combined with a Kalman filter algorithm to achieve continuous tracking of multiple personnel, providing stable target regions for subsequent feature extraction. During personnel tracking, if a new person enters or an existing person leaves the scene, the system automatically updates the tracking list to ensure that the movement trajectory of each person is accurately recorded. For each tracked person, images are captured at a preset sampling frequency and used as input data for subsequent multi-channel feature extraction, thus laying the foundation for the entire abnormal behavior detection process.

[0095] In some embodiments, in S300, the constructed feature discrimination space is constructed through the following steps:

[0096] S310, Collect personnel behavior sample data in the secure room scenario. The sample data includes normal behavior samples and abnormal behavior samples of the same person, wherein abnormal behavior samples include behavior samples of carrying a mobile phone and using a computer.

[0097] S320, Select anchor samples, positive samples, and negative samples from the sample data to construct a triplet training dataset. The triplet training dataset includes two construction modes. In mode one, the anchor samples are the normal behavior features of a person at a certain moment, the positive samples are the normal behavior features of the same person in another frame, and the negative samples are the features of the abnormal behavior category of the same person. In mode two, the anchor samples are a certain type of abnormal behavior of a person, the positive samples are the same type of abnormal behavior of the same person, and the negative samples are another type of abnormal behavior or normal behavior of the same person.

[0098] S330, the samples in the triplet training dataset are input into the multilayer perceptron encoder, and the encoder parameters are updated by minimizing the triplet loss function until the model converges, thus obtaining the feature discrimination space; wherein, the triplet loss function is... f(·) represents the feature mapping function, a represents the anchor sample, p represents the positive sample, and n represents the negative sample. This represents the Euclidean distance, and margin is the boundary parameter.

[0099] This embodiment uses a triplet loss function to drive the encoder to learn discriminative feature maps, enabling features of the same type of behavior to cluster in the discriminative space while separating features of different types of behavior. Specifically, the sample data collection covers common normal behaviors in daily work scenarios in secure rooms (such as sitting quietly at work, reviewing paper documents, and moving around in designated areas) and preset open-set abnormal behaviors (such as carrying a mobile phone to take pictures, connecting to an external computer, opening a secure cabinet without authorization, and staying in a blind spot of monitoring for more than a threshold time). Each behavior category contains at least 500 video clip samples from people of different genders, ages, and clothing to enhance the model's generalization ability. The construction of the triplet training dataset adopts a dynamic sampling strategy: Mode 1 is used to strengthen the intra-class consistency of normal behaviors, ensuring that the distance between the normal behavior features of the same person at different times and in different postures is as small as possible, and that there is at least a margin (experimentally set to 1.0) between them and their own abnormal behavior features; Mode 2 targets the subcategories of abnormal behaviors (such as mobile phone shooting divided into handheld shooting, pocket shooting, etc.), and improves the ability to distinguish unknown abnormal behaviors by aggregating the features of samples of the same type of abnormal behavior. The encoder employs a three-layer fully connected network structure. The input is a 256×4=1024-dimensional fusion feature set of personnel behavior. The first layer maps to 512 dimensions, the second to 256 dimensions, and the third outputs a 128-dimensional discriminant space feature vector. Each layer uses the LeakyReLU activation function (slope 0.1) and a dropout layer (probability 0.3) to prevent overfitting. During training, the Adam optimizer is used (initial learning rate 0.001, decaying by 10% every 10 epochs), with a batch size of 64. Training is iterated for 200 epochs, stopping when the validation set triplet loss decreases by less than 1e-4 for 10 consecutive epochs. The final 128-dimensional feature discriminant space maps personnel behavior features to vector representations with clear class boundaries, providing a quantifiable feature distance metric for open-set anomaly detection.

[0100] In some embodiments, S300, the step of acquiring the fused features of individuals with the same tracking ID at different time points, calculating the Euclidean distance between the fused features in the constructed feature discrimination space, and outputting the abnormal behavior detection result based on the comparison result of the Euclidean distance and a preset threshold includes:

[0101] S310, obtain the fusion features of people with the same tracking ID at the time of entry and exit;

[0102] S320, calculate the Euclidean distance between the fused features at the time of entry and the fused features at the time of exit;

[0103] S330, compare the Euclidean distance with a preset threshold. If the Euclidean distance is greater than the preset threshold, output the abnormal behavior detection result; if the Euclidean distance is less than or equal to the preset threshold, output the normal behavior detection result.

[0104] In practical applications, the movement trajectory of personnel within the secure room is tracked in real time. When a person completes their operation and prepares to leave the secure room, the extraction of fused features at the exit moment is automatically triggered. The fused features at the entry moment are collected and stored when a person passes through the secure room entrance detection area, ensuring that they represent the complete behavioral cycle features of the same person. The Euclidean distance is calculated based on the L2 norm of two 128-dimensional feature vectors in the feature discrimination space. The preset threshold is determined through statistical analysis of a large amount of experimental data: on a test set containing 1000 groups of normal personnel entering and exiting (without abnormal behavior) and 500 groups of personnel exhibiting abnormal behavior (such as carrying items in and out, or performing unauthorized operations), ROC curve analysis determined the optimal threshold to be 1.2. At this threshold, the system's accuracy reaches 95.3%, and the false alarm rate is controlled below 3.2%. If the calculated Euclidean distance is greater than 1.2, the system determines that the person has an abnormal behavior risk in the confidential room, immediately issues an alarm through the sound and light alarm device, and pushes the video clip of the abnormal period, the person's tracking trajectory and characteristic distance value to the monitoring terminal for security personnel to review; if the distance is less than or equal to 1.2, it is determined to be normal behavior, and the detection process ends after automatically recording the person's entry and exit information.

[0105] refer to Figure 2 The provided system architecture diagram illustrates a method for detecting abnormal behavior in secure rooms based on multi-channel feature fusion, comprising the following steps:

[0106] I. Personnel Detection and Tracking

[0107] Train target detection algorithms to detect pedestrians entering and exiting, including but not limited to commonly used detection algorithms such as YOLO, RCNN, and DETR, and combine them with tracking algorithms (including but not limited to SORT, ByteTrack, and DeepSORT) to achieve pedestrian tracking and maintain identity continuity.

[0108] II. Multi-channel feature extraction and fusion architecture

[0109] Multi-channel features are extracted from the personnel image sequence. Each channel corresponds to a feature type and outputs a feature vector with the same dimension. The multi-channel features include global appearance features, key point features, temporal dynamic features, and local region anomaly features.

[0110] This step utilizes a four-channel parallel architecture to achieve comprehensive feature extraction. The four channels are defined as shown in the table below:

[0111]

[0112] Channel 1 extracts global appearance features: extracts global visual features such as the overall outline of the human body, body proportions, and clothing appearance, which are used to capture overall appearance changes caused by carried items.

[0113] Specifically, the DINOv3 pre-trained model is used to extract global appearance features, capturing global visual information such as the overall human contour, body proportions, and clothing appearance. DINOv3 is pre-trained on a large-scale dataset through self-supervised learning, effectively extracting global semantic features and spatial structure information from images. The input is a cropped image of a person. Output 1024-dimensional high-level semantic features :

[0114] ;

[0115] The 1024-dimensional high-level semantic features are remapped and encoded into a 256-dimensional global appearance feature vector using a multilayer perceptron (MLP). :

[0116] .

[0117] Channel 2 extracts key point features: Based on human key point detection, it extracts local structural features such as joint position, limb angle, and posture configuration to identify behaviors such as operating equipment and abnormal postures.

[0118] Specifically, the HRNet model was used to detect 17 COCO standard human keypoints. Each keypoint is represented by normalized coordinates and confidence score. Each keypoint is represented as follows:

[0119] , ;

[0120] in For coordinates normalized to the range [0,1], For confidence level, Let i be the i-th key point.

[0121] Draw the key points on the blank image and connect the joints to form a skeleton diagram. The data is fed into a ResNet-18 model to extract initial features of key points. :

[0122] ;

[0123] Initial features of 1024-dimensional keypoints were generated using a multilayer perceptron (MLP). Remapped and encoded as 256-dimensional keypoint feature vectors :

[0124] ;

[0125] The ResNet-18 model was pre-tuned on a human behavior skeleton dataset for a secure room. The behavior labels included 10 categories: normal entry, normal stay, normal departure, carrying cardboard boxes, putting exam papers into a briefcase, carrying a mobile phone, using a computer, using a tablet, smoking, and picking up items.

[0126] Channel 3 extracts temporal dynamic features: detects the interaction between people and objects in the environment, including picking actions, usage behaviors, contact patterns, etc., to identify interactive behaviors such as operating computers and using mobile phones.

[0127] The 3DResNet-18 model is used to process 16 consecutive frames of images to extract spatiotemporal dynamic features and model motion patterns and behavioral changes. Video sequence preprocessing collects 16 consecutive frames of images of people entering and exiting, normalized to a fixed size of 112×112 pixels. The 3DResNet-18 model captures both spatial appearance and temporal motion information simultaneously through spatiotemporal convolutional kernels, outputting temporal initial features with a dimension of 1024. :

[0128] ;

[0129] The 1024-dimensional temporal initial features were processed using a multilayer perceptron (MLP). Re-mapped and encoded as a 256-dimensional temporal interaction feature vector :

[0130] ;

[0131] Among them, 3D ResNet-18 was pre-trained on the video classification of the confidential room behavior dataset. The action labels include: normal entry, normal stay, normal departure, carrying a cardboard box, putting the test paper into a briefcase, carrying a mobile phone, using a computer, using a tablet, smoking, and picking up something.

[0132] Channel 4: Local Area Anomaly Features: Analyzes temporal information such as motion changes, motion trajectories, and duration of behavior between consecutive frames to detect long-term lingering and abnormal action sequences.

[0133] A ROI clipping mechanism is designed for high-risk areas (hands, shoulders, waist) in the secure room to enhance the ability to detect subtle anomalies such as concealed carrying and item handover.

[0134] Seventeen COCO keypoints (numbered 0-16) detected by HRNet were identified, among which:

[0135] : Left wrist (left_wrist), :right wrist(right_wrist);

[0136] : left shoulder (left_shoulder) : right shoulder;

[0137] : left hip (left_hip) : right hip;

[0138] Define the target key detection region (ROI) based on the key point coordinates detected by HRNet:

[0139] ROI of left wrist: ;by Crop a 64x64 area from the center;

[0140] Right wrist ROI: ;by Crop a 64x64 area from the center;

[0141] Waist ROI: ;by and : A 96×96 area is cropped from the center point;

[0142] Shoulder ROI: ;by and A 96×96 area is cropped from the center point;

[0143] in To be based on the center point from The process involves cropping the Region of Interest (ROI) and then extracting local features from each key target detection region. :

[0144] ;

[0145] Regional anomaly feature fusion: Max pooling is used to fuse information from multiple regions to obtain regional anomaly features. :

[0146] ;

[0147] The MLP maps the feature vectors to 256-dimensional local anomalies. :

[0148] ;

[0149] In this example, Backbone uses ResNet18 as a local feature to extract ROI region features. The ResNet-18 model is pre-trained on a local region dataset of a secure room. The manually labeled item categories include: no items, handbag, folder, cardboard box, exam paper bag, and other items.

[0150] The 256-dimensional feature vectors of each channel are concatenated according to the channel dimension to obtain a 1024-dimensional fused feature vector. :

[0151] ;

[0152] This fusion feature comprehensively characterizes the global appearance, local structure, temporal dynamics, and regional anomaly information of human behavior, providing high-quality input for subsequent metric learning.

[0153] A metric learning training framework is constructed, and a feature mapping network is trained using a triplet loss function, wherein the triplet consists of anchor samples, positive samples, and negative samples.

[0154] Triple construction strategy:

[0155] For each sample in the training set, construct triples with two modes:

[0156] Pattern 1 (Consistent with normal behavior):

[0157] Anchor point (a): A person's normal behavioral characteristics;

[0158] Positive sample (p): Normal behavioral characteristics of the same person in another frame;

[0159] Negative samples (n): Abnormal behavioral characteristics of the same person (such as carrying a mobile phone);

[0160] Mode 2 (Distinguishing Anomalous Behaviors):

[0161] Anchor point (a): A person's abnormal behavioral characteristics (such as carrying a mobile phone);

[0162] Positive samples (p): Abnormal behavioral characteristics of the same person;

[0163] Negative samples (n): another type of abnormal behavior (such as computer use) or normal behavior of the same person;

[0164] During training, the two modes are mixed in a 7:3 ratio to construct a large-scale triplet dataset, ensuring that normal classes are cohesive, abnormal classes are cohesive, and classes are separable in the feature space.

[0165] Triple construction: Constructing a triplet training and testing dataset;

[0166] Loss optimization: Train the MLP encoder, minimize the triplet loss function, and update the parameters;

[0167] Triple loss function:

[0168] ;

[0169] Where f(·) represents the feature mapping function, which is the output of the multi-channel feature fusion mentioned above; margin is the boundary parameter, which is set to 0.5 in this example; This represents the Euclidean distance (simplified to Euclidean distance).

[0170] The triplet loss function is calculated as the Euclidean distance between the anchor sample and the positive sample minus the Euclidean distance between the anchor sample and the negative sample. If the difference is less than 0, it is set to 0; otherwise, it is the sum of the difference and the boundary parameter. This loss function enforces a constraint that the distance between the anchor point and the positive sample is at least a margin smaller than the distance between the anchor point and the negative sample, thereby pushing away out-of-class samples and bringing similar samples closer together.

[0171] The Adam optimizer was used, with an initial learning rate of 1e-4, weight decay of 1e-5, and a batch size of 128. Training consisted of two phases:

[0172] Phase 1 (Epoch 1~50): Freeze the DINOv3, HRNet and 3DResNet-18 backbone networks, train only the MLP mapping layer and feature fusion parameters, and quickly converge to the preliminary discrimination space.

[0173] Phase 2 (Epoch 51~150): Unfreeze all parameters and fine-tune end-to-end, reduce the learning rate to 1e-5, and refine the feature extraction and fusion strategies.

[0174] During the inference phase, feature vectors are extracted from individuals with the same tracking ID at different time points, and the Euclidean distance in the feature space is calculated as a measure of behavioral consistency.

[0175] Temporal feature alignment: for the same person at different time points and Extract the corresponding features based on the behavior:

[0176] Time point characteristics: ;

[0177] Time point characteristics: ;

[0178] Consistency measure, calculating the Euclidean distance in the feature space:

[0179] Distance , )= ;

[0180] Anomaly detection rules: Detection is based on a preset consistency threshold (Threshold).

[0181] IF Distance( , Threshold

[0182] THEN

[0183] Triggering an abnormal alarm

[0184] ELSE

[0185] Through testing

[0186] END IF

[0187] Threshold is the consistency threshold, which is determined by the validation set.

[0188] The threshold is determined through a validation set grid search, and in this embodiment, it is set to 1.2. Different thresholds can be set for different detection items; for example, for the "long-term stay" detection, a threshold of 1.2 is used. Judgment: For the detection of "abnormal items leaving", the characteristic distance between the time of entry and the time of exit is compared.

[0189] System Deployment and Real-time Monitoring:

[0190] A high-definition RGB camera is installed in the corner of the secure room to capture the entire indoor scene. The system processes the video stream in real time at 30fps, maintaining a feature queue for each person and storing the feature vectors of the most recent 16 frames. When a person leaves the monitored area, the system automatically compares the feature distance between the entry and exit times. If the distance exceeds a threshold, an alarm is triggered, and a tagged video archive is generated for review by supervisors.

[0191] Compared with related technologies, the present invention has the following advantages:

[0192] 1. Deep fusion of multimodal features: The four-channel feature design comprehensively depicts the global appearance, local structure, regional anomalies and temporal dynamics of human behavior. Compared with the single-modal method, the feature expression capability is improved by more than 40%, and the detection rate of subtle abnormal behaviors is significantly improved.

[0193] 2. Enhanced Discriminative Boundary: The triplet loss function constructs a clearer inter-class separation boundary through explicit distance constraint optimization. Compared with the traditional cross-entropy loss, it has a significant advantage in metric learning tasks, reducing intra-class distance by 30% and increasing inter-class distance by 50%.

[0194] 3. Open set recognition capability: The anomaly degree calculation mechanism based on feature distance can effectively detect novel anomalous behaviors not seen during training. On a test set containing 10 known anomalies and 5 unknown anomalies, the detection rate of unknown anomalies reaches more than 85%, demonstrating good generalization performance.

[0195] 4. Consistent Modeling of the Same Person: By aligning temporal features and calculating Euclidean distance, the behavioral correlation of the same person at different time points is accurately modeled, effectively identifying "behavioral mutation" anomalies, breaking through the limitations of traditional single-frame detection, and reducing the false alarm rate to below 1%.

[0196] 5. End-to-end trainable: The entire detection, tracking, feature extraction, fusion and judgment process can be embedded into existing deep learning frameworks for end-to-end training. The single forward propagation time during inference is <50ms, which meets the requirements of real-time video analysis and is suitable for intelligent monitoring scenarios in secure rooms.

[0197] This invention also provides a system for detecting abnormal behavior in a secure room based on multi-channel feature fusion, comprising:

[0198] At least one processor;

[0199] At least one memory for storing at least one program;

[0200] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0201] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0202] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any intelligent terminal, including servers, edge computing devices, smart cameras, etc.

[0203] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0204] Those skilled in the art will understand that all or some of the steps and systems disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and can include any information delivery medium.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A method for detecting abnormal behavior in a secure room based on multi-channel feature fusion, characterized in that, The method comprises the following steps: S100, acquiring a video sequence of a secret room scene, detecting and tracking personnel in the video sequence to obtain personnel image sequences with consistent tracking IDs; S200, extracting multi-channel features of the personnel image sequences, and merging and splicing the multi-channel features to form fusion features of personnel behaviors; each channel corresponds to a feature type and outputs feature vectors of the same dimension; S300, obtaining fusion features of personnel with the same tracking ID at different time points, calculating the Euclidean distance between the fusion features in a constructed feature discrimination space, and outputting an abnormal behavior detection result according to a comparison result of the Euclidean distance and a preset threshold; In S200, the extraction of the multi-channel features of the personnel image sequences and the merging and splicing of the multi-channel features to form the fusion features of personnel behaviors comprise: S210, pre-processing single-frame images in the personnel image sequences, obtaining high-level semantic features of the images through a pre-trained feature extraction network, and mapping the high-level semantic features to global appearance feature vectors of a preset dimension through a multi-layer perception machine; S220, detecting human key points in the single-frame images in the personnel image sequences to obtain coordinates and confidence of each joint of a human body, drawing a human skeleton map based on the coordinates and the confidence, inputting the human skeleton map into a pre-trained classification model for feature extraction, and mapping the extracted features to key point feature vectors of a preset dimension through a multi-layer perception machine; S230, pre-processing the personnel image sequences to obtain video clips of a fixed length, extracting features of the video clips through a pre-trained three-dimensional convolution network, and mapping the extracted features to time sequence interaction feature vectors of a preset dimension through a multi-layer perception machine; S240, determining key detection regions of hands, waists and shoulders based on the human key point detection result, extracting features of the key detection regions, fusing the extracted region features, and mapping the fused features to local region abnormal feature vectors of a preset dimension through a multi-layer perception machine; S250, splicing the global appearance feature vectors, the key point feature vectors, the time sequence interaction feature vectors and the local region abnormal feature vectors to form the personnel behavior fusion features.

2. The method of claim 1, wherein, In S210, the pre-processing of the single-frame images in the personnel image sequences, the obtaining of high-level semantic features of the images through a pre-trained feature extraction network, and the mapping of the high-level semantic features to global appearance feature vectors of a preset dimension through a multi-layer perception machine comprise: S211, performing size normalization and pixel value standardization processing on the single-frame images in the personnel image sequences; S212, inputting the pre-processed single-frame images into a pre-trained ResNet50 network to extract high-level semantic features output by a full connection layer; S213, inputting the high-level semantic features into a multi-layer perception machine, performing linear mapping and ReLU activation function processing, and outputting global appearance feature vectors of a preset dimension.

3. The method of claim 1, wherein, In S220, the human key point detection of the single-frame images in the personnel image sequences to obtain coordinates and confidence of each joint of a human body comprises: S221, adopt the HRNet model to detect the human body key points of a single frame image in the personnel image sequence, output the normalized coordinates of each joint of the human body and the corresponding confidence; S222, draw a human skeleton graph based on the coordinates and the confidence, input the human skeleton graph into a ResNet-18 model trained by a secret room human behavior skeleton graph data set, and extract to obtain key point initial features of a preset dimension; S223, map the key point initial features to key point feature vectors of a preset dimension through a multi-layer perception machine.

4. The method of claim 1, wherein, In S230, the personnel image sequence is preprocessed to obtain a fixed-length video segment, features of the video segment are extracted through a pre-trained three-dimensional convolution network, and the extracted features are mapped to time sequence interaction feature vectors of a preset dimension through a multi-layer perception machine, including: S231, a plurality of continuous frames of images are intercepted from the personnel image sequence as a video segment, and each frame of image in the video segment is normalized to a preset resolution in size; S232, the video segment is input into a pre-trained C3D network, spatio-temporal features are extracted through a three-dimensional convolution layer and a pooling layer, and time sequence initial features of a preset dimension are output; S233, the time sequence initial features are input into a multi-layer perception machine, linear mapping and batch normalization processing are performed, and time sequence interaction feature vectors of a preset dimension are output.

5. The method of claim 1, wherein, In S240, the key detection regions of the hand, waist and shoulder are determined based on the human body key point detection result, features of each key detection region are extracted, the extracted region features are fused, and the fused features are mapped to local region anomaly feature vectors of a preset dimension through a multi-layer perception machine, including: S241, human body key points of a single frame image in the personnel image sequence are detected to obtain coordinates of 17 key points defined by a COCO data set; S242, a target key detection region is determined according to the coordinates of the corresponding key points, wherein the left wrist region is a boundary box with the left wrist as the center, the right wrist region is a boundary box with the right wrist as the center, the waist region is a boundary box with the average value of the coordinates of the left hip and the right hip as the center, and the shoulder region is a boundary box with the average value of the coordinates of the left shoulder and the right shoulder as the center; S243, each target key detection region is cropped from the personnel image, and local features of each target key detection region are extracted by inputting each target key detection region into a pre-trained ResNet18 network; wherein the ResNet18 network is a model trained by a secret room artificial annotation data, and the article carrying classes of the artificial annotation personnel include no article, handbag, folder, carton and test paper bag; S244, the local features of each target key detection region are maximum-pooling fused to obtain region anomaly features; S245, the region anomaly features are input into a multi-layer perception machine to output local region anomaly feature vectors of a preset dimension.

6. The method of claim 1, wherein, In S300, the constructed feature discrimination space is constructed by the following steps: S310, collect personnel behavior sample data of the confidential room scene, the sample data including normal behavior sample and abnormal behavior sample of the same personnel, wherein the abnormal behavior sample includes behavior sample of carrying a mobile phone and using a computer; S320, select anchor sample, positive sample and negative sample from the sample data to construct a three-tuple training data set, the three-tuple training data set including two construction modes, anchor sample of mode one being normal behavior feature of a person at a time, positive sample being another frame normal behavior feature of the same person, and negative sample being feature of abnormal behavior category of the same person; anchor sample of mode two being a certain abnormal behavior of a person, positive sample being the same abnormal behavior of the same person, and negative sample being another category abnormal behavior or normal behavior of the same person; S330, inputting a sample in the triple training data set into a multilayer perception encoder, updating an encoder parameter by minimizing a triple loss function until a model converges, and obtaining a feature discrimination space; wherein the triple loss function is ; f(·) represents a feature mapping function, a represents an anchor sample, p represents a positive sample, n represents a negative sample, represents an Euclidean distance, and margin is a boundary parameter.

7. The method of claim 1, wherein, In S300, the fusion feature of the personnel with the same tracking ID at different time points is obtained, the Euclidean distance between the fusion features in the constructed feature discrimination space is calculated, and the abnormal behavior detection result is output according to the comparison result of the Euclidean distance and the preset threshold, including: S310, obtaining the fusion feature of the personnel with the same tracking ID at the entry time and the exit time; S320, calculating the Euclidean distance between the fusion feature at the entry time and the fusion feature at the exit time; S330, comparing the Euclidean distance with the preset threshold, if the Euclidean distance is greater than the preset threshold, outputting the abnormal behavior detection result; if the Euclidean distance is less than or equal to the preset threshold, outputting the normal behavior detection result.

8. A secure room abnormal behavior detection system based on multi-channel feature fusion, characterized in that, including: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor implements the method in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method in any one of claims 1 to 7.

Images