Personnel identification and behavior detection method and device and security robot

Abstract

The invention relates to a personnel identification and behavior detection method and device and a security robot, and the method achieves the whole-process intelligent processing from threat perception to precise intervention through a multi-module cooperative working mechanism and an innovative algorithm design. According to the method, the problem of balance between threat detection precision and real-time performance in the prior art is solved, and accurate three-dimensional positioning and tracking of a threat target are realized through effective integration of depth information. According to the method, through technologies of deep learning algorithm optimization, three-dimensional space positioning and the like, the threat detection capability, the response speed and the intervention accuracy of the security robot in a complex environment are comprehensively improved, powerful technical support is provided for high-risk scene safety prevention and control, and the method has remarkable practical value and popularization prospects.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to a method and device for personnel recognition and behavior detection, and a security robot. Background Art

[0002] With the increasing complexity of security threats and the continuous improvement of social security requirements, intelligent security robots, as a new type of security protection means, are gradually coming to the forefront. In the security protection of various public places, commercial centers, and important facilities, security robots can achieve continuous monitoring, rapid response, and efficient threat recognition, thus making up for the deficiencies of traditional human security. In security robots, the vision module, as the core component for perception and recognition, directly determines the security protection ability of the entire system.

[0003] In recent years, significant progress has been made in computer vision technology. In particular, the application of deep learning algorithms in fields such as object detection, face recognition, and behavior analysis has provided strong technical support for the visual perception system of security robots. However, in complex and changing actual scenarios, the existing technologies still face many challenges. Traditional security robots have significant limitations in threat recognition, personnel positioning, and multi-modal information processing in complex environments, and cannot meet the real-time prevention and control requirements in high-risk scenarios. Although certain progress has been made in existing visual analysis technologies, in complex situations unique to the security field, such as low light, partial occlusion, and fast movement, the accuracy and response speed of traditional systems are still significantly insufficient, mainly including: First, existing security robots have problems of insufficient accuracy and response delay in identifying potential threat behaviors, and it is difficult to provide effective early warnings at critical moments. Especially in high-risk environments with dense crowds and drastic light changes, traditional systems are difficult to achieve real-time and accurate recognition of threat behaviors, resulting in an increase in security risks.

[0004] Second, traditional monitoring systems have obvious deficiencies in simultaneously processing multiple threat types such as armed threats and physical violence behaviors, resulting in a high threat omission rate and being unable to provide comprehensive and reliable information support for security decision-making.

[0005] In addition, existing security robots lack the ability to accurately locate threat personnel in three-dimensional space, and cannot provide the accurate position of the threat entity in the environment, thus limiting the accuracy and effectiveness of subsequent intervention measures, and making it impossible for security robots to perform precise blocking and control actions after detecting threats. Summary of the Invention

[0006] Based on this, it is necessary to provide a personnel identification and behavior detection method, device, and security robot for the above technical problems. This method solves the key technical problems of identity confirmation, threat identification, and personnel positioning of security robots in complex environments, and improves the early warning ability and response efficiency of security robots in high-risk scenarios.

[0007] A personnel identification and behavior detection method, which includes: Perform face recognition on the obtained video frame sequence using a neural network model. When a person not in the database is recognized, an unauthenticated person access warning is generated and uploaded to the security center.

[0008] According to the video frame sequence, use a dual-channel parallel processing behavior detection system to identify dangerous items and violent behaviors, and perform weighted fusion on the dangerous item identification result and the violent behavior identification result to generate a threat behavior identification result; the behavior detection system includes a dangerous goods detection module and a violence detection module. The dangerous goods detection module is used to identify dangerous items in the picture by using visual recognition according to the video frame sequence; the violence detection module is used to model the fine-grained spatial features and efficient temporal relationships of video frames by using a new video analysis architecture that integrates spatial and temporal efficient feature modeling to achieve dynamic behavior detection of personnel.

[0009] If violent behavior is detected, start the precise personnel positioning and tracking process, use the RGB-D data obtained by the depth camera to construct a three-dimensional spatial point cloud model, and the central control unit of the security robot realizes precise positioning and tracking of personnel in the monitoring area according to the three-dimensional spatial point cloud model.

[0010] In one embodiment, performing face recognition on the video frame sequence using a neural network model includes: Perform face detection on the video frame sequence using the RetinaFace model to obtain a face image.

[0011] Extract features from the face image using the VGG-Face model and perform normalization processing to obtain face features.

[0012] Quickly compare the face features with the pre-stored database by calculating the cosine similarity to obtain the face recognition result.

[0013] In one embodiment, the visual recognition method used in the dangerous goods detection module is the YOLO11 neural network.

[0014] In one embodiment, the violence detection module includes: a Transformer encoder and a selective state space model; in the violence detection module: After performing block embedding and linear projection on each video frame, feature vectors are obtained. After position encoding the feature vectors, they are fed into the Transformer encoder to obtain spatial features.

[0015] The spatial features of all video frames are concatenated to obtain a spatial feature sequence.

[0016] The spatial feature sequence is input into the selective state space model to obtain temporal features.

[0017] The temporal features are processed using a fully connected classification head to obtain the recognition results of violent behaviors.

[0018] In one embodiment, when the spatial feature sequence is input into the selective state space model, the obtained temporal features are:

[0019] Where, is the temporal feature at time t, D is the residual connection weight, K is the state space kernel, G is the gating function, is the current input feature, is t all input features before time.

[0020] In one embodiment, using the RGB-D data obtained by a depth camera, a three-dimensional spatial point cloud model is constructed to achieve precise positioning and tracking of people in the monitoring area, including: The RGB-D data is obtained by a depth camera. A neural network is used to detect human key points in the RGB image, and then they are registered with the corresponding depth information in the depth map D to generate three-dimensional point cloud data with spatial coordinates, realizing precise real-time positioning of the target.

[0021] The three-dimensional spatial coordinate data of the target key points is output to the central control unit of the security robot. When it is determined that a certain target poses a threat, the central control unit automatically calculates the optimal approaching path through the three-dimensional spatial coordinate data of the target key points, avoids obstacles, and ensures that the target does not deviate from the monitoring range while maintaining a safe distance.

[0022] In one embodiment, when a person not in the database is recognized, an access warning for unauthenticated personnel is generated and uploaded to the security center, including: When a person not in the database is recognized, a high-definition facial image of the current person is automatically captured, the detection time and location information are recorded, an access warning for unauthenticated personnel is generated, and it is uploaded to the security center together with the collected information through a secure network transmission channel.

[0023] A personnel recognition and behavior detection device, the device comprising: A face recognition module, which uses a neural network model to perform face recognition on the acquired video frame sequence. When a person not in the database is recognized, an unauthenticated person access warning is generated and uploaded to the security center.

[0024] A threat behavior recognition module, which uses a behavior detection system with dual-channel parallel processing to identify dangerous items and violent behaviors based on the video frame sequence, and performs weighted fusion on the dangerous item recognition result and the violent behavior recognition result to generate a threat behavior recognition result; the behavior detection system includes a dangerous goods detection module and a violent detection module. The dangerous goods detection module uses visual recognition to identify dangerous items in the picture based on the video frame sequence; the violent detection module uses a new video analysis architecture that integrates spatial and temporal efficient feature modeling to model the fine-grained spatial features and efficient temporal relationships of video frames, and realizes the dynamic behavior detection of personnel.

[0025] A personnel positioning and following module, which, if a violent behavior is detected, starts an accurate personnel positioning and tracking process, uses the RGB-D data acquired by a depth camera to construct a three-dimensional spatial point cloud model, and realizes the accurate positioning and tracking of personnel in the monitoring area.

[0026] A security robot, the visual module of which uses any of the above personnel recognition and behavior detection methods to perform personnel recognition and behavior detection.

[0027] The above-mentioned personnel recognition and behavior detection method, device and security robot. Through the multi-module collaborative working mechanism and innovative algorithm design, the method realizes the full-process intelligent processing from threat perception to precise intervention. This method not only overcomes the balance problem between threat detection accuracy and real-time performance in the existing technology, but also realizes the accurate three-dimensional positioning and tracking of threat targets through the effective integration of depth information. Through technologies such as deep learning algorithm optimization and three-dimensional space positioning, this method comprehensively improves the threat detection ability, response speed and intervention accuracy of security robots in complex environments, provides strong technical support for the security prevention and control in high-risk scenarios, and has significant practical value and promotion prospects. Brief Description of the Drawings

[0028] Figure 1 It is a schematic flow chart of the personnel recognition and behavior detection method in an embodiment; Figure 2 It is a flow block diagram of the personnel recognition and behavior detection method in another embodiment; Figure 3 It is the overall network structure diagram of the violent detection module in another embodiment; Figure 4 It is a flow chart of spatial feature extraction in another embodiment; Figure 5 Structural diagram of the selective state space model for another embodiment; Figure 6 Flowchart of the method for personnel identification and behavior detection for another embodiment. Detailed implementation manners

[0029] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] In one embodiment, as Figure 1 shown, a method for personnel identification and behavior detection is provided, and the method includes the following steps: Step 100: Perform face recognition on the acquired video frame sequence by using a neural network model. When a person not in the database is recognized, an access warning for unauthenticated personnel is generated and uploaded to the security center.

[0031] Specifically, the video frame sequence is acquired through the vision module of a security robot.

[0032] The single-frame behavior analysis system performs behavior recognition and analysis based on static image frames, mainly relying on visual feature extraction and pattern recognition techniques in single-frame images. It has simple processing, low computational load, and can quickly respond to obvious abnormal behavior features. However, due to relying only on the image information at a single time point, the system has significant limitations in recognizing complex behavior patterns that require temporal context. For advanced security requirements such as progressive threat behaviors, intention prediction, and understanding of behavior sequences, single-frame analysis cannot capture the key temporal features in the evolution of behaviors, resulting in a high false alarm rate and missed reports of potential security threats, and it is difficult to meet the in-depth requirements for behavior understanding and early warning capabilities of modern security robots. To overcome the problems existing in the single-frame behavior analysis system, the present application uses a video frame sequence as the input of the method for personnel identification and behavior detection.

[0033] Perform face detection on the video frame sequence by using a neural network model. The model scans the image to find the positions of faces therein, extracts unique face features from the detected face regions, and after standardizing these features, performs a mathematical comparison with the stored feature data. By calculating the similarity between the two sets of features, the system can quickly determine whether it is the same person and achieve an efficient identity recognition function.

[0034] Face recognition only confirms the identity, and the result does not participate in threat detection.

[0035] Step 102: Based on the video frame sequence, use a behavior detection system with dual-channel parallel processing to identify dangerous items and violent behaviors, and perform weighted fusion on the dangerous item identification result and the violent behavior identification result to generate a threat behavior identification result; the behavior detection system includes a dangerous goods detection module and a violence detection module. The dangerous goods detection module is used to identify dangerous items in the picture according to the video frame sequence by using visual recognition; the violence detection module is used to model the fine-grained spatial features and efficient temporal relationships of video frames by using a new video analysis architecture that integrates spatial and temporal efficient feature modeling to achieve dynamic human behavior detection.

[0036] Specifically, a single-functional vision system usually focuses on a single vision task, such as only having the function of face recognition or simple behavior monitoring. Its core feature is to adopt a single vision processing model, with a simple architecture and single function. Although the implementation cost is low, in the face of various threat forms in complex security scenarios, it lacks comprehensive response capabilities and cannot meet the requirements of comprehensive security protection. To overcome the problems existing in the single-functional vision system, this application constructs a behavior detection system with dual-channel parallel processing, and comprehensively identifies dangerous situations in the picture by running two specialized detection modules simultaneously. One channel focuses on identifying dangerous items, such as knives, etc.; the other channel specifically analyzes human actions and captures fighting behaviors in the picture. The system performs weighted summation on the identification results of the two channels to generate the final threat behavior identification result. This fusion threat analysis method significantly improves the judgment accuracy of the system, reduces false alarms, and at the same time ensures comprehensive coverage of various security hazards.

[0037] The behavior detection system with dual-channel parallel processing adopts a dual-channel threat behavior detection mechanism, which fundamentally improves the comprehensiveness of the system's threat perception. By running two specialized detection modules in parallel, the system can monitor dangerous items and abnormal behaviors simultaneously. This multi-dimensional threat analysis method not only expands the coverage of threat types, but also improves the detection accuracy of various threats through specialized model design.

[0038] Step 104: If a violent behavior is detected, start the precise personnel positioning and tracking process, use the RGB-D data obtained by the depth camera to construct a three-dimensional space point cloud model, and the central control unit of the security robot realizes precise positioning and tracking of personnel in the monitoring area according to the three-dimensional space point cloud model.

[0039] Specifically, most existing security robots mainly adopt a two-dimensional vision analysis system, which mainly relies on RGB images for visual analysis and lacks effective utilization of three-dimensional space information. This technical route has natural limitations in key functions such as target space positioning, distance measurement, and precise tracking, and it is difficult to support the robot to perform precise intervention actions in emergency situations.

[0040] In the process of personnel positioning and tracking, this method uses the RGB-D data obtained by a depth camera to construct a three-dimensional spatial point cloud model, realizing the precise positioning and tracking of personnel in the monitoring area. Specifically, it includes: first, identifying human key points through a human joint point detection model, and then combining depth information to construct a three-dimensional position representation of the personnel, forming a complete spatial perception system.

[0041] This method, through the RGB-D depth camera and high-precision point cloud construction technology, solves the key bottleneck of the traditional system in the spatial positioning of threat targets. Through precise camera internal parameter matrix transformation, the system can accurately map two-dimensional image points into three-dimensional space coordinates, achieving centimeter-level positioning accuracy for threat personnel. This precise spatial perception ability lays the foundation for the active intervention of the robot, enabling it to calculate the optimal approach path and accurately implement intervention measures according to the precise three-dimensional position information of the target after detecting a threat.

[0042] In the above-mentioned personnel recognition and behavior detection method, through a multi-module collaborative working mechanism and innovative algorithm design, the method realizes the full-process intelligent processing from threat perception to precise intervention. This method not only overcomes the balance problem of the existing technology in threat detection accuracy and real-time performance, but also realizes the precise three-dimensional positioning and tracking of threat targets through the effective integration of depth information. Through technologies such as deep learning algorithm optimization and three-dimensional spatial positioning, this method comprehensively improves the threat detection ability, response speed, and intervention accuracy of security robots in complex environments, providing strong technical support for security prevention and control in high-risk scenarios, and having significant practical value and promotion prospects.

[0043] In one of the embodiments, face recognition is performed on the video frame sequence using a neural network model, including: performing face detection on the video frame sequence using the RetinaFace model to obtain a face image; performing feature extraction on the face image using the VGG-Face model and performing normalization processing to obtain face features; and quickly comparing the face features with a pre-stored database by calculating the cosine similarity to obtain the face recognition result.

[0044] Specifically, through the optimized application of the RetinaFace model and the VGG-Face model, the personnel recognition ability of this method is significantly improved under complex conditions. The feature pyramid network structure and multi-scale feature fusion technology enable the system to adapt to common interference factors such as low light and partial occlusion, maintaining stable face detection performance. The VGG-Face extracts features from the detected faces, and finally quickly compares the face features with the pre-stored database by calculating the cosine similarity.

[0045] The RetinaFace model is an efficient face detection model that adopts a feature pyramid network structure. By fusing multi-scale features, it improves the detection accuracy and constructs five feature levels, each responsible for detecting faces of different sizes. The VGG-Face model is a deep convolutional neural network model specifically used for face feature extraction. Based on the VGG-16 architecture, it is trained with a large-scale face dataset. This model can convert the detected face images into high-dimensional feature vectors, which capture the unique identity information of the faces, enabling effective differentiation of faces of different individuals in the feature space.

[0046] As the front-end detector, the RetinaFace model accurately predicts the face position boxes, while the VGG-Face model, as the feature extractor, extracts feature vectors from the detected face images and performs L2 normalization processing. The L2 normalization processing formula is as follows:

[0047] Where, is the feature after L2 normalization processing, F is the feature vector extracted by the VGG-Face model, is the value of the i th dimension in the face feature vector, N is the dimension of the feature vector in the VGG-Face model.

[0048] The formula for calculating the cosine similarity is:

[0049] Where, is the cosine similarity, A , B are the feature vectors of the two faces in the comparison respectively, A and B are both obtained through the L2 normalization processing formula, , are the values of the i th dimension in the two face feature vectors respectively.

[0050] Since the data has been L2-normalized, the formula can be simplified to:

[0051] The face recognition result is obtained by comparing the calculated similarity with a preset threshold.

[0052] In one embodiment, the visual recognition method adopted in the dangerous goods detection module is the YOLO11 neural network.

[0053] Specifically, in the aspect of dangerous goods detection, the latest YOLO11 object detection algorithm is adopted to specifically identify dangerous goods such as knives and firearms. Compared with traditional detection algorithms, YOLO11 achieves higher inference speed and lower computational resource consumption in an edge computing environment while maintaining excellent detection accuracy. The system also adopts enhanced data augmentation strategies and improved loss functions for situations where weapons may be partially blocked or present atypical perspectives in security inspection scenarios, enhancing the model's detection ability for dangerous goods in complex scenarios.

[0054] The YOLO11 neural network adopts a lightweight backbone architecture, which fuses multi-scale receptive field information through parameterization technology, significantly enhancing the feature expression ability while maintaining the efficiency of the model. Compared with traditional backbone networks, this structure achieves a better balance between computational resource consumption and feature extraction ability; for the characteristics of the dangerous goods detection scenario, a feature enhancement module with a multi-dimensional attention mechanism is introduced. This mechanism can adaptively strengthen the representation of target-related regions, significantly enhancing the model's detection ability for dangerous goods in small-sized objects, partially occluded objects, and low-contrast scenarios; a modern reference-free box detection strategy is adopted, simplifying the prediction process, directly predicting the key positions and their attribute information of the target. This design reduces the number of prediction parameters, reduces computational redundancy in the inference process, and improves the detection accuracy of small targets, especially suitable for identifying small-sized dangerous goods at medium and long distances common in surveillance scenarios.

[0055] In one embodiment, the violence detection module includes: a Transformer encoder and a selective state space model; in the violence detection module: after block embedding and linear projection for each video frame, a feature vector is obtained, and after position encoding of the feature vector, it is input into the Transformer encoder to obtain spatial features; the spatial features of all video frames are concatenated to obtain a spatial feature sequence; the spatial feature sequence is input into the selective state space model to obtain temporal features; the temporal features are processed using a fully connected classification head to obtain the violence behavior recognition result.

[0056] Specifically, in the aspect of violence behavior detection, traditional violence detection systems often face problems such as high computational requirements and slow response speeds, while the network structure of the violence detection module in this application successfully breaks through this limitation. Adopting a new video analysis architecture that fuses efficient spatial and temporal feature modeling, the Transformer encoder and the selective state space model are organically combined to achieve fine-grained spatial feature extraction and efficient temporal relationship modeling of video data. This design enables the system to effectively understand the behavior coherence over a long period by supporting feature learning at multiple scales and improving the flow of data in the network. These optimizations enable the system to maintain high accuracy and real-time response capabilities even on edge devices with limited resources.

[0057] In one of the embodiments, the spatial feature sequence is input into the selective state space model to obtain the temporal features; the expression of the temporal features is:

[0058] Wherein, is the temporal feature at t moment, D is the residual connection weight, K is the state space kernel, G is the gating function, is the current input feature, is t all input features before the

[0059] In a specific embodiment, the violent detection module focuses on analyzing the dynamic behavior of people in the video, captures abnormal limb movements by observing consecutive frames, reads the spatio-temporal features in the video frame sequence, and simultaneously processes the spatial information and temporal changes of the image, enabling the system to distinguish normal actions from violent behaviors.

[0060] The violent detection module adopts a new video analysis architecture that fuses spatial and temporal efficient feature modeling, aiming to enhance the spatio-temporal representation ability in the violent behavior recognition task. The core of the model architecture design of the violent detection module lies in the organic combination of the Transformer encoder and the selective state space model, realizing fine-grained spatial feature extraction and efficient temporal relationship modeling of video data. The overall network structure of the violent detection module is as Figure 3 shown: The model first performs frame splitting on the video input (with the shape of [T×3×H×W]), splitting it into T static images. For each frame, the Transformer encoder is separately used for spatial feature extraction, and the spatial feature extraction process is as Figure 4 shown.

[0061] Each frame is divided into several image patches through patch embedding, and the linear projection layer maps these patches to the N-dimensional feature space respectively. To preserve the spatial position information, the model adds spatial position encoding to each image patch, and these encodings can help the model distinguish image patches at different positions.

[0062] For each image patch x feature e the calculation formula is:

[0063] Wherein, e is the feature of the image patch x , W and bare the weights and biases of the linear projection, respectively, pos is the specific position encoding.

[0064] These embedded features are fed into the Transformer encoder, and the correlations between different image patches are calculated through the self-attention mechanism, enabling the model to capture complex spatial dependencies and long-range interactions, and enhancing the model's ability to understand spatial context. After encoding each frame, a spatial feature representation is formed, which contains rich local structural information and global semantic information.

[0065] After completing the spatial feature extraction, the model concatenates the spatial features of all frames to obtain a feature sequence in the form of [T, N].

[0066] The temporal feature extraction uses an efficient selective state space model, which can dynamically filter long temporal features, effectively capture cross-frame dependencies and suppress redundant information. The structure diagram of the selective state space model is as shown in Figure 5 shown. The core calculation process of the selective state space model is as shown in the expression of the above temporal features. The state space kernel K and the gating function G jointly determine the weights of historical inputs at different times in the aggregation. By weighted aggregation of historical inputs, the model can efficiently capture long-range temporal dependencies. The residual connection structure in the model retains the original input features, further enhancing the feature expression ability.

[0067] After the temporal feature extraction is completed, the model aggregates all spatio-temporal information and outputs the probabilities of each category through a fully connected classification head to achieve efficient recognition of violent behaviors.

[0068] The significant feature of the violence detection module lies in the separate modeling of spatial and temporal features. The Transformer encoder focuses on the efficient expression of the internal structure of a single frame and can obtain fine-grained spatial features with a limited number of parameters. Through patch embedding and spatial self-attention, multi-scale spatial feature learning is achieved; while the selective state space model models the dynamic evolution process of the video, and the dynamic filtering mechanism improves the ability to capture cross-frame temporal relationships and has the ability to efficiently process long sequences. This model is significantly superior to traditional 3D convolutional and spatio-temporal hybrid Transformer models in the task of violent behavior recognition. In addition, this architecture can significantly reduce the computational resource requirements while ensuring high detection accuracy, providing strong support for the deployment of violent behavior detection in actual scenarios and edge devices.

[0069] In one of the embodiments, RGB-D data obtained by a depth camera is used to construct a three-dimensional spatial point cloud model to achieve precise positioning and tracking of personnel in the monitoring area, including: obtaining RGB-D data through the depth camera, using a neural network to detect human key points in the RGB image, and then registering with the corresponding depth information in the depth map D to generate three-dimensional point cloud data with spatial coordinates, achieving precise real-time positioning of the target; outputting the three-dimensional spatial coordinate data of the target key points to the central control unit of the security robot. When it is determined that a certain target poses a threat, the central control unit automatically calculates the optimal approaching path through the three-dimensional spatial coordinate data of the target key points, avoids obstacles, and ensures that the target does not escape from the monitoring range while maintaining a safe distance.

[0070] Specifically, point cloud construction is the core technical link of this module. The system uses a neural network to detect human key points in the RGB image and registers with the corresponding depth information in the depth map D to generate three-dimensional point cloud data with spatial coordinates. This process involves an accurate conversion from the camera coordinate system to the world coordinate system.

[0071] First, for each detected human key point (u, v), the corresponding depth value z is obtained from the depth map D, and then through the camera intrinsic matrix K Perform projection transformation:

[0072] Among them, ( X , Y , Z ) are the three-dimensional space point coordinates, ( u , v ) are the human key points, K is the camera intrinsic matrix. The camera intrinsic matrix is a 3×3 matrix describing the internal optical characteristics of the camera, including parameters such as focal length and optical center coordinates, and defines the mapping relationship between three-dimensional world points projected onto the camera's two-dimensional image plane. The camera intrinsic matrix is: , where is the camera optical center coordinate, and are the focal lengths.

[0073] The specific conversion formula is:

[0074]

[0075]

[0076] This conversion maps the two-dimensional image points to three-dimensional space points ( X , Y ,Z ) to achieve centimeter-level positioning accuracy.

[0077] After three-dimensional point cloud construction (the process of representing objects or scenes in three-dimensional space as a set of discrete points, where each point typically contains three-dimensional coordinates and possibly additional attributes such as color and reflectivity), the system outputs the three-dimensional spatial coordinates of the target joint positions to the central control unit of the security robot in real time. These precise spatial positioning data become the key basis for the robot to execute riot prevention strategies. The central control unit adjusts the motion parameters in real time according to this coordinate information to achieve intelligent tracking of suspicious persons. When the system determines that a certain person poses a threat, the control system automatically calculates the optimal approach path through the three-dimensional spatial points of the target, avoids obstacles, and ensures that the target does not escape the monitoring range while maintaining a safe distance.

[0078] In one embodiment, when a person not in the database is identified, an access warning for unauthenticated personnel is generated and uploaded to the security center, including: when a person not in the database is identified, a high-definition facial image of the current person is automatically captured, the detection time and location information are recorded, an access warning for unauthenticated personnel is generated, and it is uploaded to the security center through a secure network transmission channel together with the collected information.

[0079] It should be understood that although Figure 1 the steps in the flowchart of Figure 1 are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover,

[0080] In one embodiment, as Figure 6 shown, the process of the person recognition and behavior detection method includes: Step T1: The system uses the RetinaFace deep learning model for face detection, and VGG-Face to obtain the face feature vector, and compares it with the local face database for cosine similarity to obtain the face comparison result; Step T2: When a person not in the database is identified in Step T1, a high-definition facial image of the current person is automatically captured, the detection time and location information are recorded, an access warning for unauthenticated personnel is generated, and it is uploaded to the security center through a secure network transmission channel together with the collected information; Step T3: In this step, a threat behavior detection system with dual-channel parallel processing is constructed, including two sub-modules: weapon detection in Step T4 and violence detection in Step T5. Comprehensive analysis is performed on the detection results of these two channels to achieve precise monitoring of violent behavior; Step T4: Analyze video frames in real time based on the YOLO11 neural network architecture to detect potential dangerous items such as knives and firearms; Step T5: By adopting an efficient deep learning network, extract spatio-temporal features from the video frame sequence to efficiently identify various violent behavior patterns such as pushing, punching, kicking, and wrestling; Step T6: When violent behavior is detected in Step T3, the system starts the precise personnel positioning and tracking process. RGB-D data is obtained through a depth camera for human key point detection. Seventeen key points including the head, shoulders, elbows, wrists, hips, knees, and ankles are identified, and the three-dimensional space coordinates of the personnel are accurately calculated using the corresponding depth information to achieve precise real-time positioning of the perpetrator.

[0081] This embodiment demonstrates the complete workflow of the security robot's vision module from face recognition, threat detection to precise intervention. Through vision analysis technology and precise three-dimensional space control capabilities, the system can achieve early identification and precise intervention of security threats in complex environments, greatly improving the intelligent level and actual combat effectiveness of security work.

[0082] In one embodiment, a personnel recognition and behavior detection device is provided, including: a face recognition module, a threat behavior recognition module, and a personnel positioning and following module, where: The face recognition module is used to perform face recognition on the acquired video frame sequence using a neural network model. When a person not in the database is recognized, an unauthenticated person access warning is generated and uploaded to the security center.

[0083] The threat behavior recognition module is used to identify dangerous items and violent behaviors according to the video frame sequence using a behavior detection system with dual-channel parallel processing, and perform weighted fusion on the dangerous item recognition result and the violent behavior recognition result to generate a threat behavior recognition result; the behavior detection system includes a dangerous goods detection module and a violence detection module. The dangerous goods detection module is used to identify dangerous items in the picture using visual recognition based on the video frame sequence; the violence detection module is used to model the fine-grained spatial features and efficient temporal relationship of the video frame using a new video analysis architecture that integrates spatial and temporal efficient feature modeling to achieve dynamic behavior detection of personnel.

[0084] The personnel positioning and following module is used to start the precise personnel positioning and tracking process if violent behavior is detected. Using the RGB-D data obtained by the depth camera, a three-dimensional spatial point cloud model is constructed to achieve precise positioning and tracking of personnel in the monitored area.

[0085] In one embodiment, the face recognition module is further configured to perform face detection on the video frame sequence using the RetinaFace model to obtain face images; perform feature extraction on the face images using the VGG-Face model and perform normalization processing to obtain face features; and perform a fast comparison between the face features and the pre-stored database by calculating the cosine similarity to obtain the face recognition result.

[0086] In one embodiment, the visual recognition method used in the dangerous goods detection module of the threat behavior recognition module is the YOLO11 neural network.

[0087] In one embodiment, the violence detection module in the threat behavior recognition module includes: a Transformer encoder and a selective state space model; in the violence detection module: after performing block embedding and linear projection on each video frame, a feature vector is obtained, and after performing position encoding on the feature vector, it is input into the Transformer encoder to obtain spatial features; the spatial features of all video frames are concatenated to obtain a spatial feature sequence; the spatial feature sequence is input into the selective state space model to obtain temporal features; and the temporal features are processed using a fully connected classification head to obtain the violence behavior recognition result.

[0088] In one embodiment, the threat behavior recognition module is further configured to input the spatial feature sequence into the selective state space model to obtain the temporal feature as:

[0089] where, is the temporal feature at t moment, D is the residual connection weight, K is the state space kernel, G is the gating function, is the current input feature, is t all input features before the

[0090] In one embodiment, the personnel positioning and following module is further configured to include: obtaining RGB-D data through a depth camera, detecting human key points from the RGB image using a neural network, and then registering with the corresponding depth information in the depth map D to generate three-dimensional point cloud data with spatial coordinates, so as to achieve precise real-time positioning of the target; outputting the three-dimensional spatial coordinate data of the target key points to the central control unit of the security robot. When it is determined that a certain target poses a threat, the central control unit automatically calculates the optimal approaching path through the three-dimensional spatial coordinate data of the target key points, avoids obstacles, and ensures that the target does not get out of the monitoring range while maintaining a safe distance.

[0091] In one embodiment, the face recognition module is further configured to automatically capture a high-definition facial image of the current person when a person not in the database is recognized, record the detection time and location information, generate an access warning for the unauthenticated person, and upload it to the security center through the secure network transmission channel together with the collected information.

[0092] For the specific limitations of the personnel identification and behavior detection device, reference can be made to the limitations of the personnel identification and behavior detection method in the above text, which will not be elaborated here. Each module in the above personnel identification and behavior detection device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0093] A security robot, the vision module of which uses any of the above personnel identification and behavior detection methods for personnel identification and behavior detection.

[0094] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no conflict in the combination of these technical features, it should be considered as the scope described in this specification.

[0095] The above-described embodiments only represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for personnel recognition and behavior detection, characterized in that The method includes: Performing face recognition on the acquired video frame sequence using a neural network model. When a person not in the database is recognized, an access warning for unauthenticated personnel is generated and uploaded to the security center; According to the video frame sequence, a behavior detection system with dual-channel parallel processing is used to identify dangerous items and violent behaviors, and the dangerous item recognition result and the violent behavior recognition result are weighted and fused to generate a threat behavior recognition result; the behavior detection system includes a dangerous item detection module and a violent detection module. The dangerous item detection module is used to identify dangerous items in the picture by using a visual recognition method according to the video frame sequence; the violent detection module is used to model the fine-grained spatial features and efficient temporal relationships of video frames by using a new video analysis architecture that integrates spatial and temporal efficient feature modeling to realize the detection of personnel's dynamic behaviors; If a violent behavior is detected, an accurate personnel positioning and tracking process is started. Using the RGB-D data obtained by the depth camera, a three-dimensional spatial point cloud model is constructed, and the central control unit of the security robot realizes the accurate positioning and tracking of personnel in the monitoring area according to the three-dimensional spatial point cloud model.

2. The personnel identification and behavior detection method according to claim 1, characterized in that Performing face recognition on the video frame sequence using a neural network model includes: Performing face detection on the video frame sequence using the RetinaFace model to obtain face images; Performing feature extraction on the face images using the VGG-Face model and performing normalization processing to obtain face features; The face features are quickly compared with the pre-stored database by calculating the cosine similarity to obtain the face recognition result.

3. The personnel identification and behavior detection method according to claim 1, characterized in that, The visual recognition method used in the dangerous item detection module is the YOLO11 neural network.

4. The personnel identification and behavior detection method according to claim 1, characterized in that The violent detection module includes: a Transformer encoder and a selective state space model; in the violent detection module: After each video frame is block-embedded and linearly projected, a feature vector is obtained. After the position encoding of the feature vector, it is input into the Transformer encoder to obtain spatial features; The spatial features of all video frames are concatenated to obtain a spatial feature sequence; The spatial feature sequence is input into the selective state space model to obtain temporal features; The temporal features are processed by a fully connected classification head to obtain the violent behavior recognition result.

5. The personnel identification and behavior detection method according to claim 4, wherein Inputting the spatial feature sequence into the selective state space model to obtain temporal features as: Among them, is the timing feature at t moment, D is the residual connection weight, K is the state space kernel, G is the gating function, is the current input feature, is t all input features before the moment.

6. The personnel identification and behavior detection method according to claim 1, characterized in that Using the RGB-D data obtained by the depth camera to construct a three-dimensional spatial point cloud model to realize the accurate positioning and tracking of personnel in the monitoring area includes: Obtaining RGB-D data through the depth camera, using a neural network to detect human key points in the RGB image, and then registering them with the corresponding depth information in the depth map D to generate three-dimensional point cloud data with spatial coordinates to realize the accurate real-time positioning of the target; Output the three-dimensional spatial coordinate data of the target key points to the central control unit of the security robot. When it is determined that a certain target poses a threat, the central control unit automatically calculates the optimal approaching path through the three-dimensional spatial coordinate data of the target key points, avoids obstacles, and ensures that the target does not go out of the monitoring range while maintaining a safe distance.

7. The personnel identification and behavior detection method according to claim 1, wherein, When a person not in the database is recognized, an access warning for unauthenticated personnel is generated and uploaded to the security center, including: When a person not in the database is recognized, a high-definition facial image of the current person is automatically captured, the detection time and location information are recorded, an access warning for unauthenticated personnel is generated, and it is uploaded to the security center together with the collected information through a secure network transmission channel.

8. A personnel recognition and behavior detection device, characterized in that, The device includes: A face recognition module for performing face recognition on the acquired video frame sequence using a neural network model. When a person not in the database is recognized, an access warning for unauthenticated personnel is generated and uploaded to the security center; A threat behavior recognition module for identifying dangerous items and violent behaviors using a behavior detection system with dual-channel parallel processing based on the video frame sequence, and performing weighted fusion on the dangerous item recognition result and the violent behavior recognition result to generate a threat behavior recognition result; the behavior detection system includes a dangerous item detection module and a violent detection module. The dangerous item detection module is used to identify the dangerous items in the picture using visual recognition based on the video frame sequence; the violent detection module is used to model the fine-grained spatial features and efficient temporal relationships of video frames using a new video analysis architecture that integrates spatial and temporal efficient feature modeling to achieve dynamic behavior detection of personnel. A personnel positioning and following module for starting an accurate personnel positioning and tracking process if a violent behavior is detected, and using the RGB-D data obtained by a depth camera to construct a three-dimensional spatial point cloud model to achieve accurate positioning and tracking of personnel in the monitoring area.

9. A security robot, characterized in that, The security robot uses the personnel recognition and behavior detection method according to any one of claims 1-7 for personnel recognition and behavior detection.

Images