A method, device, electronic device and readable medium for identifying abnormal behavior on campus

Abstract

The embodiments of the present application provide a method, device, electronic device and storage medium for identifying abnormal behavior on campus, including: obtaining a current frame video image in a surveillance video of the campus; using a pedestrian detection model to detect the current frame video image to obtain at least one pedestrian detection frame, and determining an abnormal pedestrian detection frame from the at least one pedestrian detection frame; based on the abnormal pedestrian detection frame, using a pedestrian tracking model to track a target pedestrian in the abnormal pedestrian detection frame of multiple frames of video images of the surveillance video to obtain a detection frame of the target pedestrian in the multiple frames of video images; using a key point extraction model to extract pedestrian skeleton key point information from the detection frame of the target pedestrian in the multiple frames of video images to obtain a pedestrian skeleton key point information sequence; inputting the pedestrian skeleton key point information sequence into an abnormal behavior recognition model to perform abnormal behavior recognition to obtain an abnormal behavior recognition result, thereby improving the recognition rate of multi-person behavior.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent video surveillance, and in particular to a method for identifying abnormal behavior on campus, a device for identifying abnormal behavior on campus, an electronic device and a readable medium. Background Art

[0002] In order to ensure social security, the technology of identifying and monitoring abnormal behavior has received more and more attention. Among them, the traditional method of identifying abnormal behavior on campus relies on human eyes monitoring camera images. This method cannot detect abnormal behavior in time, and is time-consuming, labor-intensive and inefficient.

[0003] With the rapid development of educational information technology in recent years, more and more schools have adopted intelligent recognition methods of abnormal behaviors based on computer vision technology to achieve real-time monitoring of campus safety.

[0004] However, the current real-time monitoring of campus security still has shortcomings. For example, the existing campus abnormal behavior recognition methods can only recognize single-person behavior, and the recognition rate of multi-person interactive behavior is low. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device and readable medium for identifying abnormal behavior on campus to solve the problem of low recognition rate of multi-person interactive behavior in the prior art.

[0006] The embodiment of the present application discloses a method for identifying abnormal behavior on campus, including: obtaining a current frame video image in a surveillance video of the campus; using a pedestrian detection model to detect the current frame video image to obtain at least one pedestrian detection frame, and determining an abnormal pedestrian detection frame from the at least one pedestrian detection frame; wherein the abnormal pedestrian detection frame is a detection frame of an abnormal pedestrian posture and / or a detection frame determined by multiple adjacent interactively abnormal pedestrian detection frames; based on the abnormal pedestrian detection frame, using a pedestrian tracking model to track a target pedestrian in the abnormal pedestrian detection frame on multiple frames of video images of the surveillance video to obtain a detection frame of the target pedestrian in the multiple frames of video images; using a key point extraction model to extract pedestrian skeleton key point information from the detection frame of the target pedestrian in the multiple frames of video images to obtain a pedestrian skeleton key point information sequence; inputting the pedestrian skeleton key point information sequence into a pre-trained abnormal behavior recognition model to perform abnormal behavior recognition to obtain an abnormal behavior recognition result.

[0007] In a possible embodiment, the pedestrian detection model is a YoLov8s model; wherein the model output result of the YoLov8s model is the position information of each pedestrian detection box in at least one pedestrian detection box and its detection confidence.

[0008] In one possible embodiment, in the case where the abnormal pedestrian detection frame is a detection frame determined by multiple adjacent interactively abnormal pedestrian detection frames, the abnormal pedestrian detection frame is determined from at least one pedestrian detection frame, including: calculating the intersection-and-union ratio of two adjacent pedestrian detection frames in the current frame video image; if the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold, calculating the minimum outer enclosing matrix of the two corresponding adjacent pedestrian detection frames, and taking the average sum of the detection confidences of the two corresponding adjacent pedestrian detection frames as the interactive detection confidence of the minimum outer enclosing matrix; taking the minimum outer enclosing matrix as the abnormal pedestrian detection frame, and taking the interactive detection confidence of the minimum outer enclosing matrix as the confidence of the abnormal pedestrian detection frame.

[0009] In a possible embodiment, based on multiple frames of video images of a surveillance video, a pedestrian tracking model is used to track a target pedestrian in an abnormal pedestrian detection frame to obtain a detection frame of the target pedestrian in the multiple frames of video images, including: based on the abnormal pedestrian detection frame and its confidence, a pedestrian tracking model is used to track the target pedestrian in the abnormal pedestrian detection frame to obtain a detection frame of the target pedestrian in the multiple frames of video images; wherein the pedestrian tracking model is a ByteTrack model.

[0010] In a possible embodiment, the key point extraction model is a yolov8s-pose model; wherein the model output result of the yolov8s-pose model is pedestrian skeleton key point information and its corresponding key point confidence.

[0011] In a possible embodiment, when the abnormal pedestrian detection frame is a detection frame determined by multiple adjacent interactively abnormal pedestrian detection frames, the pedestrian skeleton key point information sequence is obtained by weighted averaging the pedestrian skeleton key point information obtained by performing key point detection on multiple adjacent interactively abnormal pedestrian detection frames and their corresponding key point confidences.

[0012] In a possible embodiment, the abnormal behavior recognition model is a STGCN++ model; wherein the model output categories of the STGCN++ model include falling, climbing, fighting, and normal.

[0013] The embodiment of the present application also provides a device for identifying abnormal behavior on campus, including: an acquisition module, used to acquire a current frame video image in a surveillance video of the campus; a detection module, used to detect the current frame video image using a pedestrian detection model, obtain at least one pedestrian detection frame, and determine an abnormal pedestrian detection frame from the at least one pedestrian detection frame; wherein the abnormal pedestrian detection frame is a detection frame of an abnormal pedestrian posture and / or a detection frame determined by multiple adjacent interactively abnormal pedestrian detection frames; a pedestrian tracking module, used to track a target pedestrian in the abnormal pedestrian detection frame on multiple frames of video images of the surveillance video using a pedestrian tracking model based on the abnormal pedestrian detection frame, to obtain a detection frame of the target pedestrian in the multiple frames of video images; an extraction module, used to extract pedestrian skeleton key point information from the detection frame of the target pedestrian in the multiple frames of video images using a key point extraction model to obtain a pedestrian skeleton key point information sequence; an abnormal behavior recognition module, used to input the pedestrian skeleton key point information sequence into a pre-trained abnormal behavior recognition model to perform abnormal behavior recognition and obtain an abnormal behavior recognition result.

[0014] The present application also provides a storage medium in an embodiment, on which a computer program is stored. When the computer program is executed by a processor, the method described in the embodiment of the present application is executed.

[0015] The present application also provides an electronic device in an embodiment, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the method described in the embodiment of the present application is performed.

[0016] The present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the method described in the embodiments of the present application.

[0017] The embodiments of the present application include the following advantages:

[0018] The embodiments of the present application provide a method, device, electronic device and readable medium for identifying abnormal behavior on campus. The method comprises: obtaining a current frame video image in a surveillance video of the campus, and detecting the current frame video image using a pedestrian detection model to obtain at least one pedestrian detection frame, and determining an abnormal pedestrian detection frame from the at least one pedestrian detection frame, wherein the abnormal pedestrian detection frame is a detection frame of an abnormal pedestrian posture and / or a detection frame determined by multiple adjacent interactively abnormal pedestrian detection frames, and based on the abnormal pedestrian detection frame, tracking a target pedestrian in the abnormal pedestrian detection frame on multiple frames of video images of the surveillance video using a pedestrian tracking model to obtain a detection frame of the target pedestrian in the multiple frames of video images, and extracting pedestrian skeleton key point information from the detection frame of the target pedestrian in the multiple frames of video images using a key point extraction model to obtain a pedestrian skeleton key point information sequence, and inputting the pedestrian skeleton key point information sequence into a pre-trained abnormal behavior recognition model to perform abnormal behavior recognition to obtain an abnormal behavior recognition result. Compared with the prior art, the method can not only improve the recognition rate of single-person behavior recognition, but also improve the recognition rate of multi-person behavior.

[0019] In order to make the above-mentioned objectives, features and advantages to be achieved by the embodiments of the present application more obvious and understandable, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flowchart of a method for identifying abnormal behavior on campus provided in an embodiment of the present application is shown;

[0022] Figure 2 A flowchart showing the specific steps of a method for identifying abnormal behavior on campus provided in an embodiment of the present application is shown;

[0023] Figure 3 A flowchart showing the specific steps of another method for identifying abnormal behavior on campus provided by an embodiment of the present application is shown;

[0024] Figure 4 A structural block diagram of a campus abnormal behavior identification device provided in an embodiment of the present application is shown;

[0025] Figure 5 is a block diagram of an electronic device provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of a readable medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] School bullying incidents are emerging one after another, which not only harms the physical and mental health development of young people, but also leaves psychological trauma in serious cases, affecting their future all-round development. Schools and society have also paid high attention to the phenomenon of school bullying. The current measures taken are mainly to install surveillance cameras in public places in schools and arrange security personnel for real-time monitoring. However, relying on manual monitoring is prone to false detection and missed detection, which is not only inefficient but also has high labor costs.

[0029] With the development of artificial intelligence, the automatic identification of abnormal behaviors of monitoring systems through computer vision technology is not only efficient, but also can liberate labor and provide a guarantee for campus safety. In addition, the current methods for identifying abnormal behaviors on campus are divided into the following two types:

[0030] One is a recognition method based on a three-dimensional convolutional neural network of video, which samples and stacks the video frames, then inputs them into a three-dimensional convolutional neural network, and outputs whether the video clip has abnormal behavior. This method has a simple structure, but the target does not occupy a large part of the image frame in the monitoring screen, resulting in a low recognition rate. In addition, the number of three-dimensional convolution parameters and the amount of calculation are large, and the real-time performance is poor.

[0031] The other is a spatiotemporal behavior detection method based on the Slowfast model. It extracts pedestrian information through the target detection model, and then uses a three-dimensional convolutional neural network to identify the behavior categories of pedestrian sequences. This method has a high recognition rate, but is limited to single-person behavior and has a low recognition rate for multi-person interactive behavior.

[0032] Therefore, the current methods for identifying abnormal behavior on campus have problems such as being limited to the identification of a single person's behavior, high computational complexity, and poor real-time performance.

[0033] Based on this, the embodiments of the present application provide a method, device, electronic device and readable medium for identifying abnormal behavior on campus, which can be deployed in the existing school monitoring system, using monitoring cameras to analyze students' abnormal behavior in real time, and simultaneously support the identification of single-person behavior and multi-person interactive behavior across monitoring devices. When abnormal behavior is identified, the abnormal picture is captured and transmitted back in real time, an alarm is issued, and security personnel or teachers are notified to arrive at the scene in time to minimize the harm. The present invention can be deployed in campuses at all stages to provide protection for campus safety.

[0034] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0035] In order to facilitate understanding of the embodiments of the present application, the terms involved in the present application are explained as follows:

[0036] Abnormal behavior on campus: It refers to abnormal behavior on campus, and includes abnormal behaviors such as students falling, students climbing (railings, walls, etc.) and students fighting (including one-on-one, many-on-one, and many-to-many fights).

[0037] SlowFast model: It is a behavior action recognition framework based on deep learning. In addition, the model achieves a fine analysis of static and dynamic content in the video by combining a slow high-resolution CNN (Slow pathway) and a fast low-resolution CNN (Fast pathway). Among them, the Slow pathway is mainly responsible for extracting environmental features in the video (such as background, static objects, etc.), while the Fast pathway focuses on capturing action features in the video (such as character movement, behavior changes, etc.).

[0038] YoLov8s model: It uses CSPDarknet 53 as the backbone network, combined with the path aggregation network (PANet) and the neck structure of the feature pyramid network (FPN). This design enables YOLOv8s to perform well in object detection of various sizes and complexities. In addition, YOLOv8s abandons the traditional Anchor-Based method, adopts the Anchor-Free detection head, and combines BCE Loss, DFL Loss and CIoU Loss to optimize positioning and classification tasks.

[0039] YOLOv8s-Pose model: It is a lightweight model in the YOLOv8 series, suitable for scenarios that require a balance between speed and accuracy. In addition, it is optimized based on YOLOv8 and is particularly suitable for pose estimation tasks. Compared with other versions, YOLOv8s-Pose has a faster inference speed while maintaining a high accuracy rate, making it suitable for real-time application scenarios. In addition, the YOLOv8s-Pose model also has the characteristics of a lightweight design, balanced speed and accuracy, and an easy-to-use model.

[0040] COCO (Common Objects in Context) dataset: It is a large-scale image recognition dataset for object detection, segmentation, and captioning tasks. It contains more than 330,000 images, each annotated with 80 object categories and 5 captions describing scenes. As well, the COCO dataset is widely used in computer vision research and has been used to train and evaluate many state-of-the-art object detection and segmentation models.

[0041] ByteTrack: It is a detection-based multi-target tracking method that aims to solve target tracking problems in the fields of video surveillance, autonomous driving, human-computer interaction, etc. ByteTrack achieves efficient and accurate target tracking by using the similarity between detection boxes and trajectories. The core idea is to use the correlation between detection boxes and trajectories to transform the target tracking problem into a data association problem. And, it is based on the tracking-by-detection paradigm. Most multi-target tracking methods obtain target IDs by associating detection boxes with scores higher than a threshold, but targets with low detection scores, such as occluded targets, are usually simply discarded, which brings problems that cannot be ignored, including a large number of missed detections and fragmented trajectories.

[0042] STGCN++: It is an improved version of STGCN (Space-Time Graph Convolutional Network), which is mainly used to process spatiotemporal sequence data. STGCN++ is optimized on the basis of STGCN to improve the prediction performance of the model on complex spatiotemporal data. And, its core idea is to model spatiotemporal data as a graph structure and extract spatiotemporal features through graph convolution operations. The STGCN++ model combines spatial domain convolution and temporal domain convolution to better capture the spatiotemporal dependencies in spatiotemporal data. Specifically, the model extracts spatial features through spatial domain convolution and temporal features through temporal domain convolution, thereby achieving a comprehensive analysis of spatiotemporal data.

[0043] ntu-120 dataset: It is a public action recognition dataset that contains 120 common actions of pedestrians and is mainly used to study human action recognition. In addition, this dataset contains 120 common actions of pedestrians and can be divided into 4 different data forms: RGB video, depth map sequence, 3D skeleton data, and infrared (IR) video.

[0044] Reference Figure 1 , Figure 1 A flowchart of the steps of a campus abnormal behavior identification method provided in an embodiment of the present application is shown. It should be understood that the campus abnormal behavior identification method can be executed by a campus abnormal behavior identification device, and the specific device of the campus abnormal behavior identification device can be set according to actual needs, and the embodiment of the present application is not limited to this. For example, the campus abnormal behavior identification device can be a computer, or a server, etc. Specifically, the campus abnormal behavior identification method can specifically include the following steps:

[0045] Step S110, obtaining a current frame video image in the campus surveillance video.

[0046] It should be understood that the surveillance video may be captured by cameras installed in public places on campus. For example, the surveillance video may be captured by cameras installed in public areas of a teaching building, or by cameras installed in a playground on campus, or by cameras installed in laboratories (e.g., physics laboratories and chemistry laboratories).

[0047] It should also be understood that the monitoring video can be a real-time video or a video collected within a preset time range. The specific time of the preset time range can be set according to actual needs, and the embodiments of the present application are not limited thereto. For example, the preset time range can be 1 minute, or 5 minutes, etc.

[0048] It should also be understood that the current frame video image can be any frame video image in the surveillance video. For example, the surveillance video stream of the campus can be uniformly sampled to obtain multiple surveillance video images, and the current frame video image can be selected from the multiple surveillance video images.

[0049] Step S120: Use a pedestrian detection model to detect the current frame video image to obtain at least one pedestrian detection frame, and determine an abnormal pedestrian detection frame from the at least one pedestrian detection frame.

[0050] The abnormal pedestrian detection frame is a detection frame of abnormal pedestrian posture and / or a detection frame determined by multiple adjacent abnormal pedestrian detection frames. The abnormal pedestrian posture detection frame and / or a detection frame determined by multiple adjacent abnormal pedestrian detection frames include a detection frame of abnormal pedestrian posture, a detection frame of multiple adjacent abnormal pedestrian detection frames, a detection frame of abnormal pedestrian posture and a detection frame determined by multiple adjacent abnormal pedestrian detection frames.

[0051] It should be understood that the abnormal pedestrian detection frame can be a detection frame for a single person or a detection frame for multiple people.

[0052] It should also be noted that the following Figure 2 and Figure 3 Two embodiments are used to describe two situations of abnormal pedestrian detection frames involved in step S120. Figure 2 and Figure 3 Related description.

[0053] Step S130 , based on the abnormal pedestrian detection frame, a pedestrian tracking model is used to track the target pedestrian in the abnormal pedestrian detection frame on multiple video frames of the surveillance video to obtain the detection frame of the target pedestrian in the multiple video frames.

[0054] Step S140, using a key point extraction model to extract pedestrian skeleton key point information from the detection frame of the target pedestrian in the multi-frame video image to obtain a pedestrian skeleton key point information sequence.

[0055] Step S150: inputting the pedestrian skeleton key point information sequence into a pre-trained abnormal behavior recognition model to perform abnormal behavior recognition and obtain abnormal behavior recognition results.

[0056] In order to facilitate understanding of the embodiments of the present application, a description is given below using specific embodiments.

[0057] First embodiment

[0058] See also Figure 2 , Figure 2 The following is a flowchart showing the specific steps of a method for identifying abnormal behavior on campus provided in an embodiment of the present application. Figure 2 As shown, the campus abnormal behavior identification method includes:

[0059] Step S210, obtaining a current frame video image in the campus surveillance video.

[0060] Step S220, using a pedestrian detection model to detect the current frame video image, obtain at least one pedestrian detection frame, and identify the pedestrian posture in each pedestrian detection in the at least one pedestrian detection frame, and determine the detection frame with abnormal pedestrian posture as an abnormal pedestrian detection frame.

[0061] It should be understood that the specific model of the pedestrian detection model can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0062] Optionally, the pedestrian detection model can be a YoLov8s model, and the YoLov8s model is trained based on the pre-trained weights of the COCO dataset, so that the model has the ability to detect pedestrians. The size of the input image of the YoLov8s model is 384×640, and the model output of the YoLov8s model is [x, y, w, h, score, class_id], (x, y) is the coordinate of the upper left corner of the pedestrian detection box, w, h are the width and height of the pedestrian detection box respectively, socre is the detection confidence of the pedestrian detection box, and class_id represents the category of the pedestrian detection box. In other words, the model output result of the YoLov8s model is the position information and detection confidence of each pedestrian detection box in at least one pedestrian detection box.

[0063] Therefore, based on the above content, the size of the current frame video image can be adjusted to 384×640, and the resized image can be input into a pre-trained pedestrian detection model for pedestrian detection to obtain at least one pedestrian detection frame.

[0064] It should also be understood that the specific method for identifying the posture of a pedestrian in each pedestrian detection frame in at least one pedestrian detection frame can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0065] Optionally, the human skeleton key point information of the pedestrian in the current pedestrian detection frame may be detected, and based on the detected human skeleton key point information, it is determined whether the posture of the pedestrian is abnormal.

[0066] Optionally, the current pedestrian detection frame can be cut out from the current frame video image, and the cut out image can be input into the posture recognition model to obtain the pedestrian posture recognition result. If the pedestrian posture recognition result is abnormal posture, the corresponding pedestrian detection frame is determined to be an abnormal pedestrian detection frame.

[0067] Among them, the specific model of the posture recognition model can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0068] Step S230 , based on the abnormal pedestrian detection frame, a pedestrian tracking model is used to track the target pedestrian in the abnormal pedestrian detection frame on multiple video frames of the surveillance video to obtain the detection frame of the target pedestrian in the multiple video frames.

[0069] It should be understood that the specific model of the pedestrian tracking model can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0070] Optionally, the pedestrian tracking model is a ByteTrack model, so that based on the abnormal pedestrian detection frame and its confidence, the pedestrian tracking model is used to track the target pedestrian in the abnormal pedestrian detection frame to obtain the detection frame of the target pedestrian in multiple frames of video images.

[0071] For example, the abnormal pedestrian detection frame and its confidence score are input into a pre-trained pedestrian tracking model, and the input abnormal pedestrian detection frame is associated with previous and next frames in multiple video frames to detect the target pedestrian detection frame corresponding to the abnormal pedestrian detection frame from the multiple video frames.

[0072] It should also be understood that the specific number of frames of the multi-frame video images can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0073] Step S240, using a key point extraction model to extract pedestrian skeleton key point information from the detection frame of the target pedestrian in the multi-frame video image to obtain a pedestrian skeleton key point information sequence, wherein the pedestrian skeleton key point information sequence includes the pedestrian skeleton key point information corresponding to the current frame video image.

[0074] It should be understood that the specific model of the key point extraction model can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0075] Optionally, the key point extraction model is a yolov8s-pose model, and the yolov8s-pose model is trained based on the pre-trained weights of the coco data set, so that the model has the ability to extract key points of the human skeleton. The size of the input image of the yolov8s-pose model is 192×256, and the output of the yolov8s-pose model is [v,c], where v=17, representing 17 key points of the human skeleton, and c is [x,y,score], where (x,y) is the key coordinate position, and score is the key point confidence. In other words, the model output result of the yolov8s-pose model is the pedestrian skeleton key point information and its corresponding key point confidence.

[0076] Therefore, based on the above content, the yolov8s-pose model can be used to extract the human skeleton key point information of the target pedestrian detection frame, and the human skeleton key point information corresponding to multiple frames of video images can be determined as a pedestrian skeleton key point information sequence.

[0077] Step S250: inputting the pedestrian skeleton key point information sequence into a pre-trained abnormal behavior recognition model to perform abnormal behavior recognition and obtain an abnormal behavior recognition result.

[0078] It should be understood that the specific model of the abnormal behavior identification model can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0079] Optionally, the abnormal behavior recognition model is an STGCN++ model, and the STGCN++ model is trained and fine-tuned based on the pre-trained weights of the ntu-120 data set; wherein, the input of the STGCN++ model is [b, t, c, v], where b is the input batch, t is the number of frames, c is 2 (two-dimensional coordinates in this embodiment), and v is the number of key points. In this embodiment, t is compressed from the original 50 to 16, which can effectively reduce the amount of parameters and calculations; the output of the STGCN++ model is [b, n], n is the behavior category, and the output categories in this implementation are falling, climbing, fighting, and normal.

[0080] Therefore, based on the above content, the pedestrian skeleton key point information sequence can be input into the pre-trained abnormal behavior recognition model to perform abnormal behavior recognition and obtain abnormal behavior recognition results.

[0081] Second embodiment

[0082] See also Figure 3 , Figure 3 A flowchart showing the specific steps of another method for identifying abnormal behavior on campus provided by an embodiment of the present application is shown. Figure 3 As shown, the campus abnormal behavior identification method includes:

[0083] Step S310, obtaining a current frame of video image in the campus surveillance video.

[0084] Step S320, use the pedestrian detection model to detect the current frame video image, obtain at least one pedestrian detection frame, and calculate the intersection-and-union ratio of two adjacent pedestrian detection frames in the current frame video image. If the intersection-and-union ratio is greater than the preset intersection-and-union ratio threshold, calculate the minimum outer enclosing matrix of the two corresponding adjacent pedestrian detection frames, and use the average sum of the detection confidences of the two corresponding adjacent pedestrian detection frames as the interactive detection confidence of the minimum outer enclosing matrix, and use the minimum outer enclosing matrix as an abnormal pedestrian detection frame, and use the interactive detection confidence of the minimum outer enclosing matrix as the confidence of the abnormal pedestrian detection frame.

[0085] It should be understood that the relevant content of the pedestrian detection model can be found in Figure 2 The relevant description in will not be repeated here.

[0086] It should also be understood that the specific process of step S320 can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0087] Specifically, a pre-trained pedestrian detection model is used to detect the current frame video image to obtain at least one pedestrian detection frame, and all pedestrian detection frames in the at least one pedestrian detection frame are sorted according to the position of each pedestrian detection frame in the at least one pedestrian detection frame to obtain a sorting result of the pedestrian detection frame. For example, the pedestrian detection frames are sorted from left to right to obtain a sorting result; for another example, the pedestrian detection frames are sorted from top to bottom to obtain a sorting result.

[0088] Subsequently, based on the sorting result of the pedestrian detection frames, the intersection-and-union ratio of two adjacent pedestrian detection frames in the current frame video image may be calculated, and the intersection-and-union ratio may be compared with a preset intersection-and-union ratio threshold.

[0089] If the intersection-in-union ratio is less than or equal to the preset intersection-in-union ratio threshold, the two pedestrian detection frames are determined to be normal pedestrian detection frames; if the intersection-in-union ratio is greater than the preset intersection-in-union ratio threshold, the minimum outer enclosing matrix of the two corresponding adjacent pedestrian detection frames is calculated, and the average sum of the detection confidences of the two corresponding adjacent pedestrian detection frames is used as the interactive detection confidence of the minimum outer enclosing matrix, and the minimum outer enclosing matrix is ​​used as the abnormal pedestrian detection frame, and the interactive detection confidence of the minimum outer enclosing matrix is ​​used as the confidence of the abnormal pedestrian detection frame.

[0090] The calculation expression of the intersection-in-union ratio is: intersection-in-union ratio = intersection of pedestrian detection frames / union of pedestrian detection frames.

[0091] Step S330 , based on the abnormal pedestrian detection frame, a pedestrian tracking model is used to track the target pedestrian in the abnormal pedestrian detection frame of the multiple video frames of the surveillance video to obtain the detection frame of the target pedestrian in the multiple video frames.

[0092] It should be understood that the relevant content of the pedestrian tracking model can be found in Figure 2 The relevant description in the ByteTrack model is as follows: according to the information of the pedestrian detection box and pedestrian interaction box predicted in the previous frame, the Kalman filter is used to predict the position of each pedestrian prediction box and interaction prediction box; the Hungarian algorithm is used to calculate the similarity between the pedestrian detection box and the pedestrian interaction box in the current frame, and the one with the highest similarity score is selected to associate the previous and next frames.

[0093] It should also be understood that the specific process of step S330 can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0094] Specifically, the abnormal pedestrian detection frame and its confidence score can be input into a pre-trained pedestrian tracking model, and the input abnormal pedestrian detection frame can be associated with previous and next frames in multiple video frames to detect the target pedestrian detection frame corresponding to the abnormal pedestrian detection frame from the multiple video frames.

[0095] Step S340, using a key point extraction model to extract pedestrian skeleton key point information from the detection frame of the target pedestrian in the multi-frame video image to obtain a pedestrian skeleton key point information sequence, wherein the pedestrian skeleton key point information sequence includes the pedestrian skeleton key point information corresponding to the current frame video image.

[0096] It should be understood that the relevant content of the key point extraction model can be found in Figure 2 The relevant description in will not be repeated here.

[0097] It should also be understood that the pedestrian skeleton key point information sequence is obtained by performing weighted average sum of the pedestrian skeleton key point information obtained by performing key point detection on multiple adjacent pedestrian detection frames with abnormal interactions and their corresponding key point confidences, specifically:

[0098] The key points of multiple pedestrians involved in the interactive pedestrian frame are merged into one pedestrian's key point. The calculation formula for each key point is:

[0099]

[0100] In the formula, x represents the x-coordinate of the merged pedestrian key point; n represents the number of pedestrians involved in the pedestrian interaction box; score i represents the confidence of the key points of the i-th pedestrian; x i represents the x-coordinate of the key point of the i-th pedestrian; y represents the y-coordinate of the merged pedestrian key point; y i Represents the y coordinate of the key point of the i-th pedestrian.

[0101] Step S350: inputting the pedestrian skeleton key point information sequence into a pre-trained abnormal behavior recognition model to perform abnormal behavior recognition and obtain abnormal behavior recognition results.

[0102] It should be understood that the relevant content of the abnormal behavior recognition model can be found in Figure 2 The relevant description in will not be repeated here.

[0103] In summary, with the help of the above technical solution, this application has at least the following advantages:

[0104] Low cost: No depth camera or ordinary camera is required. Most schools have already installed surveillance cameras. All you need to do is read the surveillance video and perform calculations. It is easy to deploy and greatly saves costs.

[0105] High accuracy: The solution of the present invention is to align the key points of single-person and multi-person interactions, and realize a network model to simultaneously realize the behavior recognition of single-person and multi-person interactions, with higher recognition accuracy.

[0106] High throughput: The pedestrian detection model and key point extraction model of the present invention both adopt lightweight models, compress and fine-tune the input frame number of the abnormal behavior recognition model, reduce the overall computational complexity, and thus improve the throughput of the system operation.

[0107] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0108] Reference Figure 4 , Figure 4 The structural block diagram of a campus abnormal behavior identification device 400 provided in an embodiment of the present application is shown. It should be understood that the campus abnormal behavior identification device 400 is capable of executing each step in the above method embodiment. The specific functions of the campus abnormal behavior identification device 400 can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here. The campus abnormal behavior identification device 400 includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the campus abnormal behavior identification device 400. Specifically, the campus abnormal behavior identification device 400 includes:

[0109] An acquisition module 410 is used to acquire a current frame video image in a surveillance video of a campus;

[0110] The detection module 420 is used to detect the current frame video image using a pedestrian detection model to obtain at least one pedestrian detection frame, and determine an abnormal pedestrian detection frame from the at least one pedestrian detection frame; wherein the abnormal pedestrian detection frame is a detection frame of an abnormal pedestrian posture and / or a detection frame determined by multiple adjacent pedestrian detection frames with abnormal interactions;

[0111] The pedestrian tracking module 430 is used to track the target pedestrian in the abnormal pedestrian detection frame based on the abnormal pedestrian detection frame by using the pedestrian tracking model to obtain the detection frame of the target pedestrian in the multiple video frames;

[0112] An extraction module 440 is used to extract pedestrian skeleton key point information from the detection frame of the target pedestrian in the multi-frame video image using a key point extraction model to obtain a pedestrian skeleton key point information sequence;

[0113] The abnormal behavior recognition module 450 is used to input the pedestrian skeleton key point information sequence into a pre-trained abnormal behavior recognition model to perform abnormal behavior recognition and obtain abnormal behavior recognition results.

[0114] In a possible embodiment, the pedestrian detection model is a YoLov8s model; wherein the model output result of the YoLov8s model is the position information of each pedestrian detection box in at least one pedestrian detection box and its detection confidence.

[0115] In one possible embodiment, the detection module 420 is used to: when the abnormal pedestrian detection frame is a detection frame determined by multiple adjacent interactively abnormal pedestrian detection frames, calculate the intersection-and-union ratio of two adjacent pedestrian detection frames in the current frame video image; if the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold, calculate the minimum outer enclosing matrix of the two corresponding adjacent pedestrian detection frames, and use the average sum of the detection confidences of the two corresponding adjacent pedestrian detection frames as the interactive detection confidence of the minimum outer enclosing matrix; use the minimum outer enclosing matrix as the abnormal pedestrian detection frame, and use the interactive detection confidence of the minimum outer enclosing matrix as the confidence of the abnormal pedestrian detection frame.

[0116] In a possible embodiment, the pedestrian tracking module 430 is used to track the target pedestrian in the abnormal pedestrian detection frame based on the abnormal pedestrian detection frame and its confidence, using a pedestrian tracking model to obtain the detection frame of the target pedestrian in multiple frames of video images; wherein the pedestrian tracking model is a ByteTrack model.

[0117] In a possible embodiment, the key point extraction model is a yolov8s-pose model; wherein the model output result of the yolov8s-pose model is pedestrian skeleton key point information and its corresponding key point confidence.

[0118] In a possible embodiment, when the abnormal pedestrian detection frame is a detection frame determined by multiple adjacent interactively abnormal pedestrian detection frames, the pedestrian skeleton key point information sequence is obtained by weighted averaging the pedestrian skeleton key point information obtained by performing key point detection on multiple adjacent interactively abnormal pedestrian detection frames and their corresponding key point confidences.

[0119] In a possible embodiment, the abnormal behavior recognition model is a STGCN++ model; wherein the model output categories of the STGCN++ model include falling, climbing, fighting, and normal.

[0120] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0121] In addition, the present application also provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0122] Memory 503, used for storing computer programs;

[0123] The processor 501 is used to execute the program stored in the memory 503, and implements the following steps:

[0124] Get the current frame video image in the campus surveillance video;

[0125] A pedestrian detection model is used to detect the current frame video image to obtain at least one pedestrian detection frame, and an abnormal pedestrian detection frame is determined from the at least one pedestrian detection frame; wherein the abnormal pedestrian detection frame is a detection frame of an abnormal pedestrian posture and / or a detection frame determined by a plurality of adjacent interactively abnormal pedestrian detection frames;

[0126] Based on the abnormal pedestrian detection frame, the pedestrian tracking model is used to track the target pedestrian in the abnormal pedestrian detection frame of the multi-frame video images of the surveillance video to obtain the detection frame of the target pedestrian in the multi-frame video images;

[0127] A key point extraction model is used to extract pedestrian skeleton key point information from the detection frame of the target pedestrian in multiple video frames to obtain a pedestrian skeleton key point information sequence;

[0128] The pedestrian skeleton key point information sequence is input into the pre-trained abnormal behavior recognition model to perform abnormal behavior recognition and obtain the abnormal behavior recognition result.

[0129] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0130] The communication interface is used for communication between the above terminal and other devices.

[0131] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0132] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0133] like Figure 6 As shown, in another embodiment provided by the present invention, a readable medium 601 is also provided, in which instructions are stored, and when the instructions are run on a computer, the computer executes the campus abnormal behavior identification method described in the above embodiment. The readable medium refers to a computer-readable storage medium.

[0134] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the campus abnormal behavior identification method described in the above embodiment.

[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method, and will not be elaborated here.

[0136] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0137] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0138] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0140] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0141] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for identifying abnormal behavior on campus, characterized in that: include: Get the current frame video image in the campus surveillance video; The current frame video image is detected using a pedestrian detection model to obtain at least one pedestrian detection frame, and an abnormal pedestrian detection frame is determined from the at least one pedestrian detection frame; wherein the abnormal pedestrian detection frame is a detection frame of an abnormal pedestrian posture and / or a detection frame determined by a plurality of adjacent interactively abnormal pedestrian detection frames; Based on the abnormal pedestrian detection frame, a pedestrian tracking model is used to track the target pedestrian in the abnormal pedestrian detection frame on multiple video frames of the surveillance video to obtain the detection frame of the target pedestrian in the multiple video frames; A key point extraction model is used to extract pedestrian skeleton key point information from the detection frame of the target pedestrian in the multi-frame video image to obtain a pedestrian skeleton key point information sequence; The pedestrian skeleton key point information sequence is input into a pre-trained abnormal behavior recognition model to perform abnormal behavior recognition and obtain an abnormal behavior recognition result.

2. The method for identifying abnormal behavior on campus according to claim 1, characterized in that: The pedestrian detection model is a YoLov8s model; wherein, the model output result of the YoLov8s model is the position information and detection confidence of each pedestrian detection box in the at least one pedestrian detection box.

3. The method for identifying abnormal behavior on campus according to claim 2, characterized in that: In a case where the abnormal pedestrian detection frame is a detection frame determined by a plurality of adjacent interactively abnormal pedestrian detection frames, determining the abnormal pedestrian detection frame from the at least one pedestrian detection frame includes: Calculating the intersection-and-union ratio of two adjacent pedestrian detection frames in the current frame video image; If the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold, the minimum outer enclosing matrix of the two adjacent pedestrian detection frames corresponding thereto is calculated, and the average sum of the detection confidences of the two adjacent pedestrian detection frames corresponding thereto is taken as the interactive detection confidence of the minimum outer enclosing matrix; The minimum outer enclosing matrix is ​​used as the abnormal pedestrian detection frame, and the interactive detection confidence of the minimum outer enclosing matrix is ​​used as the confidence of the abnormal pedestrian detection frame.

4. The method for identifying abnormal behavior on campus according to claim 3, characterized in that: The method of tracking the target pedestrian in the abnormal pedestrian detection frame using a pedestrian tracking model based on the multiple frames of video images of the surveillance video to obtain the detection frame of the target pedestrian in the multiple frames of video images includes: Based on the abnormal pedestrian detection frame and its confidence, a pedestrian tracking model is used to track the target pedestrian in the abnormal pedestrian detection frame to obtain the detection frame of the target pedestrian in the multi-frame video image; wherein the pedestrian tracking model is a ByteTrack model.

5. The method for identifying abnormal behavior on campus according to claim 1, characterized in that: The key point extraction model is the yolov8s-pose model; wherein the model output result of the yolov8s-pose model is the pedestrian skeleton key point information and its corresponding key point confidence.

6. The method for identifying abnormal behavior on campus according to claim 5, characterized in that: In the case where the abnormal pedestrian detection frame is a detection frame determined by multiple adjacent interactively abnormal pedestrian detection frames, the pedestrian skeleton key point information sequence is obtained by weighted averaging the pedestrian skeleton key point information obtained by performing key point detection on the multiple adjacent interactively abnormal pedestrian detection frames and their corresponding key point confidences.

7. The method for identifying abnormal behavior on campus according to claim 1, characterized in that: The abnormal behavior recognition model is a STGCN++ model; wherein the model output categories of the STGCN++ model include falling, climbing, fighting and normal.

8. A device for identifying abnormal behavior on campus, characterized in that: include: An acquisition module is used to acquire a current frame video image in the campus surveillance video; A detection module, configured to detect the current frame video image using a pedestrian detection model to obtain at least one pedestrian detection frame, and determine an abnormal pedestrian detection frame from the at least one pedestrian detection frame; wherein the abnormal pedestrian detection frame is a detection frame of an abnormal pedestrian posture and / or a detection frame determined by a plurality of adjacent interactively abnormal pedestrian detection frames; A pedestrian tracking module is used to track the target pedestrian in the abnormal pedestrian detection frame based on the abnormal pedestrian detection frame by using a pedestrian tracking model on multiple video frames of the surveillance video to obtain the detection frame of the target pedestrian in the multiple video frames; An extraction module, used for extracting pedestrian skeleton key point information from the detection frame of the target pedestrian in the multi-frame video image by using a key point extraction model to obtain a pedestrian skeleton key point information sequence; The abnormal behavior recognition module is used to input the pedestrian skeleton key point information sequence into a pre-trained abnormal behavior recognition model to perform abnormal behavior recognition and obtain abnormal behavior recognition results.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the campus abnormal behavior identification method as described in any one of claims 1-7 when executing the program stored in the memory.

10. A readable medium having instructions stored thereon, which, when executed by one or more processors, causes the processors to execute the campus abnormal behavior identification method according to any one of claims 1 to 7.

Images