Image-based crowd abnormal behavior identification method

Abstract

The invention provides an image-based crowd abnormal behavior recognition method, and belongs to the technical field of crowd abnormal behavior recognition. The method comprises the following steps: acquiring a to-be-identified crowd video stream; detecting the crowd in the video frame through a target detection network to obtain a target frame and a person category label of each person; extracting human body key point skeleton coordinates in each target frame through a posture estimation model; generating a group distribution thermodynamic diagram for the whole view; performing multi-modal coding on the target frame, the personnel category labels, the skeleton key point coordinates and the thermodynamic diagram to obtain crowd feature codes; and inputting the crowd feature codes into a large language model for sequence modeling and context reasoning, and outputting a crowd behavior category. According to the method, the target frame, the personnel category, the skeleton information and the thermodynamic diagram are jointly coded through the multi-modal fusion technology, and the extraction precision and robustness of the crowd behavior characteristics in a complex scene are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of crowd abnormal behavior recognition technology, and in particular to an image-based crowd abnormal behavior recognition method. Background Technology

[0002] Crowd behavior recognition involves analyzing the dynamic features of crowds in video or image sequences to classify and understand specific behaviors. This technology primarily involves several subfields, including object detection, pose estimation, and motion analysis. At the fundamental theoretical level, object detection algorithms are used to locate individuals within a crowd, human pose estimation techniques extract key skeletal information, and optical flow analysis captures motion features.

[0003] Existing crowd behavior recognition methods suffer from the following technical shortcomings: First, methods based on single visual features (such as using only target bounding boxes or skeleton information) struggle to meet the recognition needs of complex scenes, especially in densely populated or occluded situations. Second, traditional methods are inadequate in multimodal information fusion, failing to effectively integrate multiple features such as bounding boxes, person types, skeleton key points, and heatmaps. Furthermore, existing technologies lack efficient inference mechanisms, making it difficult to extract high-level behavioral semantics from low-level features. For example, while current target detection algorithms can accurately locate individuals, they cannot distinguish between person types; pose estimation algorithms are susceptible to interference in multi-person scenarios; and while heatmap methods can reflect crowd density distribution, they cannot characterize the interaction relationships between individuals. These limitations severely restrict the practical application effectiveness of crowd behavior recognition technology. Summary of the Invention

[0004] In view of this, the present invention provides an image-based method for identifying abnormal crowd behavior, which combines target detection, human skeleton estimation, and crowd density heatmap to infer the current behavior category of the crowd through a large language model.

[0005] Therefore, the present invention provides the following technical solution: A method for identifying abnormal crowd behavior based on images, comprising: Acquire video streams of the people to be identified; The object detection network is used to detect people in video frames to obtain the bounding box and category label of each person; Within each bounding box, the skeleton coordinates of human key points are extracted using a pose estimation model; Generate a heatmap of population distribution across the entire field of view; Multimodal coding is performed on the target bounding box, personnel category labels, skeleton key point coordinates, and heatmap to obtain the crowd feature code; The group's characteristic encoding is input into a large language model for sequence modeling and contextual reasoning, and the group's behavior category is output.

[0006] Furthermore, the step of obtaining crowd feature encoding by performing multimodal encoding on the target bounding box, personnel category labels, skeleton key point coordinates, and heatmap includes: Position encoding is performed on the target bounding box to obtain its features; Personnel attribute features are obtained by embedding personnel category tags; Graph structure encoding is performed on the coordinates of key points in the skeleton to obtain skeleton features; Population density features are obtained by performing convolutional feature extraction on heatmaps; The target bounding box features, personnel attribute features, skeleton features, and crowd density features are combined as crowd feature encoding.

[0007] Furthermore, the target detection network includes: YOLOv8.

[0008] Furthermore, the pose estimation model includes a high-resolution network.

[0009] Furthermore, the personnel category labels include gender and age group; The age group labels include: children, youth, middle-aged, and elderly.

[0010] Furthermore, the step of detecting people in video frames using a target detection network to obtain the bounding box and category label for each person includes: The people in the target box are classified by gender and age group using a people attribute classification network; The personnel attribute classification network includes ResNet.

[0011] Advantages and positive effects of the present invention: This invention utilizes multimodal fusion technology to jointly encode bounding boxes, personnel categories, skeleton information, and heatmaps, significantly improving the accuracy and robustness of crowd behavior feature extraction in complex scenarios. Simultaneously, a large language model based on the Transformer architecture effectively models long-range spatiotemporal dependencies among crowds, accurately identifying individual interactions, resulting in a dual improvement in both behavior recognition accuracy and semantic understanding depth. This technical solution has significant application value in fields such as dense crowd monitoring and intelligent security. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of the image-based crowd abnormal behavior recognition method in an embodiment of the present invention; Figure 2 This is a framework diagram of an image-based method for identifying abnormal crowd behavior in an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] This invention provides an image-based method for identifying abnormal crowd behavior, which combines target detection, human skeleton estimation, and crowd density heatmaps to infer the current behavior category of the crowd through a large language model.

[0017] Combination Figure 1 As shown, an image-based method for identifying abnormal crowd behavior includes the following steps: 1. Collect video streams of the crowd.

[0018] 2. Detect crowds in input video frames or images using an object detection network to obtain the bounding box for each person; extract key points of the human skeleton within the bounding box region; generate a heatmap of crowd distribution for the entire field of view to characterize crowd density and spatial distribution features.

[0019] 3. Perform multimodal coding on the target bounding box information, personnel category labels, skeleton key point coordinates, and heatmap features to obtain the population feature code; where the personnel category labels include: gender and age group.

[0020] 4. Input the encoded features of the population into a large language model based on the Transformer architecture for sequence modeling and contextual reasoning, and output the group behavior category or behavior description.

[0021] Example 1. Target detection and classification: 1) Use a deep learning object detection model to detect crowds in the input image and output each bounding box; 2) Classify the people in the target box by gender and age group using a personnel attribute classification network.

[0022] Age group classification labels include: children, youth, middle-aged and elderly.

[0023] In this embodiment, preferably, the target detection model is YOLOv8.

[0024] In this embodiment, preferably, the personnel attribute classification network is ResNet.

[0025] 2. Human skeleton extraction: Within each bounding box, the skeleton information of human key points is extracted using a pose estimation model; Normalize the skeletal information of key human body points.

[0026] In this embodiment, preferably, the pose estimation model is HRNet (High-ResolutionNet) or OpenPose (Open Pose).

[0027] 3. Heatmap generation: A global heatmap is generated based on the detected location and density information of the crowd to reflect the distribution and aggregation of the crowd.

[0028] 4. Multimodal feature encoding: The target bounding box coordinates, personnel category labels, skeleton key points, and heatmap features are encoded into a unified vector representation.

[0029] The target bounding box is encoded to obtain target bounding box features; personnel attribute categories are embedded to obtain personnel attribute features; skeletal key points are encoded using graph structure to obtain skeletal features; and convolutional features are extracted from the heatmap to obtain crowd density features.

[0030] The target bounding box features, personnel attribute features, skeleton features, and crowd density features are combined to form crowd features.

[0031] 5. Transformer reasoning The Transformer architecture takes crowd features as sequence input, models the relationship between individuals and groups through a multi-head attention mechanism, and outputs group behavior categories or descriptive text.

[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying abnormal crowd behavior based on images, characterized in that, include: Acquire video streams of the people to be identified; The object detection network is used to detect people in video frames to obtain the bounding box and category label of each person; Within each bounding box, the skeleton coordinates of human key points are extracted using a pose estimation model; Generate a heatmap of population distribution across the entire field of view; Multimodal coding is performed on the target bounding box, personnel category labels, skeleton key point coordinates, and heatmap to obtain the crowd feature code; The group's characteristic encoding is input into a large language model for sequence modeling and contextual reasoning, and the group's behavior category is output.

2. The method according to claim 1, characterized in that, The process of obtaining crowd feature encoding by performing multimodal encoding on the target bounding box, personnel category labels, skeleton key point coordinates, and heatmap includes: Position encoding is performed on the target bounding box to obtain its features; Personnel attribute features are obtained by embedding personnel category tags; Graph structure encoding is performed on the coordinates of key points in the skeleton to obtain skeleton features; Population density features are obtained by performing convolutional feature extraction on heatmaps; The target bounding box features, personnel attribute features, skeleton features, and crowd density features are combined as crowd feature encoding.

3. The method according to claim 1, characterized in that, The target detection network includes: YOLOv8.

4. The method according to claim 1, characterized in that, The pose estimation model includes a high-resolution network.

5. The method according to claim 1, characterized in that, The personnel category labels include gender and age group; The age group labels include: children, youth, middle-aged, and elderly.

6. The method according to claim 1, characterized in that, The step of detecting people in video frames using a target detection network to obtain the bounding box and category label for each person includes: The people in the target box are classified by gender and age group using a people attribute classification network; The personnel attribute classification network includes ResNet.

Images