Public place abnormal behavior identification method and system based on deep learning

Abstract

The invention discloses a public place abnormal behavior recognition method and system based on deep learning, and relates to the technical field of machine learning, and the method comprises the steps: connecting monitoring equipment to obtain and preprocess a video stream; identifying pedestrian targets through a lightweight target detection model and distributing IDs; tracking a pedestrian ID, constructing a motion track and identifying behavior action features; and training an abnormal behavior recognition model through an abnormal behavior sample library, importing a movement track and behavior action feature recognition abnormality, and generating abnormal feedback information according to the traceability level of a result and the like. The technical problems that in public place abnormal behavior detection through a traditional means, abnormal behaviors in a complex scene are difficult to accurately recognize, missing detection and false detection are prone to occurring, information is not timely and inaccurate, and the safety management and control requirement cannot be met are solved, and the purposes of accurately recognizing the public place abnormal behaviors and improving the safety of public place abnormal behavior detection are achieved. And the abnormal feedback information which is accurate and contains the traceability level, the risk level and the timeliness is output, and the technical effect of meeting the safety management and control requirements is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a public place abnormal behavior recognition method and system based on deep learning. BACKGROUND

[0002] In real life, public place safety management is crucial, and timely identification of abnormal behavior is of great significance to public safety. In the prior art, the detection of abnormal behavior in public places mainly adopts artificial monitoring and traditional behavior analysis algorithms. These methods have played a role in environments with less people flow and simple scenes, but as the complexity of public places increases, traditional technologies gradually expose their limitations. Traditional methods are difficult to accurately identify abnormal behavior in complex scenes, are prone to missed detection and false detection, and the obtained abnormal information is not timely and accurate, which is difficult to meet the needs of precise control and rapid response of public place safety, and the application of deep learning technology in this field has become an improvement direction. SUMMARY

[0003] The present application provides a public place abnormal behavior recognition method and system based on deep learning, which is used to solve the technical problem that traditional methods are difficult to accurately identify abnormal behavior in complex scenes in the detection of abnormal behavior in public places, are prone to missed detection and false detection, and the information is not timely and accurate, which cannot meet the safety control needs.

[0004] In a first aspect, the present application provides a public place abnormal behavior recognition method based on deep learning, which comprises connecting a monitoring device, real-time acquiring a monitoring video stream, and pre-processing video frames of the video stream; identifying pedestrian targets in the video frames through a lightweight target detection model, generating a bounding box, and assigning a pedestrian ID based on the bounding box; performing pedestrian tracking on the video frame based on the assigned pedestrian ID, constructing a motion trajectory between consecutive frames, and identifying inter-frame behavior action features; establishing an abnormal behavior sample library for deep learning of multiple abnormal behavior features to obtain an abnormal behavior recognition model; importing the motion trajectory and behavior action features into the abnormal behavior recognition model for abnormal behavior recognition, outputting an abnormal behavior recognition result, and generating abnormal feedback information according to the traceability level, risk level, and timeliness of the abnormal behavior recognition result.

[0005] In a second aspect of the present application, a public place abnormal behavior recognition system based on deep learning is provided, and the system comprises: a video frame preprocessing execution module configured to connect a monitoring device, acquire a monitoring video stream in real time, and perform video frame preprocessing on the video stream; a pedestrian ID assignment execution module configured to identify a pedestrian target in a video frame through a lightweight target detection model, generate a bounding box, and assign a pedestrian ID based on the bounding box; an inter-frame behavior action feature recognition module configured to track a pedestrian in a video frame based on the assigned pedestrian ID, construct a motion trajectory between continuous frames, and identify an inter-frame behavior action feature; an abnormal behavior recognition model construction module configured to establish an abnormal behavior sample library to perform deep learning on multiple abnormal behavior features, and obtain an abnormal behavior recognition model; and an abnormal feedback information acquisition module configured to input the motion trajectory and the behavior action feature into the abnormal behavior recognition model to perform abnormal behavior recognition, output an abnormal behavior recognition result, and generate abnormal feedback information according to a traceable level, a risk level, and timeliness of the abnormal behavior recognition result.

[0006] One or more technical solutions provided in the present application have at least the following technical effects or advantages: The present application acquires a video stream by connecting a monitoring device and performs preprocessing, identifies a pedestrian target by using a lightweight target detection model and assigns an ID, tracks a pedestrian based on the ID to construct a motion trajectory and identify a behavior action feature, performs recognition in combination with an abnormal behavior recognition model trained by an abnormal behavior sample library, and generates feedback information according to a traceable level, a risk level, and timeliness of a result, so as to accurately recognize abnormal behavior in a public place, make the abnormal behavior recognition result in the public place more accurate and reliable, achieve accurate recognition of abnormal behavior in the public place, output accurate abnormal feedback information containing a traceable level, a risk level, and timeliness, and meet the technical effect of safety control requirements. BRIEF DESCRIPTION OF DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0008] Figure 1 is a flowchart of a public place abnormal behavior recognition method based on deep learning provided by the embodiments of the present application.

[0009] Figure 2 is a structural schematic diagram of a public place abnormal behavior recognition system based on deep learning provided by the embodiments of the present application.

[0010] The reference signs are explained as follows: a video frame preprocessing execution module 1, a pedestrian ID assignment execution module 2, an inter-frame behavior action feature recognition module 3, an abnormal behavior recognition model construction module 4, and an abnormal feedback information acquisition module 5. DETAILED DESCRIPTION

[0011] The application provides a public place abnormal behavior recognition method and system based on deep learning, which is used to solve the technical problem that traditional methods are difficult to accurately identify abnormal behaviors in complex scenes, are prone to missed detection and false detection, and cannot meet the safety control requirements due to untimely and inaccurate information.

[0012] The technical solutions in the embodiments of the application will be clearly and completely described in combination with the accompanying drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the application.

[0013] It should be noted that the terms "first", "second", and the like in the specification and the above drawings of the application are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices.

[0014] Embodiment one, as shown in the figure, a public place abnormal behavior recognition method based on deep learning, wherein the method comprises: Figure 1 Step A100: connecting a monitoring device, acquiring a monitoring video stream in real time, and performing video frame preprocessing on the video stream. Step A100: connecting a monitoring device, acquiring a monitoring video stream in real time, and performing video frame preprocessing on the video stream.

[0015] Specifically, a monitoring device such as a high-definition network camera is connected to acquire a monitoring video stream of a public place in real time through a network protocol. The video stream frame rate is usually 25 frames per second, and each frame contains original image information of pedestrians, environmental background, etc. in the scene. These original video frames may have noise due to factors such as light changes and camera shaking. For example, local overexposure may occur in backlight scenes, and Gaussian noise may occur in low-light environments at night. In addition, the resolutions of different cameras differ, and the common original resolution is 1920x1080 pixels.

[0016] Next, the obtained video stream is preprocessed. First, a median filtering algorithm is used to remove salt and pepper noise in the frame, and a 3x3 pixel filtering window is set to effectively reduce the interference of isolated noise points on subsequent identification. Then, the frame resolution is dynamically adjusted according to the real-time crowd density of the monitoring area. When the crowd density is less than 3 people, the frame resolution is adjusted to 1920x1080 pixels, and when the crowd density is higher than 8 people, it is reduced to 640x480 pixels, which reduces the computational load while ensuring identification accuracy. Finally, color space standardization is performed to convert the RGB format to the HSV format and normalize the pixel values of each channel to the range of 0-1, which unifies the data scale and facilitates feature extraction by the lightweight target detection model.

[0017] The above steps provide denoised, resolution-adapted, and standardized video frame data for subsequent pedestrian target identification, laying a foundation for accurate pedestrian target identification.

[0018] Step A200: Identify pedestrian targets in video frames through a lightweight target detection model, generate bounding boxes, and assign pedestrian IDs based on the bounding boxes.

[0019] Optionally, the input resolution is dynamically switched according to the real-time crowd density, the pedestrian targets in the video frames are identified using a lightweight target detection model, and the bounding boxes are generated. The group cluster area box and the individual pedestrian box are determined based on the position distance relationship and overlap degree of the bounding boxes, and then the IDs are created for pedestrian ID assignment. The specific steps are described in detail in A210-A240.

[0020] Step A300: Perform video frame pedestrian tracking based on the assigned pedestrian IDs, construct motion trajectories between consecutive frames, and identify inter-frame behavior and action features.

[0021] In an embodiment of the present application, the pedestrian ID is selected based on the identification target ID, the motion change trajectory and posture change box between video frames are obtained through tracking, and then the motion trajectory coordinates are located and the action form is identified to generate the motion trajectory and action features. The specific steps are described in detail in A310-A330.

[0022] Step A400: Establish an abnormal behavior sample library for multi-abnormal behavior feature deep learning to obtain an abnormal behavior recognition model.

[0023] In the embodiment of the present application, the abnormal behavior sample library is a multi-scene abnormal sample collection according to abnormal cases in public places, including abnormal motion features, abnormal pedestrian posture change features, pedestrian distance relationship change features, and corresponding abnormal event label sample sets.

[0024] ​​Specifically, a plurality of scene abnormal samples containing abnormal features and event labels are collected, a plurality of abnormal recognition sub-channels are learned according to the label types, and the sub-channels are integrated according to the traceable levels of abnormal events to obtain an abnormal behavior recognition model. The specific steps are described in detail in A410-A440.

[0025] Step A500: introducing the motion trajectory and behavior action feature into the abnormal behavior recognition model for abnormal behavior recognition, outputting an abnormal behavior recognition result, and generating abnormal feedback information according to the traceable level, risk level, and timeliness of the abnormal behavior recognition result.

[0026] Specifically, before the motion trajectory and behavior action feature are introduced into the abnormal behavior recognition model, the two types of features need to be preprocessed and integrated. The motion trajectory needs to be combed according to the position change between consecutive frames in time sequence, remove noise points and smooth processing to form a coherent path description; the behavior action feature extracts key action forms such as limb angle change and action continuity, and converts them into structured feature vectors that can be analyzed by the model. Then, the two are fused according to the feature correlation to ensure that the features input into the model contain both spatial motion information and behavior form details.

[0027] After the integrated features are received by the abnormal behavior recognition model, parallel analysis is performed through the internal multiple recognition sub-channels. Each sub-channel corresponds to a specific abnormal event type, corresponding to multiple events such as fighting, falling, and attacking with weapons. The input features can be compared with the typical abnormal features in the abnormal behavior sample library to calculate the matching degree. When the matching degree of a certain type of event exceeds a predetermined threshold, the model determines that the abnormal behavior occurs, and outputs the recognition result containing the event type, start time, involved target ID, and occurrence area.

[0028] After the recognition result is output, its traceable level needs to be evaluated: by analyzing the development stage and intervention possibility of the event, it is determined whether there is still room for recovery or control. If the event has ended and cannot be intervened, such as after the completion of theft, the suspect escapes, the traceable level is low; if the event is still ongoing or intervention can still alleviate the consequences, such as in the early stages of fighting, the traceable level is high; when evaluating the risk level, the potential impact of the event on personnel safety and public order is combined to determine the degree of severity of the behavior, the range involved, etc. For example, the risk of attacking with weapons is higher than that of slight pushing; the timeliness is determined according to the urgency of the event to determine the optimal response time window. Emergency events (such as sudden fainting) need to be responded to within a short period of time, while slowly developing abnormalities (such as the gradual gathering of crowds) have a relatively relaxed response window. The specific data is described in detail in step A430.

[0029] According to the evaluation results of the above three levels, abnormal feedback information is generated: for high-risk, high-timeliness, and high-traceability level events, the feedback information needs to contain event details, real-time location, and rapid disposal suggestions, and is preferentially pushed to front-line security personnel; for medium-level events, the feedback information is mainly event reminders, and is pushed to regional responsible persons; for low-traceability or low-risk events, the feedback information is only recorded and archived for subsequent review.

[0030] By standardizing feature input, accurately identifying abnormal events, and multi-dimensionally evaluating event attributes, the application generates abnormal feedback information of hierarchical classification, realizes timely response and reasonable disposal of abnormal behaviors in public places, and improves the pertinence and efficiency of safety management.

[0031] Further, the method provided in the embodiment of the application comprises the following steps: A210: dynamically switching input resolution according to real-time crowd density, identifying pedestrian targets in video frames by using a lightweight target detection model, edge marking the identified pedestrians, and generating a bounding box.

[0032] A220: identifying the distance relationship and overlap degree of the boundary position based on the bounding box.

[0033] A230: determining a group cluster area box and a personal pedestrian box according to the distance relationship and overlap degree of the position.

[0034] A240: creating an ID for each of the group cluster area box and the personal pedestrian box to perform pedestrian ID allocation.

[0035] Specifically, after connecting the monitoring device, the real-time crowd density monitoring module is used to obtain the crowd density data of the current scene, the YOLOv8 algorithm is used to realize crowd density grading, the density is divided into three levels of sparse, medium, and dense, the input resolution of the lightweight target detection model is dynamically adjusted according to the density to reduce the calculation load, and the specific division table is shown in Table 1.

[0036] Table 1: Crowd density division table ; ; Then, the YOLOv8s lightweight model is used for pedestrian target identification. The model realizes detection accuracy in dense scenes while maintaining inference speed through decoupling head design and Anchor-Free structure. The model edge marks the video frames to generate a bounding box containing the position and size of the pedestrian, for example, 30-50 bounding boxes are generated per frame in a 1920x1080 original resolution video.

[0037] Further, the construction of the YOLOv8s lightweight target detection model first based on a deep learning framework, adopts a decoupled head design to separate classification and regression tasks, and combines Anchor-Free structure to reduce the dependence on prior boxes to improve the adaptability of dense scenes; In the training stage, multiple scene pedestrian samples in public places are used as training data, which contains pedestrian images and corresponding boundary box labels under different densities and lighting conditions. Through back propagation to optimize model parameters, dynamic resolution adaptation logic is integrated to enable the model to automatically adjust the processing resolution according to the input real-time crowd density data, such as the above-mentioned sparse, medium, and dense classifications. For example, when sparse, the resolution is 1920x1080 pixels, when medium, the resolution is 1280x720 pixels, and when dense, the resolution is 640x480 pixels. The model input is pre-processed video frames and corresponding scene crowd density information, and the output is the edge label result of the pedestrian target in the video frame, i.e. the bounding box, which specifically represents the bounding box containing the pedestrian position coordinates, size width and height, and recognition confidence. On average, 30-50 bounding boxes can be generated per frame, providing accurate target positioning data for subsequent position relationship analysis and pedestrian ID allocation based on bounding boxes.

[0038] Then, the generated bounding boxes are analyzed for position distance relationship and overlap degree: First, calculate the Euclidean distance of adjacent bounding boxes. If the distance is less than 1.5 meters, it is determined to be spatially adjacent.

[0039] At the same time, CIoU (Complete IoU, Complete Intersection over Union) is used to calculate the overlap degree. When the CIoU value is greater than 0.5, it is considered to be highly overlapped. Specifically, when calculating CIoU to determine the overlap degree of the bounding box, first, the coordinate information of the two bounding boxes to be detected is determined, usually represented by the top-left corner (x1, y1) and the bottom-right corner (x2, y2) to represent the position and range of the box. From this, the area of the two boxes and the intersection area can be calculated: the intersection area is the rectangular area of the overlapping part of the two boxes, determined by the maximum x and y values of the top-left corner and the minimum x and y values of the bottom-right corner; The union area is the sum of the areas of the two boxes minus the intersection area, and the intersection over union (IoU) is the ratio of the intersection to the union. On this basis, the Euclidean distance between the centers of the two boxes is calculated, and compared with the diagonal length of the smallest bounding rectangle that can contain both boxes to get the distance penalty term; At the same time, a width-to-height ratio penalty term is introduced, which calculates the shape matching degree by the difference between the width-to-height ratios of the two boxes. Finally, CIoU is obtained by subtracting the weighted sum of the distance penalty term and the width-to-height ratio penalty term from IoU. The larger the value, the higher the overlap of the two boxes. When the value is greater than 0.5, it is determined that the two bounding boxes are highly overlapped.

[0040] The non-maximum suppression (NMS) algorithm is used to remove redundant boxes, a confidence threshold of 0.65 and an IoU threshold of 0.5 are set, and effective boxes with high confidence and no overlap are retained. Specifically, after the lightweight target detection model generates the bounding box of the pedestrian target in the video frame, the NMS algorithm first sorts all the bounding boxes in descending order according to the confidence, and selects the bounding boxes with a confidence higher than 0.65, and removes the redundant candidate boxes with a too low confidence. Then, the highest confidence bounding box is selected from the sorted boxes as the reference box, and the CIoU of the remaining retained boxes and the reference box is calculated. If the CIoU of a certain bounding box and the reference box is greater than 0.5, it is determined as a highly overlapped redundant box and is removed; if the CIoU is less than or equal to 0.5, the bounding box is retained. Then, the highest confidence box is selected from the remaining unprocessed boxes as a new reference box, and the above calculation and selection process is repeated until all retained bounding boxes are processed, and finally the effective bounding boxes with high confidence and no obvious overlap are obtained, which lays a foundation for the subsequent distinction of group clusters and individual pedestrian boxes.

[0041] Then, based on the position relationship and overlap degree of the bounding box, the DBSCAN clustering algorithm is used to determine the group cluster area box and the individual pedestrian box: the distance threshold is set to 1.5 meters and the minimum sample number is 3, and the bounding boxes meeting the conditions are merged into group cluster area boxes; the single bounding box not clustered is regarded as an individual pedestrian box. For example, in a dense scene such as a subway platform, DBSCAN can merge more than 3 adjacent pedestrian boxes into a group cluster, while in a sparse scene such as a square, most of the bounding boxes remain as individual boxes. In the clustering process, the Kalman filter is combined to predict the position of the next frame bounding box, which improves the stability of clustering.

[0042] Finally, according to the clustering result, the pedestrian ID is assigned: the group cluster area box is assigned a shared ID, for example, group001, and the number of members in the cluster is recorded; the individual pedestrian box is assigned a unique ID, such as person001, and the Hungarian algorithm is used to realize cross-frame ID tracking, ensuring the consistency of the ID of the same pedestrian in consecutive frames.

[0043] Through dynamic resolution adjustment to balance the calculation resources and detection accuracy, combined with the lightweight model and clustering algorithm, precise pedestrian recognition and ID assignment in multiple density scenes are realized, which reduces the missed detection rate in dense crowd scenes and significantly improves the accuracy and robustness of abnormal behavior recognition.

[0044] Further, the step A240 in the method provided by the embodiment of the application comprises: A241: acquiring a continuous video stream of a monitoring area, and detecting a motion target area in a video frame.

[0045] A242: based on the motion target area, calculating a group optical flow motion vector and a main color histogram to generate a feature signature.

[0046] A243: Determine a crowd cluster based on the feature signature, and create a hierarchical ID for the crowd cluster.

[0047] In the embodiments of the present application, the moving target region refers to a region where a moving target in a video frame detected from a continuous video stream of a monitoring region is located. The crowd optical flow motion vector is a normalized 12-dimensional motion direction distribution vector obtained by dividing the optical flow direction into 12 30° intervals, and calculating the proportion of each interval vector based on the motion speed as the weight.

[0048] Optionally, when acquiring the continuous video stream of the monitoring region, the video data is received in real time from the distributed monitoring device, such as a high-definition camera array, through the RTSP network protocol, and is converted into a standardized RGB format frame sequence after hardware decoding. The frame rate is kept at 25 frames per second to ensure the continuity of the motion, and the single frame resolution is dynamically adapted according to the real-time crowd density, such as 1280x720 pixels in a medium density scene. At the same time, the video streams of multiple devices are synchronized through the timestamp to ensure the consistency of the frame sequence in the time dimension.

[0049] Then, the moving target region in the video frame is detected, and a mixed Gaussian background modeling (MOG2) algorithm is used to construct a dynamic background model: the model input is the pre-processed video frame of the monitoring region for 100 continuous frames, usually in RGB or grayscale format, and the resolution is dynamically adapted according to the scene, as shown in Table 1. The construction process first initializes 3-5 Gaussian distributions for each pixel of each frame image, and each distribution includes the mean value, which describes the brightness / color center value of the pixel; the variance, which describes the fluctuation range of the pixel value; and the weight, which describes the probability that the distribution belongs to the background.

[0050] Subsequently, the pixels are updated frame by frame: the matching degree of the current pixel value with each Gaussian distribution is calculated, and if the pixel value falls within the 2.5 times variance range of a certain distribution, it is considered to be matched. When the matching is successful, the weight, mean value (fine-tuned to the current pixel value) and variance (adjusted according to the fluctuation) of the distribution are updated through the iterative formula. When the matching fails, the weight of the existing distribution is reduced and a new Gaussian distribution is created for the new pixel value. If the total number of distributions exceeds the upper limit, the distribution with the lowest weight is replaced. After 100 frames of learning, the Gaussian distributions of each pixel are sorted by weight, and the first few distributions whose cumulative weight percentage exceeds 80% are retained as part of the background model. Finally, a dynamic background model containing multiple Gaussian distribution parameters for each pixel, i.e. mean value, variance and weight, is output, which provides a basis for subsequent differentiation between moving targets (foreground) and background.

[0051] Afterwards, for each new input video frame, the matching degree of the pixel points with the background model is calculated, and when the matching degree is lower than 0.7, it is marked as a foreground pixel; the foreground pixel region is binarized with a threshold of 0.5, and a 3*3 structure element is used for morphological erosion operation to remove isolated noise points, and then through inflation operation to connect the broken foreground region, finally the connected region with an area greater than 30*30 pixels is extracted as the moving target region, and an average of 5-20 moving target regions can be detected per frame.

[0052] Next, the optical flow direction is divided into 12 30° intervals and the proportion of each interval vector is calculated to output a 12-dimensional motion direction distribution vector, the group region is converted to the HSV color space and the pixel distribution is calculated according to 8 color tones and 4 saturation regions to output a 32-dimensional color histogram vector, and then the two vectors are weighted according to the weight to obtain the feature label, and the specific steps are described in detail in A242-1-A242-3.

[0053] After obtaining the feature signature of the group, first, the similarity between the feature signatures of different moving target regions is calculated: the cosine similarity algorithm is used to compare the 44-dimensional vectors of each feature signature, i.e. 12-dimensional motion direction distribution vector + 32-dimensional color histogram vector, and if the similarity value is greater than 0.85, it is determined that the group features of the two regions are highly similar. After comparing all the feature signatures, the DBSCAN clustering algorithm is used for group cluster division, the eps (neighborhood radius) is set to 0.15 and the minimum sample number is set to 3, the feature signatures that meet the similarity condition and have a number not less than 3 are clustered into a group cluster, and the single or feature difference is large. It is temporarily classified as an independent individual or a to-be-confirmed cluster.

[0054] After determining the group cluster, a hierarchical ID is created for each group cluster: first, a primary main ID is assigned to the group cluster, which adopts the Group_XX format, XX is an Arabic numeral, such as Group_001, Group_002, etc., and the main ID uniquely identifies the group cluster; then a secondary sub-ID is assigned to each pedestrian target contained in the group cluster, which adopts the Group_XX_Person_YY format, YY is the cluster index, such as Group_001_Person_01, the sub-ID is associated with the main ID, and records the position index of the individual in the group, such as the coordinate offset relative to the center of the group.

[0055] At the same time, a dynamic updating mechanism of the hierarchical ID is established, when a pedestrian is added or removed in the group cluster, the sub-ID sequence number is automatically updated, if the group cluster is split into two sub-clusters with significant feature difference, i.e. the cosine similarity is less than 0.6, then a new main ID is assigned to the new sub-cluster, such as Group_001 splits out Group_003, to ensure that the ID system is synchronized with the dynamic changes of the group.

[0056] The group cluster is accurately divided by feature signature clustering, and the hierarchical ID structure is combined to realize the association identification of groups and individuals, which not only retains the traceability of the overall characteristics of the group, but also can be refined to the tracking of individual behaviors, and provides a clear target hierarchical relationship for subsequent abnormal behavior identification.

[0057] Further, step A250 in the method provided by the embodiment of the application includes: A242-1: dividing the optical flow direction into 12 30° intervals, calculating the proportion of each interval vector based on the weight of the motion speed, and outputting a normalized 12-dimensional motion direction distribution vector based on the proportion of each interval vector.

[0058] A242-2: converting the group region to the HSV color space, calculating the pixel distribution according to 8 hue regions and 4 saturation regions, and outputting a 32-dimensional color histogram vector.

[0059] A242-3: weighting according to the weight proportion of the motion direction distribution vector and the color histogram vector to obtain a feature label.

[0060] In the embodiment of the application, the optical flow direction is divided into 12 30° intervals.

[0061] Specifically, when processing based on the moving target region, first, the group-related pixels are extracted from the detected moving target region, and the static background and isolated small targets, such as regions with an area less than 50*50 pixels, are excluded, and the focus is on the group region with continuous motion and a pixel area ratio exceeding 60% of the total area, so as to delimit the effective range for subsequent feature calculation.

[0062] Then, when calculating the group optical flow motion vector, first, the continuous two frames of images of the group region are preprocessed to convert them into grayscale images to reduce the calculation complexity while retaining the pixel brightness change information. When the Farneback algorithm is used, it is assumed that there is a quadratic polynomial expansion in the local neighborhood of each pixel in the image, and the 3*3 neighborhood of each pixel in the two frames of images is fitted by the least square method to obtain a polynomial coefficient matrix describing the grayscale distribution of the image. Based on the polynomial coefficients of the two frames of images, the displacement vector in the neighborhood is calculated to preliminarily estimate the motion trend of the pixel; then, Gaussian smoothing processing is introduced to filter the initial displacement vector to reduce noise interference, and the displacement field is refined through iteration optimization, usually 3-5 times, so that the optical flow estimation is closer to the real motion. Finally, the algorithm outputs the motion parameters of each pixel in the group region: the motion direction (range 0-360°, accurate to 1°) and the motion speed (unit: pixel / frame, reflecting the distance of pixel movement between two frames), which provides basic data for subsequent division of the optical flow direction into 12 30° intervals and calculation of the proportion of each interval vector.

[0063] Afterwards, the light flow direction is divided into 12 intervals at intervals of 30°, 0-30°, 30-60°, and so on until 330-360°, and the proportion of the sum of the speeds of all pixels in each interval in the total speed of the group area is calculated with the speed of each pixel as the weight, to obtain the proportion of 12 interval vectors. The proportions are subjected to L1 normalization processing so that the sum of each dimension is 1, and a normalized 12-dimensional motion direction distribution vector is output.

[0064] After converting the group area into the HSV color space, the 0-360° is divided into 8 regions according to the hue H channel, each region being 45°, and the 0-1 is divided into 4 regions according to the saturation S channel, each region being 0.25, forming 8x4=32 color regions. The proportion of the number of pixels in each region in the total number of pixels of the group area is calculated, and a 32-dimensional color histogram vector is output, for example, a certain travel group occupies 30% in the red hue 0-45° region and 40% in the high saturation 0.75-1 region, corresponding to vector dimension values of 0.30 and 0.40.

[0065] Finally, the weight proportions of the motion direction distribution vector and the color histogram vector are set according to the scene requirements, such as a motion feature weight of 0.6 and a color feature weight of 0.4, the dimensions of the two vectors are weighted respectively, and then spliced to form a 44-dimensional feature label as the feature signature of the group.

[0066] By fusing the group motion direction distribution and color distribution features, a feature signature that can uniquely represent the characteristics of the group is generated, the precise differentiation of different group clusters is realized, and a reliable basis is provided for subsequent group recognition and abnormal behavior detection based on the feature signature.

[0067] Further, the step A300 in the method provided by the embodiment of the application comprises: A310: performing dynamic target screening based on the pedestrian ID to obtain an identification target ID.

[0068] A320: tracking and identifying the identification target ID between video frames to obtain a motion change trajectory and a posture change bounding box between the video frames.

[0069] A330: performing motion trajectory coordinate positioning and action form recognition based on the motion change trajectory and the posture change bounding box to generate a motion trajectory and an action feature.

[0070] Specifically, first, a motion speed threshold and a pixel area threshold are set, the motion features and the pixel features corresponding to the pedestrian ID are identified and determined, and the pedestrian IDs meeting the threshold requirements are selected as the identification target ID, and the specific steps are described in detail in A311-A312.

[0071] Then, when tracking the identified target ID between video frames, a Kalman filter algorithm is used to predict the possible position of the target in the current frame, and a Hungarian algorithm is used to match the target bounding box of the previous frame with the detection box of the current frame: the intersection over union (IoU) of the two boxes is calculated, and when the IoU is greater than 0.6, it is determined that they are the same target, and the tracking state is updated after successful matching. For consecutive video frames with a frame rate of 25 frames per second and an interval of 40 ms, the position changes of the target ID are continuously recorded, and if the matching is successful for 3 consecutive frames, the position coordinates (x, y) are concatenated in time order to form a motion change trajectory; at the same time, by changing the aspect ratio of the bounding box (such as from 1.2 to 0.8) and the vertex coordinates, the posture change box is captured to reflect the posture adjustment of the target such as bending and turning.

[0072] Then, when positioning the coordinates according to the motion change trajectory, the intrinsic parameters (focal length, pixel size) and extrinsic parameters (installation angle, height) of the monitoring camera are combined to convert the pixel coordinates into actual physical coordinates, such as 1 pixel corresponding to 0.01 meters, and pixel coordinates (100, 200) being converted into actual coordinates (1.0 meters, 2.0 meters) to generate a motion trajectory in meters with an accuracy of 0.05 meters. Action pattern recognition is achieved by extracting key points within the posture change box, such as the head, shoulders, and limb endpoints, calculating the relative displacement and angle change of adjacent frame key points, such as the elbow joint angle changing from 170° to 90° to determine a bending action, and quantifying these changes into a 128-dimensional action feature vector, which includes information such as action amplitude, speed, and duration.

[0073] By tracking consecutive motion trajectories and posture changes between frames, combining coordinate conversion and key point analysis to generate accurate motion trajectories and action features, the method provides high-quality input data for the abnormal behavior recognition model, and improves the accuracy and completeness of the behavior feature description.

[0074] Further, the step A310 in the method provided by the embodiment of the application includes: A311: setting a motion speed threshold and a pixel area threshold.

[0075] A312: using the motion speed threshold and the pixel area threshold to identify and determine the motion features and pixel features corresponding to the pedestrian ID, and screening the pedestrian ID that meets the threshold requirement as the identified target ID.

[0076] Specifically, first, the regional risk value of the identified target region is obtained, the risk region distribution range and the risk coefficient of each region are determined, and then the motion speed threshold and the pixel area threshold are dynamically adjusted, which is described in detail in A311-1-A311-3.

[0077] Next, the pedestrian IDs are identified and determined using the set motion speed threshold Vmin and pixel area threshold Pmin: for each pedestrian ID, first compare its motion speed with Vmin, if the motion speed < Vmin, the target below this value is mostly static or slowly moving static target, determine as a static target, and mark as an ID to be skipped tracking; then compare its pixel area with Pmin, if the pixel area < Pmin, the target below this value is mostly a long-distance target with a long distance from the camera and blurred details, determine as a long-distance target, and also mark as an ID to be skipped tracking. After double determination, the pedestrian IDs with motion speed ≥ Vmin and pixel area ≥ Pmin pixels are screened out, and these IDs are determined as the identified target IDs that need to be tracked. For example, if Vmin is 0.3 m / s and Pmin is 100 pixels in a certain scene, the motion speed of a certain pedestrian ID is 0.6 m / s and the pixel area is 200 pixels, which meets the double threshold requirements and is selected as an identified target ID; another pedestrian ID with a motion speed of 0.2 m / s is a static target or a long-distance target with a pixel area of 80 pixels, which is excluded.

[0078] By setting the motion speed and pixel area double thresholds, the pedestrian IDs that meet the tracking conditions are accurately screened out, the invalid tracking of static and long-distance targets is skipped, the calculation resource consumption is reduced, and the effectiveness and accuracy of the tracking target are ensured.

[0079] Further, the step A311 in the method provided by the embodiment of the application includes: A311-1: Obtain a region risk value of an identified target region.

[0080] A311-2: Determine a risk region distribution range and a risk coefficient of each region according to the region risk value.

[0081] A311-3: Dynamically adjust the motion speed threshold and the pixel area threshold according to the risk region distribution range and the risk coefficient of each region.

[0082] In one embodiment, when the region risk value of the identified target region is obtained, the function attribute, the historical abnormal event frequency and the environmental characteristics of the monitored region are combined for quantitative evaluation: by calling the monitoring data in the past 12 months, the abnormal events such as crowding, pushing and running in each region are counted, the events are weighted according to the severity, the slight event is 1 point, and the serious event is 5 points, the region risk base score is calculated; at the same time, the region function label such as the bank entrance and the children activity area is referred to, the risk addition is added to the special function region, the bank entrance adds 3 points, the children activity area adds 4 points, and finally the region risk value is standardized to the interval of 0-10, wherein the risk value of the children activity area is usually 7-9, the risk value of the bank entrance is 6-8, and the risk value of the ordinary channel is 2-4.

[0083] Next, when determining the distribution range of risk areas and the risk coefficient of each area based on the regional risk value, the risk values ​​are used as the basis for dividing the risk range: for areas with a risk value of 8-10, the distribution range is set as a circular area with a radius of 50 meters and the center of the area as the origin, with a risk coefficient of 0.8-1.0; for areas with a risk value of 4-7, the distribution range is a circular area with a radius of 30 meters and a risk coefficient of 0.4-0.7; for areas with a risk value of 0-3, the distribution range is a circular area with a radius of 20 meters and a risk coefficient of 0.1-0.3. For example, a certain children's activity area has a risk value of 9, its risk area distribution range is a circular area with a radius of 50 meters and a risk coefficient of 0.9; a certain ordinary corridor has a risk value of 2, its distribution range is 20 meters and its risk coefficient is 0.2.

[0084] Then, based on the distribution range of risk areas and the risk coefficient, the movement speed threshold and pixel area threshold are dynamically adjusted to establish a correlation formula between the threshold and the risk coefficient: Movement speed threshold = base speed threshold (2m / s) × (1 - risk coefficient × 0.5), pixel area threshold = base area threshold (300 pixels) × (1 - risk coefficient × 0.4). For example, in the children's activity area with a risk coefficient of 0.9, the movement speed threshold is adjusted to 2 × (1 - 0.9 × 0.5) = 1.1m / s, and the pixel area threshold is adjusted to 300 × (1 - 0.9 × 0.4) = 192 pixels; in the ordinary corridor with a risk coefficient of 0.2, the movement speed threshold is 2 × (1 - 0.2 × 0.5) = 1.8m / s, and the pixel area threshold is 300 × (1 - 0.2 × 0.4) = 276 pixels, achieving more sensitive target detection in high-risk areas and reducing invalid recognition in low-risk areas.

[0085] By quantifying regional risks and linking them to threshold adjustments, the motion speed and pixel area thresholds can adapt to the security requirements of different regions. This ensures timely capture of abnormal behavior in high-risk regions while reducing the false positive rate in low-risk regions, thus improving the accuracy of target selection.

[0086] Furthermore, step A400 in the method provided in this application embodiment includes: A410: Collect abnormal samples from multiple scenarios based on abnormal cases in public places, including abnormal motion features, abnormal pedestrian posture change features, pedestrian distance relationship change features, and corresponding abnormal event labels.

[0087] A420: Based on abnormal samples from multiple scenarios, multiple recognition channels are learned and converged according to the type of abnormal event label to obtain multiple abnormal recognition sub-channels.

[0088] A430: Assign weights to identify abnormal events based on their traceability level, risk level, and timeliness.

[0089] A440: integrate the plurality of abnormality recognition sub-channels based on the abnormal event recognition weight distribution to obtain the abnormal behavior recognition model.

[0090] In the embodiments of the present application, the abnormal samples include stampede events, weapon attack events, falling and fainting events, etc.

[0091] Optionally, when collecting multi-scene abnormal samples according to abnormal cases in public places, first, select multiple typical public places such as train station waiting hall, shopping mall atrium, school playground, etc., and deploy high-definition monitoring equipment with a resolution of 1920x1080 and a frame rate of 25 frames / second to collect video data for multiple high-frequency abnormal events such as stampede events, theft, fighting, weapon attack, falling and fainting, etc. At least 1000 abnormal event videos are collected for each type of scene, each video is 30-60 seconds long, covers different time periods (weekdays / weekends, daytime / nighttime) and crowd densities (sparse / medium / dense), and ensures the scene diversity and coverage of the samples. At the same time, the environmental parameters of the event occurrence are recorded synchronously, such as light intensity and whether there are obstacles, to provide scene context information for feature extraction.

[0092] When collecting abnormal motion features, the optical flow algorithm is used to calculate the motion speed and direction change rate of the target: for example, the average motion speed of the crowd in the stampede event can reach 3-5 m / s, and the direction confusion degree, i.e. the proportion of targets moving in different directions, is more than 60%; the motion speed of the thief is mostly 1-2 m / s, and the direction presents repeated detours, with a direction change rate of ≥120° / second. When extracting abnormal pedestrian posture change features, the key point detection algorithm is used to calculate the angle change of the key nodes of the head, shoulders and knees: for example, when falling and fainting, the angle between the human body trunk and the ground decreases from 90° straight to ≤30°, and the change time is ≤2 seconds; in the fighting event, the elbow joint and wrist joint of the upper limb joint have a fluctuation range of more than 60°, and the duration is ≥5 seconds. When recording the distance relationship change features of pedestrians, the Euclidean distance and change rate between targets are calculated: for example, in the weapon attack event, the distance between the attacker and the victim quickly decreases from ≥2 m to ≤0.5 m, and the distance change rate is ≥1.5 m / second; while the distance change rate of normal crowds is mostly stable at 0-0.3 m / second.

[0093] Next, corresponding abnormal event labels are added to the collected samples, and a combination of manual labeling and machine pre-labeling is used: first, the event type is preliminarily identified by a target detection model, and then reviewed by multiple professional labeling personnel in the field, to label the start / end frame, core target ID and event category, such as stampede, theft, fighting, etc., and the label consistency needs to reach more than 95%. The finally constructed abnormal behavior sample library contains labeled videos and feature data, each sample is associated with an abnormal motion feature vector, a posture change feature vector, a distance relationship feature vector and a corresponding event label, forming a structured sample data system.

[0094] Then, the samples are clustered according to the abnormal event label to construct a cluster, the time sequence features are fitted in multiple stages, positive and negative example samples are constructed, and the convergence is trained to obtain an abnormal recognition sub-channel of multiple event types, and the defense stage node determined based on the traceable level and the like is added as an output constraint to the corresponding sub-channel. The specific steps are described in detail in A421-A425.

[0095] After that, when the abnormal event recognition weight is distributed according to the traceable level, risk level, and timeliness of the abnormal event, the three indicators are quantified first: for example, the traceable level is divided into 1-10 points according to the event development stage and the possibility of intervention, and 10 points are the most valuable for intervention, i.e. the event is in continuous and intervention can significantly reduce the consequences. If the fight is in a continuous state at the initial stage, timely intervention can prevent the situation from escalating, and 9 points are obtained. After the theft is completed, the suspect has escaped, and it is impossible to recover the loss through intervention, and 3 points are obtained. If the crowd stampede is in the initial stage of spread, intervention can control the range of expansion, and 6 points are obtained; the risk level is divided into 1-10 points according to the possible personnel safety and property loss, and 10 points are the highest risk, such as stampede event with 10 points, and single fall with 3 points; timeliness is divided into 1-5 levels according to the best response time, level 1 is the most urgent response, such as fainting within 5 minutes, level 1, and delayed response for property theft, level 3.

[0096] Then, the weight proportions of the three indicators are determined: the risk level accounts for 50%, the timeliness accounts for 30%, and the traceable level accounts for 20%. The weight of each abnormal event type is calculated by the formula: comprehensive weight = risk score x 0.5 + timeliness level x 0.3 + traceable score x 0.2. For example, the comprehensive weight of armed attack (risk 10 points, timeliness level 1, traceable 9 points) is 10 x 0.5 + 1 x 0.3 + 9 x 0.2 = 5 + 0.3 + 1.8 = 7.1, and the comprehensive weight of fall and fainting (risk 3 points, timeliness level 1, traceable 6 points) is 3 x 0.5 + 1 x 0.3 + 6 x 0.2 = 1.5 + 0.3 + 1.2 = 3.0.

[0097] Finally, when integrating multiple anomaly recognition sub-channels based on abnormal event identification weight distribution, a weighted voting mechanism is adopted: the recognition result (such as yes / no for an event) of each sub-channel to the input features is accompanied by a confidence score of 0-1, and the confidence scores of each sub-channel are multiplied by the corresponding event weight and summed to obtain the total score of the event type. Set the total score threshold to 0.6, when the total score exceeds the threshold, it is determined that the abnormal event; if the total scores of multiple event types exceed the threshold, sort them from high to low, and take the top two as the candidate results. For example, the input features trigger the fighting sub-channel (confidence score 0.9, weight 5.2) and the weapon attack sub-channel (confidence score 0.7, weight 7.1) at the same time, the total scores are 0.9*5.2=4.68, 0.7*7.1=4.97, and both exceed 0.6, so the final output is the abnormal behavior of weapon attack and fighting. At the same time, the abnormal behavior recognition model retains the original output of each sub-channel, which is used for generating abnormal feedback information in the subsequent process.

[0098] By quantifying the key attributes of abnormal events and assigning weights, and then integrating multiple recognition sub-channels in a weighted manner, the abnormal behavior recognition model can not only highlight the identification priority of high-risk and high-urgency events, but also comprehensively consider multi-dimensional features to improve recognition accuracy, achieving efficient and accurate identification of multiple abnormal behaviors in public places.

[0099] Further, the step A420 in the method provided in the embodiment of the application comprises: A421: clustering the multi-scene abnormal samples based on the abnormal event label type, and constructing a multi-event type sample cluster.

[0100] A422: performing multi-stage feature fitting on the time sequence change characteristics of the multi-event type sample cluster, and establishing multi-stage time sequence characteristics.

[0101] A423: based on the multi-stage time sequence characteristics, constructing positive example samples and negative example samples according to the multi-scene abnormal samples.

[0102] A424: using the multi-stage time sequence characteristics of the positive example samples and the negative example samples for recognition model learning convergence, and obtaining multiple abnormal recognition sub-channels corresponding to multiple event types.

[0103] A425: wherein, according to the traceable level, risk level and timeliness of the abnormal event, a defense stage node is determined, and the defense stage node is added to the abnormal recognition sub-channel of the corresponding event type as a recognition output constraint stage.

[0104] In one embodiment, first, multi-scene abnormal samples are clustered based on abnormal event label types, a K-means clustering algorithm is used, the initial class center is the abnormal event label, the sample feature vector is calculated, the cosine similarity between the abnormal motion, posture, and distance relationship features is included, and the samples with a similarity greater than 0.85 are classified into the same cluster. Finally, multiple event type sample clusters are constructed, each cluster contains 5,000-10,000 samples, and the crowd density of the stampede event cluster is greater than 8 people The fighting event cluster contains more than 70% of samples with close-range body contact, ensuring the concentration of features in the same event.

[0105] Then, the multi-stage feature fitting of the time sequence change characteristics of the multi-event type sample cluster is performed, the time sequence characteristics of each sample are divided into three stages: the initial stage 0-10 seconds, the development stage 10-20 seconds, and the outbreak stage 20-30 seconds, according to the event development cycle. The features of each stage are fitted using a long short-term memory network (LSTM), and the feature mean, change rate, and other key indicators within the stage are extracted to establish a 256-dimensional multi-stage time sequence feature vector, in which the feature weight of the outbreak stage accounts for 40% to highlight the feature performance at the event exacerbation.

[0106] Subsequently, when constructing positive and negative samples based on multi-stage time sequence characteristics, the positive samples are real abnormal event samples confirmed by manual annotation, such as samples containing explicit punching and pushing actions in fighting events, accounting for 60%; the negative samples are selected from scenes similar to abnormal features but actually normal, such as crowded subway platform crowds, whose motion characteristics are similar to the initial stage of stampede but have no crushing and falling behavior, accounting for 40%, and each negative sample needs to have a time sequence similarity of 0.7-0.8 with at least one positive sample to enhance the model's ability to distinguish between easily confused scenes and avoid recognition errors.

[0107] After that, the positive and negative samples are used to learn and converge the recognition model based on multi-stage time sequence characteristics, each event type cluster is trained separately, a ResNet-18 deep residual network is used as the basic architecture, a cross-entropy loss function is used to optimize the model parameters, and the model is iteratively trained for 80 rounds. When the verification set accuracy is stable at more than 90% and the fluctuation is less than 1% for 5 consecutive rounds, the convergence is stopped, and finally multiple abnormal recognition sub-channels are obtained, each sub-channel has a recognition accuracy of 91%-95% for the corresponding event type.

[0108] Finally, defense phase nodes are determined. Based on the traceability level, risk level, and timeliness of abnormal events, and through statistical analysis of historical data, key phases for each event are identified. For example, in stampede events, when the crowd's movement speed is >2 m / s and the overlap is >60%, 80% of the events will escalate at this stage; this stage is designated as the defense phase node. In fight events, the stage where the frequency of physical contact is >3 times / second is designated as a node. These nodes are added as recognition output constraint stages to the corresponding sub-channels to ensure that the sub-channels must output recognition results at this stage, with the false alarm rate controlled below 5%.

[0109] By clustering to gather features of similar events, time-series fitting to capture the development patterns of events, training with positive and negative examples to improve discrimination capabilities, and constraining nodes to control output timing, the resulting multiple anomaly recognition sub-channels can accurately identify corresponding types of abnormal events and output results in a timely manner at key stages, thereby improving the accuracy and timeliness of abnormal behavior recognition.

[0110] In summary, the deep learning-based method for identifying abnormal behavior in public places provided in this application has the following technical effects: This application acquires and preprocesses video streams by connecting to monitoring equipment, identifies pedestrian targets and assigns IDs using a lightweight target detection model, constructs motion trajectories based on ID tracking and identifies behavioral characteristics, and performs identification using a recognition model trained on an abnormal behavior sample library. Feedback information is generated based on the traceability level of the results, thereby accurately identifying various abnormal behaviors in public places. This makes the identification results of abnormal behaviors in public places more accurate and reliable, achieving the technical effect of accurately identifying abnormal behaviors in public places and outputting accurate abnormal feedback information that includes traceability level, risk level, and timeliness, thus meeting the needs of safety management and control.

[0111] Example 2, as Figure 2 As shown, based on the same inventive concept as the aforementioned Embodiment 1, this application provides a deep learning-based system for identifying abnormal behavior in public places, the system comprising: The video frame preprocessing execution module 1 is used to connect to the monitoring equipment, acquire the monitoring video stream in real time, and perform video frame preprocessing on the video stream.

[0112] Pedestrian ID allocation execution module 2 is used to identify pedestrian targets in video frames through a lightweight target detection model, generate bounding boxes, and allocate pedestrian IDs based on the bounding boxes.

[0113] The inter-frame behavior and action feature recognition module 3 performs pedestrian tracking in video frames based on the assigned pedestrian ID, constructs motion trajectories between consecutive frames, and identifies inter-frame behavior and action features.

[0114] An abnormal behavior recognition model construction module 4 is configured to establish an abnormal behavior sample library for deep learning of multiple abnormal behavior features, and obtain an abnormal behavior recognition model.

[0115] An abnormal feedback information acquisition module 5 is configured to input the motion trajectory and behavior action feature into the abnormal behavior recognition model for abnormal behavior recognition, output an abnormal behavior recognition result, and generate abnormal feedback information according to a traceable level, a risk level, and timeliness of the abnormal behavior recognition result.

[0116] Further, the pedestrian ID allocation execution module 2 is configured to perform the following steps: According to real-time crowd density, an input resolution is dynamically switched, a lightweight target detection model is used to identify a pedestrian target in a video frame, the identified pedestrian is edge-labeled, and a bounding box is generated; a boundary position distance relationship and an overlap degree are identified based on the bounding box; a group cluster area box and a personal pedestrian box are determined according to the position distance relationship and the overlap degree; and an ID is respectively created according to the group cluster area box and the personal pedestrian box for pedestrian ID allocation.

[0117] Further, the pedestrian ID allocation execution module 2 is configured to perform the following steps: A continuous video stream of a monitoring area is acquired, and a motion target area in a video frame is detected; a group optical flow motion vector and a principal color histogram are generated based on the motion target area; and a group cluster is determined based on a feature signature, and a hierarchical ID is created for the group cluster.

[0118] Further, the pedestrian ID allocation execution module 2 is configured to perform the following steps: A light flow direction is divided into 12 30° intervals, and a proportion of a vector in each interval is calculated with motion speed as a weight; a normalized 12-dimensional motion direction distribution vector is output based on the proportion of the vector in each interval; a group area is converted to an HSV color space, a pixel distribution is calculated according to 8 hue regions and 4 saturation regions, and a 32-dimensional color histogram vector is output; and a feature label is obtained by weighting according to a weight proportion based on the motion direction distribution vector and the color histogram vector.

[0119] Further, the inter-frame behavior action feature recognition module 3 is configured to perform the following steps: A dynamic target is screened based on the pedestrian ID, and an identification target ID is obtained; a motion change trajectory and a posture change bounding box between video frames are obtained by tracking and identifying the identification target ID between the video frames; motion trajectory coordinate positioning and action form recognition are performed based on the motion change trajectory and the posture change bounding box, and a motion trajectory and an action feature are generated.

[0120] Further, the inter-frame behavior action feature recognition module 3 is configured to perform the following steps: A motion speed threshold and a pixel area threshold are set, and the motion features and the pixel features corresponding to the pedestrian IDs are identified and determined using the motion speed threshold and the pixel area threshold, so that the pedestrian IDs meeting the threshold requirements are screened as the identification target IDs.

[0121] Further, the inter-frame behavior action feature recognition module 3 is configured to perform the following steps: A region risk value of the identification target region is obtained, and a risk region distribution range and a risk coefficient of each region are determined according to the region risk value; and the motion speed threshold and the pixel area threshold are dynamically adjusted according to the risk region distribution range and the risk coefficient of each region.

[0122] Further, the abnormal behavior recognition model construction module 4 is configured to perform the following steps: Multi-scene abnormal samples are collected according to abnormal cases in public places, including abnormal motion features, abnormal pedestrian posture change features, pedestrian distance relationship change features and corresponding abnormal event labels; multi-recognition channel learning convergence is performed according to the abnormal event label types based on the multi-scene abnormal samples, to obtain a plurality of abnormal recognition sub-channels; abnormal event recognition weights are distributed according to the traceability level, risk level and timeliness of the abnormal events; and the plurality of abnormal recognition sub-channels are integrated based on the abnormal event recognition weight distribution, to obtain the abnormal behavior recognition model.

[0123] Further, the abnormal behavior recognition model construction module 4 is configured to perform the following steps: The multi-scene abnormal samples are clustered based on the abnormal event label types, to construct a multi-event type sample cluster; multi-stage feature fitting is performed on the time sequence change features of the multi-event type sample cluster, to establish multi-stage time sequence features; positive samples and negative samples are constructed based on the multi-stage time sequence features and the multi-scene abnormal samples; and the multi-stage time sequence features of the positive samples and the negative samples are used for recognition model learning convergence, to obtain a plurality of abnormal recognition sub-channels corresponding to the multi-event types; wherein, a defense stage node is determined according to the traceability level, the risk level and the timeliness of the abnormal events, and the defense stage node is added to the abnormal recognition sub-channels of the corresponding event types as an identification output constraint stage.

[0124] The public place abnormal behavior recognition system based on deep learning provided in the embodiments of the present application can perform the public place abnormal behavior recognition method based on deep learning provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0125] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, however, any number of different modules can be used and run on the user terminal and / or server, the various units and modules are only divided according to the functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific name of each functional unit is only for the convenience of mutual differentiation, and does not serve to limit the protection scope of the present application.

[0126] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application. In some cases, the actions or steps described in the present application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

Claims

1. A method for identifying abnormal behavior in public places based on deep learning, characterized in that, include: Connect to monitoring equipment, acquire monitoring video streams in real time, and perform video frame preprocessing on the video streams; A lightweight object detection model is used to identify pedestrian targets in video frames, generate bounding boxes, and assign pedestrian IDs based on the bounding boxes. Based on the assigned pedestrian ID, pedestrian tracking is performed on video frames to construct motion trajectories between consecutive frames and identify behavioral action features between frames. An abnormal behavior sample library is established to perform deep learning on multiple abnormal behavior features to obtain an abnormal behavior recognition model; The motion trajectory and behavioral features are imported into the abnormal behavior recognition model for abnormal behavior recognition, and the abnormal behavior recognition results are output. Abnormal feedback information is generated based on the traceability level, risk level, and timeliness of the abnormal behavior recognition results.

2. The method for identifying abnormal behavior in public places based on deep learning according to claim 1, characterized in that, A lightweight object detection model is used to identify pedestrian targets in video frames, generate bounding boxes, and assign pedestrian IDs based on the bounding boxes, including: The input resolution is dynamically switched according to the real-time crowd density. A lightweight target detection model is used to identify pedestrian targets in video frames, and edge marking is performed on the identified pedestrians to generate bounding boxes. Based on the bounding box, identify the boundary position distance relationship and overlap. Based on location distance and overlap, determine the group cluster area bounding box and the individual pedestrian bounding box; Based on the group cluster region frame and the individual pedestrian frame, create IDs and assign pedestrian IDs.

3. The method for identifying abnormal behavior in public places based on deep learning according to claim 2, characterized in that, Creating IDs for pedestrian ID assignment also includes: Acquire continuous video streams of the monitored area and detect moving target regions in the video frames; Based on the moving target region, the group optical flow motion vector and the principal color histogram are calculated to generate feature signatures; Based on the feature signature, a population cluster is determined, and a hierarchical ID is created for the population cluster.

4. The method for identifying abnormal behavior in public places based on deep learning according to claim 3, characterized in that, Calculate the group optical flow motion vector and the principal color histogram to generate feature signatures, including: The optical flow direction is divided into 12 30° intervals, and the proportion of the vector in each interval is calculated with the motion velocity as the weight. Based on the proportion of the vector in each interval, a normalized 12-dimensional motion direction distribution vector is output. Convert the group region to the HSV color space, calculate the pixel distribution according to 8 hue regions and 4 saturation regions, and output a 32-dimensional color histogram vector; Feature labels are obtained by weighting the motion direction distribution vector and color histogram vector according to the weight ratio.

5. The method for identifying abnormal behavior in public places based on deep learning according to claim 3, characterized in that, Based on the assigned pedestrian ID, pedestrian tracking is performed on video frames to construct motion trajectories between consecutive frames and identify inter-frame behavioral action features, including: Dynamic target filtering is performed based on the pedestrian ID to obtain the identification target ID; Based on the target ID, the video frames are tracked and identified to obtain the motion change trajectory and pose change bounding box between video frames; Based on the motion change trajectory and posture change bounding box, the motion trajectory coordinates are located and the action shape is recognized to generate the motion trajectory and action features.

6. The method for identifying abnormal behavior in public places based on deep learning according to claim 5, characterized in that, Dynamic target filtering is performed based on the pedestrian ID to obtain the target ID for identification, including: Set motion speed threshold and pixel area threshold; The motion speed threshold and pixel area threshold are used to identify and determine the motion features and pixel features corresponding to the pedestrian ID, and the pedestrian IDs that meet the threshold requirements are selected as the identification target IDs.

7. The method for identifying abnormal behavior in public places based on deep learning according to claim 6, characterized in that, Set motion speed threshold and pixel area threshold, including: Obtain the regional risk value of the target area; Based on the regional risk values, determine the distribution range of risk areas and the risk coefficient of each area; Based on the distribution range of the risk areas and the risk coefficient of each area, the motion speed threshold and pixel area threshold are dynamically adjusted.

8. The method for identifying abnormal behavior in public places based on deep learning according to claim 1, characterized in that, An abnormal behavior sample library is established, and deep learning of multiple abnormal behavior features is performed to obtain an abnormal behavior recognition model, including: Based on abnormal cases in public places, we collected abnormal samples from multiple scenarios, including abnormal motion characteristics, abnormal pedestrian posture change characteristics, pedestrian distance relationship change characteristics, and corresponding abnormal event labels. Based on abnormal samples from multiple scenarios, multiple recognition channels are learned and converged according to the abnormal event label type to obtain multiple abnormal recognition sub-channels. The weights for identifying abnormal events are allocated according to their traceability level, risk level, and timeliness. The multiple anomaly recognition sub-channels are integrated based on the anomaly event recognition weight allocation to obtain the anomaly behavior recognition model.

9. The method for identifying abnormal behavior in public places based on deep learning according to claim 8, characterized in that, Multiple anomaly detection sub-channels were obtained, including: Based on the abnormal event label type, the abnormal samples from the multiple scenarios are clustered to construct a multi-event type sample cluster; Multi-stage feature fitting is performed on the temporal variation characteristics of the multi-event type sample cluster to establish multi-stage temporal features; Based on the aforementioned multi-stage temporal characteristics, positive example samples and negative example samples are constructed according to the abnormal samples in multiple scenarios; The recognition model is learned and converged by utilizing the multi-stage temporal features of the positive and negative samples to obtain multiple anomaly recognition sub-channels corresponding to multiple event types. Specifically, defense phase nodes are determined based on the traceability level, risk level, and timeliness of abnormal events, and these defense phase nodes are added as identification output constraint phases to the abnormal identification sub-channels of the corresponding event types.

10. A deep learning-based system for identifying abnormal behavior in public places, characterized in that: The system is used to implement the deep learning-based method for identifying abnormal behavior in public places according to any one of claims 1-9, the system comprising: The video frame preprocessing execution module is used to connect to the monitoring equipment, acquire the monitoring video stream in real time, and perform video frame preprocessing on the video stream; The pedestrian ID allocation execution module is used to identify pedestrian targets in video frames using a lightweight object detection model, generate bounding boxes, and allocate pedestrian IDs based on the bounding boxes. The inter-frame behavior and action feature recognition module tracks pedestrians in video frames based on the assigned pedestrian ID, constructs motion trajectories between consecutive frames, and identifies inter-frame behavior and action features. The abnormal behavior recognition model building module is used to establish an abnormal behavior sample library, perform deep learning on multiple abnormal behavior features, and obtain an abnormal behavior recognition model. The abnormal feedback information acquisition module is used to import the motion trajectory and behavioral action features into the abnormal behavior recognition model for abnormal behavior recognition, output the abnormal behavior recognition result, and generate abnormal feedback information based on the traceability level, risk level, and timeliness of the abnormal behavior recognition result.

Images