An urban social surface abnormal behavior recognition method and system based on image analysis
By employing multi-dimensional data collection and multi-level annotation methods, combined with the YOLO network's attention mechanism, the problem of abnormal behavior identification in complex urban social scenarios has been solved, achieving high-precision, all-time, all-scenario security monitoring.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI YUETIAN ELECTRONIC TECH CO LTD
- Filing Date
- 2026-03-26
- Publication Date
- 2026-06-26
Smart Images
Figure CN122290202A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image analysis technology, specifically to a method and system for identifying abnormal behaviors in urban social settings based on image analysis. Background Technology
[0002] With the acceleration of urbanization, urban social security management faces increasingly complex challenges. Issues such as students suddenly falling in school corridors, conflicts arising from gatherings in community squares, and security guards neglecting their posts at the entrances and exits of workplaces and industrial parks all require real-time monitoring and early warning through efficient technological means. Traditional video surveillance relies on manual inspections, which has shortcomings such as low efficiency and delayed response, and cannot meet the modern society's demand for all-weather, all-coverage, and high-precision public security.
[0003] However, when existing technologies are applied in complex scenarios in open urban communities, they suffer from drawbacks such as a sharp drop in recognition accuracy in complex environments, a high rate of missed detection of abnormal behavior in dense crowds, misclassification of normal behavior as falls, and inability to quantify the degree of intrusion or exceed the threshold for the number of people. These drawbacks make it difficult to meet the needs of high-precision, all-time, all-scenario security monitoring in urban communities. Summary of the Invention
[0004] The purpose of this invention is to provide a method and system for identifying abnormal behaviors in urban social settings based on image analysis, so as to solve the problems mentioned in the background art.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a method and system for identifying abnormal behaviors in urban social settings based on image analysis, comprising the following steps: Step 1: Data collection. Obtain environmental weather data for weekdays, holidays, different light periods in the morning, noon and evening, and sunny, rainy and foggy conditions, as well as event characteristic data for school building corridors, community squares, and entrances and exits of the unit's campus, to obtain event data. Step 2: Data classification and sample construction. Based on the event data, the data is divided into positive and negative samples to obtain sample data. Step 3: Multi-level annotation. Based on the sample data, a three-level annotation system is established using professional annotation tools, and the annotated data is divided into training set, validation set and test set. Step 4: Model building and optimization. Based on the training set, validation set, and test set, the model is trained and evaluated to obtain the abnormal behavior recognition model. Step 5: Abnormal Behavior Recognition and Judgment. Input the image to be detected into the trained abnormal behavior recognition model. The abnormal behavior recognition model outputs the event type, target subject location and feature attributes. It determines whether there is abnormal behavior according to preset rules and outputs the recognition result.
[0006] Furthermore, the event feature data is obtained by combining real-world scenario collection with simulated scenario supplementation. The simulated scenarios include organizing personnel to simulate falling postures and gathering and wandering behaviors, and collecting samples of complex scenarios such as low light at night, partial obstruction, and rainy / foggy weather. The proportion of complex scenario samples in the positive samples is 40%-50%.
[0007] Furthermore, the positive samples include complete process frames and key action frames of events such as people falling, people gathering, people loitering, people staying, on-duty personnel arriving at their posts, gatekeepers leaving their posts, area intrusion, exceeding the limit for the number of people, and intrusion into restricted areas. The number of positive samples for each type of event is no less than 6,000-9,000, and the proportion of complex scene samples reaches 40-50%. The negative samples are background images without target events and images of similar interference events. The ratio of positive to negative samples is 4:1.
[0008] Furthermore, the three-level annotation system includes first-level annotation of event types, second-level annotation of the bounding box and category of the target subject, and third-level annotation of core feature attributes. When a person falls, the coordinates of key points on the body and the state of a complete fall or a partial fall need to be annotated. When an area is invaded, the intersection area between the preset restricted area boundary and the invading target needs to be annotated. When the number of people exceeds the limit, all people in the counting area need to be annotated. The training set, validation set and test set are divided in a ratio of 7:2:1.
[0009] Furthermore, the multi-level annotation includes the following steps: S1: Perform first-level annotation on the image, marking the event type as one of the following: person falls, people gather, people loiter, people stay, on-duty personnel arrive at their posts, gatekeeper leaves their posts, area intrusion, number of people exceeds limit, and restricted area intrusion; S2: Perform secondary annotation on the image, draw bounding boxes for the target subject and label the category, which includes one of the following: fallen person, crowd, on-duty personnel, and intrusion target; S3: Perform three levels of annotation on the image, annotating the core feature attributes. For example, when a person falls, annotate the coordinates of key body points and distinguish between a complete fall and a partial fall. For an intrusion into a region, annotate the area where the boundary of the preset restricted area intersects with the intrusion target. When the number of people exceeds the limit, annotate all people in the count area. S4: Data sampling inspection. The labeled data is sampled and divided into training set, validation set and test set in a 7:2:1 ratio.
[0010] Furthermore, the model construction and optimization includes the following steps: M1: Feature extraction module construction, based on the YOLO basic network, adds attention mechanism modules to each layer of the feature pyramid to enhance the feature weight of key areas, taking into account the characteristics of dense population and variable target scale in urban social scenes. M2: Model training, based on the training set, trains the model to obtain the trained model; M3: Model Validation. Based on the trained model, the validation set is input into the trained model to validate the model and obtain the validation model. M4: Model testing, based on the validation model, inputs the test set into the validation model to obtain a preliminary abnormal behavior recognition model; M5: Model optimization. Based on the initial abnormal behavior recognition model, the model is optimized to obtain the abnormal behavior recognition model.
[0011] Furthermore, the abnormal behavior identification and determination includes the following steps: A: Image input: Input the image to be detected into the abnormal behavior recognition model; A2: Feature extraction and output. The abnormal behavior recognition model outputs the event type, target subject bounding box position, and feature attributes to obtain feature data. A3: Rule matching and anomaly detection: Based on feature data, abnormal behavior is detected according to preset rules to obtain the detection result; A4: Result storage and early warning triggering: Based on the judgment result, the judgment result is associated with the event key frame and stored to trigger an early warning, thus obtaining the identification result; A5: Output the recognition results, including the recognition results, confidence level, target location, and event keyframes.
[0012] A method for identifying abnormal behavior in urban social areas based on image analysis, as described in any one of the above, includes: The data acquisition module obtains environmental weather data for weekdays, holidays, different light periods in the morning, noon and evening, and sunny, rainy and foggy conditions, as well as event characteristic data for teaching building corridors, community squares, and entrances and exits of unit parks, to obtain event data; The data classification and sample construction module divides event data into positive and negative samples to obtain sample data; The multi-level annotation module, based on sample data, uses professional annotation tools to establish a three-level annotation system and divides the annotated data into training set, validation set and test set; The model building and optimization module trains and evaluates the model based on the training set, validation set, and test set to obtain the abnormal behavior recognition model. The abnormal behavior recognition and judgment module inputs the image to be detected into the trained abnormal behavior recognition model. The abnormal behavior recognition model outputs the event type, target subject location and feature attributes, determines whether there is abnormal behavior according to preset rules, and outputs the recognition result.
[0013] Compared with the prior art, the beneficial effects of the present invention are: This image analysis-based method and system for identifying abnormal behaviors in urban social settings first collects event feature data from various scenarios, including school corridors, community squares, and entrances / exits of workplaces, under different lighting conditions and weather conditions (weekdays, holidays, and varying lighting and weather conditions). The system combines real and simulated scenarios to form event data. Next, the data is divided into positive and negative samples. Positive samples include complete event frames for nine categories, while negative samples consist of background and interference images at a 4:1 ratio. Then, a professional tool is used for three-level annotation: event type, target bounding box and category, and core feature attributes. After sampling, the data is divided into training, validation, and test sets at a 7:2:1 ratio. The model incorporates an attention mechanism to adapt to dense scenarios. The model is trained, validated, tested, and optimized using the dataset. Finally, the input image is used, and the model outputs the event type, target location, and feature attributes. Anomalies are identified according to preset rules, associated keyframes are stored, and an alert is triggered. The system outputs the recognition result. This system ensures that the model can adapt to complex spatiotemporal and environmental changes in the real world, avoiding overfitting caused by training in a single scenario and significantly improving generalization ability.
[0014] Meanwhile, fine-grained labeling is applied to different events, covering the abnormal monitoring needs of people, positions, and areas in urban management. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the overall process of the present invention. Detailed Implementation
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0017] like Figure 1As shown, the present invention provides a technical solution: a method and system for identifying abnormal behaviors in urban social areas based on image analysis, including the following steps: Step 1, data collection, acquiring environmental weather data for weekdays, holidays, different light periods in the morning, noon and evening, sunny, rainy and foggy weather, and event feature data for school building corridors, community squares, and unit park entrances and exits, to obtain event data. The event feature data is obtained by combining real-world scene collection with simulated scene supplementation. The simulated scenes include organizing people to simulate falling postures and gathering and wandering behaviors, and collecting samples of complex scenes such as low light at night, partial obstruction, and rainy and foggy weather. The proportion of complex scene samples in the positive samples reaches 40%-50%.
[0018] It is important to note that this step covers weekdays and holidays, different light times in the morning, noon and evening, weather conditions such as sunny, rainy and foggy, and key areas such as school building corridors, community squares, and entrances and exits of workplaces and campuses. The event dimension collects image data of real-world scenarios for 9 core events, such as students falling on school playgrounds, people gathering in community green belts, the on-duty status of personnel at workplace checkpoints, guards leaving their posts in guard rooms, intrusion into the boundary lines of campuses, intrusion into restricted areas of school buildings, and exceeding the capacity of canteens. Simulated scenario samples are also added to ensure comprehensive event coverage.
[0019] Step 2: Data Classification and Sample Construction. Based on the event data, the data is divided into positive and negative samples to obtain sample data. Positive samples include complete process frames and key action frames of events such as people falling, people gathering, people loitering, people staying, on-duty personnel arriving at their posts, gatekeepers leaving their posts, area intrusion, exceeding the limit for the number of people, and intrusion into restricted areas. The number of positive samples for each type of event is no less than 6,000-9,000, and the proportion of samples in complex scenes is 40-50%. Negative samples are background images without target events and images of similar interference events. The ratio of positive to negative samples is 4:1.
[0020] It is important to note that positive samples include complete process frames and key action frames of various events, such as the moment of falling or the intrusion threshold. The number of samples for each type of event is no less than 8,000, of which complex scene samples account for 45%, covering events such as falling in low light at night and gathering events with partial occlusion in crowds. Negative samples include background images of scenes without target events, such as empty corridors and deserted squares, as well as images of similar interfering events, such as samples that distinguish between normal sitting postures and falling postures, and samples that compare normal walking and wandering behaviors. The ratio of positive to negative samples is strictly controlled at 4:1. Through the above sample construction strategy, misjudgments caused by sample imbalance are effectively avoided, providing high-quality sample support for model training.
[0021] Step 3: Multi-level annotation. Based on sample data, a three-level annotation system is established using professional annotation tools, and the annotated data is divided into training set, validation set, and test set. Step 4: Model construction and optimization. Based on the training set, validation set, and test set, training and evaluation are performed to obtain the abnormal behavior recognition model. Step 5: Abnormal behavior recognition and judgment. The image to be detected is input into the trained abnormal behavior recognition model. The abnormal behavior recognition model outputs the event type, target subject location, and feature attributes. It determines whether abnormal behavior exists according to preset rules and outputs the recognition result. The three-level annotation system includes first-level annotation of event type, second-level annotation of the bounding box and category of the target subject, and third-level annotation of core feature attributes. For example, if a person falls, the coordinates of key body points and the state of a complete fall or partial fall need to be annotated. For area intrusion, the intersection area of the preset restricted area boundary and the intrusion target needs to be annotated. If the number of people exceeds the limit, all people in the counting area need to be annotated. The training set, validation set, and test set are divided in a 7:2:1 ratio. Multi-level annotation includes the following steps: First-level annotation of the image, marking the event type as one of the following: person falling, people gathering, people loitering, people staying, on-duty personnel arriving at their posts, gatekeeper leaving their posts, area intrusion, exceeding the limit for the number of people, and intrusion into a restricted area; Second-level annotation of the image, drawing bounding boxes for the target subject and labeling the category, which includes one of the following: fallen person, gathered crowd, on-duty personnel, and intrusion target; Third-level annotation of the image, labeling the core feature attributes, where falling person is labeled with the coordinates of key body points and distinguishes between a complete fall and a partial fall, area intrusion is labeled with the intersection area of the preset restricted area boundary and the intrusion target, and exceeding the limit for the number of people is labeled with all people within the count area; Data sampling, sampling the labeled data and dividing it into training set, validation set, and test set in a 7:2:1 ratio.
[0022] It is important to note that LabelStudio, a professional annotation tool, should be used to establish a three-level annotation system: event type, target subject, and feature attributes. Specific annotation standards should be developed for key events: When a person falls, the coordinates of key body points, including the head, torso, and limbs, should be annotated to distinguish between a complete fall and a partial fall; when an area is invaded, the intersection of the preset restricted area boundary and the invading target should be annotated; when the number of people exceeds the limit, all people within the counting area should be annotated to ensure no omissions.
[0023] Model construction and optimization include the following steps: Feature extraction module construction: Based on the YOLO basic network, attention mechanism modules are added to each layer of the feature pyramid to enhance the feature weights of key areas, taking into account the characteristics of dense populations and variable target scales in urban social scenarios; Model training: The model is trained based on the training set to obtain the trained model; Model validation: Based on the trained model, the validation set is input into the trained model to validate the model, resulting in the validated model; Model testing: Based on the validated model, the test set is input into the validated model to obtain the preliminary abnormal behavior recognition model; Model optimization: Based on the preliminary abnormal behavior recognition model, the model is optimized to obtain the final abnormal behavior recognition model.
[0024] Abnormal behavior recognition and judgment includes the following steps: Image input: Input the image to be detected into the abnormal behavior recognition model; Feature extraction and output: The abnormal behavior recognition model outputs the event type, target subject bounding box position, and feature attributes to obtain feature data; Rule matching and abnormal judgment: Based on the feature data, abnormal behavior is judged according to preset rules to obtain the judgment result; Result storage and early warning triggering: Based on the judgment result, the judgment result is associated with the event keyframe and stored, and an early warning is triggered to obtain the recognition result; Recognition result output: Output the recognition result, confidence level, target position, and event keyframe.
[0025] It is important to note that the image to be detected is input into the trained abnormal behavior recognition model. The model outputs the event type, the bounding box location of the target subject, and feature attributes. Abnormal behavior is determined based on preset rules: an alert is triggered when the model outputs "person has fallen" and the feature attribute is labeled "complete fall"; an abnormality is determined when the model outputs "region intrusion" and detects an intersection between the intruding target and the boundary of the restricted area; and an alarm is triggered when the model outputs "number of people exceeds the limit" and the number of people within the counting area exceeds a preset threshold. The determination results are stored in association with the event keyframes. The system synchronously outputs the recognition results, confidence level, target location, and event keyframes, completing the entire abnormal behavior recognition process.
[0026] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended embodiments and their equivalents.
Claims
1. A method for identifying abnormal behaviors in urban social settings based on image analysis, characterized in that: Includes the following steps: Step 1: Data collection. Obtain environmental weather data for weekdays, holidays, different light periods in the morning, noon and evening, and sunny, rainy and foggy conditions, as well as event characteristic data for school building corridors, community squares, and entrances and exits of the unit's campus, to obtain event data. Step 2: Data classification and sample construction. Based on the event data, the data is divided into positive and negative samples to obtain sample data. Step 3: Multi-level annotation. Based on the sample data, a three-level annotation system is established using professional annotation tools, and the annotated data is divided into training set, validation set and test set. Step 4: Model building and optimization. Based on the training set, validation set, and test set, the model is trained and evaluated to obtain the abnormal behavior recognition model. Step 5: Abnormal Behavior Recognition and Judgment. Input the image to be detected into the trained abnormal behavior recognition model. The abnormal behavior recognition model outputs the event type, target subject location and feature attributes. It determines whether there is abnormal behavior according to preset rules and outputs the recognition result.
2. The method for identifying abnormal behaviors in urban social environments based on image analysis according to claim 1, characterized in that: The event feature data was obtained by combining real-world scenario data collection with simulated scenario data collection. The simulated scenarios included organizing personnel to simulate falling postures and gathering and wandering behaviors. Samples were also collected for complex scenarios such as low light at night, partial obstruction, and rainy / foggy weather. The proportion of complex scenario samples in the positive samples was 40%-50%.
3. The method for identifying abnormal behaviors in urban social environments based on image analysis according to claim 1, characterized in that: The positive samples consist of complete process frames and key action frames including people falling, people gathering, people loitering, people staying, on-duty personnel arriving at their posts, gatekeepers leaving their posts, area intrusion, exceeding the number of people, and intrusion into restricted areas. The number of positive samples for each type of event is no less than 6,000-9,000, and the proportion of complex scene samples is 40-50%. The negative samples are background images without target events and images of similar interference events. The ratio of positive to negative samples is 4:
1.
4. The method for identifying abnormal behaviors in urban social environments based on image analysis according to claim 1, characterized in that: The three-level annotation system includes first-level annotation of event types, second-level annotation of the bounding box and category of the target subject, and third-level annotation of core feature attributes. When a person falls, the coordinates of key points on the body and the state of a complete fall or a partial fall need to be annotated. When an area is invaded, the intersection area between the boundary of the preset restricted area and the invading target needs to be annotated. When the number of people exceeds the limit, all people in the counting area need to be annotated. The training set, validation set and test set are divided in a ratio of 7:2:
1.
5. The method for identifying abnormal behaviors in urban social environments based on image analysis according to claim 1, characterized in that: The multi-level annotation includes the following steps: S1: Perform first-level annotation on the image, marking the event type as one of the following: person falls, people gather, people loiter, people stay, on-duty personnel arrive at their posts, gatekeeper leaves their posts, area intrusion, number of people exceeds limit, and restricted area intrusion; S2: Perform secondary annotation on the image, draw bounding boxes for the target subject and label the category, which includes one of the following: fallen person, crowd, on-duty personnel, and intrusion target; S3: Perform three levels of annotation on the image, annotating the core feature attributes. For example, when a person falls, annotate the coordinates of key body points and distinguish between a complete fall and a partial fall. For an intrusion into a region, annotate the area where the boundary of the preset restricted area intersects with the intrusion target. When the number of people exceeds the limit, annotate all people in the count area. S4: Data sampling inspection. The labeled data is sampled and divided into training set, validation set and test set in a 7:2:1 ratio.
6. The method for identifying abnormal behaviors in urban social environments based on image analysis according to claim 1, characterized in that: The model construction and optimization include the following steps: M1: Feature extraction module construction, based on the YOLO basic network, adds attention mechanism modules to each layer of the feature pyramid to enhance the feature weight of key areas, taking into account the characteristics of dense population and variable target scale in urban social scenes. M2: Model training, based on the training set, trains the model to obtain the trained model; M3: Model Validation. Based on the trained model, the validation set is input into the trained model to validate the model and obtain the validation model. M4: Model testing, based on the validation model, inputs the test set into the validation model to obtain a preliminary abnormal behavior recognition model; M5: Model optimization. Based on the initial abnormal behavior recognition model, the model is optimized to obtain the abnormal behavior recognition model.
7. The method for identifying abnormal behaviors in urban social environments based on image analysis according to claim 1, characterized in that: The abnormal behavior identification and determination includes the following steps: A: Image input: Input the image to be detected into the abnormal behavior recognition model; A2: Feature extraction and output. The abnormal behavior recognition model outputs the event type, target subject bounding box position, and feature attributes to obtain feature data. A3: Rule matching and anomaly detection: Based on feature data, abnormal behavior is detected according to preset rules to obtain the detection result; A4: Result storage and early warning triggering: Based on the judgment result, the judgment result is associated with the event key frame and stored to trigger an early warning, thus obtaining the identification result; A5: Output the recognition results, including the recognition results, confidence level, target location, and event keyframes.
8. A system for identifying abnormal behaviors in urban social settings based on image analysis, characterized in that: The method for identifying abnormal behavior in urban social areas based on image analysis, as described in any one of claims 1-7, includes: The data acquisition module obtains environmental weather data for weekdays, holidays, different light periods in the morning, noon and evening, and sunny, rainy and foggy conditions, as well as event characteristic data for teaching building corridors, community squares, and entrances and exits of unit parks, to obtain event data; The data classification and sample construction module divides event data into positive and negative samples to obtain sample data; The multi-level annotation module, based on sample data, uses professional annotation tools to establish a three-level annotation system and divides the annotated data into training set, validation set and test set; The model building and optimization module trains and evaluates the model based on the training set, validation set, and test set to obtain the abnormal behavior recognition model. The abnormal behavior recognition and judgment module inputs the image to be detected into the trained abnormal behavior recognition model. The abnormal behavior recognition model outputs the event type, target subject location and feature attributes, determines whether there is abnormal behavior according to preset rules, and outputs the recognition result.