Crowd abnormal behavior identification method and system

Abstract

The invention relates to the technical field of data intelligent analysis, in particular to a crowd abnormal behavior recognition method and system. Comprising the following steps: S1, acquiring video data in a monitoring range; s2, acquiring mobile phone big data information of all individuals in a monitoring range; s3, performing data processing on the video data and the mobile phone big data information; s4, obtaining a feature set of the video data, and obtaining a feature set of the mobile phone big data information; s5, performing data fusion on the feature set of the video data and the feature set of the mobile phone big data information; s6, abnormal behavior recognition is carried out, wherein abnormal behaviors comprise a gathering behavior, a fighting behavior and a trampling event; s7, displaying the abnormal behavior identified in the step S6, and sending abnormal information to a manager; through combined analysis of the video data and the mobile phone big data, the accuracy of crowd abnormal behavior recognition is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent data analysis, and in particular to a method and system for identifying abnormal behavior of a crowd. Background Art

[0002] Public safety incidents such as stampedes caused by crowds have become issues that require special attention. In particular, in recent years, some lawless elements have used improper means, such as robbery, intimidation, violent crimes, and other forms of illegal crimes, which are seriously threatening social stability and people’s lives and property safety. Therefore, strengthening urban management is currently the key means to solve the above problems.

[0003] With the development of urban management technology, there are more and more means to obtain urban monitoring data. There are a large amount of mobile data in the urban management operation system. The correlation analysis between the data is mostly done manually, which is inefficient and difficult to effectively integrate the various systems. It is difficult to identify and track simply through a certain type of data; therefore, effectively integrating various data to analyze crowd behavior has become a necessary analysis method at present.

[0004] Chinese patent CN116503801A discloses a method and device for identifying abnormal crowd behavior, wherein the identification method includes collecting a sequence of pedestrian images to be detected, annotating a sample set, preprocessing the images in the sample set, training a crowd behavior recognition model, collecting a video stream of pedestrians in a monitoring scene, inputting the video stream frame by frame into the trained crowd behavior recognition model, and judging whether there is abnormal behavior according to the abnormal behavior evaluation standard, wherein the crowd behavior recognition model includes two modeling channels: spatial modeling and temporal modeling. The above identification method only analyzes abnormal crowd behavior by processing the video, and does not combine and analyze multiple data such as video data and mobile phone big data, so the accuracy is low.

[0005] Therefore, there is an urgent need to provide a method and system for identifying abnormal crowd behavior. Compared with the existing technology, the accuracy of identifying abnormal crowd behavior can be improved by combining video data with mobile phone big data for analysis. Summary of the invention

[0006] The present invention solves the technical problems existing in the prior art and provides a method and system for identifying abnormal behavior of a crowd.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A method and system for identifying abnormal behavior of a crowd, comprising the following steps:

[0009] S1. Obtain video data within the monitoring range;

[0010] S2. Obtaining big data information on mobile phones of all individuals within the monitoring range;

[0011] S3, processing the video data obtained in step S1 and the mobile phone big data information obtained in step S2;

[0012] S4, processing the video data processed by step S3 to obtain a feature set of the video data, and processing the mobile phone big data information processed by step S3 to obtain a feature set of the mobile phone big data information;

[0013] S5, fusing the feature set of the video data with the feature set of the mobile phone big data information, and matching each individual in the mobile phone big data information with each individual in the video data;

[0014] S6. Based on the matching results processed in step S5, abnormal behaviors are identified. Abnormal behaviors include gathering behaviors, fighting behaviors, and stampede incidents.

[0015] S7. Display the abnormal behavior identified in step S6 and send abnormal information to the management personnel.

[0016] Furthermore, S4 specifically includes the following steps:

[0017] S41, processing the video data after the data processing in step S3 to obtain a feature set of the video data, setting a time step, the feature set of the video data includes a set of personnel density of all time steps in the video data, a set of average moving speeds of all time steps in the video data, a set of main moving directions of all time steps in the video data, a set of average facial expression scores of all individuals in the video data, a set of abnormal behavior scores of all individuals in the video data, and a set of position coordinates of all individuals in the video data;

[0018] S42. Process the mobile phone big data information after the data processing in step S3 to obtain a feature set of the mobile phone big data information. The feature set of the mobile phone big data information includes location features, the association relationship of each individual, and the overall comprehensive risk score of each individual.

[0019] Furthermore, in step S41, the density of people at each time step is represented by the number of individuals detected in all video frames corresponding to the corresponding time step, the average moving speed at each time step is represented by calculating the average moving speed of all individuals in all video frames corresponding to the corresponding time step, and the main moving direction of each time step is represented by the moving direction of more than half of the individuals in all video frames corresponding to the corresponding time step.

[0020] Furthermore, in step S41, the average facial expression score of each individual in each time step is obtained by the following method:

[0021] Set multiple basic emotions and multiple expression categories. Basic emotions include anger, disgust, fear, happiness, sadness, surprise, and neutrality. Expression categories include frowning, wrinkling nose, staring, smiling, blinking, drooping eyebrows, closing eyes, opening eyes, opening mouth, and no expression. Set negative emotions to anger, disgust, fear, and sadness. Set the weight matrix between expression categories and basic emotions.

[0022] There is at least one video frame in each time step. All expression categories of each individual in each video frame in each time step are identified based on facial recognition technology. According to the weight matrix between expression categories and basic emotions, weighted operations are performed to obtain the expression category score set of each individual in each video frame, and the average negative emotion score of each individual is calculated, thereby obtaining the average facial expression score of each individual in each time step.

[0023] The average negative emotion score for each individual was calculated as follows:

[0024]

[0025] In the above formula, represents the average negative emotion score of the jth negative emotion of the individual, n represents the number of video frames in each time step, and e ij represents the score value of the jth negative emotion in the i-th video frame of the individual, e ij Derived from the expression category score set;

[0026] The average facial expression score for each individual in each time step was calculated as follows:

[0027]

[0028] In the above formula, e represents the average facial expression score of each individual in each time step, and m represents the total number of negative emotions in each time step.

[0029] Furthermore, in step S42, the location features include longitude, latitude, altitude, and timestamp;

[0030] Use clustering algorithms to analyze individual call records and text message data, identify different types of communication patterns, and identify the associations of each individual.

[0031] Furthermore, the overall comprehensive risk score of each individual in step S42 is obtained by the following method:

[0032] Use sentiment analysis tools to analyze the individual's likes, comments, and sharing interactive behaviors to obtain negative emotion scores, specifically: obtain the number of likes of the individual for posts related to negative emotions, set a negative score for a single like, and obtain the negative emotion score of the like by multiplying the number of likes by the negative score for a single like; obtain the number of negative comments made by the individual, set a negative score for a single comment, and obtain the negative emotion score of the comment by multiplying the number of negative comments by the negative score for a single comment; obtain the number of shares of the individual for posts related to negative emotions, set a negative score for sharing, and obtain the negative emotion score of sharing by multiplying the number of shares by the negative score for sharing; the negative emotion score is obtained by adding the negative emotion score of likes, the negative emotion score of comments, and the negative emotion score of sharing;

[0033] Identify the individual's interest areas, set the risk areas, assign different risk scores and weight values ​​to each risk area, select the risk areas existing in the interest areas identified by each individual, multiply the corresponding risk score by the weight value for each risk area selected by the individual, and then add the results of multiplying the risk score of each selected risk area by the weight value to obtain a comprehensive risk score;

[0034] The overall comprehensive risk score = α × negative emotion score + β × comprehensive risk score; wherein α and β represent the weight values ​​of the negative emotion score and the comprehensive risk score, respectively.

[0035] Furthermore, S5 specifically includes the following steps:

[0036] S51, matching location information, calculating the Euclidean distance or Manhattan distance between the location feature in the mobile phone big data information of each individual and the location coordinates in the feature set of the video data, and selecting the individual in the video data with the closest Euclidean distance or Manhattan distance as the matching individual of the individual in the mobile phone big data information;

[0037] S52, matching timestamps, using a dynamic time adjustment algorithm, selecting two individuals with the closest time in the video data and the mobile phone big data information, adjusting the time axis, and ensuring the time consistency of the video data and the mobile phone big data information.

[0038] Furthermore, in step S6, fighting behavior is detected by the following method:

[0039] Calculate the comprehensive score of each individual. The comprehensive scores of all individuals constitute sequence data. Input the sequence data into the behavior classification model. Use the behavior classification model to identify aggressive actions. Aggressive actions include punching and kicking. The presence of aggressive actions indicates the presence of fighting behavior.

[0040] The comprehensive score of each individual is calculated by the following formula:

[0041] Comprehensive score = γ × average facial expression score + δ × abnormal behavior score + ε × overall comprehensive risk score; among them, γ, δ, and ε are the weight coefficients of the average facial expression score, abnormal behavior score, and overall comprehensive risk score, respectively.

[0042] Furthermore, in step S6, the aggregation behavior is detected by the following method:

[0043] (1) Determine the association relationship between individuals within the monitoring range to form multiple association relationship networks, each of which contains at least two individuals, and any two individuals in each association relationship network have a direct association relationship or an association relationship through one or more other individuals. Set an association number threshold. When the total number of individuals in the association relationship network with the largest number of individuals within the monitoring range is greater than the association number threshold, perform personnel density determination;

[0044] (2) By determining the density of people within the monitoring range and setting a density threshold, when the total number of individuals within a certain time step within the monitoring range is greater than the density threshold, it is determined that there is a gathering behavior and an alarm is triggered immediately;

[0045] The pedaling time is detected by the following method:

[0046] For each time step, a speed threshold is set. When the average moving speed is greater than the speed threshold and the total number of individuals in the time step is greater than the density threshold, it is determined that there is a potential trampling risk in the main moving direction within the time step.

[0047] A system for identifying abnormal behavior of a crowd includes a collection module, a preprocessing module, a data feature extraction module, a data fusion module, an identification module, a display module and a communication module. The collection module executes steps S1 and S2, the preprocessing module executes step S3, the data feature extraction module executes step S4, the data fusion module executes step S5, the identification module executes step S6, the display module executes the abnormal behavior display operation in step S7, and the communication module executes the abnormal information sending operation to the management personnel in step S7.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The present invention combines the monitoring data in the video data with the data information in the mobile phone big data, and identifies abnormal behaviors in the crowd by analyzing the facial expression information, abnormal behavior information, and interest and social activity information of each individual, thereby effectively improving the accuracy of identifying abnormal behaviors in the crowd. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention will be clearly described below in conjunction with the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention. It should be noted that the orientation or position relationship indicated by the terms "center", "up", "down", "left", "right", "vertical", "horizontal", etc. is based on the orientation or position relationship shown in the accompanying drawings. It is only for the convenience of describing the present invention and simplifying the description. It does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0052] like Figure 1 As shown, the present invention provides a method for identifying abnormal behavior of a crowd, comprising the following steps:

[0053] S1. Obtain video data within the monitoring range. Use a monitoring camera or drone with GPS function to shoot video within the monitoring range, and transmit the shot video to the video cloud platform. The shot video is EXIF ​​data, including shooting date, shooting time, and geographic location information. The geographic location information includes longitude, latitude, altitude, direction angle, direction, etc.

[0054] S2. Obtain big data information on the mobile phones of all individuals within the monitoring range. The signaling data of the mobile phones of all individuals within the monitoring range are transmitted to the acquisition and decoding layer through the base station for signaling decoding. Based on the deep detection DPI technology of data packets, the DPI system is used to perform deep detection on different network loads, such as HTTP, DNS, etc., to obtain valid information in the message, thereby obtaining the spatiotemporal data of mobile phones within the monitoring range, that is, big data information.

[0055] The spatiotemporal data of a mobile phone includes location information, communication information and social activity information; communication information includes an individual’s call records and text message data; social activity information includes an individual’s interactive behavior information such as likes, comments, and shares on social media, as well as the topics of posts published by the individual and the content of discussions in which the individual participates.

[0056] S3. The video data obtained in step S1 and the mobile phone big data information obtained in step S2 are connected to the data preprocessing module through the interface module. The interface module uses wired or wireless access. The data preprocessing module performs data cleaning, denoising and format conversion on the video data and the mobile phone big data information respectively.

[0057] S4, using the data feature extraction module to perform individual detection and identification on the data processed in step S3, and obtain the feature set of the video data and the feature set of the mobile phone big data information, specifically:

[0058] S41. Process the video data processed in step S3 to obtain a feature set of the video data. Receive the video surveillance data stream, process it frame by frame, obtain the video frame, use the YOLOv5 algorithm to detect individuals, identify and locate individuals, and be able to quickly identify and locate individuals in the video while maintaining high accuracy. For each time step, key features are extracted from the video frame. The key features include the personnel density dt, the average moving speed st, the main moving direction θt of each individual, the average facial expression score et of each individual, and the abnormal behavior score gt of each individual.

[0059] The density of people is expressed by measuring the number of individuals detected in the video frame corresponding to each time step, reflecting the degree of crowd gathering; the average moving speed is expressed by calculating the average moving speed of all individuals detected in the video frame corresponding to each time step, reflecting the movement status of the crowd;

[0060] The main movement direction of each individual is represented by determining the movement direction of more than half of the crowd in the video frame corresponding to each time step, which helps to understand the group behavior pattern.

[0061] The average facial expression score of each individual is obtained by using facial recognition technology to evaluate the average emotional state of each individual in each time step, providing emotional information.

[0062] Seven basic emotions are set, including anger, disgust, fear, happiness, sadness, surprise, and neutrality; 10 expression categories are set, including frowning, wrinkling nose, staring, smiling, blinking, drooping eyebrows, closing eyes, opening eyes wide, opening mouth, and expressionless. Expressionless means no frowning, wrinkling nose, staring, smiling, blinking, drooping eyebrows, closing eyes, opening eyes wide, and opening mouth; the weight matrix between each expression category and the basic emotions is set, as shown in Table 1:

[0063] Table 1

[0064]

[0065] There is at least one video frame in each time step. Based on the facial recognition technology, all expression categories of each individual in each video frame in each time step are identified. According to the weight matrix between the expression categories and the basic emotions, weighted operations are performed to obtain the expression category score set of each individual in each video frame, and then the average negative emotion score and the average facial expression score of each individual in each time step are calculated; anger, disgust, fear, and sadness are set as negative emotions.

[0066] The average negative sentiment score is calculated as follows:

[0067]

[0068] In the above formula, represents the average negative emotion score of the jth negative emotion of the individual, n represents the number of video frames in each time step, and e ij Represents the score value of the jth negative emotion in the i-th video frame of the individual.

[0069] The average facial expression score was calculated as follows:

[0070]

[0071] In the above formula, e represents the average facial expression score of the individual at each time step, and m represents the total number of negative emotions in each time step.

[0072] The abnormal behavior score uses behavior recognition technology to evaluate the specific behavior of each individual in the video frame corresponding to each time step and provide information at the behavior level.

[0073] Set 8 abnormal behaviors and set a score for each abnormal behavior, as shown in Table 2:

[0074] Table 2

[0075]

[0076] The abnormal behavior score of each individual is calculated by the following formula:

[0077]

[0078] In the above formula, g represents the abnormal behavior score of the individual in each time step, Y y represents the score of the individual's y-th abnormal behavior, C y represents the number of times the yth abnormal behavior of an individual occurs in each time step, and K represents the total number of abnormal behaviors that occur in each time step.

[0079] The key features extracted in each time step are integrated to form a feature vector corresponding to each time step. The feature vectors of all time steps form the feature set of the video data, which is expressed as: vt = [d, s, θ, e, g, x], where d represents the set of personnel density at all time steps in the video data, s represents the set of average moving speeds at all time steps in the video data, θ represents the set of main moving directions at all time steps in the video data, e represents the set of average facial expression scores of all individuals in the video data, g represents the set of abnormal behavior scores of all individuals in the video data, and x represents the set of position coordinates of all individuals in the video data.

[0080] In order to track individuals in the video, the DeepSORT algorithm is used on the detection results of the YOLOv5 algorithm to achieve cross-frame tracking of individuals. It can maintain the identity consistency of the individual and keep tracking even when the individual temporarily disappears or overlaps.

[0081] S42, extracting features from the mobile phone big data information to obtain a feature set of the mobile phone big data information, specifically:

[0082] S421. Use GPS data processing technology and geo-fencing technology to process location information and extract location features. The location features include longitude, latitude, altitude, timestamp, place of origin, household registration, start time of stay, end time of stay, length of stay, number of passengers on the road section, flow speed of the road section, source of passenger flow, lowest real-time number of the day, and highest real-time number of the day.

[0083] The GPS coordinate data of individuals, including longitude, latitude, altitude and timestamp, are extracted through GPS data processing technology. The GPS signal is smoothed using the Kalman filter to improve the location accuracy. The complete individual location trajectory is constructed by combining time and GPS coordinate data.

[0084] Through geo-fence technology, we extract individual fence data, set up virtual geo-fences, and automatically record the time and frequency of individuals entering and exiting the geo-fences, so as to analyze the individual's place of origin, household registration, start time of stay, end time of stay, length of stay, number of passengers on the road section, flow speed of the road section, source of passenger flow, lowest real-time number of the day, and highest real-time number of the day.

[0085] S422. Conduct in-depth analysis of communication information, extract the associations between individuals within the monitoring range, identify the associations between individuals, use clustering algorithms to analyze individual call records and text message data, identify different types of communication patterns, and identify the associations between people within the monitoring range.

[0086] S423, intelligently mine social activity information, extract each individual's negative emotions, and calculate each individual's overall comprehensive risk score; specifically:

[0087] (1) Conduct social interaction analysis and use sentiment analysis tools to analyze individual interactive behaviors such as likes, comments, and shares. Sentiment analysis tools such as TextBlob and VADER can be used to obtain individual emotional tendencies and calculate negative sentiment scores.

[0088] Specifically:

[0089] Analyze the content topics of likes, obtain the number of likes for posts related to negative emotions, set the negative score value for a single like, and the negative emotion score for likes is obtained by multiplying the number of likes by the negative score value for a single like; analyze the content of the comment text, obtain the number of negative comments, set the negative score value for a single comment, and the negative emotion score for the comment is obtained by multiplying the number of negative comments by the negative score value for a single comment; conduct a comprehensive analysis of the shared content and the sharing context, obtain the number of shares of posts related to negative emotions, set the negative score value for sharing, and the negative emotion score for sharing is obtained by multiplying the number of shares by the negative score value for sharing.

[0090] The negative sentiment score to be obtained in this step is obtained by adding the negative sentiment scores of likes, comments, and shares.

[0091] For example, if you like 3 posts related to negative emotions, set the negative score of a single like to 6 / 10. If you comment 4 posts related to negative emotions, set the negative score of a single comment to 8 / 10. If you share 2 posts related to negative emotions, set the negative score of a single share to 7 / 10. You can calculate the negative emotion score of the individual:

[0092] Negative emotion score = (3×6 / 10)+(4×8 / 10)+(2×7 / 10)=(1.8)+(3.2)+(1.4)=6.4.

[0093] (2) Conduct individual interest analysis, use topic models to analyze the topics of posts published by individuals and the content of discussions they participate in, and use algorithms such as LDA (Latent Dirichlet Allocation) or NMF (Non-negative Matrix Factorization) to conduct topic modeling on the content published by individuals. Use algorithms such as TF-IDF or TextRank to extract keywords from posts, identify the individual's interest areas, and identify dangerous areas from the individual's interest areas. Dangerous areas include sports, military, news information, music, shopping, fraud, violence, pornography, antisocial, extreme politics, crime stories, network security, technology product reviews, natural disaster response, political commentary, and religious beliefs. Based on the dangerous areas identified by the individual, a comprehensive risk score is obtained.

[0094] For each area, different risk scores and weight values ​​are given, as shown in Table 3.

[0095] Table 3

[0096]

[0097]

[0098] The comprehensive risk score is calculated by multiplying the score of each identified risk area by the corresponding weight value and then summing the results.

[0099] (3) The negative emotion score obtained from the social interaction analysis and the comprehensive risk score obtained from the individual interest analysis are integrated to calculate the overall comprehensive risk score. The specific calculation is as follows:

[0100] The overall comprehensive risk score value = α×negative emotion score + β×comprehensive risk score; wherein α and β represent the weight values ​​of the negative emotion score and the comprehensive risk score, respectively, α is preferably 0.4, and β is preferably 0.6.

[0101] From the perspective of importance, negative emotions in social interactions can spread quickly, affect the emotional state of a large number of individuals, and sometimes even inspire mass incidents, so they are of high importance. From the perspective of scope of influence, the rapid spread of negative emotions may have a negative impact on public opinion and lead to social instability. Therefore, α is 0.4. From the perspective of importance, the individual's interest areas are closely related to potential public safety hazards, especially in terms of violence, pornography, antisocial behavior, etc., so they are also of high importance. From the perspective of scope of influence, these interest areas may not only have a negative impact on individuals, but also cause harm to the entire society. Therefore, β is 0.6.

[0102] The feature set of mobile phone big data information includes location features, the correlation between individuals within the monitoring range, and the overall comprehensive risk score value of each individual within the monitoring range.

[0103] S5. Fusing the feature set of the video data with the feature set of the mobile phone big data information, and matching each individual in the mobile phone big data information with each individual in the video data, specifically:

[0104] S51. Matching location information, comparing the location coordinates of each individual in the mobile phone big data information with the location coordinates of each individual in the feature set of the video data, respectively calculating the distance between the location coordinates of each individual in the mobile phone big data information and the location coordinates of each individual in the feature set of the video data, which may be the Euclidean distance or the Manhattan distance, and selecting the individual in the video data when the Euclidean distance or the Manhattan distance is the shortest as the matching individual of the individual in the mobile phone big data information, thereby achieving matching of each individual in the mobile phone big data information.

[0105] S52, matching timestamps. Considering that the video data and mobile phone big data information come from different systems and there may be time offsets, a timestamp synchronization algorithm is used, and a dynamic time adjustment algorithm is preferably used to select the two individuals with the closest time in the video data and mobile phone big data information, and adjust the time axis to ensure the time consistency of the video data and mobile phone big data information.

[0106] S6. Identify individual abnormal behaviors; use deep learning models such as convolutional neural networks or 3D CNN to identify abnormal behavior patterns, including gathering behavior, fighting / armed fighting, and stampede incidents.

[0107] Detection of aggregation behavior:

[0108] (1) Determine the association relationship between individuals within the monitoring range, form multiple association relationship networks, each of which has at least two individuals. Any two individuals in each association relationship network have a direct association relationship or can be connected through one or more other individuals. Set an association number threshold. When the total number of individuals in the association network with the largest number of associated individuals within the monitoring range is greater than the association number threshold, perform personnel density determination.

[0109] (2) By determining the density of people within the monitoring range and calculating the total number of individuals in each time step, the density of people is reflected and a density threshold is set. When the total number of individuals in a certain time step within the monitoring range is greater than the density threshold, it is determined that there is a gathering behavior and an alarm is triggered immediately.

[0110] Fighting / armed combat behavior detection: For each individual within the monitoring range, the average facial expression score, abnormal behavior score, and overall comprehensive risk score are weighted to obtain a comprehensive score for each individual. The comprehensive scores of all individuals constitute sequence data, which are input into the behavior classification model. The behavior classification model uses the Transformer model to identify aggressive actions, including signs of physical conflict such as punching and kicking. The presence of aggressive actions indicates the presence of fighting / armed combat behavior.

[0111] The comprehensive score of each individual is calculated by the following formula:

[0112] Comprehensive score = γ × average facial expression score + δ × abnormal behavior score + ε × overall comprehensive risk score; among them, γ, δ, and ε are the weight coefficients of the average facial expression score, abnormal behavior score, and overall comprehensive risk score, and the values ​​of γ, δ, and ε are 0.2, 0.7, and 0.1, respectively.

[0113] Stampede detection: Predict potential stampede risks by analyzing the average moving speed, main moving direction, and population density of people within the monitoring range.

[0114] For each time step, a speed threshold is set. When the average moving speed is greater than the speed threshold and the total number of individuals in the time step is greater than the density threshold, it is determined that there is a potential trampling risk in the main moving direction within the time step.

[0115] S7. Display the abnormal behavior identified in step S6 through the display module, and update the abnormal behavior in real time, intuitively display it on the GIS map, mark the specific location of abnormal people or individuals, and automatically send abnormal information to managers through the communication module, including abnormal behavior patterns and specific locations, to ensure rapid response and timely intervention.

[0116] The present invention also provides a system for identifying abnormal behavior of a crowd, including a collection module, a preprocessing module, a data feature extraction module, a data fusion module, an identification module, a display module and a communication module. The collection module executes steps S1 and S2, the preprocessing module executes step S3, the data feature extraction module executes step S4, the data fusion module executes step S5, the identification module executes step S6, the display module is used to display the abnormal behavior identified in step S6, and the communication module is used to automatically send abnormal information to management personnel, including abnormal behavior patterns and specific locations, to ensure rapid response and timely intervention.

[0117] The present invention combines the monitoring data in the video data with the data information in the mobile phone big data, and identifies abnormal behaviors in the crowd by analyzing the facial expression information, abnormal behavior information, and interest and social activity information of each individual, thereby effectively improving the accuracy of identifying abnormal behaviors in the crowd.

[0118] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions of the technical solution of the present invention by ordinary technicians in this field do not deviate from the essence and scope of the technical solution of the present invention.

Claims

1. A method for identifying abnormal behavior of a crowd, characterized in that: The following steps are involved: S1. Obtain video data within the monitoring range; S2. Obtaining big data information on mobile phones of all individuals within the monitoring range; S3, processing the video data obtained in step S1 and the mobile phone big data information obtained in step S2; S4, processing the video data processed by step S3 to obtain a feature set of the video data, and processing the mobile phone big data information processed by step S3 to obtain a feature set of the mobile phone big data information; S5, fusing the feature set of the video data with the feature set of the mobile phone big data information, and matching each individual in the mobile phone big data information with each individual in the video data; S6. Based on the matching results processed in step S5, abnormal behaviors are identified. Abnormal behaviors include gathering behaviors, fighting behaviors, and stampede incidents. S7. Display the abnormal behavior identified in step S6 and send abnormal information to the management personnel.

2. A method for identifying abnormal crowd behavior according to claim 1, characterized in that: S4 specifically includes the following steps: S41, processing the video data after the data processing in step S3 to obtain a feature set of the video data, setting a time step, the feature set of the video data includes a set of personnel density of all time steps in the video data, a set of average moving speeds of all time steps in the video data, a set of main moving directions of all time steps in the video data, a set of average facial expression scores of all individuals in the video data, a set of abnormal behavior scores of all individuals in the video data, and a set of position coordinates of all individuals in the video data; S42. Process the mobile phone big data information after the data processing in step S3 to obtain a feature set of the mobile phone big data information. The feature set of the mobile phone big data information includes location features, the association relationship of each individual, and the overall comprehensive risk score of each individual.

3. A method for identifying abnormal crowd behavior according to claim 2, characterized in that: In step S41, the population density of each time step is represented by the number of individuals detected in all video frames corresponding to the corresponding time step, the average moving speed of each time step is represented by calculating the average moving speed of all individuals in all video frames corresponding to the corresponding time step, and the main moving direction of each time step is represented by the moving direction of more than half of the individuals in all video frames corresponding to the corresponding time step.

4. A method for identifying abnormal crowd behavior according to claim 2, characterized in that: In step S41, the average facial expression score of each individual in each time step is obtained by the following method: Set multiple basic emotions and multiple expression categories. Basic emotions include anger, disgust, fear, happiness, sadness, surprise, and neutral. Expression categories include frowning, wrinkling nose, staring, smiling, blinking, drooping eyebrows, closing eyes, opening eyes, opening mouth, and no expression. Set negative emotions to anger, disgust, fear, and sadness. Set the weight matrix between expression categories and basic emotions; There is at least one video frame in each time step. All expression categories of each individual in each video frame in each time step are identified based on facial recognition technology. According to the weight matrix between expression categories and basic emotions, weighted operations are performed to obtain the expression category score set of each individual in each video frame, and the average negative emotion score of each individual is calculated, thereby obtaining the average facial expression score of each individual in each time step. The average negative emotion score for each individual was calculated as follows: In the above formula, represents the average negative emotion score of the jth negative emotion of the individual, n represents the number of video frames in each time step, and e ij represents the score value of the jth negative emotion in the i-th video frame of the individual, e ij Derived from the expression category score set; The average facial expression score for each individual in each time step was calculated as follows: In the above formula, e represents the average facial expression score of each individual in each time step, and m represents the total number of negative emotions in each time step.

5. The method for identifying abnormal crowd behavior according to claim 2, characterized in that: In step S42, the location features include longitude, latitude, altitude and timestamp; Use clustering algorithms to analyze individual call records and text message data, identify different types of communication patterns, and identify the associations of each individual.

6. A method for identifying abnormal crowd behavior according to claim 2, characterized in that: The overall comprehensive risk score for each individual in step S42 is obtained by the following method: Use sentiment analysis tools to analyze the individual's likes, comments, and sharing interactive behaviors to obtain negative emotion scores, specifically: obtain the number of likes of the individual for posts related to negative emotions, set a negative score for a single like, and obtain the negative emotion score of the like by multiplying the number of likes by the negative score for a single like; obtain the number of negative comments made by the individual, set a negative score for a single comment, and obtain the negative emotion score of the comment by multiplying the number of negative comments by the negative score for a single comment; obtain the number of shares of the individual for posts related to negative emotions, set a negative score for sharing, and obtain the negative emotion score of sharing by multiplying the number of shares by the negative score for sharing; the negative emotion score is obtained by adding the negative emotion score of likes, the negative emotion score of comments, and the negative emotion score of sharing; Identify the individual's interest areas, set the risk areas, assign different risk scores and weight values ​​to each risk area, select the risk areas existing in the interest areas identified by each individual, multiply the corresponding risk score by the weight value for each risk area selected by the individual, and then add the results of multiplying the risk score of each selected risk area by the weight value to obtain a comprehensive risk score; The overall comprehensive risk score = α × negative emotion score + β × comprehensive risk score; wherein α and β represent the weight values ​​of the negative emotion score and the comprehensive risk score, respectively.

7. A method for identifying abnormal crowd behavior according to claim 2, characterized in that: S5 specifically includes the following steps: S51, matching location information, calculating the Euclidean distance or Manhattan distance between the location feature in the mobile phone big data information of each individual and the location coordinates in the feature set of the video data, and selecting the individual in the video data with the closest Euclidean distance or Manhattan distance as the matching individual of the individual in the mobile phone big data information; S52, matching timestamps, using a dynamic time adjustment algorithm, selecting two individuals with the closest time in the video data and the mobile phone big data information, adjusting the time axis, and ensuring the time consistency of the video data and the mobile phone big data information.

8. A method for identifying abnormal crowd behavior according to claim 2, characterized in that: In step S6, fighting behavior is detected by the following method: Calculate the comprehensive score of each individual. The comprehensive scores of all individuals constitute sequence data. Input the sequence data into the behavior classification model. Use the behavior classification model to identify aggressive actions. Aggressive actions include punching and kicking. The presence of aggressive actions indicates the presence of fighting behavior. The comprehensive score of each individual is calculated by the following formula: Comprehensive score = γ × average facial expression score + δ × abnormal behavior score + ε × overall comprehensive risk score; among them, γ, δ, and ε are the weight coefficients of the average facial expression score, abnormal behavior score, and overall comprehensive risk score, respectively.

9. The method for identifying abnormal crowd behavior according to claim 1, characterized in that: In step S6, the aggregation behavior is detected by the following method: (1) Determine the association relationship between individuals within the monitoring range to form multiple association relationship networks, each of which contains at least two individuals, and any two individuals in each association relationship network have a direct association relationship or an association relationship through one or more other individuals. Set an association number threshold. When the total number of individuals in the association relationship network with the largest number of individuals within the monitoring range is greater than the association number threshold, perform personnel density determination; (2) By determining the density of people within the monitoring range and setting a density threshold, when the total number of individuals within a certain time step within the monitoring range is greater than the density threshold, it is determined that there is a gathering behavior and an alarm is triggered immediately; The pedaling time is detected by the following method: For each time step, a speed threshold is set. When the average moving speed is greater than the speed threshold and the total number of individuals in the time step is greater than the density threshold, it is determined that there is a potential trampling risk in the main moving direction within the time step.

10. A system for identifying abnormal behavior of a crowd, characterized in that: A method for identifying abnormal behavior of a crowd according to any one of claims 1 to 9, comprising a collection module, a preprocessing module, a data feature extraction module, a data fusion module, an identification module, a display module and a communication module, wherein the collection module executes steps S1 and S2, the preprocessing module executes step S3, the data feature extraction module executes step S4, the data fusion module executes step S5, the identification module executes step S6, the display module executes the abnormal behavior display operation in step S7, and the communication module executes the abnormal information sending operation to the management personnel in step S7.

Images