A behavior recognition and emotion analysis method and system based on multiple scenes of subways
By using YOLO Pose, SlowFast, and LSTM models in the subway monitoring system, combined with DBSCAN and Byte Track algorithms, behavior recognition and emotion analysis in multiple subway scenarios were achieved. This solved the problems of low detection efficiency, complex backgrounds, and processing of multiple video streams, and improved the accuracy and real-time performance of subway monitoring.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUZHOU UNIV
- Filing Date
- 2024-11-29
- Publication Date
- 2026-07-31
AI Technical Summary
In existing subway monitoring systems, behavior recognition technology suffers from low detection efficiency, complex background redundancy, low detection accuracy, and inability to process multiple video streams, resulting in poor monitoring performance.
By employing YOLO Pose, SlowFast, and LSTM models, combined with DBSCAN and Byte Track algorithms, and through real-time acquisition of video images, we perform bounding box detection, key point extraction, crowd clustering, and sentiment analysis to construct individual and group behavior recognition models, enabling behavior recognition and sentiment analysis in multiple subway scenarios.
It improves detection efficiency, reduces background redundancy interference, enables accurate identification of abnormal behavior of individuals and groups, supports multi-threaded video processing and real-time display, and enhances the safety management of the subway transportation system.
Smart Images
Figure CN119888838B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence recognition technology, and in particular to a method and system for behavior recognition and emotion analysis in multiple scenarios in subways. Background Technology
[0002] In modern urban rail transit systems, subways, as an efficient and convenient public transportation tool, carry a large volume of passengers. To ensure the safety and efficiency of subway operations, effective monitoring and management of passenger behavior within the subway system is crucial. Especially within subway stations and carriages, passenger and staff behavior patterns are diverse, including normal walking, waiting for the train, conversation, and potentially abnormal behaviors such as individual actions (eating, dozing off, using mobile phones) and group behaviors (chatting in groups, horseplay). These behaviors not only affect the subway's operational order but may also pose a potential threat to passenger safety.
[0003] To achieve real-time monitoring and management of passenger behavior inside subway stations, a behavior recognition method based on high-definition cameras and advanced image processing technology has emerged. This method captures images inside the subway and uses intelligent algorithms to analyze the images, thereby achieving accurate identification of passenger behavior.
[0004] Although existing behavior recognition technologies have achieved some success in the field of subway monitoring, a series of problems still need to be solved:
[0005] Low detection efficiency: Due to the dense crowds and diverse behavioral patterns inside subways, existing behavior recognition technologies often struggle to accurately detect the behavior of all individuals within a short timeframe. This leads to the system easily missing important abnormal behavioral information in practical applications, thereby reducing the effectiveness of monitoring.
[0006] Complex background redundancy: The subway interior environment is complex, containing a large amount of background information such as billboards, seats, and handrails. This background information occupies a large proportion of the image and is easily confused with human behavior, thus increasing the difficulty of behavior recognition.
[0007] Low detection accuracy: Due to the diversity and complexity of human behavior, existing behavior recognition technologies often struggle to accurately identify all behaviors. Especially in densely populated areas or when behavioral characteristics are not readily apparent, the system's recognition accuracy often drops significantly.
[0008] Inability to detect multiple video streams simultaneously: Subway monitoring systems typically consist of multiple high-definition cameras, each capturing footage from a different area. However, existing behavior recognition technologies often can only process a single video stream, unable to handle information from multiple streams simultaneously. This limits the monitoring range and effectiveness of the system in practical applications.
[0009] In conclusion, existing behavior recognition technologies still have certain limitations in the field of subway monitoring. Summary of the Invention
[0010] To address these issues, this invention provides a method and system for behavior recognition and emotion analysis in multiple subway scenarios, which solves problems such as low detection efficiency and complex background redundancy in existing recognition technologies.
[0011] To address the aforementioned problems, embodiments of the present invention provide a behavior recognition and sentiment analysis method based on multiple scenarios in a subway system. This method includes:
[0012] Real-time acquisition of video images from multiple subway scenarios;
[0013] Each person in the captured video images is detected, and the bounding box information of each person is obtained; the bounding box of each person is cropped to obtain a time-series image; a pre-built personal behavior recognition model is trained for specific personal behaviors; the cropped time-series image is input into the trained personal behavior recognition model for behavior classification and recognition;
[0014] Key points are extracted from the body of each person in the captured video images; based on the location of the body key points, the detected people are clustered into groups with a tendency to cluster, and target tracking is performed on each cluster to obtain a set of body key point feature vectors for multiple consecutive frames; a pre-built group behavior recognition model is trained for specific group behaviors; the obtained set of body key point feature vectors for multiple consecutive frames is input into the trained group behavior recognition model for classification and recognition.
[0015] Facial key points are extracted from the body of each person in the captured video images; based on the location of the facial key points, a face image is obtained using a preset processing method; for emotion analysis, a pre-built emotion recognition model is trained; the acquired face image is input into the trained emotion recognition model to obtain the person's emotional state.
[0016] Preferably, the method for training a pre-built personal behavior recognition model for specific personal behaviors includes:
[0017] Collect positive and negative sample image datasets for the corresponding behaviors;
[0018] Label the positive sample image dataset and the negative sample image dataset;
[0019] The labeled positive and negative image datasets are input into the pre-built SlowFast personal behavior recognition model for training. When the output feature vector is very close to the expected personal behavior feature vector, the SlowFast personal behavior recognition model is trained and the trained SlowFast personal behavior recognition model is obtained.
[0020] Preferably, the method for extracting key points from the body of each person in the acquired video images includes:
[0021] The YOLO POSE detection model was used to extract key points from the body of each person in the acquired video images. Eleven key point locations were extracted from each person's body, including the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist.
[0022] Preferably, the method for clustering detected individuals into groups to form clusters with a tendency to cluster, and for tracking targets in each cluster, includes:
[0023] The DBSCAN clustering algorithm is used to cluster the detected people into groups to determine the spatial relationships between them and form clusters with a tendency to cluster together. The Byte Track algorithm is used to track the target in each cluster to ensure their continuity between consecutive frames.
[0024] Preferably, the method for training a pre-built group behavior recognition model for specific group behaviors includes:
[0025] Collect positive and negative sample image datasets for the corresponding behaviors;
[0026] Annotate key points of human bodies in positive and negative sample image datasets;
[0027] The labeled positive and negative image datasets are input into the pre-built LSTM group behavior recognition model for training, resulting in a trained LSTM group behavior recognition model.
[0028] Preferably, the method for obtaining a face image using a preset processing method based on the location of facial key points includes:
[0029] Based on the location of facial key points, select an appropriately sized rectangle to crop out the face image;
[0030] When a person's two eyes and two ears are detected, the face is framed by a square with the position of the two ears as the axis of symmetry and the distance between the two ears as the side length.
[0031] When a person's two eyes are detected, but only one ear can be detected, compensation is made based on the positions of the eyes and ear to frame the face.
[0032] Preferably, the method for training a pre-built emotion recognition model for emotion analysis includes:
[0033] Collect images of faces showing negative emotions and images of faces showing positive emotions;
[0034] Annotate facial images showing negative and positive emotions;
[0035] The labeled images of faces with negative emotions and images of faces with positive emotions are input into the pre-built MobileNetSmall emotion recognition model for training, resulting in a trained MobileNetSmall emotion recognition model.
[0036] This invention also provides a behavior recognition and sentiment analysis system based on multiple scenarios in a subway system. This system is used to implement the aforementioned behavior recognition and sentiment analysis method based on multiple scenarios in a subway system, specifically including:
[0037] The video image acquisition module is used to acquire video images in real time from multiple scenes in the subway.
[0038] The personal behavior recognition module is used to detect each person in the captured video images and obtain the bounding box information of each person; crop the bounding box of each person to obtain a time-series image; train a pre-built personal behavior recognition model for specific personal behaviors; and input the cropped time-series image into the trained personal behavior recognition model for behavior classification and recognition.
[0039] The group behavior recognition module is used to extract key points from the body of each person in the captured video images; based on the position of the body key points, the detected people are clustered into groups with a tendency to cluster, and target tracking is performed on each cluster to obtain a set of body key point feature vectors for multiple consecutive frames; a pre-built group behavior recognition model is trained for specific group behaviors; the obtained set of body key point feature vectors for multiple consecutive frames is input into the trained group behavior recognition model for classification and recognition;
[0040] The emotion recognition module is used to extract facial key points from the body of each person in the captured video images; based on the position of the facial key points, it uses a preset processing method to obtain the face image; for emotion analysis, it trains a pre-built emotion recognition model; and inputs the acquired face image into the trained emotion recognition model to obtain the person's emotional state.
[0041] This invention also provides an electronic device, which includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to realize the above-described behavior recognition and emotion analysis method based on multiple subway scenarios.
[0042] This invention also provides a computer storage medium storing a computer software product, the computer software product including several instructions to cause a computer device to execute the above-described behavior recognition and emotion analysis method based on multiple subway scenarios.
[0043] As can be seen from the above technical solutions, this invention application has the following beneficial effects:
[0044] This invention provides a method and system for behavior recognition and emotion analysis in multiple scenarios within a subway system, significantly improving detection efficiency and effectively reducing interference from redundant background information. By focusing on the bounding box area of the target person and analyzing only the content within the box, this invention achieves accurate identification of abnormal behaviors of individuals and groups, including inappropriate behavior by subway staff and violations by passengers. Simultaneously, by combining advanced YOLO Pose, SlowFast, and LSTM models, this invention enables multi-threaded video processing and real-time display and warnings of abnormal behaviors, solving the problems of slow detection rate, instability, low accuracy, and inability to simultaneously detect multiple video streams in existing technologies, thus providing strong support for the safety management of subway transportation systems. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Referring to the accompanying drawings will provide a clearer understanding of the features and advantages of the present invention. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. Wherein:
[0046] Figure 1 This is a flowchart illustrating a behavior recognition and sentiment analysis method based on multiple scenarios in a subway, as provided in this embodiment.
[0047] Figure 2 This is a schematic diagram illustrating the detection of actual staff members using mobile phones in the embodiment.
[0048] Figure 3 This is a schematic diagram illustrating the detection of actual staff members dozing off as part of the individual behavior described in the example.
[0049] Figure 4This is a schematic diagram illustrating the detection of actual passengers eating in the embodiment.
[0050] Figure 5 This is a schematic diagram illustrating the detection of group chat behavior in the embodiment.
[0051] Figure 6 This is a schematic diagram illustrating the detection of group behavior such as mob fighting in the embodiment.
[0052] Figure 7 This is a schematic diagram of actual emotion detection in the emotion recognition example.
[0053] Figure 8 This is a block diagram of a behavior recognition and emotion analysis system based on multiple scenarios in a subway, as provided in the embodiment. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0055] Example 1
[0056] To address the issues of low detection efficiency and complex background redundancy in existing recognition methods, such as... Figure 1 As shown in the figure, this invention proposes a behavior recognition and sentiment analysis method based on multiple scenarios in a subway system. The method includes:
[0057] Real-time acquisition of video images from multiple subway scenarios;
[0058] Each person in the captured video images is detected, and the bounding box information of each person is obtained; the bounding box of each person is cropped to obtain a time-series image; a pre-built personal behavior recognition model is trained for specific personal behaviors; the cropped time-series image is input into the trained personal behavior recognition model for behavior classification and recognition;
[0059] Key points are extracted from the body of each person in the captured video images; based on the location of the body key points, the detected people are clustered into groups with a tendency to cluster, and target tracking is performed on each cluster to obtain a set of body key point feature vectors for multiple consecutive frames; a pre-built group behavior recognition model is trained for specific group behaviors; the obtained set of body key point feature vectors for multiple consecutive frames is input into the trained group behavior recognition model for classification and recognition.
[0060] Facial key points are extracted from the body of each person in the captured video images; based on the location of the facial key points, a face image is obtained using a preset processing method; for emotion analysis, a pre-built emotion recognition model is trained; the acquired face image is input into the trained emotion recognition model to obtain the person's emotional state.
[0061] As can be seen from the above technical solution, this invention provides a behavior recognition and emotion analysis method based on multiple scenarios in subways. It acquires video images in real time, performs bounding box detection and cropping for each person, obtains temporal images, and trains an individual behavior recognition model for classification and recognition. Simultaneously, it extracts body key points for crowd clustering and target tracking, training a group behavior recognition model to identify group behavior. Furthermore, it extracts facial key points to obtain face images and trains an emotion recognition model to analyze emotional states. This method focuses on the target person region, effectively reducing interference from redundant background information, improving detection efficiency and accuracy, and providing an efficient and precise solution for behavior recognition and emotion analysis in subway scenarios.
[0062] This embodiment uses a high-definition camera to capture real-time images of the subway interior, obtaining real-time video images of the subway interior.
[0063] In this embodiment, for individual behaviors (including playing on a mobile phone, eating, and dozing off), the present invention uses the YOLO algorithm to detect each person in the acquired video images and obtain the bounding box information of each person; then, the bounding box of each person is cropped to obtain a time-series image; next, for specific individual behaviors, a pre-built SlowFast individual behavior recognition model is trained; finally, the cropped time-series image is input into the trained SlowFast individual behavior recognition model for behavior classification and recognition.
[0064] The YOLO algorithm, short for You Only Look Once, is an object detection system based on a single neural network, particularly suitable for real-time object detection tasks. The following is a detailed explanation of the YOLO algorithm:
[0065] Algorithm principle: The YOLO algorithm treats object detection as a regression problem, using a single convolutional neural network (CNN) to simultaneously predict multiple bounding boxes and class probabilities. It does not require finding the object's location first and then determining its type; instead, it directly predicts the object's type and location.
[0066] Algorithm Features: (1) Single-stage detection: Unlike traditional two-stage detection algorithms (such as the R-CNN series), YOLO uses a single-stage object detection method. It integrates object localization and classification into a single neural network, directly predicting the bounding box and class probability of the object in the image, which greatly improves the processing speed. (2) Fully convolutional network: YOLO uses a fully convolutional network (FCN) to process images, which means that the entire network structure is composed of convolutional layers and no fully connected layers are used. The advantage of doing so is that it can accept input images of any size and can maintain a fast processing speed. (3) Grid partitioning: When processing images, YOLO divides the image into an S×S grid. Each grid cell is responsible for predicting the bounding box and class of the object centered in that cell. This design allows YOLO to detect multiple objects in the image at the same time. (4) Anchor box mechanism: In order to improve the accuracy of detection, YOLO introduces the anchor box mechanism. Anchor boxes are a series of predefined bounding boxes of fixed size and shape used to predict the position and size of the object. In this way, the algorithm can better adapt to targets of different shapes and sizes. (5) Multi-scale training: In order to better detect targets of different sizes, YOLO adopts a multi-scale training strategy. During training, the network will perform forward and backward propagation at different input image sizes, which helps the model learn features at more scales.
[0067] Algorithm flow: (1) Image preprocessing: Adjust the input image to a fixed size (e.g., 416×416) and perform normalization. (2) Feature extraction: Extract features from the image using a convolutional neural network. (3) Bounding box and class prediction: Predict multiple bounding boxes and their confidence and class probabilities for each grid cell. (4) Post-processing: Remove overlapping bounding boxes using the non-maximum suppression (NMS) algorithm to obtain the final detection result.
[0068] Furthermore, methods for training pre-built personal behavior recognition models for specific individual behaviors include:
[0069] Collect positive and negative sample image datasets for the corresponding behaviors;
[0070] Label the positive sample image dataset and the negative sample image dataset;
[0071] The labeled positive and negative image datasets are input into the pre-built SlowFast personal behavior recognition model for training. When the output feature vector is very close to the expected personal behavior feature vector, the SlowFast personal behavior recognition model is trained and the trained SlowFast personal behavior recognition model is obtained.
[0072] The SlowFast personal behavior recognition model refers to a deep learning model framework for personal behavior recognition. The following is a detailed explanation of the SlowFast personal behavior recognition model:
[0073] The SlowFast personal behavior recognition model employs a unique dual-path network structure, consisting of a slow pathway and a fast pathway. This design aims to simultaneously capture spatial and temporal information from the video, thereby improving the ability and efficiency of recognizing complex actions.
[0074] The SlowFast personal behavior recognition model works based on a two-stream network structure, simultaneously processing both slow and fast paths. At certain stages of the network, features from the fast and slow paths are fused to enhance the model's discriminative ability. Furthermore, the SlowFast model utilizes 3D convolutions to process video data, thereby better understanding information in the temporal dimension.
[0075] In this embodiment, for group behaviors (including chatting and horseplay), the present invention uses the YOLO Pose detection model to extract key points from the body of each person in the acquired video images; the DBSCAN clustering algorithm is used to cluster the detected people according to the position of the body key points, forming clusters with a tendency to gather, and the ByteTrack algorithm is used to track the target in each cluster, obtaining a set of body key point feature vectors for multiple consecutive frames; for specific group behaviors, a pre-built LSTM group behavior recognition model is trained; the obtained set of body key point feature vectors for multiple consecutive frames is input into the trained LSTM group behavior recognition model for classification and recognition.
[0076] The YOLO Pose detection model is a human pose estimation model based on deep neural networks. It is primarily used for real-time, high-precision identification of the positions of key points on the human body in images or videos, thereby estimating human pose. The following is a detailed explanation of the YOLO Pose detection model:
[0077] The YOLO Pose detection model is based on the YOLO (You Only Look Once) object detection framework, particularly YOLOv5. YOLO Pose incorporates pose estimation techniques, enabling the model to simultaneously detect human targets and estimate their pose. Trained on extensive image data, the YOLO Pose detection model accurately identifies the locations of key points on the human body in images, such as the head, shoulders, elbows, wrists, hips, knees, and ankles, achieving high-precision pose estimation. Thanks to the efficient architecture of YOLOv5, the YOLO Pose detection model maintains high accuracy while achieving fast inference speeds, meeting the demands of real-time applications.
[0078] YOLO Pose Detection Model Neural Network Structure: YOLO Pose uses YOLOv5 as its basic architecture and incorporates pose estimation techniques. It uses CSP-darknet53 as the backbone, PANet to fuse features from different scales within the backbone, and finally, decoupled heads for predicting box and keypoints. Loss Function: During training, the YOLO Pose model uses Object Keypoint Similarity Loss to optimize model performance. Additionally, it uses CIoU Loss for bounding box supervision to improve object detection accuracy.
[0079] Furthermore, methods for extracting key points from the body of each person in the acquired video images include:
[0080] The YOLO Pose detection model was used to extract key points from the body of each person in the acquired video images. Eleven key point locations were extracted from each person's body, including the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist.
[0081] Furthermore, the method for clustering detected individuals into groups to form clusters with a tendency to cluster, and then tracking the target within each cluster, includes:
[0082] The DBSCAN clustering algorithm is used to cluster the detected people into groups to determine the spatial relationships between them and form clusters with a tendency to cluster together. The Byte Track algorithm is used to track the target in each cluster to ensure their continuity between consecutive frames.
[0083] DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm. This algorithm can divide regions with sufficiently high density into clusters and can discover clusters of arbitrary shapes in noisy spatial databases. The basic idea of the DBSCAN algorithm is to distinguish clusters from noise by the density of sample points. The algorithm randomly selects an unprocessed point from the dataset D. If this point is a core point, it finds all objects density-connected to this point, forming a cluster. If the point is not a core point (i.e., a boundary point or a noise point), it skips this point and continues processing the next point. This process continues until all points have been processed.
[0084] The Byte Track algorithm is a multi-target tracking algorithm based on the Tracking-by-Detection paradigm. It first obtains the target's location information through a target detector, and then matches the detected target with an existing trajectory through a data association algorithm, thereby achieving multi-target tracking.
[0085] Specifically, for group chat behavior, this invention calculates the Euclidean distance between each person. When the Euclidean distance between two people is higher than a threshold of 500 pixels, one of them is marked as the core object. Core objects are identified to form temporary group chat clusters. These temporary group chat clusters are then merged to obtain trending group chat clusters. The ByteTrack algorithm is used to match the same trending group chat cluster across different frames, obtaining the action image of each trending group chat cluster in the time series.
[0086] Specifically, for group behavior involving mob fighting, this invention calculates the intersection-union ratio (IU) between the bounding boxes of each pair of people. When the IU between the bounding boxes of two people is higher than a set threshold, one of them is marked as the core object. Temporary mob fighting clusters are formed by finding these core objects and merging them to obtain a cluster of people exhibiting a trend of mob fighting. The Byte Track algorithm is used to match the same cluster of people exhibiting a trend of mob fighting in different frames, obtaining the action images of each cluster of people exhibiting a trend of mob fighting in the time series.
[0087] Furthermore, methods for training pre-built group behavior recognition models for specific group behaviors include:
[0088] Collect positive and negative sample image datasets for the corresponding behaviors;
[0089] Annotate key points of human bodies in positive and negative sample image datasets;
[0090] The labeled positive and negative image datasets are input into the pre-built LSTM group behavior recognition model for training, resulting in a trained LSTM group behavior recognition model.
[0091] The LSTM crowd behavior recognition model refers to a model that uses a Long Short-Term Memory (LSTM) network for crowd behavior recognition. LSTM is a special type of recurrent neural network (RNN) that can effectively capture semantic relationships between long sequences, mitigating the vanishing or exploding gradient phenomenon. It is particularly suitable for processing and predicting important events with long intervals and delays in time series. In crowd behavior recognition, the LSTM model can process video frames or sensor data to extract features related to crowd behavior, thereby achieving the recognition, classification, and prediction of crowd behavior. The LSTM model can monitor and analyze crowd behavior in surveillance videos in real time, promptly detecting abnormal behavior and issuing alerts.
[0092] In this embodiment, for emotion analysis, the present invention uses the YOLO Pose detection model to extract facial key points (nose, left eye, right eye, left ear, right ear) of each person's body in the acquired video images; based on the position of the facial key points, a preset processing method is used to obtain the face image; for emotion analysis, a pre-built MOBILENET SMALL emotion recognition model is trained; the acquired face image is input into the trained MOBILENET SMALL emotion recognition model to obtain the person's emotional state, i.e., negative emotion or positive emotion.
[0093] Furthermore, methods for obtaining facial images using preset processing methods based on the location of facial key points include:
[0094] Based on the location of facial key points, select an appropriately sized rectangle to crop out the face image;
[0095] If a person's two eyes cannot be detected, it is assumed that the person is facing sideways or with their back to the camera, and face detection cannot be performed.
[0096] When a person's two eyes and two ears are detected, the face is framed by a square with the position of the two ears as the axis of symmetry and the distance between the two ears as the side length.
[0097] When both eyes are detected, but only one ear can be detected, compensation is made based on the positions of the eyes and ear to frame the face. For example, if only the left ear can be detected, then one-third of the distance from the left ear to the right eye is compensated to the right, and a square is drawn using the endpoints of the two lines as axes to frame the face.
[0098] Furthermore, for sentiment analysis, methods for training pre-built sentiment recognition models include:
[0099] Collect images of faces showing negative emotions and images of faces showing positive emotions;
[0100] Annotate facial images showing negative and positive emotions;
[0101] The labeled images of faces with negative emotions and images of faces with positive emotions are input into the pre-built MOBILENET SMALL emotion recognition model for training, resulting in a trained MOBILENET SMALL emotion recognition model.
[0102] The MobileNet Small emotion recognition model refers to a lightweight emotion recognition network model based on the MobileNet architecture. MobileNet is a deep learning model known for its high computational performance and low memory requirements, making it particularly suitable for mobile devices and real-time applications. Emotion recognition technology, also known as sentiment analysis or emotion identification, is an artificial intelligence technology that automatically identifies and classifies emotional states by analyzing various forms of information expressed by humans, such as text, speech, and images. In the MobileNet Small emotion recognition model, "Small" usually refers to the relatively small size or complexity of the model, which allows the model to maintain a certain level of recognition accuracy while having faster processing speed and lower resource consumption. This lightweight model is particularly suitable for emotion recognition in resource-constrained environments, such as mobile devices and embedded systems. The working principle of the MobileNet Small emotion recognition model is similar to other emotion recognition models, identifying emotions by analyzing features in input data (such as images and speech). Specifically, the model extracts key features from the input data, such as facial expressions, tone of voice, and vocabulary, and then uses these features to determine the human emotional state.
[0103] Based on the above method, the behavior of passengers and staff can be monitored in real time, referring to... Figures 2 to 7 This invention can not only effectively reduce the accident rate and improve the safety of subway operations, but also optimize subway operation efficiency and improve the passenger travel experience.
[0104] Example 2
[0105] like Figure 8 As shown, this invention provides a behavior recognition and sentiment analysis system based on multiple scenarios in a subway system. This system is used to implement the behavior recognition and sentiment analysis method based on multiple scenarios in a subway system described in Embodiment 1 above, specifically including:
[0106] The video image acquisition module is used to acquire video images in real time from multiple scenes in the subway.
[0107] The personal behavior recognition module is used to detect each person in the captured video images and obtain the bounding box information of each person; crop the bounding box of each person to obtain a time-series image; train a pre-built personal behavior recognition model for specific personal behaviors; and input the cropped time-series image into the trained personal behavior recognition model for behavior classification and recognition.
[0108] The group behavior recognition module is used to extract key points from the body of each person in the captured video images; based on the position of the body key points, the detected people are clustered into groups with a tendency to cluster, and target tracking is performed on each cluster to obtain a set of body key point feature vectors for multiple consecutive frames; a pre-built group behavior recognition model is trained for specific group behaviors; the obtained set of body key point feature vectors for multiple consecutive frames is input into the trained group behavior recognition model for classification and recognition;
[0109] The emotion recognition module is used to extract facial key points from the body of each person in the captured video images; based on the position of the facial key points, it uses a preset processing method to obtain the face image; for emotion analysis, it trains a pre-built emotion recognition model; and inputs the acquired face image into the trained emotion recognition model to obtain the person's emotional state.
[0110] This embodiment presents a behavior recognition and emotion analysis system based on multiple subway scenarios, used to implement the aforementioned behavior recognition and emotion analysis method based on multiple subway scenarios. Therefore, the specific implementation of the behavior recognition and emotion analysis system based on multiple subway scenarios can be found in the previous embodiment section of the behavior recognition and emotion analysis method based on multiple subway scenarios. To avoid redundancy, it will not be repeated here.
[0111] Example 3
[0112] This invention provides an electronic device, which includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to realize the above-mentioned behavior recognition and emotion analysis method based on multiple subway scenarios.
[0113] Example 4
[0114] This invention provides a computer storage medium storing a computer software product, which includes several instructions to cause a computer device to execute the aforementioned behavior recognition and emotion analysis method based on multiple subway scenarios.
[0115] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0116] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0118] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. A behavior recognition and emotion analysis method based on multiple scenarios of subway, characterized in that, include: Real-time acquisition of video images from multiple subway scenarios; Each person in the captured video images is detected, and the bounding box information of each person is obtained; The bounding boxes of each person are cropped to obtain time-series images; positive and negative sample image datasets corresponding to the individual behaviors are collected and labeled; the labeled datasets are input into a pre-built SlowFast individual behavior recognition model for training; training is completed when the matching degree between the output feature vector and the expected individual behavior feature vector reaches a preset threshold, resulting in a trained SlowFast individual behavior recognition model; the cropped time-series images are then input into the trained individual behavior recognition model for behavior classification and recognition; The YOLO Pose detection model was used to extract key points from the bodies of each person in the captured video images. Eleven key point locations were extracted for each person's body, including the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist. Based on the location of the body key points, the DBSCAN clustering algorithm was used to cluster the detected people into groups to determine the spatial relationships between people and form clusters with a tendency to cluster together. The ByteTrack algorithm was used to track the target in each cluster to ensure their continuity between consecutive frames and obtain a set of body key point feature vectors for consecutive frames. Collect positive and negative sample image datasets corresponding to group behavior, annotate the key points of the human body in the positive and negative sample image datasets, input the annotated dataset into a pre-built LSTM group behavior recognition model for training, and obtain the trained LSTM group behavior recognition model; input the obtained body key point feature vector group of multiple consecutive frames into the trained group behavior recognition model for classification and recognition. Facial key points are extracted from the body of each person in the acquired video images. Based on the position of the facial key points, a rectangle of appropriate size is selected to crop the face image. When two eyes and two ears are detected, a square is bounded with the position of the two ears as the axis of symmetry and the distance between the two ears as the side length. When two eyes are detected but only one ear is detected, compensation is made based on the position of the eyes and ears to bound the face. Images of faces with negative emotions and images of faces with positive emotions are collected and labeled. The labeled images are input into a pre-built MobileNetSmall emotion recognition model for training to obtain a trained MobileNetSmall emotion recognition model. The acquired face images are input into the trained emotion recognition model to obtain the person's emotional state.
2. A behavior recognition and emotion analysis system based on multiple scenarios of subways, characterized in that, The system is used to implement the behavior recognition and sentiment analysis method based on multiple scenarios in the subway as described in claim 1, specifically including: The video image acquisition module is used to acquire video images in multiple scenarios in the subway in real time; The personal behavior recognition module is used to detect each person in the captured video images and obtain the bounding box information of each person; crop the bounding box of each person to obtain a time-series image; train a pre-built personal behavior recognition model for specific personal behaviors; and input the cropped time-series image into the trained personal behavior recognition model for behavior classification and recognition. The group behavior recognition module is used to extract key points from the body of each person in the captured video images; based on the position of the body key points, the detected people are clustered into groups with a tendency to cluster, and target tracking is performed on each cluster to obtain a set of body key point feature vectors for multiple consecutive frames; a pre-built group behavior recognition model is trained for specific group behaviors; the obtained set of body key point feature vectors for multiple consecutive frames is input into the trained group behavior recognition model for classification and recognition; The emotion recognition module is used to extract facial key points from the body of each person in the captured video images; based on the position of the facial key points, it uses a preset processing method to obtain the face image; for emotion analysis, it trains a pre-built emotion recognition model; and inputs the acquired face image into the trained emotion recognition model to obtain the person's emotional state.
3. An electronic device, comprising: The electronic device includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the behavior recognition and emotion analysis method based on multiple subway scenarios as described in claim 1.
4. A computer storage medium, characterized in that, The computer storage medium stores a computer software product, which includes several instructions to cause a computer device to execute the behavior recognition and emotion analysis method based on multiple subway scenarios as described in claim 1.