Behavior recognition and emotion analysis method and system based on multiple scenes of subway

Abstract

The invention provides a behavior recognition and emotion analysis method and system based on multiple scenes of a subway, and relates to the technical field of artificial intelligence recognition, and the method comprises the steps: collecting videos in real time, detecting and cutting out a time sequence image of each person, and carrying out the behavior classification through a pre-trained individual behavior recognition model. Meanwhile, body key points are extracted to cluster and track people, and group behaviors are recognized through a group behavior recognition model. In addition, face key points are extracted to obtain a face image, and an emotion recognition model is utilized to analyze an emotion state. According to the technology, background redundancy is effectively reduced, the detection efficiency and accuracy are improved, and an efficient solution is provided for behavior recognition and emotion analysis in a subway scene.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence recognition technology, and in particular to a behavior recognition and emotion analysis method and system based on multiple subway scenarios. Background Art

[0002] In the modern urban rail transit system, the subway, as an efficient and convenient public transportation tool, carries a large number of people. In order to ensure the safety and efficiency of subway operations, it is particularly important to effectively monitor and manage the behavior of people inside the subway. Especially in subway stations and carriages, the behavior patterns of passengers and staff are diverse, including normal walking, waiting for the train, talking, and possible abnormal behaviors, such as individual behavior (eating, dozing off, playing with mobile phones) and group behavior (chatting in groups, playing around), etc. These behaviors not only affect the operation order of the subway, but also pose a potential threat to the safety of passengers.

[0003] In order to achieve real-time monitoring and management of the behavior of people inside the subway, a behavior recognition method based on high-definition cameras and advanced image processing technology has emerged. This method captures the internal images of the subway and uses intelligent algorithms to analyze the images, thereby achieving accurate recognition of people's behavior.

[0004] Although the existing behavior recognition technology has achieved certain application results in the field of subway monitoring, there are still a series of problems that need to be solved:

[0005] Low detection efficiency: Due to the dense population inside the subway and the complex and diverse behavior patterns, existing behavior recognition technologies often find it difficult to accurately detect the behavior of all people in a short period of time. This results in the system easily missing important abnormal behavior information in actual applications, thereby reducing the effectiveness of monitoring.

[0006] Complex background redundant information: The subway internal environment is complex, and there is a lot of background information, such as billboards, seats, handrails, etc. This background information occupies a large proportion in the image and is easily confused with human behavior, thus increasing the difficulty of behavior recognition.

[0007] Low detection accuracy: Due to the diversity and complexity of human behavior, existing behavior recognition technology often finds it difficult to accurately recognize all behaviors. Especially in situations where there are many people or the behavior characteristics are not obvious, the recognition accuracy of the system often drops significantly.

[0008] Unable to detect multiple video streams at the same time: Subway monitoring systems are usually composed of multiple high-definition cameras, each of which captures images of different areas. However, existing behavior recognition technologies can only process a single video stream and cannot process information from multiple video streams at the same time. This limits the monitoring range and effect of the system in practical applications.

[0009] In summary, the existing behavior recognition technology still has certain limitations in the field of subway monitoring. Summary of the invention

[0010] To this end, an embodiment of the present invention provides a method and system for behavior recognition and emotion analysis based on multiple scenarios in subways, which is used to solve the problems of low detection efficiency of recognition technology in the prior art and complex redundant information of the detection image background.

[0011] In order to solve the above problems, an embodiment of the present invention provides a behavior recognition and emotion analysis method based on multiple scenarios in subways, the method comprising:

[0012] Real-time acquisition of video images in multiple scenes of the subway;

[0013] Detect each person in the collected video image to obtain the bounding box information of each person; crop the bounding box of each person to obtain a time-series image; train a pre-built personal behavior recognition model for specific personal behaviors; input the cropped time-series image into the trained personal behavior recognition model for behavior classification and recognition;

[0014] Extract key points of each person's body in the collected video images; cluster the detected people according to the positions of the key points of the body to form clusters with a tendency to gather, and track the target of each cluster to obtain a group of body key point feature vectors of multiple consecutive frames; train a pre-built group behavior recognition model for specific group behaviors; input the obtained group of body key point feature vectors of multiple consecutive frames into the trained group behavior recognition model for classification and recognition;

[0015] The facial key points of each person in the captured video image are extracted; according to the position of the facial key points, a preset processing method is used to obtain a face image; for emotion analysis, a pre-built emotion recognition model is trained; the acquired face image is input into the trained emotion recognition model to obtain the person's emotional state.

[0016] Preferably, the method for training a pre-built personal behavior recognition model for a specific personal behavior includes:

[0017] Collect positive sample image datasets and negative sample image datasets of corresponding behaviors;

[0018] Label the positive sample image dataset and the negative sample image dataset;

[0019] The labeled positive sample image dataset and negative sample image dataset are input into the pre-built SlowFast personal behavior recognition model for training. When the output feature vector is extremely close to the expected personal behavior feature vector, the training of the SlowFast personal behavior recognition model is completed, and a trained SlowFast personal behavior recognition model is obtained.

[0020] Preferably, the method for extracting key points of each person's body in the collected video image comprises:

[0021] The YOLO POSE detection model is used to extract key points of each character's body in the collected video images. Eleven key point positions are extracted for each character's body, including nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist.

[0022] Preferably, the method of clustering the detected persons to form clusters with a crowding tendency and tracking the target for each cluster includes:

[0023] The DBSCAN clustering algorithm is used to cluster the detected characters to determine the spatial relationship between the characters and form clusters with a tendency to gather; the Byte Track algorithm is used to track the target of each cluster to ensure their continuity between consecutive frames.

[0024] Preferably, the method for training a pre-built group behavior recognition model for a specific group behavior comprises:

[0025] Collect positive sample image datasets and negative sample image datasets of corresponding behaviors;

[0026] Label the key points of the human body in the positive sample image dataset and the negative sample image dataset;

[0027] The labeled positive sample image dataset and negative sample image dataset are input into the pre-built LSTM group behavior recognition model for training to obtain a trained LSTM group behavior recognition model.

[0028] Preferably, the method of obtaining a face image using a preset processing method according to the positions of facial key points includes:

[0029] According to the location of facial key points, select a rectangular frame of appropriate size to crop the face image;

[0030] When the two eyes and two ears of a person are detected, the position of the two ears is used as the symmetry axis, and the square with the distance between the two ears as the side length is used to frame the face;

[0031] When both eyes of a person are detected but only one ear is detected, compensation is performed based on the positions of the eyes and ears to frame the face.

[0032] Preferably, the method for training a pre-built emotion recognition model for emotion analysis comprises:

[0033] Collect negative emotion images and positive emotion images of human faces;

[0034] Label negative emotion face images and positive emotion face images;

[0035] The labeled negative emotion face images and positive emotion face images are input into the pre-built MobileNetSmall emotion recognition model for training to obtain the trained MobileNet Small emotion recognition model.

[0036] The embodiment of the present invention further provides a system for behavior recognition and emotion analysis based on multiple subway scenarios, which is used to implement the above-mentioned method for behavior recognition and emotion analysis based on multiple subway scenarios, and specifically includes:

[0037] Video image acquisition module, used to collect video images in multiple scenes of the subway in real time;

[0038] The personal behavior recognition module is used to detect each person in the collected video images and obtain the bounding box information of each person; crop the bounding box of each person to obtain a time-series image; train a pre-built personal behavior recognition model for specific personal behaviors; and input the cropped time-series image into the trained personal behavior recognition model for behavior classification and recognition;

[0039] The group behavior recognition module is used to extract key points of each person's body in the collected video images; according to the position of the body key points, the detected people are clustered to form clusters with a tendency to gather, and each cluster is tracked to obtain a group of body key point feature vectors of continuous multiple frames; for specific group behaviors, a pre-built group behavior recognition model is trained; the obtained group of body key point feature vectors of continuous multiple frames is input into the trained group behavior recognition model for classification and recognition;

[0040] The emotion recognition module is used to extract facial key points of each person's body in the collected video images; obtain the face image using a preset processing method according to the position of the facial key points; train a pre-built emotion recognition model for emotion analysis; input the obtained face image into the trained emotion recognition model to obtain the person's emotional state.

[0041] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned behavior recognition and emotion analysis method based on multiple subway scenarios.

[0042] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes a number of instructions for enabling a computer device to execute the above-mentioned behavior recognition and emotion analysis method based on multiple subway scenarios.

[0043] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0044] The embodiment of the present invention provides a method and system for behavior recognition and emotion analysis based on multiple scenes in the subway, which significantly improves the detection efficiency and effectively reduces the interference of redundant background information. By focusing on the framed area of ​​the target person and only analyzing the content within the frame, the present invention realizes the accurate recognition of abnormal behaviors of individuals and groups, including improper behaviors of subway staff and illegal behaviors of passengers. At the same time, combined with advanced YOLO Pose, SlowFast and LSTM models, the present invention realizes multi-threaded video processing and real-time display and warning of abnormal behaviors, solving the problems of slow detection rate, instability, low accuracy and inability to detect multiple video streams simultaneously in the prior art, and provides strong support for the safety management of subway transportation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0046] Figure 1 A flowchart of a method for behavior recognition and emotion analysis based on multiple subway scenarios provided in an embodiment;

[0047] Figure 2 This is a schematic diagram of the actual staff member playing with the mobile phone detection of personal behavior in the embodiment;

[0048] Figure 3 A schematic diagram of actual staff dozing detection for individual behavior in the embodiment;

[0049] Figure 4Schematic diagram of actual passenger eating detection of individual behavior in the embodiment;

[0050] Figure 5 A schematic diagram of group chat detection of group behavior in an embodiment;

[0051] Figure 6 Schematic diagram of group behavior detection of a crowd fighting in the embodiment;

[0052] Figure 7 is a schematic diagram of actual emotion detection in emotion recognition in an embodiment;

[0053] Figure 8 The block diagram of a behavior recognition and emotion analysis system based on multiple subway scenarios provided in an embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Embodiment 1

[0056] In order to solve the problems of low detection efficiency of recognition methods in the prior art, complex redundant information of the detection image background, etc. Figure 1 As shown, the embodiment of the present invention proposes a behavior recognition and emotion analysis method based on multiple scenarios in subways, the method comprising:

[0057] Real-time acquisition of video images in multiple scenes of the subway;

[0058] Detect each person in the collected video image to obtain the bounding box information of each person; crop the bounding box of each person to obtain a time-series image; train a pre-built personal behavior recognition model for specific personal behaviors; input the cropped time-series image into the trained personal behavior recognition model for behavior classification and recognition;

[0059] Extract key points of each person's body in the collected video images; cluster the detected people according to the positions of the key points of the body to form clusters with a tendency to gather, and track the target of each cluster to obtain a group of body key point feature vectors of multiple consecutive frames; train a pre-built group behavior recognition model for specific group behaviors; input the obtained group of body key point feature vectors of multiple consecutive frames into the trained group behavior recognition model for classification and recognition;

[0060] The facial key points of each person in the captured video image are extracted; according to the position of the facial key points, a preset processing method is used to obtain a face image; for emotion analysis, a pre-built emotion recognition model is trained; the acquired face image is input into the trained emotion recognition model to obtain the person's emotional state.

[0061] From the above technical solution, it can be seen that the present invention provides a method for behavior recognition and emotion analysis based on multiple scenes in subways. By real-time acquisition of video images, bounding box detection and cropping of each person are performed, time-series images are obtained, and a personal behavior recognition model is trained for classification and recognition; at the same time, body key points are extracted for crowd clustering and target tracking, and a group behavior recognition model is trained to identify group behavior; in addition, facial key points are extracted to obtain face images, and an emotion recognition model is trained to analyze emotional states. This method focuses on the target person area, effectively reduces the interference of background redundant information, improves detection efficiency and accuracy, and provides an efficient and accurate solution for behavior recognition and emotion analysis in subway scenes.

[0062] This embodiment uses a high-definition camera to capture the interior of the subway in real time, thereby obtaining a real-time video image of the interior of the subway.

[0063] In this embodiment, for personal behaviors (including playing with mobile phones, eating, and dozing off), the present invention uses the YOLO algorithm to detect each person in the collected video images and obtain the bounding box information of each person; then, the bounding box of each person is cropped to obtain a time series image; then, for specific personal behaviors, a pre-built SlowFast personal behavior recognition model is trained; finally, the cropped time series image is input into the trained SlowFast personal behavior recognition model for behavior classification and recognition.

[0064] The YOLO algorithm, which stands for You Only Look Once, is a target detection system based on a single neural network, which is particularly suitable for real-time target detection tasks. The following is a detailed explanation of the YOLO algorithm:

[0065] Algorithm principle: The YOLO algorithm treats the target detection task as a regression problem, and simultaneously predicts multiple bounding boxes and category probabilities through a single convolutional neural network (CNN). It does not need to find the location of the object first and then determine the type of the object, but directly predicts the type and location of the object.

[0066] Algorithm features: (1) Single-stage detection: Unlike traditional two-stage detection algorithms (such as the R-CNN series), YOLO uses a single-stage target detection method. It integrates target positioning and classification into a single neural network, directly predicting the target's bounding box and category probability in the image, greatly improving the processing speed. (2) Fully convolutional network: YOLO uses a fully convolutional network (FCN) to process images, which means that the entire network structure is composed of convolutional layers and no fully connected layers are used. The advantage of this is that it can accept input images of any size and can maintain a relatively fast processing speed. (3) Grid division: When processing an image, YOLO divides the image into an S×S grid. Each grid cell is responsible for predicting the bounding box and category of the target centered in the cell. This design enables YOLO to detect multiple targets in an image at the same time. (4) Anchor box mechanism: In order to improve the accuracy of detection, YOLO introduces the anchor box mechanism. Anchor boxes are a series of pre-defined bounding boxes of fixed size and shape that are used to predict the location and size of the target. In this way, the algorithm can better adapt to objects of different shapes and sizes. (5) Multi-scale training: In order to better detect objects of different sizes, YOLO adopts a multi-scale training strategy. During the training process, the network performs forward propagation and back propagation on different input image sizes, which helps the model learn features of more scales.

[0067] Algorithm flow: (1) Image preprocessing: Resize the input image to a fixed size (such as 416×416) and perform normalization. (2) Feature extraction: Extract features from the image using a convolutional neural network. (3) Bounding box and category prediction: Each grid cell predicts multiple bounding boxes and their confidence and category probabilities. (3) Post-processing: Remove overlapping bounding boxes using the non-maximum suppression (NMS) algorithm to obtain the final detection result.

[0068] Furthermore, for specific personal behaviors, the method of training a pre-built personal behavior recognition model includes:

[0069] Collect positive sample image datasets and negative sample image datasets of corresponding behaviors;

[0070] Label the positive sample image dataset and the negative sample image dataset;

[0071] The labeled positive sample image dataset and negative sample image dataset are input into the pre-built SlowFast personal behavior recognition model for training. When the output feature vector is extremely close to the expected personal behavior feature vector, the training of the SlowFast personal behavior recognition model is completed, and a trained SlowFast personal behavior recognition model is obtained.

[0072] The SlowFast personal behavior recognition model refers to a deep learning model framework for personal behavior recognition. The following is a detailed explanation of the SlowFast personal behavior recognition model:

[0073] The SlowFast personal action recognition model uses a unique dual-pathway network structure, including a slow pathway and a fast pathway. This design aims to capture both spatial and temporal information in the video, thereby improving the recognition ability and efficiency of complex actions.

[0074] The working principle of the SlowFast personal behavior recognition model is based on a two-stream network structure, that is, processing slow and fast processing paths at the same time. At certain stages of the network, the features of the fast path and the slow path are fused to enhance the model's discrimination ability. In addition, the SlowFast personal behavior recognition model also uses 3D convolution to process video data to better understand information in the time dimension.

[0075] In this embodiment, for group behaviors (including group chatting and group fighting), the present invention uses the YOLO Pose detection model to extract key points of each person's body in the collected video images; uses the DBSCAN clustering algorithm to cluster the detected people according to the positions of the body key points to form clusters with a gathering trend, and uses the ByteTrack algorithm to track the target of each cluster to obtain a continuous multi-frame body key point feature vector group; for specific group behaviors, train a pre-constructed LSTM group behavior recognition model; and inputs the obtained continuous multi-frame body key point feature vector group into the trained LSTM group behavior recognition model for classification and recognition.

[0076] The YOLO Pose detection model is a human posture estimation model based on a deep neural network. It is mainly used to identify the positions of key points of the human body in an image or video in real time and with high precision, thereby estimating the human posture. The following is a detailed explanation of the YOLO Pose detection model:

[0077] The YOLO Pose detection model is based on the YOLO (You Only Look Once) target detection series framework, especially the YOLOv5 version. YOLO Pose combines the relevant technologies of posture estimation, enabling the model to simultaneously detect human targets and estimate their postures. The YOLO Pose detection model is trained with a large amount of image data and can accurately identify the positions of key points of the human body in the image, such as the head, shoulders, elbows, wrists, hips, knees, and ankles, to achieve high-precision posture estimation. Thanks to the efficient architecture of YOLOv5, the YOLO Pose detection model can achieve fast inference speed while maintaining high accuracy to meet the needs of real-time applications.

[0078] YOLO Pose detection model neural network structure: YOLO Pose uses YOLOv5 as the basic architecture and combines related technologies of pose estimation. It uses CSP-darknet53 as the backbone, uses PANet to fuse features of different scales from the backbone, and finally uses Decoupled Heads to predict boxes and keypoints. Loss function: During the training process, the YOLO Pose model uses the Object KeypointSimilarity Loss to optimize the model performance. In addition, it also uses CIoU Loss for bounding box supervision to improve the accuracy of target detection.

[0079] Furthermore, the method for extracting key points of each person's body in the collected video image includes:

[0080] The YOLO Pose detection model is used to extract key points of each character's body in the collected video images. Eleven key point positions are extracted for each character's body, including nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist.

[0081] Furthermore, the method of clustering the detected persons to form clusters with a tendency to gather people and tracking the target of each cluster includes:

[0082] The DBSCAN clustering algorithm is used to cluster the detected characters to determine the spatial relationship between the characters and form clusters with a tendency to gather; the Byte Track algorithm is used to track the target of each cluster to ensure their continuity between consecutive frames.

[0083] DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm. The algorithm can divide areas with sufficiently high density into clusters, and can find clusters of any shape in a noisy spatial database. The basic idea of ​​the DBSCAN algorithm is to distinguish clusters from noise by the density of sample points. The algorithm randomly selects an unprocessed point from the data set D. If the point is a core point, all objects connected by density from the point are found to form a cluster. If the point is not a core point (i.e., a boundary point or a noise point), skip the point and continue to process the next point. This process continues until all points have been processed.

[0084] The Byte Track algorithm is a multi-target tracking algorithm based on the Tracking-by-Detection paradigm, that is, the target location information is first obtained through the target detector, and then the detected target is matched with the existing trajectory through the data association algorithm to achieve multi-target tracking.

[0085] Specifically, for the group chat behavior, the present invention calculates the Euclidean distance between each character. When the Euclidean distance between two characters is higher than a threshold of 500 pixels, one of the characters is marked as a core object, and the core object is found to form a temporary group chat cluster, and the temporary group chat cluster is merged to obtain a group chat cluster with a trend. The ByteTrack algorithm is used to match the same group chat cluster with a trend in different frames to obtain the action image of each group chat cluster with a trend in the time series.

[0086] Specifically, for group behaviors of mobbing and fighting, the present invention calculates the intersection-and-union ratio between the bounding boxes of each pair of human bodies; when the intersection-and-union ratio between the bounding boxes of two human bodies is higher than a set threshold, one of the people is marked as a core object, and the core object is found to form a temporary mobbing and fighting cluster, and the temporary mobbing and fighting cluster is merged to obtain a trend of people mobbing and fighting cluster. The Byte Track algorithm is used to match the same trend of people mobbing and fighting cluster in different frames to obtain the action image of each trend of people mobbing and fighting cluster in the time series.

[0087] Furthermore, for a specific group behavior, a method of training a pre-built group behavior recognition model includes:

[0088] Collect positive sample image datasets and negative sample image datasets of corresponding behaviors;

[0089] Label the key points of the human body in the positive sample image dataset and the negative sample image dataset;

[0090] The labeled positive sample image dataset and negative sample image dataset are input into the pre-built LSTM group behavior recognition model for training to obtain a trained LSTM group behavior recognition model.

[0091] The LSTM group behavior recognition model refers to a model that uses a long short-term memory (LSTM) network to perform group behavior recognition. LSTM is a special recurrent neural network (RNN) that can effectively capture semantic associations between long sequences and alleviate gradient vanishing or explosion phenomena. It is particularly suitable for processing and predicting important events with long intervals and delays in time series. In group behavior recognition, the LSTM model can extract features related to group behavior by processing video frames or sensor data, thereby realizing the recognition, classification and prediction of group behavior. The LSTM model can monitor and analyze group behavior in surveillance videos in real time, detect abnormal behavior in a timely manner and issue an alarm.

[0092] In this embodiment, for emotion analysis, the present invention uses a YOLO Pose detection model to extract facial key points (nose, left eye, right eye, left ear, right ear) of each character's body in the captured video image; according to the positions of the facial key points, a preset processing method is used to obtain a face image; for emotion analysis, a pre-built MOBILENET SMALL emotion recognition model is trained; the acquired face image is input into the trained MOBILENET SMALL emotion recognition model to obtain the person's emotional state, that is, negative emotion or positive emotion.

[0093] Furthermore, according to the positions of facial key points, the method of obtaining a facial image using a preset processing method includes:

[0094] According to the location of facial key points, select a rectangular frame of appropriate size to crop the face image;

[0095] When a person's eyes cannot be detected, it is considered that the person is facing in profile or facing away from the camera, and face detection cannot be performed;

[0096] When the two eyes and two ears of a person are detected, the position of the two ears is used as the symmetry axis, and the square with the distance between the two ears as the side length is used to frame the face;

[0097] When both eyes of a person are detected, but only one ear can be detected, compensation is made based on the positions of the eyes and ears to frame the face. For example, when only the left ear can be detected, one-third of the distance from the left ear to the right eye is compensated to the right of the right eye, and a square is drawn with the endpoints of the two lines as the axis to frame the face.

[0098] Further, for sentiment analysis, the method of training a pre-built sentiment recognition model includes:

[0099] Collect negative emotion images and positive emotion images of human faces;

[0100] Label negative emotion face images and positive emotion face images;

[0101] The labeled negative emotion face images and positive emotion face images are input into the pre-built MOBILENET SMALL emotion recognition model for training to obtain a trained MOBILENET SMALL emotion recognition model.

[0102] The MobileNet Small emotion recognition model refers to a lightweight emotion recognition network model based on the MobileNet architecture. MobileNet is a deep learning model known for its efficient computing performance and low memory requirements, and is particularly suitable for mobile devices and real-time applications. Emotion recognition technology, also known as sentiment analysis or emotion recognition, is an artificial intelligence technology that automatically identifies and classifies human emotional states by analyzing various forms of information such as text, speech, and images expressed by humans. In the MobileNet Small emotion recognition model, "Small" usually refers to the relatively small size or complexity of the model, which enables the model to have faster processing speed and lower resource consumption while maintaining a certain recognition accuracy. This lightweight model is particularly suitable for emotion recognition in resource-constrained environments, such as mobile devices and embedded systems. The working principle of the MobileNet Small emotion recognition model is similar to other emotion recognition models, which recognize emotions by analyzing features in input data (such as images, speech, etc.). Specifically, the model extracts key features from the input data, such as facial expressions, intonation, vocabulary, etc., and then uses these features to judge the emotional state of humans.

[0103] According to the above method, the behavior of passengers and staff can be monitored in real time. Figures 2 to 7 The present invention can not only effectively reduce the accident rate and improve the safety of subway operation, but also optimize the subway operation efficiency and improve the travel experience of passengers.

[0104] Embodiment 2

[0105] like Figure 8 As shown, the present invention provides a behavior recognition and emotion analysis system based on multiple subway scenarios, which is used to implement the behavior recognition and emotion analysis method based on multiple subway scenarios in the first embodiment, and specifically includes:

[0106] Video image acquisition module, used to collect video images in multiple scenes of the subway in real time;

[0107] The personal behavior recognition module is used to detect each person in the collected video images and obtain the bounding box information of each person; crop the bounding box of each person to obtain a time-series image; train a pre-built personal behavior recognition model for specific personal behaviors; and input the cropped time-series image into the trained personal behavior recognition model for behavior classification and recognition;

[0108] The group behavior recognition module is used to extract key points of each person's body in the collected video images; according to the position of the body key points, the detected people are clustered to form clusters with a tendency to gather, and each cluster is tracked to obtain a group of body key point feature vectors of continuous multiple frames; for specific group behaviors, a pre-built group behavior recognition model is trained; the obtained group of body key point feature vectors of continuous multiple frames is input into the trained group behavior recognition model for classification and recognition;

[0109] The emotion recognition module is used to extract facial key points of each person's body in the collected video images; obtain the face image using a preset processing method according to the position of the facial key points; train a pre-built emotion recognition model for emotion analysis; input the obtained face image into the trained emotion recognition model to obtain the person's emotional state.

[0110] The present embodiment is a behavior recognition and emotion analysis system based on multiple subway scenarios, which is used to implement the aforementioned behavior recognition and emotion analysis method based on multiple subway scenarios. Therefore, the specific implementation method of the behavior recognition and emotion analysis system based on multiple subway scenarios can be found in the embodiment part of the behavior recognition and emotion analysis method based on multiple subway scenarios in the previous text. In order to avoid redundancy, it will not be repeated here.

[0111] Embodiment 3

[0112] An embodiment of the present invention provides an electronic device, which includes a processor, a memory and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned behavior recognition and emotion analysis method based on multiple subway scenarios.

[0113] Embodiment 4

[0114] An embodiment of the present invention provides a computer storage medium, which stores a computer software product. The computer software product includes a number of instructions for enabling a computer device to execute the above-mentioned behavior recognition and emotion analysis method based on multiple subway scenarios.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A behavior recognition and emotion analysis method based on multiple subway scenarios, characterized in that: include: Real-time acquisition of video images in multiple scenes of the subway; Detect each person in the captured video image and obtain the bounding box information of each person; The bounding box of each person is cropped to obtain a time-series image; a pre-built personal behavior recognition model is trained for specific personal behaviors; the cropped time-series image is input into the trained personal behavior recognition model for behavior classification and recognition; Extract key points of each person's body in the collected video images; cluster the detected people according to the positions of the key points of the body to form clusters with a tendency to gather, and track the target of each cluster to obtain a group of body key point feature vectors of multiple consecutive frames; train a pre-built group behavior recognition model for specific group behaviors; input the obtained group of body key point feature vectors of multiple consecutive frames into the trained group behavior recognition model for classification and recognition; The facial key points of each person in the captured video image are extracted; according to the position of the facial key points, a preset processing method is used to obtain a face image; for emotion analysis, a pre-built emotion recognition model is trained; the acquired face image is input into the trained emotion recognition model to obtain the person's emotional state.

2. The method for behavior recognition and emotion analysis based on multiple subway scenarios according to claim 1 is characterized in that: The method for training a pre-built personal behavior recognition model for a specific personal behavior includes: Collect positive sample image datasets and negative sample image datasets of corresponding behaviors; Label the positive sample image dataset and the negative sample image dataset; The labeled positive sample image dataset and negative sample image dataset are input into the pre-built SlowFast personal behavior recognition model for training. When the output feature vector is extremely close to the expected personal behavior feature vector, the training of the SlowFast personal behavior recognition model is completed, and a trained SlowFast personal behavior recognition model is obtained.

3. The behavior recognition and emotion analysis method based on multiple scenes in subway according to claim 1 is characterized in that: The method for extracting key points of each person's body in the collected video image comprises: The YOLO Pose detection model is used to extract key points of each character's body in the collected video images. Eleven key point positions are extracted for each character's body, including nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist.

4. The method for behavior recognition and emotion analysis based on multiple subway scenarios according to claim 1 is characterized in that: The method of clustering the detected people to form clusters with a crowd-gathering tendency and tracking the target of each cluster includes: The DBSCAN clustering algorithm is used to cluster the detected characters to determine the spatial relationship between the characters and form clusters with a tendency to gather; the Byte Track algorithm is used to track the target of each cluster to ensure their continuity between consecutive frames.

5. The method for behavior recognition and emotion analysis based on multiple subway scenarios according to claim 1 is characterized in that: The method for training a pre-built group behavior recognition model for a specific group behavior includes: Collect positive sample image datasets and negative sample image datasets of corresponding behaviors; Label the key points of the human body in the positive sample image dataset and the negative sample image dataset; The labeled positive sample image dataset and negative sample image dataset are input into the pre-built LSTM group behavior recognition model for training to obtain a trained LSTM group behavior recognition model.

6. The method for behavior recognition and emotion analysis based on multiple subway scenarios according to claim 1 is characterized in that: The method for obtaining a face image using a preset processing method according to the positions of facial key points includes: According to the location of facial key points, select a rectangular frame of appropriate size to crop the face image; When the two eyes and two ears of a person are detected, the position of the two ears is used as the symmetry axis, and the square with the distance between the two ears as the side length is used to frame the face; When both eyes of a person are detected but only one ear is detected, compensation is performed based on the positions of the eyes and ears to frame the face.

7. The method for behavior recognition and emotion analysis based on multiple subway scenarios according to claim 1 is characterized in that: The method for training a pre-built emotion recognition model for emotion analysis includes: Collect negative emotion facial images and positive emotion facial images; Label negative emotion face images and positive emotion face images; The labeled negative emotion face images and positive emotion face images are input into the pre-built MobileNet Small emotion recognition model for training to obtain a trained MobileNet Small emotion recognition model.

8. A behavior recognition and emotion analysis system based on multiple subway scenarios, characterized in that: The system is used to implement the behavior recognition and emotion analysis method based on multiple subway scenarios as described in any one of claims 1 to 7, specifically including: Video image acquisition module, used to collect video images in multiple scenes of subway in real time; The personal behavior recognition module is used to detect each person in the collected video images and obtain the bounding box information of each person; crop the bounding box of each person to obtain a time-series image; train a pre-built personal behavior recognition model for specific personal behaviors; and input the cropped time-series image into the trained personal behavior recognition model for behavior classification and recognition; The group behavior recognition module is used to extract key points of each person's body in the collected video images; according to the position of the body key points, the detected people are clustered to form clusters with a tendency to gather, and each cluster is tracked to obtain a group of body key point feature vectors of continuous multiple frames; for specific group behaviors, a pre-built group behavior recognition model is trained; the obtained group of body key point feature vectors of continuous multiple frames is input into the trained group behavior recognition model for classification and recognition; The emotion recognition module is used to extract facial key points of each person's body in the collected video images; obtain the face image using a preset processing method according to the position of the facial key points; train a pre-built emotion recognition model for emotion analysis; input the obtained face image into the trained emotion recognition model to obtain the person's emotional state.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the behavior recognition and emotion analysis method based on multiple subway scenarios as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer software product, which includes a number of instructions for enabling a computer device to execute the behavior recognition and emotion analysis method based on multiple subway scenarios as described in any one of claims 1 to 7.

Images