Indoor personnel video behavior process identification system based on target detection and MS-TCN + + algorithm

Abstract

The invention discloses an indoor personnel video behavior process identification system based on target detection and an MS-TCN + + algorithm, accurate detection and identification of a personnel behavior process in an indoor free scene are realized through cooperative processing of target detection, behavior triggering and time sequence identification, and the system utilizes adaptive switching of high and low frame intervals to realize accurate identification of the personnel behavior process in an indoor free scene. Details when behaviors occur are fully captured, and the calculation amount in a non-behavior period is remarkably reduced, so that automatic segmentation and positioning of behavior segments in a real-time video stream are realized; through key limb detection and cross-frame integration based on YOLOv7, continuity and detection stability of target tracking are guaranteed, and detection errors caused by shielding, posture changes and multi-person interaction are effectively coped with; and in combination with multi-scale time sequence modeling and inter-frame consistency constraint of the MS-TCN + + model, the system can extract behavior laws in long time sequence data and suppress the influence of noise frames, so that the recognition accuracy and robustness are improved while the real-time performance is ensured.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an indoor people video behavior process recognition system based on target detection and MS-TCN++ algorithm. Background Technology

[0002] To achieve automatic identification and early warning of the entire process of human behavior in indoor free-roaming scenarios, computer vision and video understanding technologies are widely used for real-time analysis in scenarios such as home security, public safety monitoring, intelligent elderly care, and industrial inspection. However, due to limitations such as target occlusion, lighting changes, multi-person object interactions, and differences in the duration of behavior in indoor environments, how to stably, accurately, and with low latency analyze the continuous human behavior process in long-term video streams remains an important research direction and engineering challenge in this field.

[0003] Existing machine vision-based video behavior recognition solutions are mostly based on "object detection + simple temporal statistics / counting" and often rely on standardized controlled action environments. In complex indoor free-roaming scenes, these methods face multiple challenges: First, online judgment of real-time video streams is required, making it difficult to balance computational cost and latency; second, the start and end points of behaviors in long videos are difficult to pinpoint accurately, and a large number of non-behavioral segments interfere with recognition; third, the high degree of freedom and complex interactions of human activities make trigger-based recognition difficult, leading to problems such as a significant decrease in recall, the coexistence of false positives and false negatives, and a decrease in recognition accuracy as video length increases, making it difficult to meet practical deployment requirements.

[0004] Therefore, in indoor video behavior recognition, how to accurately locate the start and end of behavior under conditions of free movement and complex interaction, balance the detection rate and false detection rate in real-time processing of long videos, and improve the stability and timeliness of recognition have become urgent problems to be solved. Summary of the Invention

[0005] This application provides an indoor people video behavior process recognition system based on target detection and MS-TCN++ algorithm.

[0006] Firstly, an indoor people video behavior process recognition system based on target detection and the MS-TCN++ algorithm, the system comprising: The system comprises a key limb target detection module, a behavior trigger discrimination module, and an MS-TCN++ behavior process recognition module, which are connected in series. The key limb target detection module is used to perform single-frame detection of human body and key limbs in the real-time video stream of the indoor scene at a preset frame interval, and obtain a set of detection boxes containing target coordinates and categories; and to perform spatiotemporal correlation and merging of cross-frame detection boxes according to the intersection-union ratio to generate target trajectory. The behavior triggering discrimination module is used to associate the head, hands, and feet as limb targets with human targets on a human body basis, and to judge possible behavior frames of video frames based on preset spatial overlap rules and motion speed rules. It generates behavior flags and performs state machine-style adaptive adjustment of frame intervals accordingly. When the condition is continuously met, it switches to a low frame interval, and when the condition is continuously not met, it switches to a high frame interval. It outputs the target area and corresponding frame sequence that trigger the behavior. The MS-TCN++ behavior process recognition module is used to cut continuous frame segments from the original video based on the trigger target region, and after preprocessing the segments by standardizing the length and unifying the size, input the preprocessed frame sequence into the MS-TCN++ model for temporal classification. The loss function containing temporal consistency constraints is used to constrain the frame-level classification to obtain the confidence vector of each behavior category, and output the recognition result containing behavior category, time interval and confidence according to the preset threshold.

[0007] Optionally, in the above scheme, the key limb target detection module uses the YOLOv7 target detection algorithm to detect the whole human body and at least one of the upper limbs, head, and feet, and sets detection thresholds and constraints for the detection score and target size to output a list of detection boxes.

[0008] Optionally, in the above scheme, the spatiotemporal association of the cross-frame detection boxes includes: tiling the detection boxes of several consecutive interval frames into a one-dimensional list, calculating the cross-over-union ratio (CUI), and merging them according to the maximum bounding box principle when the CUI is greater than a preset threshold, thereby generating a cross-frame integrated target trajectory list.

[0009] Optionally, in the above scheme, the limb target association of the behavior triggering discrimination module includes: The test results are categorized into lists of human bodies, heads, hands, and feet. For each human target, calculate the intersection relationship and intersection ratio with each limb target, and select the top-k intersection ratio to associate with the human target, so that each human target is associated with 1 head target, 1 or 2 hand targets and several foot targets.

[0010] Optionally, in the above scheme, the rule judgment of the behavior triggering judgment module includes: Spatial rule: When the intersection-union ratio between the detection frames of the upper limb and the head is greater than a preset threshold, it is determined that there is limb contact; Motion rules: The displacement velocity is calculated based on the spatial displacement of the detection box center in adjacent frames and the frame interval time. When the displacement velocity of several consecutive frames meets the condition with respect to the preset threshold, a judgment of static or violent motion is given. It outputs possible behavior flags based on the spatial rules and the motion rules.

[0011] Optionally, in the above scheme, the state machine-based adaptive adjustment of the frame interval includes: In the high frame interval state, detection is performed at the first preset interval. When the behavior flags of several consecutive frames meet the triggering conditions, the system switches to the low frame interval state. In the low frame interval state, detection is performed at the second preset interval. When the behavior flags of several consecutive frames meet the regression conditions, the system switches back to the high frame interval state.

[0012] Optionally, in the above scheme, the extraction and preprocessing of the behavioral fragments includes: Based on the frame location information of the triggered target area, cut out continuous frame segments containing the target from the original video; When the length of a segment exceeds a preset length threshold, the segment is divided into two parts. The video clips are standardized in length and clipped spatially, and the image frame size is unified to a preset pixel size.

[0013] Optionally, in the above scheme, the MS-TCN++ behavior process recognition module uses Temporal Focal Loss, which includes temporal consistency constraints, to constrain frame-level classification during training or inference. The loss function includes parameter settings for class balance weights, focus parameters, base weights, temporal consistency strength, inconsistency sensitivity, and temporal neighborhood radius.

[0014] Secondly, a method for recognizing indoor people's video behavior processes, the method comprising: S1, perform single-frame detection of human body and key limbs in real-time video stream at high frame intervals, and perform intersection-union comparison correlation and merging on cross-frame detection boxes to obtain the target trajectory; S2 associates the head, hands, and feet as human body units, calculates behavior flags based on spatial overlap and motion speed rules, and adaptively adjusts the frame interval according to the state machine to output the triggered target area and corresponding frame sequence. S3. Based on the triggered target region, crop behavior segments from the original video, preprocess the segments in terms of length and size, and then input them into the MS-TCN++ model for temporal classification. Based on the loss function containing temporal consistency constraints, obtain the confidence of each category, and output the recognition result containing category, time interval and confidence according to the preset threshold.

[0015] Compared with the prior art, this application has at least the following beneficial effects: Based on further analysis and research of existing technical problems, this application recognizes that in indoor video behavior process recognition, how to accurately locate the start and end points of behavior under conditions of free movement and complex interaction, balance the detection rate and false detection rate in real-time processing of long videos, and improve the stability and timeliness of recognition have become urgent problems to be solved. Through the collaborative processing of target detection, behavior triggering and temporal recognition, the accurate detection and recognition of personnel behavior processes in indoor free scenes are realized, which can effectively solve the problems of difficulty in locating the start and end points of behavior, the decrease in recognition accuracy caused by free movement, and the difficulty of triggering recognition in long videos in existing technologies. The system utilizes adaptive switching between high and low frame intervals to fully capture details of actions occurring during the action, significantly reducing computational load during non-action periods, thus enabling automatic segmentation and localization of action segments in real-time video streams. Through key limb detection and cross-frame integration based on YOLOv7, it ensures the continuity of target tracking and detection stability, effectively addressing detection errors caused by occlusion, pose changes, and multi-person interactions. Furthermore, combined with multi-scale temporal modeling and inter-frame consistency constraints using the MS-TCN++ model, the system can extract behavioral patterns from long-term data and suppress the influence of noisy frames, thereby improving recognition accuracy and robustness while maintaining real-time performance. Overall, the system achieves high-precision detection and recognition of continuous human behavior in complex indoor scenes, balancing recognition rate, real-time performance, and stability. Attached Figure Description

[0016] Figure 1 A block diagram of the module architecture of an indoor people video behavior process recognition system based on target detection and MS-TCN++ algorithm provided in one embodiment of this application; Figure 2 This is a system architecture diagram of an indoor people video behavior process recognition system based on target detection and MS-TCN++ algorithm, provided as an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).

[0019] Recognizing the behavior of people indoors is difficult due to issues such as real-time judgment of video streams, difficulty in locating the starting point of behavior in long videos, and the degree of freedom of movement of people.

[0020] The machine vision-based indoor personnel video behavior process recognition system refers to the identification and early warning of the entire process of continuous personnel behavior in real-time monitoring video collected by home-use resolution monitoring devices in free indoor scenes, using machine vision technology.

[0021] Existing machine vision-based video behavior recognition is often based on target detection and time-series analysis counting, and is widely used in action constraint environments where personnel actions are required to be standardized. This leads to problems such as limited personnel freedom, complex recognition conditions, reduced recognition accuracy in long videos, and difficulty in trigger-based recognition in long videos in actual application. As a result, the recognition recall rate drops significantly in practical applications, and the recognition accuracy also decreases with the length of the video.

[0022] To address this problem, this embodiment proposes an indoor personnel video behavior process recognition system based on target detection and the MS-TCN++ algorithm. 1) The system uses a serial processing approach combining the target detection algorithm and the MS-TCN++ algorithm to detect and recognize video behavior processes; 2) The system first uses the YOLO target detection algorithm to detect human bodies and key limb targets in indoor scenes, and then identifies possible behavior processes by designing limb contact and movement rules; 3) Then, the MS-TCN++ video action recognition algorithm is used to identify and filter the detected possible behavior process video segments. The system achieves accurate detection and recognition of personnel video behavior processes in indoor free scenes through a two-stage progressive recognition process.

[0023] The system decomposes the task of recognizing video behavior processes in indoor free-movement scenes into a data processing algorithm flow from coarse to fine. It uses target detection algorithms and discrimination rules to detect as many possible behavior frames as possible in the real-time video stream, reducing the system's false negative rate. Then, it uses video action recognition algorithms to identify and filter the detected behavior videos, reducing the system's false positive rate and making the system's behavior process detection results more accurate. This results in a high-precision, low-false-positive indoor people video behavior process recognition system.

[0024] The problem addressed in this embodiment is to provide an indoor personnel video behavior process recognition system based on target detection and the MS-TCN++ algorithm. Addressing issues such as the free and unrestrained movement of people in indoor real-time monitoring scenarios and the interference of long videos lacking behavioral segments on accuracy, a progressive video behavior process detection and filtering algorithm is designed, resulting in an intelligent personnel behavior process recognition system with high recognition accuracy and low false detection rate.

[0025] In this embodiment, as Figure 1 and Figure 2 As shown, an indoor people video behavior process recognition system based on target detection and MS-TCN++ algorithm is provided. The system includes a key limb target detection module, a behavior trigger discrimination module, and an MS-TCN++ behavior process recognition module, which are connected in series. The key limb target detection module is used to perform single-frame detection of human body and key limbs in the real-time video stream of the indoor scene at a preset frame interval, and obtain a set of detection boxes containing target coordinates and categories; and to perform spatiotemporal correlation and merging of cross-frame detection boxes according to the intersection-union ratio to generate target trajectory. The behavior triggering discrimination module is used to associate the head, hands, and feet as limb targets with human targets on a human body basis, and to judge possible behavior frames of video frames based on preset spatial overlap rules and motion speed rules. It generates behavior flags and performs state machine-style adaptive adjustment of frame intervals accordingly. When the condition is continuously met, it switches to a low frame interval, and when the condition is continuously not met, it switches to a high frame interval. It outputs the target area and corresponding frame sequence that trigger the behavior. The MS-TCN++ behavior process recognition module is used to cut continuous frame segments from the original video based on the trigger target region, and after preprocessing the segments by standardizing the length and unifying the size, input the preprocessed frame sequence into the MS-TCN++ model for temporal classification. The loss function containing temporal consistency constraints is used to constrain the frame-level classification to obtain the confidence vector of each behavior category, and output the recognition result containing behavior category, time interval and confidence according to the preset threshold.

[0026] This embodiment provides an indoor personnel video behavior process recognition system based on target detection and MS-TCN++ algorithm. It mainly consists of a key limb target detection module, a behavior trigger discrimination module, and an MS-TCN++ behavior process recognition module. The modules adopt a serial data transmission structure to realize real-time detection and recognition of the entire process of personnel behavior in indoor free scenes.

[0027] In its implementation, the system first uses a key limb target detection module to perform interval sampling and detection on the input video stream. This module employs the YOLOv7 target detection algorithm to detect targets appearing in indoor scenes, including the entire human body and key limbs (such as the head, hands, and feet). During the detection phase, the YOLOv7 model extracts features from the input image frames and generates a set of predicted bounding boxes through a multi-scale feature fusion network. The system sets a detection threshold of thresh_detec=0.5 based on the behavior recognition requirements to filter target detection boxes with high confidence. The detection result output is a list of detection boxes, detec_objs, containing the target's location coordinates (x, y, w, h) and category information.

[0028] To ensure the continuity and stability of detection, this invention performs cross-frame correlation on the detection results of adjacent frames. Specifically, the system flattens the detection results of three consecutive frames and calculates the Intersection over Union (IOU) of any two detection boxes. When the IOU value is greater than 0.7, these detection boxes are merged using the maximum bounding box principle to form a unified target trajectory list tracked_objs. This process can effectively eliminate the problem of missed or false detections in a single frame caused by changes in illumination, pose, etc., and provide stable temporal data input for subsequent behavior judgment.

[0029] The second part of the system is the behavior trigger discrimination module. This module performs limb association and behavior judgment on the target detection results. First, based on the detection box category, the targets are divided into a human body list (person_list), a head list (head_list), a hand list (hand_list), and a foot list (foot_list). Then, for each human body target, the intersection relationship and intersection ratio with each limb target are calculated. The top-k limb targets with the highest intersection ratios are selected and associated with the human body target, thus forming a structured human body-limb combination dataset.

[0030] Based on this, the system determines behavior triggers according to preset spatial and motion rules. Spatial rules determine the presence of contact behavior, such as patting the head or scratching the ear, by checking if the intersection-union ratio (IU) of the upper limb and head detection boxes exceeds a threshold (e.g., 0.3). Motion rules calculate the displacement velocity v = Δd / Δt by measuring the displacement of the center of the same target detection box in adjacent frames and the frame interval, and distinguish between static and violent behaviors based on a threshold v_thresh = 10 pixels / frame. When the behavior conditions are met for three consecutive frames, the system generates a behavior flag behavior_flag = 1; otherwise, it is 0.

[0031] To balance real-time performance and recognition accuracy, the behavior triggering and discrimination module further employs a state machine-based adaptive frame interval adjustment mechanism. The system initially operates in a high frame interval state (H-mode), detecting once every 30 frames by default. When the behavior flag is 1 for 3 consecutive frames, the system switches to a low frame interval state (L-mode), detecting once every 8 frames to capture behavior details. When the behavior flag is 0 for 5 consecutive frames, the system returns to the high frame interval state. This dynamic adjustment mechanism increases the sampling density when a behavior occurs and reduces the computational load after the behavior ends, ensuring both real-time performance and recognition integrity. The final output is a list of target regions that trigger the behavior (trigger_objs) and the corresponding frame sequence.

[0032] The third part of the system is the MS-TCN++ behavior process recognition module, which is used for temporal modeling and classification of continuous video segments that trigger target regions. First, the system extracts continuous frame segments containing the target from the original video based on the time interval information recorded in `trigger_objs`. For segments exceeding a set length threshold, the system performs secondary segmentation to ensure that the length of each segment matches the network input requirements. Subsequently, preprocessing is performed on the extracted segments, including length normalization, spatial cropping, and frame size unification (224×224 pixels). The preprocessed frame sequence is organized into a normalized video data list `behavior_clips`.

[0033] During the recognition phase, the system inputs each behavior_clips into the MS-TCN++ model. This model is based on a multi-scale temporal convolutional structure, which can simultaneously extract short-term and long-term dependency features. To improve the stability of temporal recognition and its robustness to noisy frames, this invention replaces the traditional frame-level cross-entropy loss with temporal focal loss. This loss function introduces inter-frame consistency constraints on top of the standard Focal Loss, enabling the model to consider the continuity of behavior between adjacent frames during training and inference. The model output is a behavior category confidence vector, such as: "waving" 0.92, "slapping head" 0.88, "head-banging" 0.85, etc. The system sets a confidence threshold prob_thresh=0.65 and outputs recognition results with confidence scores higher than the threshold, including the behavior category, occurrence time interval, and confidence score.

[0034] In summary, the entire system adopts a progressive structure of "detection-triggering-recognition" to realize a closed-loop processing flow from real-time video input to the final behavior recognition result. Each module is sequentially connected through data interfaces to realize the detection and recognition of behavior throughout the entire process in complex indoor scenarios.

[0035] This embodiment introduces a hierarchical algorithm design in three stages: target detection, behavior triggering, and temporal recognition, which can effectively solve several key problems mentioned in the background technology.

[0036] First, addressing the challenge of pinpointing the start and end points of actions, an adaptive detection mechanism with high and low frame intervals enables the system to automatically determine the start and end of actions without requiring manual annotation of action times. A state machine mechanism ensures increased sampling frequency during action occurrences and reduced detection overhead during non-action periods, thereby enabling action triggering and segmentation recognition in long video streams and avoiding recognition errors caused by ambiguous start and end points.

[0037] Secondly, to address the issue of decreased recognition accuracy due to the high degree of freedom and diverse movements of people, this invention employs the YOLOv7 algorithm in the target detection module and utilizes a cross-frame IOU integration strategy to improve the stability of human body and key limb detection. This ensures that target tracking consistency is maintained even under complex conditions such as posture changes and partial occlusion, providing reliable input for subsequent behavior judgment.

[0038] Third, to address the issues of decreased accuracy in long video recognition and difficulties in trigger-based recognition, a two-stage structure of "initial detection of behavior rules + in-depth temporal verification" is adopted to achieve recognition filtering. In the first stage, spatial and motion rules are used to quickly determine the likelihood of behavior and reduce redundant frame input. In the second stage, the MS-TCN++ temporal network is used to capture long-term dependency information, ensuring accurate classification of long-sequence repetitive actions or periodic behaviors, and significantly improving recognition stability and accuracy.

[0039] Fourth, to address the issues of limited real-time processing capabilities and excessive system computation, a dynamic sampling strategy and lightweight network architecture are employed to effectively reduce the computational load while ensuring detection and recognition performance. The YOLOv7 detector maintains high accuracy even at high frame rates, while the MS-TCN++ model reduces the number of parameters by 40% compared to traditional TCN, enabling the system to possess excellent real-time performance and deployability.

[0040] Finally, under conditions such as multi-target interaction, low contrast, and occlusion in complex indoor environments, the temporal consistency constraint loss function of this invention can reduce the interference of isolated noise frames on the overall classification results, making the recognition results more stable and reliable.

[0041] In summary, this system can accurately detect and identify human behavior in long-term video in free-movement scenarios, effectively improving the positioning accuracy and recognition reliability of the behavior process, and significantly reducing the false detection rate and false negative rate. Thus, it can operate stably and meet real-time requirements in various scenarios such as intelligent care, public safety, and industrial monitoring.

[0042] In this embodiment, the key limb target detection module uses the YOLOv7 target detection algorithm to detect the whole human body and at least one of the upper limbs, head, and feet, and sets detection thresholds and constraints for the detection score and target size to output a list of detection boxes.

[0043] In this embodiment, the spatiotemporal association of the cross-frame detection boxes includes: tiling the detection boxes of several consecutive interval frames into a one-dimensional list, calculating the cross-union ratio (CUI), and merging them according to the maximum bounding box principle when the CUI is greater than a preset threshold, thereby generating a cross-frame integrated target trajectory list.

[0044] In this embodiment, the limb target association of the behavior triggering discrimination module includes: The test results are categorized into lists of human bodies, heads, hands, and feet. For each human target, calculate the intersection relationship and intersection ratio with each limb target, and select the top-k intersection ratio to associate with the human target, so that each human target is associated with 1 head target, 1 or 2 hand targets and several foot targets.

[0045] In this embodiment, the rule judgment of the behavior triggering discrimination module includes: Spatial rule: When the intersection-union ratio between the detection frames of the upper limb and the head is greater than a preset threshold, it is determined that there is limb contact; Motion rules: The displacement velocity is calculated based on the spatial displacement of the detection box center in adjacent frames and the frame interval time. When the displacement velocity of several consecutive frames meets the condition with respect to the preset threshold, a judgment of static or violent motion is given. It outputs possible behavior flags based on the spatial rules and the motion rules.

[0046] In this embodiment, the state machine-based adaptive adjustment of the frame interval includes: In the high frame interval state, detection is performed at the first preset interval. When the behavior flags of several consecutive frames meet the triggering conditions, the system switches to the low frame interval state. In the low frame interval state, detection is performed at the second preset interval. When the behavior flags of several consecutive frames meet the regression conditions, the system switches back to the high frame interval state.

[0047] In this embodiment, the extraction and preprocessing of the behavioral fragments includes: Based on the frame location information of the triggered target area, cut out continuous frame segments containing the target from the original video; When the length of a segment exceeds a preset length threshold, the segment is divided into two parts. The video clips are standardized in length and clipped spatially, and the image frame size is unified to a preset pixel size.

[0048] In this embodiment, the MS-TCN++ behavior process recognition module uses Temporal Focal Loss, which includes temporal consistency constraints, to constrain frame-level classification during training or inference. The loss function includes parameter settings for class balance weights, focus parameters, base weights, temporal consistency strength, inconsistency sensitivity, and temporal neighborhood radius.

[0049] In one embodiment, an indoor personnel video behavior process recognition system based on target detection and MS-TCN++ algorithm is provided. It consists of three algorithm modules: key limb target detection, behavior trigger discrimination, and video behavior recognition. Through a serial processing architecture, it achieves the detection, trigger judgment, and accurate recognition of the entire behavior process of personnel in a free indoor scene. As shown in Figure 1, the overall system flow is as follows: First, the target detection algorithm is used to detect human bodies and their key limb targets in real-time video of the indoor scene at high frame intervals. Then, possible behavior frames are judged according to preset rules. When multiple judgments indicate the presence of behavior, it is considered the start of behavior, triggering low-frame-interval target detection and judgment. When multiple judgments indicate no behavior, it is considered the end of behavior, and the system returns to high-frame-interval detection, completing the screening of possible behavior processes. Finally, the MS-TCN++ video action recognition algorithm is used to identify and filter the detected possible behavior process video segments, forming a two-stage progressive and efficient recognition framework. The overall process follows a progressive strategy of "initial screening based on behavior triggers and detailed examination based on temporal action recognition," ensuring a balance between detection efficiency and recognition accuracy.

[0050] Key Limb Target Detection Module: This module is responsible for detecting human bodies and key limb targets in single frames of indoor real-time video, providing basic positioning information for subsequent behavior rule judgment, and reducing the computational complexity of behavior detection in real-time video by adaptively adjusting the low frame interval sparse sampling, while ensuring the integrity of behavior detection target detection.

[0051] 1) Target detection in interval image frames: The module uses the YOLOv7 target detection algorithm as its core. Based on behavioral needs, it performs single or multi-class detection on the whole body and key limbs such as the upper limbs and head, targeting high-frequency limb contact behaviors (taking self-stimulation behaviors of children as an example, such as waving, patting the head, and bumping the body). Compared with similar target detection algorithms, the YOLOv7 algorithm achieves good performance in both detection and classification. It is currently the only detector that can still exceed 30 FPS at high accuracy (optimal accuracy 56.8% AP). The algorithm can adaptively constrain the size range of the detected target to accelerate detection and reduce false detections. It is also more lightweight and efficient than similar detection algorithms. This allows the key limb target detection module to quickly locate dynamic targets at high and low frame intervals (the system defaults to an initial high interval of 30 frames and a low interval of 8 frames), and output a list of detection boxes containing the target coordinates (x, y, w, h) and limb category: detec_objs:[obj1,obj2,...,objn].

[0052] The specific process is as follows: extract image frames (video_frames:[framef1,framef2,framef3,...]) from the input video sequence at set intervals, perform target detection on each frame with a detection threshold (thresh_detec=0.5), and generate a set of detection boxes (frame_objs:[[obj1_framef1,obj2_framef2],[obj1_framef2,obj2_framef2],...]) for each frame.

[0053] 2) Cross-frame detection box integration: To address the potential for missed target detections across different frame intervals, the module utilizes Intersection over Union (IOU) to perform spatiotemporal correlation of cross-frame detection boxes. Detection boxes from three consecutive frame intervals are flattened into a one-dimensional list, and pairwise IOU values ​​are calculated. Detection boxes with IOU > 0.7 are merged according to the maximum bounding box principle, generating a cross-frame integrated target trajectory list `tracked_objs` to ensure consistent localization of the same target across time series.

[0054] Behavior Trigger Judgment Module: Based on the target detection results, the behavior trigger judgment module determines whether the behavior process detection process is triggered through preset limb contact rules and motion patterns, and dynamically adjusts the frame interval to balance efficiency and accuracy. The main processes of the module include "possible behavior frame judgment" and "adaptive adjustment of frame interval".

[0055] 1) Judgment of physical contact and movement rules

[0056] The behavior frame determination process can be roughly divided into two steps: limb target association and behavior trigger determination, as described in detail below: ①Physical goal association The limb target association process is mainly responsible for associating and combining the whole human body and key limb targets such as the upper limbs and head detected in the key limb target detection process on a human body basis. This forms a data foundation that can be directly judged for the behavior trigger judgment process. The specific combination steps are as follows: A. Human Target Classification: First, the recognition results of key limb targets are classified by category (human body, two hands, one hand, foot, head) into: human body list person_list, hand list hand_list (including categories: two hands, one hand), head list head_list, and foot list foot_list; B. Human Target Association: This function is responsible for associating detected discrete limb targets with human targets one by one. For each human body, only one head, one to two hands, and feet are associated. The specific processing is as follows: For each human body target in the person_list, perform the following consistent processing on the limb lists x_list (head, hands, feet, etc.): Calculate the intersection relationship (inter_flag) between the current human target and each limb target in the limb list (x_list), and the proportion of the intersection area to the limb target (inter_prop). From all limb targets that have a bounding box relationship with the human body, select Topk (the upper limit of the number of limb targets that can be associated) in the intersection ratio inter_prop as the association result, add the limb target pair to the "x_info" field (such as head_info, hand_info, foot_info) of the human body detection target, and remove it from x_list.

[0057] ②Behavior trigger judgment: The system associates limb targets with human body detection target information and determines the relationships between limb targets during human behavior to achieve preliminary detection of abnormal behavior. The two types of triggering conditions defined for the behavioral process are as follows: A. Spatial rules: For each human target in the person_list, determine whether different limb detection boxes overlap (e.g., the IOU of the upper limb and head detection boxes is >0.3), and determine whether there is limb contact behavior (e.g., patting the head, scratching the ear). B. Motion Rules: Combining the images of the previous and next frames of the current frame, the displacement velocity v=Δd / Δt of the target detection box is calculated for each human body target and limb target tracked_objs in the person_list of the previous and next frames based on their relative spatial position (Δd is the distance between the centers of the detection boxes in adjacent frames, and Δt is the frame interval time). When the velocity is lower than the threshold v_thresh=10 pixels / frame for 3 consecutive frames, it is judged as static behavior (such as continuous hand waving). When it is higher than the threshold, it is judged as violent motion behavior (such as body collision).

[0058] The process performs rule matching on the integrated detection bounding box tracked_objs for each frame and outputs a behavior_flag (1 indicates that a behavior may exist, and 0 indicates that no behavior exists).

[0059] 2) Adaptive adjustment of frame interval: In behavior detection, the frame interval adjustment uses a state machine mechanism to dynamically adjust the detection interval. A. High frame interval state (H-mode): Initial state, detection interval is 30 frames. When behavior_flag=1 for 3 consecutive frames, low frame interval state (L-mode) is triggered, and the interval is switched to 1 frame (detection frame by frame). B. Low frame interval state (L-mode): Detection at 8-frame intervals. When behavior_flag=0 for 5 consecutive frames, revert to H-mode.

[0060] This mechanism performs intensive sampling to capture details when the behavior occurs and sparse sampling after the behavior ends to reduce computational load. By acquiring image frames at high and low frame intervals and determining the behavior rules of the interval image frames, the behavior process starts from low interval triggering to high interval regression, and integrates the output frame interval records and the target region list trigger_objs.

[0061] MS-TCN++ Behavior Process Recognition Module: As the core of the system's detailed inspection phase, the behavior process recognition module uses the MS-TCN++ video action recognition algorithm to model the temporal features of video segments in the target area that triggers the behavior, thereby achieving accurate recognition and screening of personnel video behavior processes.

[0062] 1) Behavioral segment extraction and preprocessing: Based on the frame sequence and the frame location information of trigger_objs, continuous frame segments containing the target are cropped from the original video, and segments exceeding the segment length threshold by 50% are further divided. The segment preprocessing process includes: ① Length standardization: The sequence is unified by using the video interpolation frame interpolation method with image height and width as interpolation dimensions. The system defaults to a video length of 10 seconds, totaling 248 frames; ② Size standardization: Video segments are spatially cropped based on the range of personnel movement to reduce interference from useless information in the original image, and the image frame size is adjusted to 224×224 pixels to generate a list of segments to be identified, behavior_clips:[b_video1,b_video2,...,b_videon].

[0063] 2) Temporal video behavior classification and recognition: The video action recognition process is based on the MS-TCN++ temporal video action recognition algorithm, which classifies and identifies various behavioral processes from initially detected potential action segments (taking self-stimulating behaviors as an example, including waving, spinning, head-patting, and head-banging). The MS-TCN++ algorithm captures long-term temporal dependencies (such as the periodicity of repetitive actions) through multi-scale temporal convolution kernels and attention mechanisms. It achieves 89.2% accuracy on the UCF101 dataset while reducing the number of parameters by 40% compared to traditional TCN, meeting real-time requirements. To address the class imbalance problem, the method improves upon the MS-TCN++ algorithm by replacing the cross-entropy loss for frame-level classification with temporal attention loss (Temporal Focal Loss). This introduces temporal consistency constraints on top of the standard Focal Loss, considering not only the classification difficulty of the current frame but also the consistency of its adjacent frames. For example, an isolated "abnormal" frame surrounded by preceding and following "normal" frames can have its weight reduced, as it may be noise. The Temporal Focal Loss function is described as follows:

[0064] Where p_t is the model's predicted probability of the true class, α_t is the class balancing weight, γ is the focusing parameter, β is the basic weight, λ is the temporal consistency strength, σ is the inconsistency sensitivity, and K is the temporal neighborhood radius.

[0065] The model takes RGB sequences of video clips from behavior_clips as input and outputs a confidence vector containing the behavior category (e.g., "waving" 0.92, "head-banging" 0.85, "normal" 0.11). The system sets a recognition threshold prob_thresh=0.65 and makes a final judgment on behavior categories with confidence scores higher than the threshold, outputting a result res_behavior:[(class,score,[t_start,t_end]),...] containing the behavior category, time interval, and confidence score. For example, the recognition result might be "head-banging behavior, confidence score 0.88, occurring in frames 120-145".

[0066] The system has a two-stage collaboration mechanism: 1) First stage (detection trigger): By connecting the target detection module and the behavior trigger discrimination module, the system achieves lightweight processing of "high frame interval coarse detection → rule judgment → dynamic frame interval adjustment", which can quickly locate target areas that may have self-stimulatory behavior and trigger dense detection. 2) Second stage (behavior recognition): The MS-TCN++ module performs deep temporal feature analysis on the triggered behavior segments, and utilizes the algorithm's lightweight nature and long-term temporal modeling capabilities to achieve accurate classification while maintaining real-time performance.

[0067] The two stages are seamlessly connected through the data interfaces of trigger_objs and behavior_clips, forming a closed loop of "detection trigger - accurate recognition", which effectively improves the detection efficiency and recognition accuracy of human behavior processes in complex indoor scenarios.

[0068] This embodiment presents an indoor personnel video behavior process recognition system based on target detection and the MS-TCN++ algorithm. It decomposes the task of recognizing personnel behavior processes in complex indoor scenes step by step. By integrating algorithms such as YOLO target detection, dynamic frame interval behavior triggering rules, and MS-TCN++ temporal action recognition, it constructs a progressive behavior analysis architecture of "coarse detection-fine triggering-deep recognition". This enables efficient detection and accurate recognition of various indoor personnel behavior processes (such as abnormal actions, repetitive operations, and interactive behaviors), providing universal behavior analysis technology support for scenarios such as industrial inspection and monitoring, public safety early warning, and intelligent elderly care.

[0069] The system's core capabilities are target localization, motion state discrimination, and time-series feature modeling. Through a processing mechanism of "initial detection of behavior rules and deep network verification," it improves the behavior detection rate in complex scenarios such as multiple overlapping people and rapid movement, while effectively reducing the misjudgment rate of similar behaviors (such as irrelevant movements of non-target personnel). It also enables the system to operate stably in edge devices with limited computing power and cloud collaborative architecture, and can be applied to outdoor intelligent monitoring scenarios with greater changes and more complex environments.

[0070] In one embodiment, a method for recognizing indoor people's video behavior is provided, the method comprising: S1, perform single-frame detection of human body and key limbs in real-time video stream at high frame intervals, and perform intersection-union comparison correlation and merging on cross-frame detection boxes to obtain the target trajectory; S2 associates the head, hands, and feet as human body units, calculates behavior flags based on spatial overlap and motion speed rules, and adaptively adjusts the frame interval according to the state machine to output the triggered target area and corresponding frame sequence. S3. Based on the triggered target region, crop behavior segments from the original video, preprocess the segments in terms of length and size, and then input them into the MS-TCN++ model for temporal classification. Based on the loss function containing temporal consistency constraints, obtain the confidence of each category, and output the recognition result containing category, time interval and confidence according to the preset threshold.

[0071] This embodiment achieves accurate detection and recognition of human behavior processes in indoor free scenes through the collaborative processing of target detection, behavior triggering, and temporal recognition. The system utilizes adaptive switching between high and low frame intervals to fully capture the details of behavior occurrences and significantly reduce the computational load during non-behavioral periods, thereby enabling automatic segmentation and localization of behavior segments in real-time video streams. By using YOLOv7-based key limb detection and cross-frame integration, the continuity of target tracking and detection stability are ensured, effectively addressing detection errors caused by occlusion, posture changes, and multi-person interactions. Furthermore, by combining multi-scale temporal modeling and inter-frame consistency constraints of the MS-TCN++ model, the system can extract behavioral patterns from long-term data and suppress the influence of noisy frames, thereby improving recognition accuracy and robustness while ensuring real-time performance.

[0072] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed, relates to all or part of the processes in the system described above.

[0073] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. An indoor personnel video behavior process recognition system based on target detection and MS-TCN++ algorithm, characterized in that, The system includes a key limb target detection module, a behavior trigger discrimination module, and an MS-TCN++ behavior process recognition module, which are connected in series. The key limb target detection module is used to perform single-frame detection of human body and key limbs in the real-time video stream of the indoor scene at a preset frame interval, and obtain a set of detection boxes containing target coordinates and categories; and to perform spatiotemporal correlation and merging of cross-frame detection boxes according to the intersection-union ratio to generate target trajectory. The behavior triggering discrimination module is used to associate the head, hands, and feet as limb targets with human targets on a human body basis, and to judge possible behavior frames of video frames based on preset spatial overlap rules and motion speed rules. It generates behavior flags and performs state machine-style adaptive adjustment of frame intervals accordingly. When the condition is continuously met, it switches to a low frame interval, and when the condition is continuously not met, it switches to a high frame interval. It outputs the target area and corresponding frame sequence that trigger the behavior. The MS-TCN++ behavior process recognition module is used to cut continuous frame segments from the original video based on the trigger target region, and after preprocessing the segments by standardizing the length and unifying the size, input the preprocessed frame sequence into the MS-TCN++ model for temporal classification. The loss function containing temporal consistency constraints is used to constrain the frame-level classification to obtain the confidence vector of each behavior category, and output the recognition result containing behavior category, time interval and confidence according to the preset threshold.

2. The system of claim 1, wherein, The key limb target detection module uses the YOLOv7 target detection algorithm to detect the whole human body and at least one of the upper limbs, head, and feet. It sets detection thresholds and constraints for the detection score and target size to output a list of detection boxes.

3. The system of claim 1, wherein, The spatiotemporal association of the cross-frame detection boxes includes: tiling the detection boxes of several consecutive interval frames into a one-dimensional list, calculating the cross-union ratio (CUI), and merging them according to the maximum bounding box principle when the CUI is greater than a preset threshold, thereby generating a cross-frame integrated target trajectory list.

4. The system of claim 1, wherein, The limb target association of the behavior triggering discrimination module includes: The test results are categorized into lists of human bodies, heads, hands, and feet. For each human target, calculate the intersection relationship and intersection ratio with each limb target, and select the top-k intersection ratio to associate with the human target, so that each human target is associated with 1 head target, 1 or 2 hand targets and several foot targets.

5. The system of claim 1, wherein, The rule judgment of the behavior triggering judgment module includes: Spatial rule: When the intersection-union ratio between the detection frames of the upper limb and the head is greater than a preset threshold, it is determined that there is limb contact; Motion rules: The displacement velocity is calculated based on the spatial displacement of the detection box center in adjacent frames and the frame interval time. When the displacement velocity of several consecutive frames meets the condition with respect to the preset threshold, a judgment of static or violent motion is given. It outputs possible behavior flags based on the spatial rules and the motion rules.

6. The system of claim 1, wherein, The state machine-based adaptive adjustment of the frame interval includes: In the high frame interval state, detection is performed at the first preset interval. When the behavior flags of several consecutive frames meet the triggering conditions, the system switches to the low frame interval state. In the low frame interval state, detection is performed at the second preset interval. When the behavior flags of several consecutive frames meet the regression conditions, the system switches back to the high frame interval state.

7. The system of claim 1, wherein, The extraction and preprocessing of the behavioral segments include: Based on the frame location information of the triggered target area, cut out continuous frame segments containing the target from the original video; When the length of a segment exceeds a preset length threshold, the segment is divided into two parts. The video clips are standardized in length and clipped spatially, and the image frame size is unified to a preset pixel size.

8. The system of claim 1, wherein, The MS-TCN++ behavior process recognition module uses Temporal Focal Loss, which includes temporal consistency constraints, to constrain frame-level classification during training or inference. The loss function includes parameter settings for class balance weights, focus parameters, base weights, temporal consistency strength, inconsistency sensitivity, and temporal neighborhood radius.

9. A method for indoor personnel video behavior process recognition, applied to the system of any one of claims 1-8, characterized in that, The method includes: S1, perform single-frame detection of human body and key limbs in real-time video stream at high frame intervals, and perform intersection-union comparison correlation and merging on cross-frame detection boxes to obtain the target trajectory; S2 associates the head, hands, and feet as human body units, calculates behavior flags based on spatial overlap and motion speed rules, and adaptively adjusts the frame interval according to the state machine to output the triggered target area and corresponding frame sequence. S3. Based on the triggered target region, crop behavior segments from the original video, preprocess the segments in terms of length and size, and then input them into the MS-TCN++ model for temporal classification. Based on the loss function containing temporal consistency constraints, obtain the confidence of each category, and output the recognition result containing category, time interval and confidence according to the preset threshold.

10. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the method of claim 9.

Images