Taking behavior recognition method and device and medium

Abstract

The invention discloses a fighting behavior recognition method and device and a medium, and the method comprises the steps: obtaining a real-time video stream of a monitoring region, carrying out the multi-modal feature extraction of each image frame in the real-time video stream, and obtaining the multi-modal features of at least two target individuals in each image frame; on the basis of the multi-modal features of the current image frame, according to a preset single-frame judgment rule, generating single-frame fight state judgment for judging whether fight suspicion exists between the at least two target individuals or not; for a plurality of continuous image frame sequences in the real-time video stream, associating and tracking identification information of at least two target individuals to obtain a time sequence state and a space trajectory of the same target individual pair in the image frame sequences; and on the basis of the time sequence state and the space trajectory, according to a preset continuous frame verification rule, generating continuous fighting behavior judgment, and in response to the continuous fighting behavior judgment, triggering fighting behavior early warning and generating corresponding alarm evidence.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, device and medium for recognizing fighting behavior. Background Technology

[0002] Existing fight behavior recognition solutions mainly rely on computer vision technology to analyze surveillance video streams, but they generally suffer from insufficient recognition accuracy and poor scene adaptability, resulting in a high false alarm rate and making it difficult to meet the strict requirements for early warning reliability in practical applications.

[0003] At the feature utilization level, existing methods are mostly limited to simple combinations of single or a few features. Most schemes rely solely on the range of limb movements or contact distance reflected by key points on the human body for judgment. However, everyday human activities such as hugging, dancing, and carrying objects share similar limb forms with fighting, making misjudgment highly likely based on this single feature. Furthermore, these methods generally lack effective utilization of three-dimensional spatial information, especially the crucial social signal of facial orientation. Traditional facial orientation estimation based on two-dimensional images is easily affected by the shooting angle, failing to accurately quantify the relative orientation and antagonistic relationship between individuals, thus making it difficult to distinguish between face-to-face conflict and non-confrontational close contact. In addition, although facial expressions are an important basis for judging intent, existing technologies rarely integrate them deeply with limb and orientation features and design joint judgment criteria, resulting in insufficient system interpretation of behavioral intent.

[0004] At the temporal analysis level, existing methods utilize continuous frame information in a simplistic and one-sided manner. Many schemes trigger alarms by checking only a few frames or short time windows for suspected actions, lacking consideration of the persistence and dynamic evolution of behavior. This approach is overly sensitive to brief, non-malicious physical contact, generating numerous invalid alarms. A deeper problem lies in the failure of existing methods to incorporate individual spatiotemporal trajectory changes into a comprehensive verification system. Real fighting often involves entanglement, movement, and positional changes between the parties, while simple static contact or stationary disputes may fall under non-fighting conflicts. Ignoring the analysis of spatial characteristics such as the range and direction of movement trajectories prevents the system from effectively distinguishing between persistent attacks and brief contact—two fundamentally different behavioral patterns. Summary of the Invention

[0005] This application provides a method, device, and medium for identifying fighting behavior to solve the aforementioned technical problems.

[0006] On the one hand, embodiments of this application provide a method for identifying fighting behavior, including: The system acquires real-time video streams of the monitored area and extracts multimodal features from each image frame in the real-time video stream to obtain multimodal features of at least two target individuals in each image frame; the multimodal features include limb contact features, face orientation features, and facial expression features. Based on the multimodal features of the current image frame, and according to the preset single-frame determination rules, a single-frame fighting status determination is generated to determine whether there is a fighting suspicion between the at least two target individuals. For a series of consecutive image frames in the real-time video stream, the identification information of at least two target individuals is associated and tracked to obtain the temporal state and spatial trajectory of the same target individual in the image frame sequence; Based on the temporal state and the spatial trajectory, a continuous fighting behavior determination is generated according to the preset continuous frame verification rules. In response to the continuous fighting behavior determination, a fighting behavior warning is triggered, and corresponding alarm evidence is generated. The alarm evidence includes event time information, spatial location information, and associated video clips.

[0007] In one implementation of this application, multimodal feature extraction is performed on each image frame in the real-time video stream to obtain multimodal features of at least two target individuals in each image frame, specifically including: Human key point detection processing is performed on each image frame in the real-time video stream to obtain the two-dimensional coordinates of multiple preset human key points of at least two target individuals in the image frame. Based on the two-dimensional coordinates of the multiple preset human body key points, the distance between specific human body key points belonging to different target individuals is calculated, and the distance is compared with a preset distance threshold to generate limb contact features; the limb contact features are used to indicate whether there is limb contact between different target individuals. Perform 3D facial landmark recognition processing on each image frame to obtain the 3D coordinates of multiple preset 3D facial landmarks of the at least two target individuals. Based on the three-dimensional coordinates of the multiple preset three-dimensional facial key points, the facial orientation vector of each target individual is calculated, and the angle between the facial orientation vectors of different target individuals is calculated to generate three-dimensional facial orientation features. Perform facial expression recognition and classification processing on each image frame to obtain the facial expression classification of the at least two target individuals in the image frame and the confidence level corresponding to the facial expression classification; The facial expression classification is matched with a preset aggressive expression category, and the confidence level is compared with a second preset threshold to generate aggressive expression features.

[0008] In one implementation of this application, based on the multimodal features of the current image frame and according to a preset single-frame determination rule, a single-frame fighting state determination is generated to determine whether there is a fighting suspicion between the at least two target individuals. Specifically, this includes: Read the limb contact features in the current image frame to determine whether there is limb contact between the target individuals; Read the three-dimensional face orientation features for the target individual pair to determine whether the vector angle in the three-dimensional face orientation features falls within a preset angle range threshold. Read the aggressive facial expression features of the target individual pair to determine whether both parties in the target individual pair exhibit aggressive facial expressions; If there is contact between the target individuals, the included angle of the vectors falls within the preset included angle range threshold, and both parties in the target individuals exhibit aggressive expressions, a positive single-frame fighting state determination for the target individuals in the current image frame is generated; otherwise, a negative single-frame fighting state determination is generated.

[0009] In one implementation of this application, for a series of consecutive image frames in the real-time video stream, the identification information of at least two target individuals is associated and tracked to obtain the temporal state and spatial trajectory of the same target individual in the image frame sequence, specifically including: For a series of consecutive image frames in the real-time video stream, obtain the detection bounding boxes of all pedestrian targets in each frame of the image frame sequence; Based on the appearance and motion features of the detection box, data association is performed between multiple consecutive image frames, a unique identifier ID is assigned and maintained for each pedestrian target, and pedestrian trajectory tracking data is generated. From the pedestrian trajectory tracking data, the continuous position coordinates of target individuals with the same unique identifier ID in the image frame sequence are extracted to form a spatial trajectory; Based on the unique identifier ID, the single-frame fighting state determination of the corresponding target individual pair in the image frame sequence is aggregated to form the temporal state of the target individual pair.

[0010] In one implementation of this application, based on the temporal state and the spatial trajectory, a continuous fighting behavior determination is generated according to a preset continuous frame verification rule, specifically including: The temporal state is analyzed, the frequency of occurrence of affirmative single-frame fighting state judgments in a continuous image frame sequence of a preset length is counted, and the proportion of affirmative single-frame fighting state judgments to the total number of frames is calculated. The ratio is compared with a third preset threshold. If the ratio is greater than or equal to the third preset threshold, a high-frequency coincidence determination is generated. Analyze the spatial trajectory and calculate the positional change of each individual in the target individual pair in the continuous image frame sequence; The position change of each individual is compared with the fourth preset threshold. If the position change of both individuals is greater than or equal to the fourth preset threshold, a motion determination is generated. When the high-frequency conformity determination and the accompanying movement determination are obtained simultaneously, a positive continuous fighting behavior determination is generated.

[0011] In one implementation of this application, in response to the determination of continuous fighting behavior, a fighting behavior warning is triggered, and corresponding alarm evidence is generated, specifically including: In response to the affirmative determination of a continuous fighting behavior, the alarm interface is invoked to send an alarm command containing a unique event identifier to the monitoring platform, and the time information of generating the affirmative determination of a continuous fighting behavior is recorded as event time information; Based on the position coordinates of the target individual in the image frame corresponding to the determination of the positive and continuous fighting behavior, the center point coordinates or region coordinates of the target individual are calculated as spatial position information. Based on the event time information, a video stream containing a first preset duration before the current time and a second preset duration after the current time is extracted from the cache of the real-time video stream, and the extracted video stream is encapsulated into an associated video segment; The event time information, the spatial location information, and the associated video clips are bound and stored to form structured alarm evidence.

[0012] In one implementation of this application, human key point detection processing is performed on each image frame in the real-time video stream to obtain the two-dimensional coordinates of multiple preset human key points of at least two target individuals in the image frame, specifically including: A human pose estimation model based on a deep convolutional neural network is used to perform forward inference calculations on image frames in the real-time video stream, and multi-level visual feature maps of the image frames are extracted through the feature extraction network in the human pose estimation model. Based on the multi-level visual feature map, the key point localization head in the human pose estimation model is used to regress and calculate the two-dimensional coordinates of multiple preset human key points in the image frame; the preset human key points include at least the joints of the wrist, elbow, shoulder and knee.

[0013] In one implementation of this application, three-dimensional facial landmark recognition processing is performed on each image frame to obtain the three-dimensional coordinates of multiple preset three-dimensional facial landmarks of the at least two target individuals, specifically including: A 3D face reconstruction algorithm based on monocular images is used to process the face regions detected in the image frames of the real-time video stream, and the parameterized 3D face model is fitted to the 2D image of the face region through the 3D face reconstruction algorithm. By optimizing the shape parameters, pose parameters, and expression parameters of the parametric 3D face model, the projection of the parametric 3D face model onto the 2D image plane is aligned with the 2D image of the face region. On the parameterized 3D face model, multiple preset 3D face key points are located, and the 3D coordinates of the multiple preset 3D face key points in 3D space are output.

[0014] On the other hand, this application embodiment also provides a fighting behavior recognition device, the device comprising: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform a fighting behavior recognition method as described above.

[0015] On the other hand, this application embodiment also provides a non-volatile computer storage medium storing computer-executable instructions, which, when executed, implement a fighting behavior recognition method as described above.

[0016] This application provides a method, device, and medium for identifying fighting behavior, which has at least the following beneficial effects: By deeply integrating physical contact features, 3D facial orientation features, and aggressive expression features, a multi-dimensional evidence chain is constructed for single-frame judgment. This overcomes the inherent flaw of relying on single physical action features, which can easily misjudge friendly or neutral behaviors such as hugging or dancing as fighting. By simultaneously requiring the fulfillment of three conditions—physical contact, opposing orientation, and aggressive expression—the system establishes a higher and more comprehensive standard for judging suspected fighting within a single frame. By introducing a spatiotemporal trajectory verification mechanism based on continuous frame sequences, the system statistically analyzes the frequency with which the same target individual meets the single-frame fighting state in multiple consecutive frames and analyzes the accompanying trajectory movement range. This allows the system to effectively distinguish between genuine, continuous fighting behavior and brief, non-malicious physical contact. From feature extraction and single-frame judgment to trajectory tracking, continuous verification, and finally alarm evidence generation, a complete, closed-loop, and highly automated processing flow is formed. This not only improves the logical rigor and robustness of the entire system, enabling it to adapt to complex and ever-changing environmental interference and target occlusion in monitoring scenarios, but also achieves fully automated processing from raw video streams to structured alarm evidence, greatly improving the intelligence level and response efficiency of public safety monitoring. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a method for recognizing fighting behavior provided in an embodiment of this application; Figure 2 This is a schematic diagram of the internal structure of a fighting behavior recognition device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a method for recognizing fighting behavior provided in an embodiment of this application.

[0021] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0022] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server. This application does not make any specific limitations on this.

[0023] like Figure 1 As shown in the embodiment of this application, a method for recognizing fighting behavior includes: Step 101: Obtain the real-time video stream of the monitored area, and extract multimodal features from each image frame in the real-time video stream to obtain the multimodal features of at least two target individuals in each image frame.

[0024] It should be noted that the multimodal features in the embodiments of this application include limb contact features, face orientation features, and facial expression features.

[0025] Multimodal feature extraction is designed as a parallel or efficient pipelined process. It can be understood that "multimodal" here specifically refers to parsing and extracting multiple feature representations reflecting different physical and semantic levels of behavior from a single physical signal source—visual. These include physical contact features representing physical interaction, 3D facial orientation features representing spatial confrontation, and aggressive facial expression features representing emotional intent. For example, for an input RGB image frame, the system may simultaneously or sequentially activate three relatively independent but shared feature extraction submodules that perform some preprocessing steps (such as face detection).

[0026] Specifically, the system needs to perform human keypoint detection processing on image frames. This is typically accomplished using a human pose estimation model based on a deep convolutional neural network. It's worth noting that human pose estimation is a mature task in computer vision, aiming to locate the positions of major joints of the human body from an image. In this implementation, the model's workflow (e.g., using architectures like HRNet or OpenPose) consists of two main stages. The first stage is feature extraction: the model uses its deep convolutional neural network backbone to perform forward inference calculations on the entire input image, gradually abstracting and fusing visual information at different scales, ultimately outputting a set of semantically rich, multi-level visual feature maps. These feature maps retain rich information from low-level edges to high-level human body part structures. The second stage is keypoint localization: based on these feature maps, the model's keypoint localization head (usually a convolutional layer or a small network) performs heatmap regression or direct coordinate regression on each preset human keypoint (such as joints at the wrist, elbow, shoulder, and knee), thereby calculating the two-dimensional coordinates of multiple preset human keypoints in the image frame.

[0027] After obtaining the 2D keypoint coordinates of at least two target individuals, the system calculates the distance between specific human keypoints belonging to different target individuals based on the 2D coordinates of multiple preset human keypoints. For example, it calculates the Euclidean pixel distance between the right wrist joint of individual A and the center of the torso region of individual B (which can be defined by its hip and shoulder keypoints). Then, the distance is compared with a preset distance threshold. This threshold can be converted into an approximate physical distance (e.g., 0.5 meters) based on camera calibration information to adapt to different scenarios. The comparison result, i.e., whether there is at least one pair of keypoints across individuals with a distance less than the threshold, is encoded as a limb contact feature, which is Boolean or confidence data used to indicate whether close limb interaction exists.

[0028] Simultaneously, the system performs 3D facial landmark recognition processing, aiming to obtain the precise geometry of the face in 3D space from the 2D image. Specifically, the system first uses a lightweight face detector to locate the region of each face in the image. Then, for each detected face region, a 3D face reconstruction algorithm based on monocular images is used to process the detected face regions in the image frames of the real-time video stream, and the 3D face reconstruction algorithm is used to fit a parameterized 3D face model to the 2D image of the face region. It should be noted that a parameterized 3D face model (such as 3DMM) is a mathematical model that represents any face as an average face plus a series of shape basis and expression basis linear combinations. The fitting process is to optimize the shape parameters, pose parameters, and expression parameters of the parameterized 3D face model so that the projection of the parameterized 3D face model onto the 2D image plane is aligned with the 2D image of the face region. This can be understood as a numerical optimization process.

[0029] The algorithm iteratively adjusts shape parameters (controlling face shape), pose parameters (controlling 3D rotation and translation), and expression parameters (controlling facial muscle movements) to minimize the differences between the contours and feature point positions of the 3D model's 2D projection onto the virtual camera and the input real face image. Once the optimization converges and alignment is complete, a 3D digital model that highly matches the current face appearance and pose is obtained.

[0030] Next, multiple pre-defined 3D facial key points are located on the parametric 3D face model, and their 3D coordinates in 3D space are output. Key points such as the corners of the eyes, the tip of the nose, and the corners of the mouth have predefined 3D positions on the model, and their (x,y,z) coordinates can be directly read. Based on these 3D coordinates, the system can calculate the face orientation vector for each target individual. A common method is to fit a plane representing the front of the face using these key points; the normal vector of this plane, after being appropriately oriented (e.g., pointing forward), becomes the face orientation vector. Finally, the angle between the face orientation vectors of different target individuals is calculated. This angle α, calculated in 3D space, directly reflects whether the two faces are facing each other, back to back, or at a certain angle, thus generating accurate 3D face orientation features.

[0031] In this embodiment, the expression recognition processing typically shares the same face detection result with the 3D face reconstruction to improve efficiency. The system performs expression recognition and classification processing on each image frame, usually through a dedicated expression classification model trained on a convolutional neural network. This model receives a cropped and aligned face region image as input, performs forward propagation, and outputs a probability distribution vector representing the confidence level of the face belonging to various preset expression categories, such as anger, disgust, fear, happiness, sadness, surprise, and neutral. The system selects the category with the highest confidence level as the facial expression classification and records its corresponding confidence level.

[0032] The system then matches facial expression categories with preset aggressive expression categories and compares the confidence level with a second preset threshold. For example, a set of aggressive expression categories can be predefined, such as categories including anger, roaring, and ferocity, or anger and disgust can be used as proxies. If the identified category belongs to this set and its confidence level is higher than the second preset threshold—a probability threshold used to ensure recognition reliability, such as 0.5—then the individual is considered to have exhibited an aggressive expression in the current frame. For at least two target individuals, the system needs to perform this judgment separately and combine the judgment results from both to generate the final aggressive expression feature. This feature indicates the composite condition of whether both individuals exhibit a high-confidence aggressive expression.

[0033] Step 102: Based on the multimodal features of the current image frame, generate a single-frame fighting status determination for whether there is a fighting suspicion between at least two target individuals according to the preset single-frame determination rules.

[0034] In this embodiment, the process of generating a single-frame fighting state determination is essentially an execution of multi-dimensional evidence fusion and joint reasoning based on strict rules. It is understood that only when strong supporting evidence is simultaneously collected from the three most distinguishing dimensions of fighting behavior—physical interaction, spatial confrontation, and emotional expression—is it considered that there is a clear suspicion of fighting between the target individuals at the instant of the current image frame. For example, this process is specifically implemented as a sequential conditional check and logical AND operation.

[0035] Specifically, the first step in this process is to read the limb contact features in the current image frame to determine whether there is limb contact between the target individuals. It's important to note that "reading" here refers to a data processing action; the system retrieves limb contact features generated specifically for the current image frame and the target individuals (e.g., two people identified by the tracker as ID_A and ID_B) from storage or a data stream. The limb contact features themselves are conclusive data from the underlying calculations, derived from the processing of human keypoint detection results. This involves calculating the distance between preset keypoints belonging to different individuals, such as the wrist of ID_A and the torso of ID_B, and comparing this distance with a preset distance threshold to generate a binary or confidence-based judgment. The judgment operation confirms whether this feature data indicates the existence of contact. For example, in a street scene, a handshake or a light tap on the shoulder might trigger a contact judgment due to the proximity of keypoints; however, even if two people are standing side-by-side very close, this judgment will not be triggered if there is no preset close relationship between limb keypoints, thus ensuring the necessary physical interaction basis for a fight.

[0036] Next, the system reads the 3D facial orientation features of the target individuals to determine whether the angle between the vectors in the 3D facial orientation features falls within a preset angle range threshold, thus verifying evidence of spatial relationship. The 3D facial orientation features read by the system include the spatial angle α between the facial orientation vectors V_A and V_B obtained based on 3D facial reconstruction. The preset angle range threshold is an angle interval set by prior knowledge, with a lower limit typically significantly greater than 90 degrees and an upper limit of 180 degrees (i.e., completely opposite). This interval aims to define what relative orientations constitute confrontation or opposition. Determining whether the angle α falls within this interval is equivalent to determining whether the two individuals are in a basically face-to-face spatial situation. For example, in a real conflict, both parties usually face each other directly for attack or defense (α is close to 180 degrees); while in friendly conversation or cooperation, although the two parties may face each other, the angle range is more flexible and may not completely fall within the preset strict confrontation range; as for back-to-back or standing in the same direction, the angle is very small and is naturally excluded, thus effectively distinguishing between spatial orientations with antagonistic intentions and close-range orientations without antagonistic intentions.

[0037] Subsequently, the aggressive facial expression features of the target individual pair are read to determine whether both parties in the pair exhibit aggressive expressions, thus focusing on evidence at the level of emotion and intention. The aggressive facial expression features read by the system are generated based on the results of facial expression recognition and classification processing. They include information such as the facial expression category of each target individual (e.g., anger, seriousness, pain) and whether the confidence level of that category exceeds a preset threshold. First, it is confirmed whether the facial expression categories of ID_A and ID_B both belong to the preset aggressive facial expression category set; second, it is confirmed whether the recognition confidence levels of both meet the credibility standard. It should be noted that the judgment is twofold, and the judgment requires both parties to exhibit the specified facial expression category. This excludes scenarios where one party is angry while the other is fearful or calm (possibly due to one party's provocation or the other's unilateral fright), and also excludes scenarios where both parties simultaneously exhibit painful expressions due to non-conflict reasons (e.g., jointly viewing painful images), as these scenarios are not valid in terms of physical contact and opposing orientation. Only when both parties in the conflict exhibit aggressive or intense negative emotions in their emotional signals does it constitute emotional evidence supporting a fight determination.

[0038] Finally, the three independent judgment results are fed into the final decision logic. If there is contact between the target individuals, the angle between their vectors falls within a preset threshold range, and both parties exhibit aggressive expressions, a positive single-frame fighting state judgment is generated for the target individuals in the current image frame; otherwise, a negative single-frame fighting state judgment is generated. This is a strict logical AND gate. Only when all three conditions—physical contact (condition one), spatial opposition (condition two), and aggressive emotions from both parties (condition three)—are simultaneously met, does the system comprehensively determine that a complete chain of evidence constituting a suspected fight has been captured in the static image of the current frame. This results in a positive single-frame fighting state judgment for the target individuals, which is then stored in association with the frame number and individual ID.

[0039] If any one of the conditions is not met—for example, if contact is detected and both parties are angry but they are actually back to back (condition two is not met), or if contact is detected and they are face to face but one of them has a calm expression (condition three is not met)—the system will generate a negative single-frame fighting state judgment. This multi-dimensional and strict joint judgment mechanism greatly reduces the false alarm space, allowing similar but non-fighting behaviors such as dancing, hugging, carrying heavy objects, and heated conversations to be effectively filtered out because they are unlikely to meet all the stringent conditions simultaneously. This provides extremely reliable and clean initial state input for subsequent time-series-based analysis of the dynamic persistence of behavior.

[0040] Step 103: For multiple consecutive image frame sequences in the real-time video stream, associate and track the identification information of at least two target individuals to obtain the temporal state and spatial trajectory of the same target individual in the image frame sequence.

[0041] In this embodiment, the process of acquiring temporal state and spatial trajectory is essentially about maintaining identity consistency across frames and fusing multi-dimensional data. It is understandable that without stable and accurate individual tracking, feature extraction and state determination for a single frame would be isolated and fragmented, failing to reveal the temporal evolution of behavior and its spatial movement patterns. For example, this process is specifically implemented as a coherent data processing flow encompassing detection, association, extraction, and aggregation.

[0042] Specifically, for a sequence of multiple consecutive image frames in a real-time video stream, the system obtains detection boxes for all pedestrian targets in each frame of the sequence. It's important to note that the image frame sequence here refers to a temporally consecutive set of image frames extracted from the real-time video stream, and the number of frames is a configurable parameter. For each frame in the sequence, the system needs to perform pedestrian detection, typically using a pre-trained general-purpose object detection model based on deep learning (such as the YOLO series, Faster R-CNN, etc.). This model performs forward inference on the input single-frame image and outputs the coordinates of all regions of interest (ROIs) identified as pedestrians in the image, usually represented by bounding boxes, i.e., detection boxes. This step provides the most basic object unit for all subsequent tracking and analysis, ensuring that potential conflict participants in each frame can be initially located. For example, in surveillance footage, regardless of whether a pedestrian is stationary or moving, this step can define their position in every frame they appear in.

[0043] Next, based on the appearance and motion features of the detection boxes, data is correlated across multiple consecutive image frames to assign and maintain a unique identifier ID for each pedestrian target, generating pedestrian trajectory tracking data. This step is the core of cross-frame identity correlation and is typically performed by multi-target tracking algorithms. Understandably, the challenge lies in correctly determining which detection box in the previous frame corresponds to which detection box in the current frame, and which corresponds to the same physical pedestrian. To address this, the algorithm utilizes both types of features for data correlation.

[0044] First, appearance features refer to deep feature vectors extracted from image regions within the detection box using deep learning models (such as ReID networks) that characterize an individual's visual appearance (e.g., clothing color, texture, body shape). The appearance features of the same individual in different frames should exhibit high similarity. Second, motion features, based on the target's motion patterns across consecutive frames, estimate the target's expected position in the current frame using prediction models such as Kalman filtering. Matching the actual detection box in the current frame with the predicted position based on the trajectory of the previous frame allows for the association of smoothly moving targets. By comprehensively comparing appearance similarity and motion consistency, the algorithm addresses potential issues such as intersections and occlusions, successfully assigning a unique identifier ID (e.g., TrackID_1, TrackID_2…) to each pedestrian appearing for the first time in the entire image frame sequence, and correctly associating this ID with a new detection box in each subsequent visible frame. The resulting pedestrian trajectory tracking data not only contains the detection box coordinates corresponding to each ID in each frame but also implicitly includes the complete link maintained across frames.

[0045] Subsequently, based on the stable trajectory tracking data, the system executes two data extraction and construction tasks in parallel. First, it constructs a spatial trajectory by extracting the continuous position coordinates of target individuals with the same unique identifier (ID) within an image frame sequence from the pedestrian trajectory tracking data. For example, for a continuously tracked target individual (such as TrackID_A), the system extracts the center point coordinates (x_i, y_i) of the detection box associated with that ID in each frame of the sequence, in chronological order, from its trajectory tracking data. This series of coordinate points, ordered by timestamp, constitutes the individual's motion path on the two-dimensional image plane during that time period, i.e., its spatial trajectory. This trajectory clearly reflects the individual's speed, direction changes, and activity range. For a pair of interacting individuals (such as TrackID_A and TrackID_B), their respective spatial trajectories jointly describe the dynamic changes in their relative positions, such as whether they are moving closer, intertwined and rotating, or remaining stationary.

[0046] Secondly, the system constructs a temporal state. Based on unique identifiers (IDs), it aggregates single-frame fighting state judgments for corresponding target individual pairs in the image frame sequence to constitute the temporal state of the target individual pairs. The system has already generated single-frame fighting state judgments for all possible target individual pairs for each frame. Now, using the unique identifiers (IDs) determined and maintained by multi-target tracking, the system can accurately backtrack and collect these judgments. For example, for the individual pair TrackID_A and TrackID_B, the system traverses each frame in the image frame sequence, checking whether a judgment result (positive or negative) targeting these two IDs has been generated in that frame. These judgment results extracted in chronological order (e.g., [Frame 1: Negative, Frame 2: Positive, Frame 3: Positive, Frame 4: Negative, ...]) are aggregated to form a Boolean sequence describing whether a fighting suspicion appears on the timeline, i.e., the temporal state of the target individual pair. This sequence visually demonstrates the start and end, duration, and intensity of the conflict suspicion.

[0047] Understandably, spatial trajectory and temporal state together constitute a complete spatiotemporal profile of a specific individual's behavioral patterns within a specific time period. The former describes behavior from the perspective of physical displacement, while the latter records behavior from the perspective of event occurrence. The combination of the two provides indispensable, multi-dimensional, and continuous evidence for the next step of determining whether the behavior is a continuous fight or a brief contact.

[0048] Step 104: Based on the temporal state and spatial trajectory, and according to the preset continuous frame verification rules, generate a continuous fighting behavior judgment, and in response to the continuous fighting behavior judgment, trigger a fighting behavior warning and generate corresponding alarm evidence; the alarm evidence includes event time information, spatial location information and related video clips.

[0049] In this embodiment, the process of generating a determination of persistent fighting behavior is a process of applying continuous frame verification rules to perform in-depth analysis of temporal state and spatial trajectory data. It is understood that the purpose of this rule is to strictly distinguish between real fighting behavior with persistent harm and brief, non-malicious physical contact or misjudgment. For example, this verification rule includes two independent verification dimensions that must be satisfied simultaneously: the temporal persistence density of the behavior, and the spatial significance of the accompanying displacement of the behavior.

[0050] Specifically, the first dimension of verification involves analyzing the temporal state, counting the frequency of affirmative single-frame fighting state judgments within a pre-defined sequence of consecutive image frames, and calculating the proportion of affirmative single-frame fighting state judgments to the total number of frames. It should be noted that the pre-defined sequence of consecutive image frames is the video segment used to construct the spatiotemporal data. The system iterates through the corresponding temporal state sequence (an array of affirmative and negative judgments) and counts the number of affirmative judgments. Then, this count is divided by the total number of frames in the sequence to obtain a proportion (e.g., 0.7 represents 70% of the frames conforming to a single-frame fighting state). This proportion quantifies the density and continuity of suspected fighting within that time period.

[0051] Next, the system compares the proportion with a third preset threshold. If the proportion is greater than or equal to the third preset threshold, a high-frequency conformity judgment is generated. This third preset threshold is a relatively high percentage threshold, such as 60%. The logic behind this comparison is that real, sustained fighting behavior usually doesn't flash by, but rather repeatedly and densely exhibits a combination of aggressive contact, opposing orientation, and angry expressions over a period of time. Only when the proportion of the sequential state exceeds this high threshold, indicating a highly persistent and consistent suspected state, does the system generate a high-frequency conformity judgment. This effectively filters out isolated signals generated by momentary misjudgments or accidental intense contact (such as immediate separation after a rapid collision).

[0052] Simultaneously, the system performs the second dimension of verification in parallel, analyzing the spatial trajectory and calculating the positional change of each individual in the target individual pair within a continuous sequence of image frames. It can be understood that the spatial trajectory here refers to the continuous sequence of positional coordinates of each member in the target individual pair within the image sequence. Calculating the positional change is to quantify the overall displacement of the individual within that time period. A common calculation method is to take the diagonal length of the outer envelope rectangle of all coordinate points in the individual's trajectory sequence, or to calculate the Euclidean distance between its positional coordinates in the first and last frames. This change reflects whether the individual has remained essentially stationary or has undergone significant movement.

[0053] The system then compares the positional change of each individual with a fourth preset threshold. If the positional change of both individuals is greater than or equal to the fourth preset threshold, a "companion movement judgment" is generated. This fourth preset threshold is a distance threshold representing significant displacement. This comparison requires both parties in the conflict to have undergone significant displacement. This design is based on the fact that many real-life fights involve pushing, pulling, dodging, chasing, or entanglement, resulting in significant changes in the positions of both parties. While static arguments, pushing, or stationary physical conflicts may meet the high-frequency characteristic in terms of time, they lack this accompanying, significant entanglement-like movement. Therefore, the system only generates a "companion movement judgment" when the displacement of both parties exceeds the threshold, thus distinguishing between "continuous fighting" and "continuous but essentially static conflict."

[0054] Ultimately, the system executes a final logical decision. When both a high-frequency conformity judgment and an accompanying movement judgment are obtained simultaneously, a positive determination of continuous fighting behavior is generated. This can be understood as a logical AND operation. This means that for an action to be ultimately classified as "continuous fighting" requiring an alarm, it must pass the aforementioned rigorous verification in both the time and spatial dimensions. Meeting only the time continuity requirement (e.g., two people arguing face-to-face for a long time with angry expressions but almost no movement) or only the spatial movement requirement (e.g., two people quickly brushing past each other in a crowd with one instance of physical contact) is insufficient to trigger the final determination. This dual verification mechanism is one of the core guarantees for the low false alarm rate achieved by this patent.

[0055] In this embodiment, once a positive determination of a persistent fighting behavior is generated, the system will immediately and automatically trigger a series of preset response actions to complete the early warning and evidence fixation. Specifically, the first step of the response is to call the alarm interface to send an alarm command containing a unique event identifier to the monitoring platform in response to the positive determination of a persistent fighting behavior, and record the time information of generating the positive determination of a persistent fighting behavior as event time information. It should be noted that calling the alarm interface is an inter-system communication action, meaning that this identification system sends a structured alarm message to the upper-level monitoring center or security platform through a predefined API or protocol. This message contains at least one unique event identifier (such as a UUID) for uniquely tracking this alarm event on the platform side. At the same time, the system will accurately record the system timestamp that generated the final determination as event time information, which identifies the time when the algorithm confirms that the fighting behavior occurred.

[0056] Next, the system needs to determine the location of the incident. Based on the position coordinates of the target individuals in the image frame corresponding to the determination of a persistent fighting behavior, the system calculates the center point coordinates or region coordinates of the target individuals as spatial location information. For example, the system retrieves the image frame that triggered the final determination (usually the last frame of the analysis time window or the frame with the most typical state) and reads the bounding box coordinates of the pair of target individuals involved (such as TrackID_A and TrackID_B) in that frame. Then, it can calculate the average coordinates of the center points of these two bounding boxes as the center point coordinates of the incident; or, it can calculate the coordinates of the smallest rectangular area that simultaneously covers both bounding boxes as the region coordinates. This spatial location information allows security personnel to quickly locate the incident on the surveillance screen or map.

[0057] Meanwhile, to provide the most direct behavioral evidence, the system uses event time information as a benchmark to extract a video stream from the real-time video stream cache, containing a first preset duration before the current moment and a second preset duration after the current moment, and then encapsulates the extracted video stream into a related video segment. Understandably, to fully record the cause, process, and brief aftermath of an event, the system automatically extracts a video segment from the continuously cached historical video data. For example, it extracts video from a period before the judgment moment (e.g., 10 seconds, to observe how the conflict occurred) and a period after (e.g., 20 seconds, to observe how the conflict developed or ended), splicing or encapsulating them into a single video file or data stream—the related video segment. This video is the most direct evidence for reviewing police reports and understanding the situation at the scene.

[0058] Finally, the system binds and stores the event's time information, spatial location information, and associated video clips, forming structured alarm evidence. It's important to note that binding means linking these discrete information items (time, location, video) through a unique event identifier, packaging them into a complete data packet or database record. "Structured" refers to the data packet having a predefined, machine-readable format (such as JSON, XML, or a specific binary format). This structured alarm evidence can then be stored in a local or remote database and sent to the monitoring platform along with or asynchronously with the alarm command, allowing security personnel to review, verify, and archive it, thus completing a fully automated closed loop from intelligent sensing to the output of a complete evidence chain.

[0059] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a fighting behavior recognition device, the structure of which is as follows: Figure 2 As shown.

[0060] Figure 2 This is a schematic diagram of the internal structure of a fighting behavior recognition device provided in an embodiment of this application. Figure 2As shown, the device includes: At least one processor; And, a memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable at least one processor to: The system acquires real-time video streams of the monitored area and extracts multimodal features from each image frame in the real-time video stream to obtain multimodal features of at least two target individuals in each image frame. The multimodal features include limb contact features, face orientation features, and facial expression features. Based on the multimodal features of the current image frame, and according to the preset single-frame judgment rules, a single-frame fighting status judgment is generated to determine whether there is a fighting suspicion between at least two target individuals. For a sequence of multiple consecutive image frames in a real-time video stream, associate and track the identification information of at least two target individuals to obtain the temporal state and spatial trajectory of the same target individual in the image frame sequence; Based on temporal state and spatial trajectory, and according to preset continuous frame verification rules, a continuous fighting behavior judgment is generated, and in response to the continuous fighting behavior judgment, a fighting behavior warning is triggered, and corresponding alarm evidence is generated; the alarm evidence includes event time information, spatial location information and related video clips.

[0061] This application also provides a non-volatile computer storage medium storing computer-executable instructions, which, when executed, can: The system acquires real-time video streams of the monitored area and extracts multimodal features from each image frame in the real-time video stream to obtain multimodal features of at least two target individuals in each image frame. The multimodal features include limb contact features, face orientation features, and facial expression features. Based on the multimodal features of the current image frame, and according to the preset single-frame judgment rules, a single-frame fighting status judgment is generated to determine whether there is a fighting suspicion between at least two target individuals. For a sequence of multiple consecutive image frames in a real-time video stream, associate and track the identification information of at least two target individuals to obtain the temporal state and spatial trajectory of the same target individual in the image frame sequence; Based on temporal state and spatial trajectory, and according to preset continuous frame verification rules, a continuous fighting behavior judgment is generated, and in response to the continuous fighting behavior judgment, a fighting behavior warning is triggered, and corresponding alarm evidence is generated; the alarm evidence includes event time information, spatial location information and related video clips.

[0062] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0063] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0064] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0069] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0070] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0071] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0072] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for recognizing fighting behavior, characterized in that, The method includes: The system acquires real-time video streams of the monitored area and extracts multimodal features from each image frame in the real-time video stream to obtain multimodal features of at least two target individuals in each image frame; the multimodal features include limb contact features, face orientation features, and facial expression features. Based on the multimodal features of the current image frame, and according to the preset single-frame determination rules, a single-frame fighting status determination is generated to determine whether there is a fighting suspicion between the at least two target individuals. For a series of consecutive image frames in the real-time video stream, the identification information of at least two target individuals is associated and tracked to obtain the temporal state and spatial trajectory of the same target individual in the image frame sequence; Based on the temporal state and the spatial trajectory, a continuous fighting behavior determination is generated according to the preset continuous frame verification rules. In response to the continuous fighting behavior determination, a fighting behavior warning is triggered, and corresponding alarm evidence is generated. The alarm evidence includes event time information, spatial location information, and associated video clips.

2. The method for identifying fighting behavior according to claim 1, characterized in that, Multimodal feature extraction is performed on each image frame in the real-time video stream to obtain multimodal features of at least two target individuals in each image frame, specifically including: Human key point detection processing is performed on each image frame in the real-time video stream to obtain the two-dimensional coordinates of multiple preset human key points of at least two target individuals in the image frame. Based on the two-dimensional coordinates of the multiple preset human body key points, the distance between specific human body key points belonging to different target individuals is calculated, and the distance is compared with a preset distance threshold to generate limb contact features; the limb contact features are used to indicate whether there is limb contact between different target individuals. Perform 3D facial landmark recognition processing on each image frame to obtain the 3D coordinates of multiple preset 3D facial landmarks of the at least two target individuals. Based on the three-dimensional coordinates of the multiple preset three-dimensional facial key points, the facial orientation vector of each target individual is calculated, and the angle between the facial orientation vectors of different target individuals is calculated to generate three-dimensional facial orientation features. Perform facial expression recognition and classification processing on each image frame to obtain the facial expression classification of the at least two target individuals in the image frame and the confidence level corresponding to the facial expression classification; The facial expression classification is matched with a preset aggressive expression category, and the confidence level is compared with a second preset threshold to generate aggressive expression features.

3. The method for identifying fighting behavior according to claim 1, characterized in that, Based on the multimodal features of the current image frame, and according to a preset single-frame determination rule, a single-frame fighting state determination is generated to determine whether there is a fighting suspicion between the at least two target individuals, specifically including: Read the limb contact features in the current image frame to determine whether there is limb contact between the target individuals; Read the three-dimensional face orientation features for the target individual pair to determine whether the vector angle in the three-dimensional face orientation features falls within a preset angle range threshold. Read the aggressive facial expression features of the target individual pair to determine whether both parties in the target individual pair exhibit aggressive facial expressions; If there is contact between the target individuals, the included angle of the vectors falls within the preset included angle range threshold, and both parties in the target individuals exhibit aggressive expressions, a positive single-frame fighting state determination for the target individuals in the current image frame is generated; otherwise, a negative single-frame fighting state determination is generated.

4. The method for identifying fighting behavior according to claim 1, characterized in that, For a series of consecutive image frames in the real-time video stream, the identification information of at least two target individuals is associated and tracked to obtain the temporal state and spatial trajectory of the same target individual in the image frame sequence, specifically including: For a series of consecutive image frames in the real-time video stream, obtain the detection bounding boxes of all pedestrian targets in each frame of the image frame sequence; Based on the appearance and motion features of the detection box, data association is performed between multiple consecutive image frames, a unique identifier ID is assigned and maintained for each pedestrian target, and pedestrian trajectory tracking data is generated. From the pedestrian trajectory tracking data, the continuous position coordinates of target individuals with the same unique identifier ID in the image frame sequence are extracted to form a spatial trajectory; Based on the unique identifier ID, the single-frame fighting state determination of the corresponding target individual pair in the image frame sequence is aggregated to form the temporal state of the target individual pair.

5. The method for identifying fighting behavior according to claim 4, characterized in that, Based on the temporal state and the spatial trajectory, and according to preset continuous frame verification rules, a continuous fighting behavior determination is generated, specifically including: The temporal state is analyzed, the frequency of occurrence of affirmative single-frame fighting state judgments in a continuous image frame sequence of a preset length is counted, and the proportion of affirmative single-frame fighting state judgments to the total number of frames is calculated. The ratio is compared with a third preset threshold. If the ratio is greater than or equal to the third preset threshold, a high-frequency coincidence determination is generated. Analyze the spatial trajectory and calculate the positional change of each individual in the target individual pair in the continuous image frame sequence; The position change of each individual is compared with the fourth preset threshold. If the position change of both individuals is greater than or equal to the fourth preset threshold, a motion determination is generated. When the high-frequency conformity determination and the accompanying movement determination are obtained simultaneously, a positive continuous fighting behavior determination is generated.

6. The method for identifying fighting behavior according to claim 1, characterized in that, In response to the determination of continuous fighting behavior, a fighting behavior warning is triggered, and corresponding alarm evidence is generated, specifically including: In response to the affirmative determination of a continuous fighting behavior, the alarm interface is invoked to send an alarm command containing a unique event identifier to the monitoring platform, and the time information of generating the affirmative determination of a continuous fighting behavior is recorded as event time information; Based on the position coordinates of the target individual in the image frame corresponding to the determination of the positive and continuous fighting behavior, the center point coordinates or region coordinates of the target individual are calculated as spatial position information. Based on the event time information, a video stream containing a first preset duration before the current time and a second preset duration after the current time is extracted from the cache of the real-time video stream, and the extracted video stream is encapsulated into an associated video segment; The event time information, the spatial location information, and the associated video clips are bound and stored to form structured alarm evidence.

7. The method for identifying fighting behavior according to claim 2, characterized in that, Human key point detection processing is performed on each image frame in the real-time video stream to obtain the two-dimensional coordinates of multiple preset human key points of at least two target individuals in the image frame, specifically including: A human pose estimation model based on a deep convolutional neural network is used to perform forward inference calculations on image frames in the real-time video stream, and multi-level visual feature maps of the image frames are extracted through the feature extraction network in the human pose estimation model. Based on the multi-level visual feature map, the key point localization head in the human pose estimation model is used to regress and calculate the two-dimensional coordinates of multiple preset human key points in the image frame; the preset human key points include at least the joints of the wrist, elbow, shoulder and knee.

8. The method for identifying fighting behavior according to claim 2, characterized in that, Perform 3D facial landmark recognition processing on each image frame to obtain the 3D coordinates of multiple preset 3D facial landmarks of the at least two target individuals, specifically including: A 3D face reconstruction algorithm based on monocular images is used to process the face regions detected in the image frames of the real-time video stream, and the parameterized 3D face model is fitted to the 2D image of the face region through the 3D face reconstruction algorithm. By optimizing the shape parameters, pose parameters, and expression parameters of the parametric 3D face model, the projection of the parametric 3D face model onto the 2D image plane is aligned with the 2D image of the face region. On the parameterized 3D face model, multiple preset 3D face key points are located, and the 3D coordinates of the multiple preset 3D face key points in 3D space are output.

9. A fighting behavior recognition device, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform a fighting behavior recognition method as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, a fighting behavior recognition method as described in any one of claims 1-8 is implemented.

Images