System for complex spatio-temporal multi-target behavior recognition and understanding oriented to surveillance videos

Abstract

The application relates to the technical field of video image recognition, and discloses a complex space-time multi-target behavior recognition and understanding system for monitoring video, which comprises the following modules: a video stream analysis module, which acquires frame sequences and determines feature aliasing area data; a feature information extraction module, which extracts human body key points, defines reference geometric nodes and constructs a space-time relation graph; a three-dimensional kinematics constraint module, which determines human body bone length and joint rotation limits as boundary parameters and uses a geometric constraint equation set to obtain the space geometry solution of a hidden point in a three-dimensional space; and a behavior semantic reasoning module, which outputs a behavior semantic label. The application solves the feature aliasing problem caused by target physical interaction by using three-dimensional kinematics rules, deduces a hidden position by using objective geometric constraints, maintains target identity sequence continuity, and improves the reliability of behavior understanding in a complex scene.

Description

Technical Field

[0001] This invention relates to a system for recognizing and understanding complex spatiotemporal multi-target behaviors in surveillance videos, belonging to the field of video image recognition technology. Background Technology

[0002] In the current field of surveillance video processing, multi-target behavior recognition typically employs a technical framework that combines target detection and feature tracking. This framework utilizes the appearance features of targets to construct similarity matching relationships over time to determine the behavioral semantics of the targets. This approach demonstrates good processing efficiency in scenarios where the target's motion trajectory is smooth, serving as a fundamental technology for current security monitoring systems. However, under stressful conditions such as high-density crowds or complex physical interactions, the continuity of visual features is compromised. When multiple targets physically overlap, resulting in impaired appearance features, the system, due to its implicit assumption of the integrity of the appearance space, is prone to pixel-level semantic pollution. Furthermore, because the system lacks the ability to perceive the structural attributes of physical entities, the recognition process generates identity switching and feature vector drift, leading to the failure of upper-level behavior understanding logic.

[0003] To mitigate recognition errors caused by occlusion, traditional improvement approaches typically involve adding hardware acquisition sources or using generative models to calculate predicted values ​​for incomplete pixels. However, adding hardware significantly increases system deployment costs and data processing latency. Furthermore, relying solely on probabilistic pixel prediction paths leads to erroneous semantic inferences when dealing with nonlinear or sudden actions due to a lack of objective physical constraints. Such solutions, which complete the visual pixel level rather than inferring from a physical dimension, struggle to address the topological breaks inherent in the human body as a physical assembly during interaction. Hardware-based visual path completion is limited, and the technological focus has shifted to utilizing algorithmic models to analyze the logic of damaged features. For example, Chinese invention patent application CN116403277A discloses an occluded human posture estimation corrector and correction method based on key point interconnection. It establishes an implicit interconnection mapping between key points and uses high-accuracy nodes to perform posterior correction of the heat map distribution of the occluded area. Although this alleviates the loss of pixel information, the underlying technology is anchored to probabilistic correlation in a two-dimensional plane and does not introduce the topological constraints of the three-dimensional physical entity of the human body. In complex multi-target interactions, visual appearance features overlap deeply. Relying solely on the mathematical correlation between heat maps is prone to producing false solutions due to the lack of spatial and physical consistency, causing irreversible drift in the behavioral semantic inference logic.

[0004] Therefore, how to use kinematic assembly constraint mechanisms to achieve deterministic repair of damaged spatiotemporal features and ensure the continuity of behavioral understanding logic in high-density physical interaction scenarios has become the technical problem to be solved by this invention. Summary of the Invention

[0005] To address the problems mentioned in the background art, the technical solution of the present invention is as follows: A system for complex spatiotemporal multi-target behavior recognition and understanding in surveillance video, the system comprising:

[0006] The video stream parsing module is used to acquire the sequence of monitoring video frames and determine the feature aliasing region data generated by multi-target interactions;

[0007] The feature information extraction module is used to extract human key points from the sequence of surveillance video frames, define human key points as reference geometric nodes, and construct a spatiotemporal relationship map based on the reference geometric nodes.

[0008] The 3D kinematic constraint module is used to determine the physical length of the human skeleton and the joint rotation limits as human kinematic boundary parameters in the 3D human motion space. The target position is calculated through the following processes: Step S101, associating the reference geometric nodes and feature aliasing region data with the 3D human motion space to determine the spatial topological reference; Step S102, constructing a set of geometric constraint equations to restrict the degrees of freedom of human motion based on the human kinematic boundary parameters; Step S103, solving the set of geometric constraint equations in the 3D human motion space to calculate the spatial geometric solution of the occluded key points in the 3D human motion space that conforms to the laws of human kinematics.

[0009] The behavioral semantic reasoning module is used to reconstruct the spatiotemporal behavioral sequence of the target based on the baseline geometric nodes, spatiotemporal relationship graph and spatial geometric solution, and output the corresponding behavioral semantic labels.

[0010] Preferably, the three-dimensional kinematic constraint module further includes a constraint parameter correction unit, which is used to adjust the parameters through the following processes: Step S201, obtain the depth estimation data of the target in the current video frame, and determine the Z-axis occlusion order between the reference geometric nodes based on the depth estimation data; Step S202, calculate the transient kinetic energy value of each reference geometric node based on the displacement vector between adjacent time frames; Step S203, compare the transient kinetic energy value with a preset kinetic energy threshold to determine the dynamic weighting coefficient used to adjust the joint rotation limit; Step S204, update the geometric constraint equation set using the dynamic weighting coefficient to change the search range of the spatial geometric solution.

[0011] Preferably, the system also includes a graph vector compensation module, which is used to superimpose the feature aliasing region data determined by the video stream parsing module with the trajectory edge in the spatiotemporal relationship graph. When target overlap is detected, the feature propagation gradient of adjacent reference geometric nodes is extracted, and the compensation feature vector of the occluded key point is calculated in reverse by combining the preset spatial coordinate mapping matrix.

[0012] Preferably, the system also includes a momentum verification module, which is used to retrieve the motion state of the reference geometric nodes stored in the previous 3 to 5 time frames, calculate the matching degree between the current spatial geometric solution candidate value and the historical momentum vector, and select the analytical solution with the highest matching degree as the compensation result of the current time series.

[0013] Preferably, the system also includes a scale adjustment module, which is used to calculate the pixel area occupied by the reference geometric node in the feature map, and adjust the sampling kernel size in the feature aggregation stage based on the ratio of the pixel area to the preset sampling area.

[0014] Preferably, the behavioral semantic reasoning module further includes an interaction modeling unit, which is used to determine the relative position potential energy field between multiple targets in the three-dimensional human motion space, and to determine the trigger probability of physical contact between targets based on the rate of change of the Euclidean distance between two adjacent reference geometric nodes.

[0015] Preferably, the reference geometric nodes include the main torso center node, the two shoulder nodes, the two elbow nodes, and the two knee nodes, and the human kinematic boundary parameters include the fixed physical length between the two shoulder nodes and the two elbow nodes.

[0016] Preferably, when the missing duration of the occluded key point exceeds 15 frames, the three-dimensional kinematic constraint module stops searching for pixel features and predicts the uniform linear trajectory of the target position in the three-dimensional human motion space based on the historical momentum vector.

[0017] Preferably, the system also includes an early warning output module, which is used to send an enhanced video stream containing spatial geometric solutions to an external monitoring terminal when the behavioral semantic label successfully matches the feature items in the preset abnormal behavior library.

[0018] Compared with the prior art, the beneficial effects of the present invention are:

[0019] 1. In complex spatiotemporal multi-target behavior recognition, a technical solution is provided to improve the recognition accuracy of complex interactive behaviors. By introducing physical constraints of geometric assembly dimensions during image sequence processing, the processing method that relies solely on pixel feature statistical matching is changed. When multiple recognition targets physically overlap, resulting in damage to appearance features, the system uses a preset human skeleton dynamics topology library, taking visible nodes as reference parts, and obtains the spatial coordinates of hidden nodes by solving a set of geometric equations including constant link length conditions and joint rotation limit conditions. This processing logic ensures that the recognition process is no longer limited by the integrity of appearance features, realizing deterministic deduction of the target topology under extreme occlusion conditions, and avoiding semantic recognition interruption caused by feature loss.

[0020] 2. Construct a feature map with self-healing capabilities to ensure the continuity of the target identity sequence. The spatial location vector separated by the video feature parsing unit is coupled with the trajectory edges in the spatiotemporal relationship map to form multi-level logical associations. In the stage where target interaction causes feature contamination, the behavioral semantic reasoning unit extracts the feature propagation gradient of adjacent nodes, combines it with the global spatial coordinate mapping matrix, calculates the compensation feature vector of the hidden node in reverse, and injects it into the multi-level spatiotemporal relationship map. This multi-mechanism collaborative working mode enables the system to maintain the integrity of the overall logic by using local effective information, reduces the probability of frequent identity switching, and improves the monitoring quality in densely populated scenes.

[0021] 3. By utilizing a kinematic momentum verification mechanism, the interference of transient noise on the judgment results is reduced. When the system performs screening in the geometric solution space, it retrieves the node motion states stored in multiple previous time slices, calculates the matching degree between the current candidate coordinates and the historical momentum vectors, and selects the analytical solution with the highest matching degree as the compensation result of the current time series. The system effectively eliminates illegal fluctuations caused by sudden changes in lighting, extreme artifacts, or similar clothing textures. This secondary verification process based on physical laws enhances the system's ability to defend against interference from complex environments, ensures that the output behavioral semantic labels conform to objective motion laws, and guarantees a high degree of certainty in the output of abnormal behavior warning commands. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the data interaction and processing of the complex spatiotemporal multi-target behavior recognition system of the present invention.

[0023] Figure 2 This is a diagram showing the core functional module architecture of the multi-target behavior recognition and understanding system of the present invention.

[0024] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] A system for complex spatiotemporal multi-target behavior recognition and understanding in surveillance video, the system comprising:

[0027] The video stream parsing module is used to acquire the sequence of monitoring video frames and determine the feature aliasing region data generated by multi-target interactions;

[0028] The feature information extraction module is used to extract human key points from the sequence of surveillance video frames, define human key points as reference geometric nodes, and construct a spatiotemporal relationship map based on the reference geometric nodes.

[0029] The 3D kinematic constraint module is used to determine the physical length of the human skeleton and the joint rotation limits as human kinematic boundary parameters in the 3D human motion space. The target position is calculated through the following processes: Step S101, associating the reference geometric nodes and feature aliasing region data with the 3D human motion space to determine the spatial topological reference; Step S102, constructing a set of geometric constraint equations to restrict the degrees of freedom of human motion based on the human kinematic boundary parameters; Step S103, solving the set of geometric constraint equations in the 3D human motion space to calculate the spatial geometric solution of the occluded key points in the 3D human motion space that conforms to the laws of human kinematics.

[0030] The behavioral semantic reasoning module is used to reconstruct the spatiotemporal behavioral sequence of the target based on the baseline geometric nodes, spatiotemporal relationship graph and spatial geometric solution, and output the corresponding behavioral semantic labels.

[0031] Preferably, the three-dimensional kinematic constraint module further includes a constraint parameter correction unit, which is used to adjust the parameters through the following processes: Step S201, obtain the depth estimation data of the target in the current video frame, and determine the Z-axis occlusion order between the reference geometric nodes based on the depth estimation data; Step S202, calculate the transient kinetic energy value of each reference geometric node based on the displacement vector between adjacent time frames; Step S203, compare the transient kinetic energy value with a preset kinetic energy threshold to determine the dynamic weighting coefficient used to adjust the joint rotation limit; Step S204, update the geometric constraint equation set using the dynamic weighting coefficient to change the search range of the spatial geometric solution.

[0032] Preferably, the system also includes a graph vector compensation module, which is used to superimpose the feature aliasing region data determined by the video stream parsing module with the trajectory edge in the spatiotemporal relationship graph. When target overlap is detected, the feature propagation gradient of adjacent reference geometric nodes is extracted, and the compensation feature vector of the occluded key point is calculated in reverse by combining the preset spatial coordinate mapping matrix.

[0033] Preferably, the system also includes a momentum verification module, which is used to retrieve the motion state of the reference geometric nodes stored in the previous 3 to 5 time frames, calculate the matching degree between the current spatial geometric solution candidate value and the historical momentum vector, and select the analytical solution with the highest matching degree as the compensation result of the current time series.

[0034] Preferably, the momentum verification module uses the following formula to determine the matching degree: Where M is the matching degree, This represents the predicted momentum vector corresponding to the candidate values ​​of the spatial geometric solution. It is the weighted average of the historical momentum vectors.

[0035] Preferably, the system also includes a scale adjustment module, which is used to calculate the pixel area occupied by the reference geometric node in the feature map, and adjust the sampling kernel size in the feature aggregation stage based on the ratio of the pixel area to the preset sampling area.

[0036] Preferably, the behavioral semantic reasoning module further includes an interaction modeling unit, which is used to determine the relative position potential energy field between multiple targets in the three-dimensional human motion space, and to determine the trigger probability of physical contact between targets based on the rate of change of the Euclidean distance between two adjacent reference geometric nodes.

[0037] Preferably, the reference geometric nodes include the main torso center node, the two shoulder nodes, the two elbow nodes, and the two knee nodes, and the human kinematic boundary parameters include the fixed physical length between the two shoulder nodes and the two elbow nodes.

[0038] Preferably, when the missing duration of the occluded key point exceeds 15 frames, the three-dimensional kinematic constraint module stops searching for pixel features and predicts the uniform linear trajectory of the target position in the three-dimensional human motion space based on the historical momentum vector.

[0039] Preferably, the system also includes an early warning output module, which is used to send an enhanced video stream containing spatial geometric solutions to an external monitoring terminal when the behavioral semantic label successfully matches the feature items in the preset abnormal behavior library.

[0040] Example 1: In a monitoring scenario of an urban rail transit hub with a personnel density of 4 people per square meter, multiple individuals engage in physical conflict at the turnstile entrance, resulting in pixel-level overlap and occlusion of facial features within the 2D monitoring video. Traditional visual feature tracking algorithms, under such overlapping conditions of more than 15 consecutive frames, output results showing target identity switching and temporal semantic breaks. The video stream parsing module acquires the monitoring video frame sequence and identifies the feature aliasing region data generated by multi-target interaction. The feature information extraction module extracts visible human key points from the monitoring video frame sequence, defining these key points as reference geometric nodes in the 3D human motion space. Based on these reference geometric nodes, a spatiotemporal relationship map reflecting the initial physical topology is constructed. For the invisible state of target nodes within the feature aliasing region data, 3D kinematics... Within the 3D human motion space, the constraint module reads the fixed physical lengths between the shoulder nodes and elbow nodes, as well as the corresponding joint rotation limits, defining the physical lengths and joint rotation limits of the human skeleton as the kinematic boundary parameters. The 3D kinematic constraint module associates the reference geometric nodes and feature aliasing region data with the 3D human motion space to determine the spatial topological reference. Based on the kinematic boundary parameters, it constructs a set of geometric constraint equations that restrict the degrees of freedom of human motion using simultaneous nonlinear algebraic equations. Based on the pinhole imaging model and perspective projection principle, the system performs pre-calibration of the skeleton scale in the initial stage when the target enters the field of view in an unobstructed state. It extracts the pixel coordinates of 2D keypoints of each target within the preceding 30 consecutive time frames and calculates the actual physical lengths between the shoulder nodes and elbow nodes of a specific target based on the camera's intrinsic focal length matrix and depth values. Updating the human kinematic boundary parameters, based on the inverse kinematic linkage principle, the system transforms the solution of the geometric constraint equations into a process of minimizing the objective function. The objective function is set as the spatial coordinates of the occluded key points to be determined, based on the reference geometric nodes and the actual physical length. The system derives the 3D Euclidean distance between the coordinates of the end nodes in a forward manner. It then employs a finite-memory quasi-Newton method to iteratively optimize the objective function within the 3D human motion space, restricting the search vector for each iteration step to within the angle range defined by the joint rotation limits. When the Euclidean distance decays less than [a certain value] between two adjacent iterations... When the number of millimeter or global iterations reaches 50, the termination condition is triggered, and the converged coordinate set is extracted as the result output. Based on the human kinematic boundary parameters, a set of geometric constraint equations is constructed by combining nonlinear algebraic equations to restrict the degree of freedom of human movement.

[0041] Within the 3D human motion space, the geometric constraint equations are solved to calculate the spatial geometric candidate values ​​of the occluded keypoints that conform to human kinematics. The momentum verification module retrieves the motion states of the reference geometric nodes stored in the previous 5 time frames and calculates the matching degree between the current spatial geometric candidate values ​​and the historical momentum vectors. The specific calculation formula is as follows: Where M is the matching degree, This represents the predicted momentum vector corresponding to the candidate values ​​of the spatial geometric solution. The weighted average of historical momentum vectors is used; the momentum verification module selects the analytical solution with the largest matching degree as the compensation result for the current time series and injects the compensation result into the spatiotemporal relationship map; this processing mechanism uses the kinematic momentum vector in the time dimension to constrain the geometric solution boundary in the spatial dimension, eliminating discrete coordinate fluctuations induced by damage to a single visual feature; the behavioral semantic reasoning module reconstructs the spatiotemporal behavior sequence of the target based on the reference geometric node, the spatiotemporal relationship map, and the spatial geometric solution; the interactive modeling unit in the behavioral semantic reasoning module determines the relative position potential energy field between multiple targets in the three-dimensional human motion space. This potential energy field is defined as a spatial repulsion weight region with a radius of 2000mm and the target center node as the origin. The weight value decreases linearly with the increase of Euclidean distance, and the decrease ratio is fixed at 0.05 for every 100mm increase in weight; when two different targets When the reference geometric nodes enter each other's weighted regions and the rate of change of Euclidean distance exceeds 2.0 m / s for 50 ms, the trigger probability calculation logic is activated, mapping the relative displacement and weight product to a trigger probability value between 0 and 1.0. For feature vectors that disappear due to feature contamination, the system compensates by extracting the feature propagation gradient of adjacent visible nodes. This gradient is obtained by calculating the cosine offset of the feature vector in the high-dimensional space within two adjacent sampling periods, multiplying it by a gain compensation coefficient of 1.2, and then back-injecting it into the feature space of the node to be determined in the current frame. Based on the rate of change of Euclidean distance between two adjacent reference geometric nodes, the trigger probability of physical contact between targets is determined. The behavior semantic reasoning module finally outputs the corresponding behavior semantic label. Under the condition of continuous overlapping occlusion, the system outputs continuous spatiotemporal trajectory expression of the target and stable behavior semantic label.

[0042] Example 2: In a scenario of analyzing surveillance video of dense crowds with frequent target overlap, the test data comes from a high-density spatiotemporal behavior dataset of crowds collected collaboratively by multiple visual sensors. The acquisition device for this dataset has a fixed sampling rate of 60Hz and a single-pixel spatial resolution. To test the system's anti-disturbance properties in complex physical environments, Gaussian white noise with a signal-to-noise ratio of 15dB is actively superimposed onto the original surveillance video frame sequence. At the same time, 50Hz dynamic shadow artifacts caused by light source flicker are superimposed as environmental disturbance sources. The feature time window length required for the video stream parsing module to extract frame sequence features needs to balance the integrity of motion feature capture with the hardware computational load. When the average displacement rate of the target individual in the surveillance image is greater than a certain threshold, in order to avoid signal aliasing and loss of motion details under the Nyquist sampling theorem, the feature time window length tends to the lower limit of the value range. Based on the spatiotemporal evolution law at this time, 5 consecutive video frames are selected as the preferred benchmark value of the feature time window.

[0043] A control group relying on 2D bounding box matching logic and an experimental group using 3D human motion space modeling were established. The crowd density was divided into three progressive test gradients: 2 people per square meter, 4 people per square meter, and 6 people per square meter. A sequence of surveillance video frames carrying disturbance sources was input to the system. When the crowd density was at a low disturbance level of 2 people per square meter, the feature extraction nodes of both systems remained stable. As the crowd density climbed to the physical interference threshold of 4 people per square meter, the video stream parsing module output large areas of feature aliasing data. The control group experienced up to 45 target identity marker jumps within a single frame due to 2D appearance feature damage. The 3D kinematic constraint module of the experimental group associated the baseline geometric nodes and feature aliasing data with the 3D human motion space, simultaneously establishing a set of geometric constraint equations including the physical lengths of the shoulders and elbows, and calculating three sets of spatial geometric solution candidate values. The momentum verification module then applied the formula... Calculate the matching degree between each spatial geometric solution candidate value and the historical state, where M is the matching degree. This represents the predicted momentum vector corresponding to the candidate values ​​of the spatial geometric solution. The weighted average of the historical momentum vectors stored within the preceding time frame is used. The momentum verification module selects the maximum analytical solution with an M value of 0.89 as the current compensation result and injects it into the spatiotemporal relationship map. The physical momentum conservation property is used to suppress coordinate abrupt changes induced by Gaussian white noise and dynamic shadows.

[0044] The behavioral semantic reasoning module receives a spatiotemporal relationship graph and reconstructs the target spatiotemporal behavioral sequence. Test data shows that when the population density reaches 4 people per square meter, the behavioral semantic label output accuracy of the control group drops non-linearly to 62.4%. The sample group of this invention maintains the behavioral semantic label output accuracy at 91.5% by relying on physical space constraints and momentum verification logic. When the population density further exceeds the limit of 6 people per square meter, the spatial geometric solution computation of the sample group of this invention reaches the hardware computing power saturation limit, and its behavioral classification accuracy decreases slightly with a gentle slope and stabilizes at 85.3%, showing an overload performance inflection point that conforms to the law of engineering evolution. This test data confirms that by introducing a set of geometric constraint equations containing human kinematic boundary parameters and momentum matching degree calculation, the system can output continuous and stable spatiotemporal behavioral understanding results by replacing probabilistic pixel matching with deterministic physical topology calculations under the conditions of appearance feature contamination and high-frequency environmental disturbances.

[0045] Example 3: In a monitoring scenario with visual blind spots and target movement speed greater than 2m / s, the limbs of different individuals in the two-dimensional image cause physical occlusion in the depth direction, and the static geometric constraint equation set exhibits kinematic boundary failure. The constraint parameter correction unit obtains the depth estimation data of the target in the current video frame. Based on the camera intrinsic parameter matrix, the constraint parameter correction unit converts the two-dimensional pixel values ​​in the depth estimation data into depth coordinates in three-dimensional physical space, compares the depth coordinate values ​​of different reference geometric nodes in the overlapping area, determines the node with the smaller depth coordinate value as the occluded foreground node, and determines the node with the larger depth coordinate value as the occluded background node, thus determining the Z-axis occlusion order among the reference geometric nodes.

[0046] For occluded background nodes, the constraint parameter correction unit obtains the three-dimensional spatial displacement vector between adjacent time frames and calculates the corresponding three-dimensional spatial velocity vector based on the video sampling rate; the system retrieves a preset human skeleton dynamics topology library and reads the relative mass constants of each reference geometric node; the constraint parameter correction unit then applies the formula... Calculate the transient kinetic energy values ​​of each reference geometric node, where, This is the transient kinetic energy value. It is a relative mass constant. The magnitude of the velocity vector in three-dimensional space is used; the system selects a calibrated kinetic energy value that reflects the upper limit of the kinematic characteristics of normal walking as the preset kinetic energy threshold.

[0047] When the transient kinetic energy value is greater than the preset kinetic energy threshold, the constraint parameter correction unit adjusts according to the formula. Determine the dynamic weighting coefficients used to adjust the joint rotation limits, where, These are dynamic weighting coefficients. Based on the weight constant, This is the transient kinetic energy value. The system presets a kinetic energy threshold. The constraint parameter correction unit multiplies the dynamic weighting coefficients as multipliers with the boundary parameters that limit the joint rotation in the geometric constraint equations. This proportionally expands the 3D angle search range of the occluded background node in the 3D human motion space and updates the geometric constraint equations to change the search range of the spatial geometric solution. The system quantifies the intensity of the target's motion based on the transient kinetic energy value and uses the dynamic weighting coefficients to relax the geometric constraint boundaries, so that the spatial coordinates of the high-speed moving limbs fall within the expanded 3D angle search range, and outputs continuous and stable behavioral semantic labels.

[0048] Example 4: This example combines Figures 1 to 2 This describes a system for recognizing and understanding complex spatiotemporal multi-target behaviors in surveillance video, such as... Figure 1As shown, the surveillance video frame sequence serves as the input source, entering both the video stream parsing module and the feature information extraction module. The video stream parsing module acquires the surveillance video frame sequence and determines the feature aliasing region data. Simultaneously, the feature information extraction module extracts key human body points as reference geometric nodes and constructs a spatiotemporal relationship graph. The feature aliasing region data output by the video stream parsing module and the reference geometric nodes output by the feature information extraction module converge to the three-dimensional kinematic constraint module. This module uses human kinematic boundary parameters to construct a set of geometric constraint equations and performs calculations of the spatial geometric solution. Finally, the behavioral semantic reasoning module reconstructs the target spatiotemporal behavior sequence based on the reference geometric nodes, the spatiotemporal relationship graph, and the spatial geometric solution, thereby achieving accurate output of behavioral semantic labels.

[0049] like Figure 2 As shown, the core functional module architecture of the system is composed of six functional branches. Among them, the video stream parsing module covers two sub-items: acquiring the monitoring video frame sequence and determining the feature aliasing region data. The feature information extraction module is responsible for constructing the spatiotemporal relationship map and extracting human key points. The three-dimensional kinematic constraint module is specifically refined into four steps: using human kinematic boundary parameters, determining the spatial topological benchmark, constructing a set of geometric constraint equations, and calculating the spatial geometric solution. The constraint parameter correction unit, which works in conjunction with it, is responsible for acquiring depth estimation data, calculating transient kinetic energy values, and determining dynamic weighting coefficients accordingly. The momentum verification module ensures data accuracy by retrieving the motion state of the benchmark geometric nodes, calculating the matching degree between the candidate values ​​of the spatial geometric solution and the historical momentum vector, and selecting the analytical solution with the highest matching degree. The behavior semantic reasoning module, by reconstructing the spatiotemporal behavior sequence of the target, determining the relative position potential energy field between multiple targets, and determining the trigger probability of physical contact between targets, jointly supports the complete operating logic of the complex spatiotemporal multi-target behavior recognition and understanding system for monitoring videos.

[0050] Example 5: In an offline calibration scenario for constructing a human skeletal dynamics topology library, a sensor array collects a 3D human gait capture dataset containing samples of specific ages and body types; the system acquires the absolute mass data of each limb segment of the test subject, calculates the quotient of the absolute mass data of a single limb segment and the overall mass value of the test subject, and determines the relative mass constant corresponding to each reference geometric node. The system inputs the relative mass constants of multiple test objects. The expected value is calculated, and a relative mass constant distribution matrix is ​​generated and written into non-volatile memory. The feature information extraction module extracts the spatial coordinate data of the limb end nodes of the test object in the normal walking state in the 3D human gait capture dataset. The 3D kinematic constraint module calculates the maximum transient kinetic energy within the gait cycle based on the corresponding video sampling rate, extracts the statistical boundary value containing a specific confidence interval, and sets the statistical boundary value as the preset kinetic energy threshold. Simultaneously, the data is saved to the human skeletal dynamics topology library, forming the underlying data benchmark for kinematic constraint solving. In the specific calibration process, the preset kinetic energy threshold Eth value is taken from the kinetic energy maximum value in the 95% confidence interval of the 10-minute normal walking gait sample library, and the quantization setting is 15.0J. The calibration of the basic weight constant is completed through the self-test process at the initial power-on stage of the system. The system automatically identifies the preset 1000mm standard reference object in the scene and calculates the deviation ratio between its actual imaging pixel size and the theoretical projected pixel size. If the deviation ratio is in the range of 0.95 to 1.05, the basic weight constant is set to the initial benchmark value of 1.0. If the deviation ratio is offset due to the installation tilt angle exceeding 15 degrees, the value of the basic weight constant is adjusted linearly according to the ratio to counteract the perspective shortening effect.

[0051] When faced with on-site deployment conditions where the viewing angle and optical focal length parameters of the monitoring equipment undergo physical changes, the system accesses a calibration test video sequence containing spatial standard reference objects; the video stream parsing module acquires a preset calibration target image with known fixed three-dimensional dimensions, and the feature information extraction module extracts the two-dimensional coordinates of the reference geometric nodes of the calibration target; the constraint parameter correction unit calculates the baseline geometric constraint error of the preset calibration target under the current camera perspective projection relationship; the system retrieves the perspective distortion magnification factor corresponding to the current focal length parameter, multiplies the perspective distortion magnification factor by the baseline geometric constraint error, and calculates the basic weight constant for compensating for optical imaging deformation. The basic weight constant Updated to the local configuration register; basic weight constants The parameter compensation logic enables the calculation parameters of the dynamic weighting coefficients to adaptively match the physical optical mapping characteristics of different surveillance cameras, maintaining the stability of the output spatial geometric solution of the three-dimensional kinematic constraint module in the urban rail transit monitoring environment with multi-view cross coverage.

[0052] Example 6: When the system faces the initial deployment conditions of a monitoring scenario, the interaction patterns of the target individual in the three-dimensional human motion space exhibit environment-specific physical characteristics. The semantic labels output by the behavioral semantic reasoning model without parameter optimization show overlapping classification boundaries. The system accesses a local working condition video dataset containing known behavioral labels. This local working condition video dataset covers normal passage behavior and abnormal physical contact behavior. The feature information extraction module extracts the baseline geometric node sequence in the local working condition video dataset and inputs it into the behavioral semantic reasoning module. The behavioral semantic reasoning module includes a spatial graph convolutional layer and a temporal convolutional layer. The spatial graph convolutional layer calculates the spatial Euclidean distance distribution features between multiple baseline geometric nodes in a single video frame. The temporal convolutional layer calculates the displacement change rate features of the corresponding baseline geometric nodes between adjacent time frames along the time axis. The behavioral semantic reasoning module fuses the spatial Euclidean distance distribution features and the displacement change rate features and outputs a high-dimensional behavioral feature vector.

[0053] The system extracts all high-dimensional behavioral feature vectors corresponding to normal traffic behavior from the local working condition video dataset, calculates the coordinates of the cluster center points of the corresponding high-dimensional behavioral feature vectors, and extracts the minimum Mahalanobis distance values ​​of abnormal physical contact behavior samples relative to the coordinates of the cluster center points in the local working condition video dataset, based on the formula... Set a judgment threshold, where, To preset the semantic judgment threshold, The system will preset a semantic determination threshold to obtain the minimum Mahalanobis distance value. Write to read-only memory; when the system is in real-time monitoring mode, the behavior semantic reasoning module calculates the real-time Mahalanobis distance between the high-dimensional behavior feature vector of the monitored target within the current feature time window and the coordinates of the cluster center point. ,in, For real-time Mahalanobis distance, when real-time Mahalanobis distance Greater than the preset semantic judgment threshold When the target interaction state deviates from the normal data distribution cluster, the system outputs a behavioral semantic label indicating abnormal physical contact. The system relies on the pre-extracted cluster center point coordinates and the preset semantic judgment threshold to form boundary criteria, and outputs behavioral classification results based on spatial Euclidean distance and displacement change rate features.

[0054] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A system for recognizing and understanding complex spatiotemporal multi-target behaviors in surveillance video, characterized in that, The system includes: The video stream parsing module is used to acquire the sequence of monitoring video frames and determine the feature aliasing region data generated by multi-target interactions; The feature information extraction module is used to extract human key points from the sequence of surveillance video frames, define human key points as reference geometric nodes, and construct a spatiotemporal relationship map based on the reference geometric nodes. The 3D kinematic constraint module is used to determine the physical length of the human skeleton and the joint rotation limits as human kinematic boundary parameters in the 3D human motion space. The target position is calculated through the following processes: Step S101, associating the reference geometric nodes and feature aliasing region data with the 3D human motion space to determine the spatial topological reference; Step S102, constructing a set of geometric constraint equations to restrict the degrees of freedom of human motion based on the human kinematic boundary parameters; Step S103, solving the set of geometric constraint equations in the 3D human motion space to calculate the spatial geometric solution of the occluded key points in the 3D human motion space that conforms to the laws of human kinematics. The behavioral semantic reasoning module is used to reconstruct the spatiotemporal behavioral sequence of the target based on the baseline geometric nodes, spatiotemporal relationship graph and spatial geometric solution, and output the corresponding behavioral semantic labels. Furthermore, the 3D kinematic constraint module also includes a constraint parameter correction unit, which adjusts the parameters through the following processes: Step S201, acquiring the depth estimation data of the target in the current video frame, and determining the Z-axis occlusion order between reference geometric nodes based on the depth estimation data; Step S202, calculating the transient kinetic energy value of each reference geometric node based on the displacement vector between adjacent time frames; Step S203, comparing the transient kinetic energy value with a preset kinetic energy threshold, and when the transient kinetic energy value is greater than the preset kinetic energy threshold, the constraint parameter correction unit adjusts the parameters according to the formula... Determine the dynamic weighting coefficients used to adjust the joint rotation limits, where, These are dynamic weighting coefficients. Based on the weight constant, This is the transient kinetic energy value. A preset kinetic energy threshold is set; in step S204, the geometric constraint equations are updated using dynamic weighting coefficients to change the search range of the spatial geometric solution.

2. The system for complex spatiotemporal multi-target behavior recognition and understanding of surveillance video according to claim 1, characterized in that, The system also includes a graph vector compensation module, which is used to superimpose the feature aliasing region data determined by the video stream parsing module with the trajectory edges in the spatiotemporal relationship graph. When target overlap is detected, the feature propagation gradient of adjacent reference geometric nodes is extracted, and the compensation feature vector of the occluded key point is calculated in reverse by combining the preset spatial coordinate mapping matrix.

3. The system for complex spatiotemporal multi-target behavior recognition and understanding of surveillance video according to claim 1, characterized in that, The system also includes a momentum verification module, which is used to retrieve the motion state of the reference geometric nodes stored in the previous 3 to 5 time frames, calculate the matching degree between the current candidate values ​​of spatial geometric solutions and the historical momentum vectors, and select the analytical solution with the highest matching degree as the compensation result of the current time series.

4. The system for complex spatiotemporal multi-target behavior recognition and understanding of surveillance video according to claim 1, characterized in that, The system also includes a scaling module, which is used to calculate the pixel area occupied by the reference geometric node in the feature map and adjust the sampling kernel size in the feature aggregation stage based on the ratio of the pixel area to the preset sampling area.

5. A system for complex spatiotemporal multi-target behavior recognition and understanding of surveillance video according to claim 1, characterized in that, The behavioral semantic reasoning module also includes an interaction modeling unit, which is used to determine the relative position potential energy field between multiple targets in the three-dimensional human motion space, and to determine the trigger probability of physical contact between targets based on the rate of change of the Euclidean distance between two adjacent reference geometric nodes.

6. A system for complex spatiotemporal multi-target behavior recognition and understanding of surveillance video according to claim 1, characterized in that, The reference geometric nodes include the main trunk center node, shoulder nodes, elbow nodes, and knee nodes, and the human kinematic boundary parameters include the fixed physical length between the shoulder nodes and elbow nodes.

7. A system for complex spatiotemporal multi-target behavior recognition and understanding of surveillance video according to claim 3, characterized in that, When the missing duration of occluded key points exceeds 15 frames, the 3D kinematic constraint module stops searching for pixel features and predicts the uniform linear trajectory of the target position in the 3D human motion space based on the historical momentum vector.

8. A system for complex spatiotemporal multi-target behavior recognition and understanding of surveillance video according to claim 1, characterized in that, The system also includes an early warning output module, which is used to send an enhanced video stream containing spatial geometric solutions to an external monitoring terminal when the behavioral semantic label successfully matches the feature items in the preset abnormal behavior library.

Images