An AI-based real-time video behavior analysis method and system

Abstract

The application discloses a kind of real-time video behavior analysis method and system based on AI, including the following steps: continuous video stream data is collected, and pre-processing is executed;Perform object detection and feature extraction, generate multi-modal feature set;Multi-view spatiotemporal atlas is constructed;Multi-view spatiotemporal atlas is executed structure recursive update processing, and dynamic spatiotemporal atlas is generated;Perform behavior inference, and generate preliminary identification result;Analysis is carried out to semantic deviation, and semantic clustering and latent cognitive correction are executed;Perform spatiotemporal jump attention processing, and key frame selection and jump fusion are carried out to time sequence feature, and generate efficient spatiotemporal feature representation;Based on efficient spatiotemporal feature representation, generate final behavior identification result.The application fuses multi-modal feature and dynamic spatiotemporal atlas structure, proposes structure recursive update and semantic self-repair mechanism, realizes the accurate identification and efficient real-time analysis of video behavior.

Description

Technical Field

[0001] This invention relates to the field of video behavior analysis, and in particular to an AI-based real-time video behavior analysis method and system. Background Technology

[0002] Existing real-time video behavior analysis technologies mainly rely on convolutional neural networks, temporal modeling, or graph convolutional networks. They achieve behavior category recognition by extracting features from video frame sequences and modeling spatiotemporal relationships. However, these methods generally suffer from problems such as single feature dimensions, insufficient spatiotemporal dependence, and weak semantic understanding. In complex scenes, the models often struggle to handle semantic interactions between multiple objects, and the recognition accuracy drops significantly when there are changes in lighting, occlusion, or interruptions in action continuity.

[0003] In addition, existing methods mostly use fixed graph structures or static weight mechanisms, which cannot adaptively adjust the connection relationship between nodes according to dynamic changes in the scene. The semantic layer is usually only used for label classification and lacks the ability to correct and self-correct semantic drift, resulting in unstable behavioral semantic expression. At the same time, traditional temporal modeling fails to achieve efficient keyframe screening and skip-attention fusion, resulting in high computational overhead and insufficient real-time performance. Summary of the Invention

[0004] One objective of this invention is to propose an AI-based real-time video behavior analysis method and system. This invention integrates multimodal features and dynamic spatiotemporal graph structures, and proposes a structural recursive update and semantic self-correction mechanism to achieve accurate identification and efficient real-time analysis of video behavior.

[0005] According to an embodiment of the present invention, a real-time video behavior analysis method based on AI includes the following steps:

[0006] Acquire continuous video stream data and perform preprocessing to generate a standardized video frame sequence;

[0007] For each frame in a standardized video frame sequence, object detection and feature extraction are performed to generate a multimodal feature set containing object pose features, motion trajectory features, optical flow features, and semantic features;

[0008] Based on a multimodal feature set, spatial nodes, temporal nodes, and semantic nodes are constructed, and spatial connection edges, temporal connection edges, and semantic connection edges are established respectively to form a multi-view spatiotemporal graph;

[0009] Perform recursive structural update processing on the multi-view spatiotemporal graph, dynamically adjust the connection relationships and connection weights between nodes, and generate a dynamic spatiotemporal graph.

[0010] Behavioral reasoning is performed on a dynamic spatiotemporal graph, and the spatiotemporal features of each node are aggregated and feature propagation is performed to generate preliminary identification results of the object's behavioral state.

[0011] The semantic bias of the initial behavioral state identification results is analyzed, semantic clustering and latent cognition correction are performed, and the semantic mapping parameters are updated to perform semantic self-correction.

[0012] After semantic self-correction, spatiotemporal jump attention processing is performed to select keyframes and perform jump-style fusion of temporal features, complete sparse feature aggregation, and generate efficient spatiotemporal feature representation.

[0013] The final behavior recognition result is generated based on efficient spatiotemporal feature representation.

[0014] Optionally, the preprocessing includes frame segmentation, noise removal, illumination equalization, time synchronization, and grayscale normalization.

[0015] Optionally, the generation of the multimodal feature set specifically includes:

[0016] Perform object detection processing on each frame of the standardized video frame sequence, extract candidate regions containing objects, and generate a set of object bounding boxes;

[0017] Perform pose feature extraction on the image region corresponding to the bounding box of the object, extract the key point position features, limb angle features and pose structure features of the object, and generate the object pose feature vector;

[0018] In adjacent frames, object matching and temporal association are performed based on object identifier numbers. The displacement change information of the object is calculated, and the velocity features, trajectory direction features and acceleration features of the object are extracted to generate motion trajectory feature vectors.

[0019] Optical flow feature extraction is performed on a continuous frame sequence to calculate the pixel motion trend within the object region, extract local dynamic texture features and orientation gradient features, and generate optical flow feature vectors.

[0020] Semantic encoding is performed on the bounding box region of the object, semantic feature extraction is performed, and semantic feature vectors are generated;

[0021] The pose feature vector, motion trajectory feature vector, optical flow feature vector, and semantic feature vector of the same object are synchronized and concatenated in chronological order to generate the object's multimodal feature vector;

[0022] The multimodal feature vectors of all objects are integrated to form a multimodal feature set.

[0023] Optionally, the generation of the multi-view spatiotemporal map specifically includes:

[0024] The multimodal feature set extracted from the standardized video frame sequence is organized in chronological order. The multimodal features of all objects in the same frame are aggregated. A spatial node set is constructed based on the coordinate position distribution of the objects in the image plane. The pose feature vector, motion trajectory feature vector, optical flow feature vector, and semantic feature vector of the objects are bound to the spatial nodes.

[0025] Perform matching association on spatial nodes of the same object between consecutive frames, establish a set of time nodes, and form a time node link according to the frame index order;

[0026] Perform clustering on the semantic feature vectors of all objects to generate a set of semantic nodes;

[0027] Based on the correspondence between the spatial node set, the temporal node set, and the semantic node set, a spatial connection edge set, a temporal connection edge set, and a semantic connection edge set are established respectively.

[0028] Assign connection weights to spatial, temporal, and semantic connection edges;

[0029] By unifying and integrating the sets of spatial nodes, temporal nodes, and semantic nodes, as well as the corresponding sets of spatial, temporal, and semantic connecting edges, a multi-view spatiotemporal graph is obtained.

[0030] Optionally, the generation of the dynamic spatiotemporal map specifically includes:

[0031] Extract the node feature vectors and corresponding connection weight parameters of all spatial nodes, temporal nodes and semantic nodes from the multi-view spatiotemporal graph to form a node feature set and a connection weight set.

[0032] Perform difference calculation on the node feature vectors of the same object in the time node set at adjacent time steps to obtain the time feature change value, calculate the average amplitude of the time feature change value, and generate a set of time change amplitudes;

[0033] The feature difference between adjacent spatial nodes is calculated based on the set of spatial nodes to generate a set of spatial feature changes; the semantic feature difference between adjacent semantic nodes is calculated based on the set of semantic nodes to generate a set of semantic feature changes; the set of temporal change magnitude, the set of spatial feature changes, and the set of semantic feature changes are integrated to form a node feature change matrix.

[0034] The dynamic similarity value between nodes is calculated based on the node feature change matrix, and a dynamic weight adjustment coefficient matrix is ​​generated based on the dynamic similarity value. The connection weight parameters of spatial connection edges, temporal connection edges and semantic connection edges are updated using the dynamic weight adjustment coefficient matrix to form an updated set of connection weights.

[0035] The updated set of connection weights is normalized to generate a normalized set of connection weights.

[0036] Based on the normalized connection weight set, a weighted aggregation calculation is performed on each node, and the feature vectors of adjacent nodes are weighted and summed according to the normalized connection weights to generate a new node feature set.

[0037] The new set of node features is input into the recursive update process of the dynamic spatiotemporal graph structure. Based on the changing trend of node features and the adjustment results of connection weights, the connection relationships between spatial nodes, temporal nodes and semantic nodes are updated topologically to generate the updated dynamic spatiotemporal graph.

[0038] Optionally, the generation of the preliminary behavioral state identification result specifically includes:

[0039] Extract the node feature vectors of all spatial nodes, temporal nodes, and semantic nodes, as well as the updated connection weights, from the dynamic spatiotemporal graph after recursive structural updates.

[0040] Based on the connection weights, spatial feature weighted aggregation calculations are performed on each spatial node to generate spatial feature aggregation results; temporal feature weighted aggregation calculations are performed on each temporal node to generate temporal feature aggregation results; and semantic feature weighted aggregation calculations are performed on each semantic node to generate semantic feature aggregation results.

[0041] The spatial feature aggregation results, temporal feature aggregation results, and semantic feature aggregation results are aligned in dimension and normalized according to node number to construct a multi-dimensional feature aggregation set;

[0042] Feature propagation computation is performed on the multidimensional feature aggregation set. Taking each node as the center, the features of its first-order and second-order adjacent nodes are passed and fused layer by layer to obtain the node feature propagation result set.

[0043] Spatiotemporal semantic fusion computation is performed based on the set of node feature propagation results. A temporal constraint matrix is ​​constructed based on the feature change sequence between time nodes, a spatial constraint matrix is ​​constructed based on the relative positional relationship between spatial nodes, and a semantic constraint matrix is ​​constructed based on the semantic similarity distribution between semantic nodes. The three types of constraint matrices are then weighted and summed according to a set ratio to generate a fusion constraint matrix.

[0044] The fusion constraint matrix is ​​applied to the set of node feature propagation results, and a weighted mapping calculation is performed on the comprehensive feature vector of each node to generate a set of node behavior feature representations;

[0045] The set of node behavior feature representations is input into a multilayer perceptron to perform behavior recognition inference, and the probability distribution vector of each object's behavior category and the corresponding node behavior state vector are obtained. The category label corresponding to the highest probability and the corresponding node behavior state vector are taken as the preliminary recognition result of the object's behavior state.

[0046] Optionally, the semantic self-correction specifically includes:

[0047] Using the set of node behavior state vectors in the preliminary behavior state recognition result set as input, calculate the difference between the set of node behavior state vectors and the set of node semantic vectors of the corresponding node in the dynamic spatiotemporal map, and generate a semantic deviation matrix.

[0048] Semantic clustering is performed based on the semantic deviation matrix. Similarity is aggregated according to the semantic difference between node behavior state vectors. Node behavior state vectors with similar semantic deviation features are divided into several semantic clustering groups to form a set of semantic clustering results.

[0049] Based on the semantic clustering result set, perform latent cognitive correction, perform parameter correction operation on the corresponding node behavior state vector, and generate a latent cognitive correction vector set;

[0050] The semantic mapping parameters are updated by using the latent cognition correction vector set, and the corrected node behavior state vectors are remapped to the semantic space to form the updated semantic mapping parameter set.

[0051] Based on the updated semantic mapping parameter set, the semantic consistency score between the node behavior state vector and the semantic node is recalculated, and the semantic weight of the node behavior state vector is adjusted according to the semantic consistency score to generate a semantic self-correction result set.

[0052] The semantic self-correction result set is fed back to the semantic node attributes of the dynamic spatiotemporal graph, and the semantic feature information of the corresponding node is replaced, so as to realize semantic layer self-correction and behavioral semantic adaptive adjustment.

[0053] Optionally, the generation of the efficient spatiotemporal feature representation specifically includes:

[0054] Extract the semantic correction vector of each node and its corresponding time index parameter, and sort the semantic correction vectors by time according to the time index parameter to form a temporal semantic correction sequence;

[0055] Calculate the feature change rate between adjacent frames for the temporal semantic correction sequence. When the change rate exceeds a preset threshold, mark the corresponding frame as a key frame to form a key frame index set. Extract the semantic correction vector corresponding to the key frame to form a key frame feature set.

[0056] Spatiotemporal jump sampling is performed in the keyframe feature set with a fixed jump step size. The sampling interval is set to twice the average interval of the time index parameter to form a jump sampling feature sequence.

[0057] Multi-scale coding is performed on each feature vector in the skip sampling feature sequence. The inter-frame feature change rate is calculated on the temporal scale, and the spatial adjacency score is obtained on the spatial scale based on the spatial connection edges between nodes. The inter-frame feature change rate and the spatial adjacency score are weighted to generate a set of multi-scale spatiotemporal coding results.

[0058] Skip attention weights are calculated on the multi-scale spatiotemporal coding result set. The inter-frame feature change rate and spatial adjacency score of each node are weighted to obtain the initial attention value, which is then normalized to form the skip attention weight set.

[0059] The multi-scale spatiotemporal coding result set is weighted and aggregated using a set of jump attention weights to generate sparse feature aggregation results, which are then used as efficient spatiotemporal feature representations.

[0060] Optionally, the final behavior recognition result is generated by using a multilayer perceptron and a long short-term memory network in series.

[0061] According to an embodiment of the present invention, an AI-based real-time video behavior analysis system includes:

[0062] The video acquisition and preprocessing module is used to acquire continuous video stream data and perform frame segmentation, noise removal, illumination equalization and time synchronization processing to generate a standardized video frame sequence.

[0063] The multimodal feature extraction module is used to perform object detection and feature extraction on standardized video frame sequences to generate a multimodal feature set;

[0064] The spatiotemporal map construction module is used to construct multi-view spatiotemporal maps based on multimodal feature sets;

[0065] The recursive structure update module is used to perform recursive structure update processing on multi-view spatiotemporal graphs, dynamically adjust the connection relationships and connection weights between nodes, and generate dynamic spatiotemporal graphs.

[0066] The behavior reasoning module is used to perform feature aggregation and feature propagation on a dynamic spatiotemporal graph to generate preliminary identification results of the object's behavior state;

[0067] The semantic self-correction module is used to analyze the semantic bias of the initial identification results of behavioral states, perform semantic clustering and latent cognition correction, and update semantic mapping parameters to achieve semantic layer self-correction.

[0068] The spatiotemporal jump attention module is used to perform keyframe selection and jump fusion after semantic self-correction, complete sparse feature aggregation, and generate efficient spatiotemporal feature representation;

[0069] The behavior recognition module is used to generate the final behavior recognition result based on efficient spatiotemporal feature representation.

[0070] The beneficial effects of this invention are:

[0071] This invention constructs a multi-dimensional fused dynamic spatiotemporal graph, enabling joint modeling and adaptive updating of objects across spatial, temporal, and semantic dimensions. This overcomes the problems of fixed structure, single features, and missing semantic information in traditional video behavior analysis methods. By introducing a multimodal feature set, the pose features, motion trajectory features, optical flow features, and semantic features of objects are extracted and fused simultaneously. This allows the model to maintain stable semantic expression and robust recognition performance even when facing complex backgrounds, changing viewpoints, and fluctuations in action continuity. The system dynamically adjusts the multi-view spatiotemporal graph structure based on the changing trends of node features and connection weights through a recursive structural update mechanism, achieving adaptive modeling of multi-object interaction relationships and improving the spatiotemporal dependency modeling capability in scenarios with concurrent multi-behavior and occlusion.

[0072] This invention further introduces a semantic self-correction mechanism. Based on the initial recognition results of behavioral states, it uses semantic clustering and latent cognitive correction strategies to adaptively adjust semantic deviations and dynamically update semantic mapping parameters, effectively solving the problems of semantic drift and misclassification, and ensuring the stability of behavioral semantic consistency. Through a spatiotemporal jump attention mechanism, key frames are selected and feature aggregation is performed in a jump manner, reducing the computational overhead of invalid frames and realizing efficient sparse feature representation. This allows the system to maintain recognition accuracy while significantly reducing the computational burden of real-time processing. Combined with the deep structure of multilayer perceptron and long short-term memory network, it achieves high-precision and high-real-time behavior recognition result output. Attached Figure Description

[0073] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0074] Figure 1 This is a flowchart of an AI-based real-time video behavior analysis method proposed in this invention;

[0075] Figure 2 This is a schematic diagram of the semantic self-correction and latent cognitive correction mechanism of the AI-based real-time video behavior analysis method proposed in this invention. Detailed Implementation

[0076] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0077] refer to Figures 1-2 A real-time video behavior analysis method based on AI includes the following steps:

[0078] Acquire continuous video stream data and perform preprocessing to generate a standardized video frame sequence;

[0079] For each frame in a standardized video frame sequence, object detection and feature extraction are performed to generate a multimodal feature set containing object pose features, motion trajectory features, optical flow features, and semantic features;

[0080] Based on a multimodal feature set, spatial nodes, temporal nodes, and semantic nodes are constructed, and spatial connection edges, temporal connection edges, and semantic connection edges are established respectively to form a multi-view spatiotemporal graph;

[0081] Perform recursive structural update processing on the multi-view spatiotemporal graph, dynamically adjust the connection relationships and connection weights between nodes, and generate a dynamic spatiotemporal graph.

[0082] Behavioral reasoning is performed on a dynamic spatiotemporal graph, and the spatiotemporal features of each node are aggregated and feature propagation is performed to generate preliminary identification results of the object's behavioral state.

[0083] The semantic bias of the initial behavioral state identification results is analyzed, semantic clustering and latent cognition correction are performed, and the semantic mapping parameters are updated to perform semantic self-correction.

[0084] After semantic self-correction, spatiotemporal jump attention processing is performed to select keyframes and perform jump-style fusion of temporal features, complete sparse feature aggregation, and generate efficient spatiotemporal feature representation.

[0085] The final behavior recognition result is generated based on efficient spatiotemporal feature representation.

[0086] In this embodiment, the preprocessing includes frame segmentation, noise removal, illumination equalization, time synchronization, and grayscale normalization.

[0087] In this embodiment, the generation of the multimodal feature set specifically includes:

[0088] For each frame of a standardized video frame sequence, object detection processing is performed to extract candidate regions containing objects, generate a set of object bounding boxes, each object bounding box contains the center coordinates, width and height parameters of the object in the image plane, and assigns a unique object identifier number to each object;

[0089] Perform pose feature extraction on the image region corresponding to the bounding box of the object, extract the key point position features, limb angle features and pose structure features of the object, and generate the object pose feature vector;

[0090] The keypoint location features, limb angle features, and posture structure features are used to describe the object's posture state, geometric shape, and structural configuration in a video frame, respectively. The keypoint location features are a set of coordinates of the main keypoints, used to reflect the relative positional distribution of various parts of the object's body in space. The limb angle features are a set of angle parameters calculated based on the line relationships between keypoints, used to describe the geometric angles, directional relationships, and posture change trends between joints. The posture structure features are topological structure information formed based on the connection relationships between keypoints, used to represent the hierarchical connection patterns and overall configuration constraints between key parts of the object's body. The three together constitute the object's posture feature vector, used to comprehensively represent the object's spatial posture state and structural shape in a single frame of video.

[0091] In adjacent frames, object matching and temporal association are performed based on object identifier numbers. The displacement change information of the object is calculated, and the velocity features, trajectory direction features and acceleration features of the object are extracted to generate motion trajectory feature vectors to describe the continuous motion behavior of the object in the time dimension.

[0092] Optical flow feature extraction is performed on a continuous frame sequence to calculate the pixel motion trend within the object region, extract local dynamic texture features and orientation gradient features, and generate optical flow feature vectors to reflect the instantaneous motion pattern of the object in a local range.

[0093] Semantic encoding is performed on the bounding box region of the object, semantic feature extraction is performed, and semantic feature vectors are generated to describe the semantic relationship between the object and the behavior in the semantic dimension.

[0094] The semantic features are high-dimensional semantic representation vectors composed of semantic embedding features, category features, and behavioral semantic features, used to describe the category attributes, contextual semantic relationships, and behavioral meanings of the detected object in the semantic space;

[0095] The pose feature vector, motion trajectory feature vector, optical flow feature vector, and semantic feature vector of the same object are synchronized and concatenated in time order to generate the object's multimodal feature vector. Each object's multimodal feature vector contains four sub-dimensions: pose feature, trajectory feature, optical flow feature, and semantic feature at the same time step.

[0096] The multimodal feature vectors of all objects are integrated to form a multimodal feature set.

[0097] In this embodiment, the generation of the multi-view spatiotemporal map specifically includes:

[0098] The multimodal feature set extracted from the standardized video frame sequence is organized in chronological order. The multimodal features of all objects in the same frame are aggregated. A spatial node set is constructed based on the coordinate position distribution of the objects in the image plane. Each spatial node corresponds to an object instance. The object's pose feature vector, motion trajectory feature vector, optical flow feature vector, and semantic feature vector are bound in the spatial node to uniformly describe the object's appearance, motion trend, and semantic attributes in the spatial dimension. This ensures that the spatial node set of each frame can fully reflect the spatial distribution characteristics of each object in the scene.

[0099] The spatial nodes of the same object are matched and associated between consecutive frames to establish a set of time nodes. The time node links are formed according to the frame index order. Each time node records the object's time index parameters, displacement parameters, velocity parameters and acceleration parameters to describe the continuous motion characteristics of the object in the time dimension.

[0100] The time nodes are used to characterize the continuous dynamic changes of an object in the time dimension. The time index parameters, displacement parameters, velocity parameters, and acceleration parameters recorded at each time node together constitute the temporal state description of the object at a specific time step. The time index parameters are used to identify the specific time position of the object in the standardized video frame sequence to ensure the traceability and sequential consistency of the time dimension. The displacement parameters are used to reflect the amount of spatial position change of the object between adjacent frames and are used to characterize the motion path of the object in the time dimension. The velocity parameters are used to represent the displacement rate of the object per unit time and are an important indicator for measuring the speed of the object's motion. The acceleration parameters are used to reflect the trend of the object's velocity change over time and can describe the dynamic behavior change characteristics of the object during motion, such as from stationary to accelerated and from accelerated to decelerated. By jointly recording the above parameters at the time nodes, a refined modeling of the object's motion state can be achieved in the time dimension, enabling the multi-view spatiotemporal map to have a comprehensive ability to express abrupt changes in motion patterns, velocity changes, and temporal continuity in dynamic behavior recognition.

[0101] Perform clustering on the semantic feature vectors of all objects to generate a set of semantic nodes;

[0102] The generation of the semantic node set specifically includes: performing normalization processing on the semantic feature vector of each object to form a standardized semantic feature set; calculating a semantic similarity matrix based on the cosine similarity between any two semantic feature vectors in the standardized semantic feature set; generating semantic aggregation subsets by merging object features with similarity higher than a preset semantic similarity threshold level by level, each semantic aggregation subset representing a set of objects with similar semantic distributions; then performing mean operation on the semantic feature vectors in each semantic aggregation subset to generate a semantic center vector, and establishing an index relationship between the semantic center vector and the corresponding semantic aggregation subset to form a preliminary semantic node set; calculating the internal semantic variance of each semantic node and determining the semantic weight parameter based on the variance distribution, so that nodes with lower semantic differences have higher weight values, thereby enhancing the discriminative power of semantic clustering; and structurally integrating the semantic center vectors, semantic weight parameters, and their associated spatial node numbers of all semantic nodes to form a semantic node set.

[0103] Based on the correspondence between the spatial node set, the temporal node set, and the semantic node set, a spatial connection edge set, a temporal connection edge set, and a semantic connection edge set are established respectively.

[0104] The construction of spatial, temporal, and semantic connection edges specifically includes: For the set of spatial nodes, calculating the positional distance between any two spatial nodes in the image plane within the same frame, determining whether a connection relationship exists based on a preset spatial proximity threshold, establishing a spatial connection edge when the positional distance is less than the spatial proximity threshold, and recording the spatial relative direction and spatial center coordinates of the connection edge to generate a set of spatial connection edges; For the set of temporal nodes, matching the time nodes of the same object in adjacent time steps based on the time index parameter, calculating the change in position, velocity, and acceleration of the object between adjacent time steps, determining whether there is a temporal correlation based on the temporal continuity score, establishing a temporal connection edge between adjacent time nodes when the temporal continuity score is higher than a preset time threshold, and recording the time interval parameter and motion continuity parameter of the connection edge to generate a set of temporal connection edges; For the set of semantic nodes, calculating the similarity between the semantic center vectors of any two semantic nodes, determining the semantic correlation based on a preset semantic similarity threshold, establishing a semantic connection edge when the similarity is higher than the semantic similarity threshold, and recording the semantic similarity value and semantic category index information of the connection edge to generate a set of semantic connection edges.

[0105] Assign connection weights to spatial, temporal, and semantic connection edges;

[0106] The specific allocation of connection weight parameters includes: for spatial connection edges, calculating the planar position difference between the spatial nodes at both ends of the connection edge to obtain the spatial distance value between the nodes; simultaneously calculating the spatial relative direction angle based on the coordinate relationship between the nodes; weighting and fusing the spatial distance value and the relative direction angle according to a preset weight ratio to generate a spatial association value; and allocating spatial connection edge weights based on the spatial association value. For temporal connection edges, calculating the time interval, displacement difference, velocity difference, and acceleration difference between adjacent time nodes, and weighting and summing them according to a preset ratio to obtain a comprehensive score for time change; comparing the comprehensive score for time change with a set time continuity threshold; when the comprehensive score for time change is less than the time continuity threshold, the object's motion is considered to be stable. The temporal connection edge is assigned a weight of 1. When the comprehensive score of time change equals the time continuity threshold, the temporal connection edge is assigned a weight of 0.5. When the comprehensive score of time change is greater than the time continuity threshold, the continuity of object movement is considered to be weakened, and the temporal connection edge is assigned a weight of 0.1. For semantic connection edges, the semantic feature similarity between semantic nodes is calculated. The difference between the semantic feature vectors of each pair of semantic nodes is statistically averaged to obtain the semantic similarity value. The semantic connection edge weights are assigned according to the magnitude of the semantic similarity value. The weights of spatial connection edges, temporal connection edges, and semantic connection edges are normalized to ensure that the weights of the three types of connection edges are distributed within a uniform numerical range, preventing the weight of any dimension from being too large or too small, which would lead to an imbalance in the graph structure.

[0107] By unifying and integrating the sets of spatial nodes, temporal nodes, and semantic nodes, as well as the corresponding sets of spatial, temporal, and semantic connecting edges, a multi-view spatiotemporal graph is obtained. In the multi-view spatiotemporal graph, all nodes and connecting edges have unique index numbers and feature attributes, forming a three-dimensional relational topology that simultaneously covers the spatial, temporal, and semantic dimensions.

[0108] In this embodiment, the generation of the dynamic spatiotemporal map specifically includes:

[0109] The node feature vectors and corresponding connection weight parameters of all spatial nodes, temporal nodes and semantic nodes are extracted from the multi-view spatiotemporal map to form a node feature set and a connection weight set, which are used as input data for the recursive structural update process.

[0110] Perform difference calculation on the node feature vectors of the same object in the time node set at adjacent time steps to obtain the time feature change value, calculate the average amplitude of the time feature change value, and generate a set of time change amplitudes to characterize the dynamic change trend of the object in the time dimension;

[0111] Based on the set of spatial nodes, the feature difference between adjacent spatial nodes is calculated to generate a set of spatial feature changes; based on the set of semantic nodes, the semantic feature difference between adjacent semantic nodes is calculated to generate a set of semantic feature changes. The set of temporal change amplitude, the set of spatial feature changes, and the set of semantic feature changes are integrated to form a node feature change matrix, which is used to describe the comprehensive change state of nodes in multidimensional space.

[0112] The dynamic similarity value between nodes is calculated based on the node feature change matrix, and a dynamic weight adjustment coefficient matrix is ​​generated based on the dynamic similarity value. The connection weight parameters of spatial connection edges, temporal connection edges and semantic connection edges are updated using the dynamic weight adjustment coefficient matrix to form an updated set of connection weights.

[0113] The process of forming the updated connection weight set specifically includes: reading the node feature change matrix, where each row of the node feature change matrix corresponds to a node's comprehensive feature change vector in the spatial, temporal, and semantic dimensions; for any two nodes, calculating the feature difference in each dimension using their feature change vectors and taking the absolute value, summing the differences in the three dimensions to obtain the comprehensive feature difference value between nodes, then taking the reciprocal of the comprehensive feature difference value and performing normalization to obtain the dynamic similarity matrix between nodes; based on the dynamic similarity matrix, performing a linear proportional mapping on each similarity value to generate a dynamic weight adjustment coefficient matrix, where the linear proportional mapping function... The system outputs weight adjustment coefficients ranging from 0.9 to 1.0. When the dynamic similarity value is between the lower and upper thresholds, the output weight adjustment coefficient is between 0.5 and 0.9. When the dynamic similarity value is below the lower threshold, the output weight adjustment coefficient is between 0 and 0.5, forming a normalized dynamic weight adjustment coefficient matrix with a defined numerical range. This matrix is ​​then applied to the connection weight parameters of spatial, temporal, and semantic connections, respectively. Element-wise multiplication is performed on the original connection weight parameters of the three types of connections to obtain an updated set of connection weight parameters. Finally, the three types of updated weight parameters are merged into a unified updated set of connection weights.

[0114] Normalization is performed on the updated set of connection weights so that the sum of the weights of each node in all its connections remains one, generating a normalized set of connection weights to maintain the connection balance of the multi-view spatiotemporal graph.

[0115] Based on the normalized connection weight set, a weighted aggregation calculation is performed on each node, and the feature vectors of adjacent nodes are summed according to the normalized connection weights to generate a new node feature set, which serves as the updated node state description.

[0116] The new set of node features is input into the recursive update process of the dynamic spatiotemporal graph structure. Based on the changing trend of node features and the adjustment results of connection weights, the connection relationships between spatial nodes, temporal nodes and semantic nodes are updated topologically to generate the updated dynamic spatiotemporal graph.

[0117] In this embodiment, the generation of the preliminary behavioral state identification result specifically includes:

[0118] Extract the node feature vectors of all spatial nodes, temporal nodes, and semantic nodes, as well as the updated connection weights, from the dynamic spatiotemporal graph after recursive structural updates.

[0119] Based on connection weights, spatial feature weighted aggregation calculation is performed on each spatial node. The node feature vectors of adjacent spatial nodes are weighted and summed according to the connection weights of the spatial connection edges to generate spatial feature aggregation results. Temporal feature weighted aggregation calculation is performed on each temporal node. The feature vectors of adjacent temporal nodes are weighted and summed according to the connection weights of the temporal connection edges to generate temporal feature aggregation results. Semantic feature weighted aggregation calculation is performed on each semantic node. The feature vectors of adjacent semantic nodes are weighted and summed according to the connection weights of the semantic connection edges to generate semantic feature aggregation results.

[0120] The spatial feature aggregation results, temporal feature aggregation results, and semantic feature aggregation results are aligned in dimension and normalized according to node number to construct a multi-dimensional feature aggregation set;

[0121] Feature propagation computation is performed on the multidimensional feature aggregation set. Taking each node as the center, the features of its first-order and second-order adjacent nodes are passed and fused layer by layer to obtain the node feature propagation result set.

[0122] The generation of the node feature propagation result set specifically includes: reading the node feature vectors of all nodes from the multidimensional feature aggregation set, and simultaneously reading the connection weights of spatial, temporal, and semantic connection edges between nodes from the dynamic spatiotemporal graph; taking any node as the center node, retrieving its first-order and second-order neighbor node sets in the dynamic spatiotemporal graph, where the first-order neighbor node set refers to the set of nodes directly connected to the center node through a single connection edge, and the second-order neighbor node set refers to the set of nodes indirectly connected to the center node through two connection edges; for first-order neighbor nodes, performing a weighted summation of the node feature vectors of the neighbor nodes according to the connection weight parameters of the corresponding connection edges to generate a first-order propagation feature vector; for second-order neighbor nodes, multiplying the connection weight parameters on the two connection paths to obtain the path propagation weight. The node feature vectors of the second-order adjacent nodes are weighted and summed using the path propagation weights as coefficients to generate second-order propagation feature vectors. The first-order propagation feature vectors and second-order propagation feature vectors are then weighted and fused according to a proportional coefficient to generate the fused propagation feature vector of the central node. The fused propagation feature vector of the central node is then linearly combined with its original node feature vectors according to a normalized ratio to generate the updated feature vector of the central node. The above calculation process is repeated, and the same feature propagation operation is performed on all nodes in the multidimensional feature aggregation set in sequence to finally form a node feature propagation result set. In the node feature propagation result set, the feature vector of each node is fused with its own feature information and the feature information of its first-order and second-order adjacent nodes, which fully describes the dynamic interaction state of the node in the three dimensions of space, time and semantics.

[0123] Spatiotemporal semantic fusion computation is performed based on the set of node feature propagation results. A temporal constraint matrix is ​​constructed based on the feature change sequence between time nodes, a spatial constraint matrix is ​​constructed based on the relative positional relationship between spatial nodes, and a semantic constraint matrix is ​​constructed based on the semantic similarity distribution between semantic nodes. The three types of constraint matrices are then weighted and summed according to a set ratio to generate a fusion constraint matrix.

[0124] The generation of the fusion constraint matrix specifically includes: reading the node feature propagation result set, extracting the node feature vector of each time node in chronological order, calculating the feature change difference sequence between adjacent time nodes, converting the feature change difference sequence into a time change rate matrix, calculating the time similarity coefficient based on the change amplitude of each node pair in the time change rate matrix, and generating a time constraint matrix; for the spatial dimension, extracting the spatial coordinates of all spatial nodes from the node feature propagation result set, calculating the Euclidean distance value and relative direction angle between each spatial node pair, normalizing the Euclidean distance value to generate a spatial distance matrix, and combining it with the direction angle calculation... The directional consistency score matrix is ​​calculated, and a spatial constraint matrix is ​​generated based on the weighted result of the distance matrix and the directional consistency score matrix. For the semantic dimension, the semantic feature vectors of all semantic nodes are extracted from the node feature propagation result set, and the semantic similarity value between each semantic node pair is calculated. The semantic similarity value is obtained by measuring the cosine similarity of the semantic feature vectors, and a semantic stability matrix is ​​constructed based on the standard deviation of the similarity distribution. The weighted superposition result of the semantic similarity matrix and the semantic stability matrix is ​​used as the semantic constraint matrix. The temporal constraint matrix, spatial constraint matrix, and semantic constraint matrix are weighted and summed according to a set proportional coefficient to generate a fusion constraint matrix.

[0125] The fusion constraint matrix is ​​applied to the set of node feature propagation results, and a weighted mapping calculation is performed on the comprehensive feature vector of each node to generate a set of node behavior feature representations;

[0126] The generation of the node behavior feature representation set specifically includes: establishing a corresponding index between the fusion constraint matrix and the node feature propagation result set according to the node number, so that each row of the fusion constraint matrix corresponds one-to-one with a node in the node feature propagation result set; for the comprehensive feature vector of each node, according to the constraint weight components of the corresponding row in the fusion constraint matrix, performing a weighted product operation on the node feature vectors of all related nodes, and summing the weighted results under the same node number to obtain the weighted fusion feature vector of that node; combining the weighted fusion feature vectors of all nodes in numerical order to form the node behavior feature representation set; and performing normalization processing on each node feature vector in the node behavior feature representation set to form a normalized node behavior feature representation set.

[0127] The set of node behavior feature representations is input into a multilayer perceptron to perform behavior recognition inference, and the probability distribution vector of each object's behavior category and the corresponding node behavior state vector are obtained. The category label corresponding to the highest probability and the corresponding node behavior state vector are taken as the preliminary recognition result of the object's behavior state.

[0128] In this embodiment, the semantic self-correction specifically includes:

[0129] Using the set of node behavior state vectors in the preliminary behavior state identification result set as input, the difference between the set of node behavior state vectors and the set of node semantic vectors of the corresponding node in the dynamic spatiotemporal graph is calculated to generate a semantic deviation matrix, which is used to quantify the semantic difference between behavior state and semantic node.

[0130] Semantic clustering is performed based on the semantic deviation matrix. Similarity is aggregated according to the semantic difference between node behavior state vectors. Node behavior state vectors with similar semantic deviation features are divided into several semantic clustering groups to form a set of semantic clustering results. Each semantic clustering group corresponds to a class of semantically similar node behavior state vectors.

[0131] Based on the semantic clustering result set, perform latent cognitive correction, perform parameter correction operation on the corresponding node behavior state vector, and generate a latent cognitive correction vector set;

[0132] The generation of the latent cognitive correction vector set specifically includes: calculating the average semantic deviation value of all node behavior state vectors within the semantic cluster as the baseline semantic deviation within the group; calculating the difference between each node behavior state vector and the baseline semantic deviation within the group to obtain the node semantic correction offset; calculating the standard deviation of the semantic deviation values ​​within the cluster as the semantic deviation fluctuation amplitude; determining the correction coefficient based on the ratio of the semantic correction offset to the semantic deviation fluctuation amplitude to control the correction intensity; multiplying the node semantic correction offset by the correction coefficient to obtain the node correction increment vector, and then weighting and fusing it with the original node behavior state vector at a preset ratio to generate the corrected node behavior state vector; and summing all the corrected node behavior state vectors to form the latent cognitive correction vector set.

[0133] The semantic mapping parameters are updated by using the latent cognition correction vector set, and the corrected node behavior state vectors are remapped to the semantic space to form an updated semantic mapping parameter set.

[0134] The generation of the updated semantic mapping parameter set specifically includes: semantically aligning each corrected node behavior state vector in the latent cognitive correction vector set with the corresponding semantic node vector, calculating the average semantic difference between the two as the semantic mapping offset of the node; updating the original semantic mapping parameters according to the semantic mapping offset, and adjusting the semantic mapping parameters of each node proportionally; performing weighted fusion of the adjusted semantic mapping parameters and the node behavior state vector to express the mapping relationship of the node in the semantic space in a new parameter combination form; normalizing the mapping relationship of all nodes; summarizing the normalized mapping results of all nodes to form the updated semantic mapping parameter set. The semantic mapping parameters are a set of parameters used to establish the correspondence between the node behavior state vector and the semantic node vector, and are used to describe the position, direction and semantic association strength of the node in the semantic space.

[0135] Based on the updated semantic mapping parameter set, the semantic consistency score between the node behavior state vector and the semantic node is recalculated, and the semantic weight of the node behavior state vector is adjusted according to the semantic consistency score to generate a semantic self-correction result set.

[0136] The generation of the semantic self-correction result set specifically includes: for each node's behavior state vector, calling the corresponding semantic mapping parameters to calculate the mapped coordinate value of the node's behavior state vector in the semantic space; performing a difference operation between the mapped coordinate value and the corresponding semantic node vector to obtain a semantic deviation vector, and calculating the Euclidean distance of the semantic deviation vector as the initial semantic consistency score; performing normalization processing on the initial semantic consistency scores of all nodes to form a semantic consistency score set; adjusting the semantic weight parameters of each node according to its semantic consistency score: when the semantic consistency score is greater than 0.8, the semantic weight parameters are taken as 0.9 times the original value; when the semantic consistency score is between 0.5 and 0.8, the semantic weight parameters are taken as 0.7 times the original value; when the semantic consistency score is less than 0.5, the semantic weight parameters are taken as 0.5 times the original value; performing element-wise weighted fusion of the adjusted semantic weight parameters and the node's behavior state vector to generate a semantic correction vector; summing up the semantic correction vectors of all nodes to form a semantic self-correction result set.

[0137] The semantic self-correction result set is fed back to the semantic node attributes of the dynamic spatiotemporal graph, and the semantic feature information of the corresponding node is replaced, so as to realize semantic layer self-correction and behavioral semantic adaptive adjustment.

[0138] In this embodiment, the generation of the efficient spatiotemporal feature representation specifically includes:

[0139] Extract the semantic correction vector of each node and its corresponding time index parameter, and sort the semantic correction vectors by time according to the time index parameter to form a temporal semantic correction sequence;

[0140] Calculate the feature change rate between adjacent frames for the temporal semantic correction sequence. When the change rate exceeds a preset threshold, mark the corresponding frame as a key frame to form a key frame index set. Extract the semantic correction vector corresponding to the key frame to form a key frame feature set.

[0141] Spatiotemporal jump sampling is performed in the keyframe feature set with a fixed jump step size. The sampling interval is set to twice the average interval of the time index parameter to form a jump sampling feature sequence.

[0142] Multi-scale coding is performed on each feature vector in the skip sampling feature sequence. The inter-frame feature change rate is calculated on the temporal scale, and the spatial adjacency score is obtained on the spatial scale based on the spatial connection edges between nodes. The inter-frame feature change rate and the spatial adjacency score are weighted to generate a multi-scale spatiotemporal coding result set. The adjacency score is the coordinate distance value between two spatial nodes.

[0143] Skip attention weights are calculated on the multi-scale spatiotemporal coding result set. The inter-frame feature change rate and spatial adjacency score of each node are weighted to obtain the initial attention value, which is then normalized to form the skip attention weight set.

[0144] The multi-scale spatiotemporal coding result set is weighted and aggregated using a set of jump attention weights to generate sparse feature aggregation results, which are then used as efficient spatiotemporal feature representations.

[0145] In this embodiment, the final behavior recognition result is generated by using a multilayer perceptron and a long short-term memory network in series.

[0146] An AI-based real-time video behavior analysis system includes:

[0147] The video acquisition and preprocessing module is used to acquire continuous video stream data and perform frame segmentation, noise removal, illumination equalization and time synchronization processing to generate a standardized video frame sequence.

[0148] The multimodal feature extraction module is used to perform object detection and feature extraction on standardized video frame sequences to generate a multimodal feature set;

[0149] The spatiotemporal map construction module is used to construct multi-view spatiotemporal maps based on multimodal feature sets;

[0150] The recursive structure update module is used to perform recursive structure update processing on multi-view spatiotemporal graphs, dynamically adjust the connection relationships and connection weights between nodes, and generate dynamic spatiotemporal graphs.

[0151] The behavior reasoning module is used to perform feature aggregation and feature propagation on a dynamic spatiotemporal graph to generate preliminary identification results of the object's behavior state;

[0152] The semantic self-correction module is used to analyze the semantic bias of the initial identification results of behavioral states, perform semantic clustering and latent cognition correction, and update semantic mapping parameters to achieve semantic layer self-correction.

[0153] The spatiotemporal jump attention module is used to perform keyframe selection and jump fusion after semantic self-correction, complete sparse feature aggregation, and generate efficient spatiotemporal feature representation;

[0154] The behavior recognition module is used to generate the final behavior recognition result based on efficient spatiotemporal feature representation.

[0155] Example 1:

[0156] To verify the feasibility of this invention in practice, it was applied to a public safety monitoring scenario. This scenario involves hundreds of high-definition cameras continuously collecting video stream data from densely populated areas. The scenario presents challenges such as complex occlusion, significant lighting variations, diverse action types, and high similarity in individual behaviors. Traditional video behavior recognition algorithms often misjudge or miss behaviors in such complex environments, especially under conditions of multi-person interaction, frequent occlusion, and changing camera perspectives, failing to maintain continuous and stable recognition results. This invention, by introducing a multimodal feature fusion mechanism and a dynamic spatiotemporal graph structure, achieves high-precision modeling and real-time recognition of object behaviors in videos, effectively solving the problem of low recognition accuracy in complex scenarios using traditional methods.

[0157] In practical applications, the monitoring system first acquires continuous video streams and performs frame-level standardization processing to generate a standardized video frame sequence. The system then performs object detection and feature extraction on each frame to form a multimodal feature set that includes pose features, motion trajectory features, optical flow features, and semantic features. Subsequently, based on these features, the present invention constructs spatial nodes, temporal nodes, and semantic nodes, establishes spatial connection edges, temporal connection edges, and semantic connection edges, and forms a multi-view spatiotemporal graph. The system performs a recursive structural update on this graph, so that the connection relationships and weights between nodes can be dynamically adjusted according to real-time feature changes, thereby obtaining a dynamic spatiotemporal graph that reflects the behavioral evolution process.

[0158] Based on this, the system executes the behavior reasoning process, generating preliminary behavior state recognition results through feature aggregation and propagation between nodes. To address the semantic ambiguity and recognition bias issues in complex scenarios, the system further executes a semantic self-correction mechanism to perform cluster analysis and latent cognitive correction on the semantic bias of the preliminary recognition results. The corrected semantic mapping parameters enable the model to be more accurate in the next step of judgment. Subsequently, the system introduces a jump attention mechanism in the spatiotemporal dimension to select and fuse key frames, complete sparse feature aggregation, and obtain an efficient spatiotemporal feature representation. Based on this representation, the system outputs behavior recognition results in real time and can automatically issue alarms when abnormal behavior is detected.

[0159] To verify the performance of the present invention, it was compared with the traditional method. The comparison results are shown in Table 1.

[0160] Table 1. Performance Comparison of the Invention and Traditional Methods

[0161]

[0162] As can be seen from the comparison data in Table 1, the AI-based real-time video behavior analysis method proposed in this invention outperforms traditional algorithms in terms of recognition accuracy, temporal stability, and multi-target interference suppression.

[0163] Traditional convolutional neural network methods can only perform feature recognition on single-frame images and lack the ability to dynamically model information in the temporal dimension. Therefore, they are prone to misjudgment in scenarios with strong behavioral continuity. While traditional two-stream optical flow methods have certain advantages in action recognition, their ability to express semantic features is insufficient, making it difficult to accurately distinguish similar behaviors in complex scenes. This invention introduces a multimodal feature set and a dynamic spatiotemporal map structure, and comprehensively utilizes the synergistic information of posture features, motion trajectory features, optical flow features, and semantic features. This enables the model to establish dynamic associations in the spatial, temporal, and semantic dimensions simultaneously, thereby improving the overall recognition accuracy to 92.3%.

[0164] Furthermore, the semantic self-correction mechanism proposed in this invention can automatically detect recognition deviations and perform latent cognitive corrections, maintaining semantic consistency between consecutive frames. This improves the semantic consistency score from 0.72 in traditional methods to 0.91. The improvement in the temporal stability coefficient also indicates that the dynamic spatiotemporal graph and weight self-update mechanism effectively avoid node feature drift and temporal breakage problems. The keyframe response latency is reduced from over 200ms in traditional algorithms to 118ms, demonstrating that this invention has significant advantages in real-time performance and can meet the online monitoring needs of complex scenarios. At the same time, the multi-target interference misjudgment rate is reduced from over 13% to 6.6%, verifying the effectiveness of the jump attention mechanism in eliminating feature interference between objects.

[0165] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A real-time video behavior analysis method based on AI, characterized in that, Includes the following steps: Acquire continuous video stream data and perform preprocessing to generate a standardized video frame sequence; For each frame in a standardized video frame sequence, object detection and feature extraction are performed to generate a multimodal feature set containing object pose features, motion trajectory features, optical flow features, and semantic features; Based on a multimodal feature set, spatial nodes, temporal nodes, and semantic nodes are constructed, and spatial connection edges, temporal connection edges, and semantic connection edges are established respectively to form a multi-view spatiotemporal graph; Perform recursive structural update processing on the multi-view spatiotemporal graph, dynamically adjust the connection relationships and connection weights between nodes, and generate a dynamic spatiotemporal graph. Behavioral reasoning is performed on a dynamic spatiotemporal graph, and the spatiotemporal features of each node are aggregated and feature propagation is performed to generate preliminary identification results of the object's behavioral state. The semantic bias of the initial behavioral state identification results is analyzed, semantic clustering and latent cognition correction are performed, and the semantic mapping parameters are updated to perform semantic self-correction. After semantic self-correction, spatiotemporal jump attention processing is performed to select keyframes and perform jump-style fusion of temporal features, complete sparse feature aggregation, and generate efficient spatiotemporal feature representation. The final behavior recognition result is generated based on efficient spatiotemporal feature representation; The semantic self-correction specifically includes: Using the set of node behavior state vectors in the preliminary behavior state recognition result set as input, calculate the difference between the set of node behavior state vectors and the set of node semantic vectors of the corresponding node in the dynamic spatiotemporal map, and generate a semantic deviation matrix. Semantic clustering is performed based on the semantic deviation matrix. Similarity is aggregated according to the semantic difference between node behavior state vectors. Node behavior state vectors with similar semantic deviation features are divided into several semantic clustering groups to form a set of semantic clustering results. Based on the semantic clustering result set, perform latent cognitive correction, perform parameter correction operation on the corresponding node behavior state vector, and generate a latent cognitive correction vector set; The semantic mapping parameters are updated by using the latent cognition correction vector set, and the corrected node behavior state vectors are remapped to the semantic space to form an updated semantic mapping parameter set. Based on the updated semantic mapping parameter set, the semantic consistency score between the node behavior state vector and the semantic node is recalculated, and the semantic weight of the node behavior state vector is adjusted according to the semantic consistency score to generate a semantic self-correction result set. The semantic self-correction result set is fed back to the semantic node attributes of the dynamic spatiotemporal graph, and the semantic feature information of the corresponding node is replaced, so as to realize semantic layer self-correction and behavioral semantic adaptive adjustment. The generation of the efficient spatiotemporal feature representation specifically includes: Extract the semantic correction vector of each node and its corresponding time index parameter, and sort the semantic correction vectors by time according to the time index parameter to form a temporal semantic correction sequence; Calculate the feature change rate between adjacent frames for the temporal semantic correction sequence. When the change rate exceeds a preset threshold, mark the corresponding frame as a key frame to form a key frame index set. Extract the semantic correction vector corresponding to the key frame to form a key frame feature set. Spatiotemporal jump sampling is performed in the keyframe feature set with a fixed jump step size. The sampling interval is set to twice the average interval of the time index parameter to form a jump sampling feature sequence. Multi-scale coding is performed on each feature vector in the skip sampling feature sequence. The inter-frame feature change rate is calculated on the temporal scale, and the spatial adjacency score is obtained on the spatial scale based on the spatial connection edges between nodes. The inter-frame feature change rate and the spatial adjacency score are weighted to generate a set of multi-scale spatiotemporal coding results. Skip attention weights are calculated on the multi-scale spatiotemporal coding result set. The inter-frame feature change rate and spatial adjacency score of each node are weighted to obtain the initial attention value, which is then normalized to form the skip attention weight set. The multi-scale spatiotemporal coding result set is weighted and aggregated using a set of jump attention weights to generate sparse feature aggregation results, which are then used as efficient spatiotemporal feature representations.

2. The AI-based real-time video behavior analysis method according to claim 1, characterized in that, The preprocessing includes frame segmentation, noise removal, illumination equalization, time synchronization, and grayscale normalization.

3. The AI-based real-time video behavior analysis method according to claim 1, characterized in that, The generation of the multimodal feature set specifically includes: Perform object detection processing on each frame of the standardized video frame sequence, extract candidate regions containing objects, and generate a set of object bounding boxes; Perform pose feature extraction on the image region corresponding to the bounding box of the object, extract the key point position features, limb angle features and pose structure features of the object, and generate the object pose feature vector; In adjacent frames, object matching and temporal association are performed based on object identifier numbers. The displacement change information of the object is calculated, and the velocity features, trajectory direction features and acceleration features of the object are extracted to generate motion trajectory feature vectors. Optical flow feature extraction is performed on a continuous frame sequence to calculate the pixel motion trend within the object region, extract local dynamic texture features and orientation gradient features, and generate optical flow feature vectors. Semantic encoding is performed on the bounding box region of the object, semantic feature extraction is performed, and semantic feature vectors are generated; The pose feature vector, motion trajectory feature vector, optical flow feature vector, and semantic feature vector of the same object are synchronized and concatenated in chronological order to generate the object's multimodal feature vector; The multimodal feature vectors of all objects are integrated to form a multimodal feature set.

4. The AI-based real-time video behavior analysis method according to claim 1, characterized in that, The generation of the multi-view spatiotemporal map specifically includes: The multimodal feature set extracted from the standardized video frame sequence is organized in chronological order. The multimodal features of all objects in the same frame are aggregated. A spatial node set is constructed based on the coordinate position distribution of the objects in the image plane. The pose feature vector, motion trajectory feature vector, optical flow feature vector, and semantic feature vector of the objects are bound to the spatial nodes. Perform matching association on spatial nodes of the same object between consecutive frames, establish a set of time nodes, and form a time node link according to the frame index order; Perform clustering on the semantic feature vectors of all objects to generate a set of semantic nodes; Based on the correspondence between the spatial node set, the temporal node set, and the semantic node set, a spatial connection edge set, a temporal connection edge set, and a semantic connection edge set are established respectively. Assign connection weights to spatial, temporal, and semantic connection edges; By unifying and integrating the sets of spatial nodes, temporal nodes, and semantic nodes, as well as the corresponding sets of spatial, temporal, and semantic connecting edges, a multi-view spatiotemporal graph is obtained.

5. The AI-based real-time video behavior analysis method according to claim 1, characterized in that, The generation of the dynamic spatiotemporal map specifically includes: Extract the node feature vectors and corresponding connection weight parameters of all spatial nodes, temporal nodes and semantic nodes from the multi-view spatiotemporal graph to form a node feature set and a connection weight set. Perform difference calculation on the node feature vectors of the same object in the time node set at adjacent time steps to obtain the time feature change value, calculate the average amplitude of the time feature change value, and generate a set of time change amplitudes; The feature difference between adjacent spatial nodes is calculated based on the set of spatial nodes to generate a set of spatial feature changes; the semantic feature difference between adjacent semantic nodes is calculated based on the set of semantic nodes to generate a set of semantic feature changes; the set of temporal change magnitude, the set of spatial feature changes, and the set of semantic feature changes are integrated to form a node feature change matrix. The dynamic similarity value between nodes is calculated based on the node feature change matrix, and a dynamic weight adjustment coefficient matrix is ​​generated based on the dynamic similarity value. The connection weight parameters of spatial connection edges, temporal connection edges and semantic connection edges are updated using the dynamic weight adjustment coefficient matrix to form an updated set of connection weights. The updated set of connection weights is normalized to generate a normalized set of connection weights. Based on the normalized connection weight set, a weighted aggregation calculation is performed on each node, and the feature vectors of adjacent nodes are weighted and summed according to the normalized connection weights to generate a new node feature set. The new set of node features is input into the recursive update process of the dynamic spatiotemporal graph structure. Based on the changing trend of node features and the adjustment results of connection weights, the connection relationships between spatial nodes, temporal nodes and semantic nodes are updated topologically to generate the updated dynamic spatiotemporal graph.

6. The AI-based real-time video behavior analysis method according to claim 1, characterized in that, The generation of the preliminary behavioral state identification result specifically includes: Extract the node feature vectors of all spatial nodes, temporal nodes, and semantic nodes, as well as the updated connection weights, from the dynamic spatiotemporal graph after recursive structural updates. Based on the connection weights, spatial feature weighted aggregation calculations are performed on each spatial node to generate spatial feature aggregation results; temporal feature weighted aggregation calculations are performed on each temporal node to generate temporal feature aggregation results; and semantic feature weighted aggregation calculations are performed on each semantic node to generate semantic feature aggregation results. The spatial feature aggregation results, temporal feature aggregation results, and semantic feature aggregation results are aligned in dimension and normalized according to node number to construct a multi-dimensional feature aggregation set; Feature propagation computation is performed on the multidimensional feature aggregation set. Taking each node as the center, the features of its first-order adjacent nodes and second-order adjacent nodes are passed and fused layer by layer to obtain the node feature propagation result set. Spatiotemporal semantic fusion computation is performed based on the set of node feature propagation results. A temporal constraint matrix is ​​constructed based on the feature change sequence between time nodes, a spatial constraint matrix is ​​constructed based on the relative positional relationship between spatial nodes, and a semantic constraint matrix is ​​constructed based on the semantic similarity distribution between semantic nodes. The three types of constraint matrices are then weighted and summed according to a set ratio to generate a fusion constraint matrix. The fusion constraint matrix is ​​applied to the set of node feature propagation results, and a weighted mapping calculation is performed on the comprehensive feature vector of each node to generate a set of node behavior feature representations; The set of node behavior feature representations is input into a multilayer perceptron to perform behavior recognition inference, and the probability distribution vector of each object's behavior category and the corresponding node behavior state vector are obtained. The category label corresponding to the highest probability and the corresponding node behavior state vector are taken as the preliminary recognition result of the object's behavior state.

7. The AI-based real-time video behavior analysis method according to claim 1, characterized in that, The final behavior recognition result is generated by using a multilayer perceptron and a long short-term memory network in series.

8. An AI-based real-time video behavior analysis system, executing the AI-based real-time video behavior analysis method according to any one of claims 1 to 7, characterized in that, include: The video acquisition and preprocessing module is used to acquire continuous video stream data and perform frame segmentation, noise removal, illumination equalization and time synchronization processing to generate a standardized video frame sequence. The multimodal feature extraction module is used to perform object detection and feature extraction on standardized video frame sequences to generate a multimodal feature set; The spatiotemporal map construction module is used to construct multi-view spatiotemporal maps based on multimodal feature sets; The recursive structure update module is used to perform recursive structure update processing on multi-view spatiotemporal graphs, dynamically adjust the connection relationships and connection weights between nodes, and generate dynamic spatiotemporal graphs. The behavior reasoning module is used to perform feature aggregation and feature propagation on a dynamic spatiotemporal graph to generate preliminary identification results of the object's behavior state; The semantic self-correction module is used to analyze the semantic bias of the initial identification results of behavioral states, perform semantic clustering and latent cognition correction, and update semantic mapping parameters to achieve semantic layer self-correction. The spatiotemporal jump attention module is used to perform keyframe selection and jump fusion after semantic self-correction, complete sparse feature aggregation, and generate efficient spatiotemporal feature representation; The behavior recognition module is used to generate the final behavior recognition result based on efficient spatiotemporal feature representation; The implementation of semantic layer self-correction specifically includes: Using the set of node behavior state vectors in the preliminary behavior state recognition result set as input, calculate the difference between the set of node behavior state vectors and the set of node semantic vectors of the corresponding node in the dynamic spatiotemporal map, and generate a semantic deviation matrix. Semantic clustering is performed based on the semantic deviation matrix. Similarity is aggregated according to the semantic difference between node behavior state vectors. Node behavior state vectors with similar semantic deviation features are divided into several semantic clustering groups to form a set of semantic clustering results. Based on the semantic clustering result set, perform latent cognitive correction, perform parameter correction operation on the corresponding node behavior state vector, and generate a latent cognitive correction vector set; The semantic mapping parameters are updated by using the latent cognition correction vector set, and the corrected node behavior state vectors are remapped to the semantic space to form an updated semantic mapping parameter set. Based on the updated semantic mapping parameter set, the semantic consistency score between the node behavior state vector and the semantic node is recalculated, and the semantic weight of the node behavior state vector is adjusted according to the semantic consistency score to generate a semantic self-correction result set. The semantic self-correction result set is fed back to the semantic node attributes of the dynamic spatiotemporal graph, and the semantic feature information of the corresponding node is replaced, so as to realize semantic layer self-correction and behavioral semantic adaptive adjustment. The generation of the efficient spatiotemporal feature representation specifically includes: Extract the semantic correction vector of each node and its corresponding time index parameter, and sort the semantic correction vectors by time according to the time index parameter to form a temporal semantic correction sequence; Calculate the feature change rate between adjacent frames for the temporal semantic correction sequence. When the change rate exceeds a preset threshold, mark the corresponding frame as a key frame to form a key frame index set. Extract the semantic correction vector corresponding to the key frame to form a key frame feature set. Spatiotemporal jump sampling is performed in the keyframe feature set with a fixed jump step size. The sampling interval is set to twice the average interval of the time index parameter to form a jump sampling feature sequence. Multi-scale coding is performed on each feature vector in the skip sampling feature sequence. The inter-frame feature change rate is calculated on the temporal scale, and the spatial adjacency score is obtained on the spatial scale based on the spatial connection edges between nodes. The inter-frame feature change rate and the spatial adjacency score are weighted to generate a set of multi-scale spatiotemporal coding results. Skip attention weights are calculated on the multi-scale spatiotemporal coding result set. The inter-frame feature change rate and spatial adjacency score of each node are weighted to obtain the initial attention value, which is then normalized to form the skip attention weight set. The multi-scale spatiotemporal coding result set is weighted and aggregated using a set of jump attention weights to generate sparse feature aggregation results, which are then used as efficient spatiotemporal feature representations.

Images