A behavior recognition and abnormality early warning method and system

Abstract

The application discloses a behavior recognition and abnormality early warning method and system. The method comprises the following steps: acquiring a monitoring video stream and preprocessing; extracting skeleton features and visual features in parallel, generating space attention guide information by the skeleton features to modulate the extraction of the visual features, fusing the features to recognize behaviors and output structured information; extracting visual tokens, sending the visual tokens to the cloud together with scene context text information after compression; decoding by the cloud, performing cross-modal fusion reasoning by a large multi-modal model, and outputting an analysis result containing a structured evidence chain; and generating and pushing early warning information when an abnormality is determined. The method and system are accurate, efficient, interpretable, and suitable for complex monitoring scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method and system for behavior recognition and anomaly warning. Background Technology

[0002] In higher education settings, lecture halls, serving as venues for large-scale lectures, centralized examinations, and academic reports, present significant challenges for classroom management and examination discipline due to their large size, high student density, and the resulting obstruction of views caused by the tiered seating arrangement. Traditional management methods, relying primarily on visual inspections by teachers or basic security monitoring, are inefficient and ill-suited to the sophisticated monitoring needs of complex scenarios.

[0003] With the development of intelligent technology, automated monitoring solutions using computer vision technology have become a research and application trend. However, existing solutions still have many key shortcomings and are difficult to adapt to the complex scenario requirements of lecture halls.

[0004] 1. Lack of semantic understanding, unable to determine behavioral attributes based on context: Mainstream solutions, such as models based on convolutional neural networks (CNNs), are essentially "action classifiers," capable only of recognizing pixel change patterns in images, but unable to deeply understand the true semantics of actions in specific teaching scenarios. For example, such models can capture the action features of "passing an object with a hand," but cannot distinguish whether the action is a violation like "passing answers during an exam" or a compliant action like "passing materials during a class discussion." The core reason is that existing models do not introduce and integrate "contextual information" (such as course type, whether an exam is in progress, time information, etc.), lacking cross-modal fusion reasoning capabilities, leading to biased behavior judgments.

[0005] 2. Poor adaptability to complex environments and insufficient accuracy in recognizing key behaviors: Some solutions introduce visual attention mechanisms (such as Transformer) to improve scene adaptability, attempting to enable the model to autonomously focus on key areas of the image. However, in environments with high density of people and severe mutual occlusion in lecture halls, attention mechanisms that rely solely on pixel information are easily distracted and cannot stably and accurately lock onto key body parts such as the head and hands of specific students. Furthermore, they cannot effectively capture subtle interactive behaviors such as "using a mobile phone" or "cheating with one's head down," resulting in recognition accuracy and robustness that cannot meet actual regulatory needs.

[0006] 3. The model is "black box" in nature, lacking interpretability and traceability: Most existing anomaly warning solutions only output behavioral categories such as "suspected violation" and their confidence levels, without providing any basis for judgment. When managers receive a warning, they cannot know which video clip, which area of ​​the screen, or which behavioral characteristics the system used to make the judgment, nor can they trace the reasoning logic. This results in low credibility of the warning results, making it difficult to use as an effective basis for management decisions and limiting the practical application of the technology.

[0007] 4. The system architecture is poorly designed, making it difficult to balance real-time performance and accuracy: From the perspective of deployment architecture, if all high-definition video streams are continuously uploaded to the cloud for in-depth analysis, it will generate huge network bandwidth consumption and operating costs; if all processing is completed locally in the classroom (on the device side), it will be limited by the computing resources of the device side, making it impossible to deploy large models that require multimodal deep fusion and complex reasoning, thus making it impossible to achieve high-precision semantic understanding and anomaly detection, forming an inherent contradiction between "accuracy and efficiency".

[0008] In summary, existing technologies for behavior recognition and anomaly warning in tiered classrooms generally suffer from deficiencies such as insufficient semantic understanding, weak anti-interference capability in complex environments, lack of interpretability of warning results, and poor system architecture adaptability. They cannot simultaneously meet the actual needs of "accurately identifying key behaviors in occluded environments, achieving deep reasoning by integrating scene context, providing a visual and traceable evidence chain, and balancing real-time performance and deployment costs".

[0009] Therefore, it is urgent to propose a behavior recognition and anomaly warning method that is adapted to the complex scenario of lecture halls and has high precision, strong robustness, interpretability and efficient architecture. This is particularly necessary to solve the problems of insufficient semantic understanding, weak anti-interference ability in complex environments, lack of interpretability of warning results, poor system architecture adaptability and inability to meet actual regulatory needs in this scenario. Summary of the Invention

[0010] The purpose of this invention is to provide a behavior recognition and anomaly warning method and system to address the problems mentioned above, such as insufficient semantic understanding in complex scenarios, weak anti-interference ability in complex environments, lack of interpretability of warning results, and poor system architecture adaptability.

[0011] The technical solution adopted in this invention is as follows:

[0012] A behavior recognition and anomaly warning method includes the following steps:

[0013] S1. Video Acquisition and Preprocessing: Acquire the monitoring video stream of the monitored area, and perform frame extraction and image quality enhancement processing to obtain a preprocessed video frame sequence;

[0014] S2. Multimodal Behavior Recognition: For the video frame sequence, extract in parallel skeleton features to characterize human posture and motion patterns, as well as visual features to characterize objects and scene context in the scene; use the skeleton features to generate spatial attention guidance information to modulate the extraction and analysis process of the visual features; fuse the modulated visual features and the skeleton features, and identify the target's behavior category based on the fused features to obtain the identified behavior, and output structured behavior information containing the identified behavior;

[0015] S3. Visual evidence compression and transmission: Based on the structured behavioral information, a set of visual tokens for characterizing the core visual content of the identified behavior are extracted from the associated video frame sequence segments, and the visual tokens are compressed and encoded; the compressed visual token data and the scene context text information describing the current scene state are sent to the cloud server together.

[0016] S4. Cloud-based Explainable Anomaly Reasoning: On a cloud server, the visual token data and scene context text information are decoded and reconstructed, and then input into a large-scale multimodal model. Through the cross-modal attention mechanism within the large-scale multimodal model, deep fusion and joint reasoning are performed to output an analysis result containing anomaly judgment conclusions and a structured evidence chain, wherein the structured evidence chain is used to explain the basis for the judgment conclusions.

[0017] S5. Warning Generation and Push: Based on the analysis results, if an anomaly is determined, a warning message will be automatically generated and pushed.

[0018] Furthermore, step S2 specifically includes:

[0019] S21. Skeleton Feature Extraction: Human skeleton key point detection is performed on the input video frame sequence to obtain a human skeleton key point sequence; the human skeleton key point sequence is constructed into a spatiotemporal graph and processed using a graph convolutional network to extract the skeleton features.

[0020] S22. Visual feature extraction: The video frame sequence is processed using a temporal visual transformer network to extract the visual features;

[0021] S23. Skeleton-guided attention fusion: A spatial attention mask is generated using the extracted skeleton features; the spatial attention mask is integrated into the visual feature extraction process to guide computational resources to focus on the region of interest defined by the key points of the human skeleton.

[0022] S24. Behavior classification: The attention-guided visual features are fused with the skeleton features, and the fused features are classified to obtain the identified behaviors. The structured behavior information containing the identified behaviors is then output.

[0023] Furthermore, in step S23, a spatial attention mask is generated using the extracted skeleton features. Specifically, a Gaussian response map centered on each human skeleton key point is generated in the visual feature map space based on the image coordinates of the key points of the human skeleton. After fusing and performing morphological operations on the Gaussian response maps of multiple people, the data is normalized to obtain the spatial attention mask matrix.

[0024] Furthermore, in step S23, the spatial attention mask is integrated into the visual feature extraction process, specifically by modulating the self-attention calculation of the visual features. The modulation causes the computing resources to be tilted toward the human skeleton key point association region indicated by the spatial attention mask.

[0025] Furthermore, in step S24, before fusing the attention-guided visual features with the skeleton features, a step of temporal attention enhancement is included for the visual features, wherein the temporal attention enhancement adopts a second-order temporal attention mechanism.

[0026] Furthermore, step S3 specifically includes:

[0027] S31. Visual token extraction: Locate the time window in which the identified behavior occurs, and select M of the most representative feature vectors from the visual features of the video frame sequence segment corresponding to the time window as visual tokens to represent the core visual content of the identified behavior; M is a preset positive integer.

[0028] S32. End-side compression coding: The M visual tokens are compressed using a pre-trained vector quantization codebook to obtain the corresponding codeword index sequence;

[0029] S33. Data transmission: Encapsulate the codeword index sequence and the scene context text information and send them to the cloud server.

[0030] Furthermore, step S4 specifically includes:

[0031] S41. Data Decoding: Receive and parse the data from the visual evidence compression and transmission steps, and reconstruct the visual token using the same vector quantization codebook based on the codeword index sequence; simultaneously, convert the scene context text information into a text token sequence.

[0032] S42, Cross-modal deep fusion: The visual token and text token sequences are input into the fusion module of the large multimodal model; the fusion module enables the visual token and text token to perform multiple rounds of bidirectional interaction and feature updates through its internal cross-modal attention layer;

[0033] S43. Generate interpretable results: Based on the features after deep fusion, perform anomaly judgment and simultaneously generate the structured evidence chain; the structured evidence chain includes at least the anomaly level, anomaly time window, a list of key visual token identifiers, and natural language reasoning text.

[0034] Furthermore, in step S42, the cross-modal attention layer adopts an alternating update strategy, specifically: in odd-numbered layers, visual tokens are used as queries, and text tokens are used as keys and values; in even-numbered layers, text tokens are used as queries, and visual tokens are used as keys and values.

[0035] Furthermore, step S5 specifically includes:

[0036] S51. Warning Judgment and Generation: Parse the anomaly judgment conclusion and structured evidence chain in the analysis results, and generate a warning message of the corresponding level according to the anomaly level. The warning message integrates the identified behavior, target location, anomaly time window and evidence summary.

[0037] S52. Warning push: The warning message is pushed to one or more designated management terminals in real time through a preset communication interface.

[0038] This invention also discloses a behavior recognition and anomaly early warning system, comprising: a video acquisition and preprocessing module for acquiring monitoring video streams of a monitored area and performing frame extraction and image quality enhancement processing; a behavior recognition module connected to the video acquisition and preprocessing module for extracting skeleton features and visual features in parallel from the processed video frame sequence, generating spatial attention guidance information using the skeleton features to modulate the extraction and analysis process of the visual features, and fusing the modulated visual features and the skeleton features to identify the behavior category of the target and output structured behavior information; a visual token processing and communication module connected to the behavior recognition module for extracting and compressing visual tokens based on the structured behavior information, and sending the compressed visual token data and scene context text information to the cloud; a cloud multimodal analysis module deployed on a cloud server for decoding and reconstructing the received visual token data and scene context text information, and performing cross-modal deep fusion reasoning through a built-in large-scale multimodal model to output analysis results including anomaly judgment conclusions and structured evidence chains; and an early warning service module connected to the cloud multimodal analysis module for generating and pushing early warning information based on the analysis results.

[0039] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0040] To address the issues of low accuracy in recognizing key behaviors and difficulty focusing on critical parts (head, hands) in crowded and heavily occluded environments like lecture halls, this paper proposes a core mechanism of "skeleton feature-guided visual feature extraction." This mechanism utilizes skeletal information robust to human occlusion to generate spatial attention guidance signals, forcing computational resources to focus on key parts of the human body and core motion areas. This significantly improves the accuracy and robustness of behavior recognition in complex occluded environments.

[0041] To address the shortcomings of existing technologies in semantic understanding, which can only recognize actions but cannot determine behavioral attributes (such as "passing answers in an exam" versus "passing materials in class"), an edge-cloud collaborative architecture is adopted. This architecture involves "extracting and compressing visual tokens, and then uploading them along with contextual text information (course type, exam status, etc.) to the cloud for multimodal reasoning." A large-scale multimodal model (LMM) is deployed in the cloud to achieve deep fusion and joint reasoning of visual evidence and textual descriptions, thereby realizing a deep understanding of behavioral semantics and contextualized judgment.

[0042] To address the problem that the "black box" nature of existing models leads to unreliable early warning results and a lack of judgment criteria, this paper mandates that the model output "analysis results that include anomaly judgment conclusions and structured evidence chains." This provides each early warning with a traceable time window, key visual features, location information, and natural language reasoning logic. Combined with the early warning push mechanism, the judgment criteria are clearly presented, making the judgment process transparent and traceable, fundamentally improving the credibility and usability of early warning results.

[0043] To address the inherent contradiction between accuracy and efficiency in existing system architectures, the aforementioned edge-cloud collaborative design achieves high-precision behavior recognition and efficient compression of visual evidence on the edge (transmitting only core visual information of the behavior, significantly reducing the bandwidth consumption of high-definition video streams), while complex multimodal reasoning is performed in the cloud. This approach ensures the depth of analysis while greatly reducing network bandwidth requirements, achieving a balance between "lightweight processing on the edge to ensure real-time performance and powerful computing power in the cloud to support complex reasoning and ensure accuracy."

[0044] Ultimately, it achieves high-precision, robust, interpretable, and efficient behavior recognition and anomaly warning in complex scenarios such as lecture halls, fully adapting to the refined supervision needs of higher education scenarios and providing managers with transparent and credible decision-making basis. Attached Figure Description

[0045] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0047] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0048] Example 1

[0049] This invention discloses a behavior recognition and anomaly warning method, which can be widely applied to various scenarios with dense crowds, clear behavior monitoring needs, and complex visual environments, including but not limited to teaching scenarios (lecture halls, large auditoriums, open course classrooms, etc.), public venue scenarios (libraries, conference centers, sports stadiums, etc.), and security monitoring scenarios (station waiting halls, shopping mall atriums, industrial production workshops, etc.). To clearly illustrate the technical solution, this invention uses a lecture hall in a teaching scenario as an example to describe the specific implementation process of the invention in detail.

[0050] like Figure 1 As shown, this invention discloses a behavior recognition and anomaly warning method, comprising the following steps:

[0051] S1. Video Acquisition and Preprocessing: Acquire the monitoring video stream of the monitored area, and perform frame extraction and image quality enhancement processing to obtain a preprocessed video frame sequence; the monitored area is a lecture hall.

[0052] S2. Multimodal Behavior Recognition: For the video frame sequence, extract in parallel skeleton features to characterize human posture and motion patterns, as well as visual features to characterize objects and scene context in the scene; use the skeleton features to generate spatial attention guidance information to modulate the extraction and analysis process of the visual features; fuse the modulated visual features and the skeleton features, and identify the target's behavior category based on the fused features to obtain the identified behavior, and output structured behavior information containing the identified behavior;

[0053] S3. Visual Evidence Compression and Transmission: Based on the structured behavioral information, a set of visual tokens representing the core visual content of the identified behavior are extracted from the associated video frame sequence segments, and the visual tokens are compressed and encoded; the compressed visual token data and scene context text information describing the current scene state are sent to the cloud server; the scene context includes one or more of the following: course status, time information, subject information, and teacher location; the course status includes regular teaching, in-class quizzes, and final exams.

[0054] S4. Cloud-based Explainable Anomaly Reasoning: On a cloud server, after decoding and reconstructing the visual token data and scene context text information, the data is input into a large multimodal model (LMM). Through the cross-modal attention mechanism within the large multimodal model, deep fusion and joint reasoning are performed to output an analysis result containing anomaly judgment conclusions and a structured evidence chain, which is used to explain the basis of the judgment conclusions.

[0055] S5. Warning Generation and Push: Based on the analysis results, if an anomaly is determined, a warning message will be automatically generated and pushed.

[0056] To address the issues of low accuracy in recognizing key behaviors and difficulty focusing on critical parts (head, hands) in crowded and heavily occluded environments like lecture halls, this paper proposes a core mechanism of "skeleton feature-guided visual feature extraction." This mechanism utilizes skeletal information robust to human occlusion to generate spatial attention guidance signals, forcing computational resources to focus on key parts of the human body and core motion areas. This significantly improves the accuracy and robustness of behavior recognition in complex occluded environments.

[0057] To address the shortcomings of existing technologies in semantic understanding, which can only recognize actions but cannot determine behavioral attributes (such as "passing answers in an exam" versus "passing materials in class"), an edge-cloud collaborative architecture is adopted. This architecture involves "extracting and compressing visual tokens, and then uploading them along with contextual text information (course type, exam status, etc.) to the cloud for multimodal reasoning." A large-scale multimodal model (LMM) is deployed in the cloud to achieve deep fusion and joint reasoning of visual evidence and textual descriptions, thereby realizing a deep understanding of behavioral semantics and contextualized judgment.

[0058] To address the problem that the "black box" nature of existing models leads to unreliable early warning results and a lack of judgment criteria, this paper mandates that the model output "analysis results that include anomaly judgment conclusions and structured evidence chains." This provides each early warning with a traceable time window, key visual features, location information, and natural language reasoning logic. Combined with the early warning push mechanism, the judgment criteria are clearly presented, making the judgment process transparent and traceable, fundamentally improving the credibility and usability of early warning results.

[0059] To address the inherent contradiction between accuracy and efficiency in existing system architectures, the aforementioned edge-cloud collaborative design achieves high-precision behavior recognition and efficient compression of visual evidence on the edge (transmitting only core visual information of the behavior, significantly reducing the bandwidth consumption of high-definition video streams), while complex multimodal reasoning is performed in the cloud. This approach ensures the depth of analysis while greatly reducing network bandwidth requirements, achieving a balance between "lightweight processing on the edge to ensure real-time performance and powerful computing power in the cloud to support complex reasoning and ensure accuracy."

[0060] Ultimately, it achieves high-precision, robust, interpretable, and efficient behavior recognition and anomaly warning in complex scenarios such as lecture halls, fully adapting to the refined supervision needs of higher education scenarios and providing managers with transparent and credible decision-making basis.

[0061] Furthermore, in step S1, the image quality enhancement processing includes denoising, illumination correction, and histogram equalization; the monitoring video stream is acquired by multiple wide-angle high-definition cameras deployed in the front, back, and middle areas of the lecture hall to achieve blind-spot-free monitoring; the video frame extraction frame rate is 25fps, and the denoising uses a median filtering algorithm.

[0062] Furthermore, step S2 specifically includes:

[0063] S21. Skeleton Feature Extraction: Human skeleton keypoint detection is performed on the input video frame sequence to obtain a human skeleton keypoint sequence; the human skeleton keypoint sequence is constructed into a spatiotemporal graph and processed using a graph convolutional network (GCN) to extract the skeleton features.

[0064] S22. Visual Feature Extraction: The video frame sequence is processed using a temporal visual transformer network to extract the visual features; the temporal visual transformer network is any one of the SwingTransformer model, 3D-CNN and self-attention combination model;

[0065] S23. Skeleton-guided attention fusion: A spatial attention mask is generated using the skeleton features extracted in step S21; the spatial attention mask is integrated into the visual feature extraction process in step S22 to guide computational resources to focus on the region of interest defined by the key points of the human skeleton.

[0066] S24. Behavior classification: The attention-guided visual features are fused with the skeleton features, and the fused features are classified to obtain the identified behaviors. The structured behavior information containing the identified behaviors is then output.

[0067] This invention utilizes graph convolutional networks to process spatiotemporal graphs to extract skeleton features, making the features more robust to changes in lighting, clothing, and occlusion. Visual features are extracted through a temporal visual transformer network, which fully captures the spatiotemporal information of appearance and scene. The skeleton features guide the visual feature extraction process, ensuring that the model's computational resources are effectively directed to key areas of the human skeleton, thus laying a solid feature foundation for subsequent high-precision classification.

[0068] Furthermore, in step S23, a spatial attention mask is generated using the skeleton features extracted in step S21. Specifically, based on the image coordinates of the key points of the human skeleton, a Gaussian response map centered on each key point of the human skeleton is generated in the visual feature map space. After fusing and performing morphological operations on the Gaussian response maps of multiple people, the spatial attention mask matrix is ​​obtained by normalization.

[0069] This process transforms abstract skeleton features into a continuous-value mask matrix aligned with the visual feature map space, making the skeleton's guidance of visual attention operable and quantifiable. This specific method ensures that the generated mask accurately covers potential interaction regions near keypoints, providing reliable input for subsequent precise attention modulation.

[0070] In step S23, the specific process of generating a spatial attention mask using the skeleton features extracted in step S21 further includes:

[0071] When generating the Gaussian response map, the standard deviation of the Gaussian response map is adaptively adjusted according to the range of motion of the joints corresponding to the key points of the human skeleton (such as the elbow joint, shoulder joint and other joints with large range of motion, the standard deviation value is larger and the coverage is wider).

[0072] Multi-person Gaussian response map fusion: Gaussian response maps generated separately for multiple people in the scene are fused by weighted averaging according to the detection confidence of each human target (the higher the detection confidence of the human body, the greater the weight of its corresponding Gaussian response map, to avoid interference from occlusion and blurred targets on the fusion result).

[0073] Morphological and smoothing processing: Convolutional smoothing and morphological dilation operations are performed sequentially on the fused Gaussian response map. Convolutional smoothing is used to eliminate noise interference generated during the fusion process, and morphological dilation is used to enhance the continuity of key parts of the human body and avoid the breakage of key areas due to local occlusion.

[0074] Normalization: The processed image is normalized using the Sigmoid or Softmax function to obtain the spatial attention mask matrix. ; Where R is the set of real numbers, representing a matrix where all elements are real numbers, H is the height of the visual feature map, and W is the width of the visual feature map. The value range is [0,1], ensuring that the mask value can be directly used to modulate the attention weights of visual features.

[0075] Furthermore, in step S23, the spatial attention mask is integrated into the visual feature extraction process of step S22, specifically by modulating the self-attention calculation of visual features. The modulation causes computing resources to be tilted toward the human skeleton key point association region indicated by the spatial attention mask.

[0076] By defining the integration of the spatial attention mask into the visual feature extraction process of step S22, specifically through the modulation of the self-attention calculation of visual features, the core focus of the skeleton's guiding role is clarified. This process allows the spatial attention mask to act as a bias term, directly intervening in and reshaping the weight distribution of visual features in the self-attention mechanism. When calculating the internal correlations of visual features, the model automatically tilts more "attention" towards the key body parts (such as the hand) indicated by the mask and their surrounding areas, thereby suppressing background noise and strengthening key information in the early stages of feature extraction, achieving a targeted enhancement of the visual perception process by the skeleton information.

[0077] Furthermore, the modulation of the self-attention weights is calculated using the following formula:

[0078]

[0079] in, Let K represent the modulated self-attention weight matrix; Q is the query matrix obtained from the current visual feature mapping, and K is the key matrix obtained from the current visual feature mapping. middle The matrix transpose symbol is used to convert the key matrix K from column vector form to row vector form, so that Q and Q are in the same row vector form. Inner product calculations can be performed to characterize feature similarity; d represents the feature dimension of the key vector; For mask strength coefficient, >0; The space attention mask matrix is ​​represented; the Softmax function is used to normalize each query row.

[0080] Furthermore, in step S24, before fusing the attention-guided visual features with the skeleton features, a step of temporal attention enhancement is included for the visual features, wherein the temporal attention enhancement adopts a second-order temporal attention mechanism.

[0081] By introducing a step of "temporal attention enhancement for visual features and employing a second-order temporal attention mechanism" before feature fusion, the model not only focuses on key spatial regions but also models the contextual dependencies of actions in the temporal dimension. This overcomes the drawback of losing action continuity that may occur with single-frame or short-time modeling. It can more accurately understand behaviors that require multiple consecutive frames to fully represent, such as "taking out a phone" or "looking down to read," improving the system's accuracy in recognizing complex and continuous behaviors and its temporal localization precision.

[0082] In the second-order temporal attention mechanism, the visual features of the t-th frame after temporal attention enhancement are represented as follows: It can be calculated using the following formula:

[0083]

[0084] Where t is the index of the current frame. For neighborhood frame indexing; and These represent the first-order and second-order temporal neighborhood frame index sets of the t-th frame, respectively; That is, the set of indices corresponding to the previous and next frames of the current frame; , which is the set of indices corresponding to the two frames before and the two frames after the current frame; ∪ represents the set merging operation, which includes the indices of both first-order and second-order neighboring frames. For the first The visual feature vector of a frame (i.e., the visual features after attention guidance). The composite weight coefficients are the result of integrating the first-order and second-order attention weights.

[0085] The composite weighting coefficient The calculation process includes: firstly, calculating the first-order domain attention weights. and second-order domain attention weights The formulas are as follows:

[0086]

[0087]

[0088] in, Let be the query vector for the t-th frame. For the first The key vector of a frame; middle The vector transpose symbol is used to transpose a query vector. Convert from column vector to row vector, and Inner product calculations can be performed to characterize the similarity of features between adjacent frames; d is the feature dimension;

[0089] The composite weight coefficients are then obtained by weighted summation. The formula is:

[0090]

[0091] in, , and , For first-order temporal neighborhood ( The contribution coefficient of attention weights, For second-order temporal neighborhood ( The contribution coefficient of the attention weight is used to independently control the contribution of different order temporal neighborhood features to the enhancement of visual features in the current frame.

[0092] In step S24, the calculation formula for feature fusion is:

[0093]

[0094] in, Let be the final fused feature of frame t; For learnable weight matrix, For learnable bias terms; The visual features of frame t after temporal attention enhancement; The skeleton features of frame t; This indicates a feature concatenation operation, used to superimpose the dimensions of visual features and skeleton features; This is an activation function used to introduce nonlinear feature transformations and enhance the model's expressive power.

[0095] Furthermore, step S3 specifically includes:

[0096] S31. Visual token extraction: Locate the time window in which the identified behavior occurs, and select M of the most representative feature vectors from the visual features of the video frame sequence segment corresponding to the time window as visual tokens to represent the core visual content of the identified behavior; M is a preset positive integer; the value range is 256 to 1024, and the feature dimension of each visual token is 512 to 1024.

[0097] S32. End-side compression coding: The M visual tokens are compressed using a pre-trained vector quantization codebook to obtain the corresponding codeword index sequence; the vector quantization codebook is a pre-trained codebook used to map high-dimensional visual tokens to low-dimensional codeword indices.

[0098] S33. Data transmission: Encapsulate the codeword index sequence and the scene context text information and send them to the cloud server.

[0099] By selecting the most representative feature vectors from visual features as visual tokens, core evidence is extracted from massive video data. Compression encoding using a pre-trained vector quantization codebook maps high-dimensional feature vectors into extremely low-bit codeword index sequences. The combination of these two processes reduces the amount of data to be uploaded by several orders of magnitude without losing core visual semantics, significantly lowering the uplink network bandwidth requirements and making real-time, large-scale edge-cloud collaborative applications feasible.

[0100] Furthermore, step S4 specifically includes:

[0101] S41. Data Decoding: Receive and parse the data from the visual evidence compression and transmission steps, and reconstruct the visual tokens based on the codeword index sequence using the same vector quantization codebook; at the same time, convert the scene context text information into a text token sequence; the number of text token sequences ranges from 16 to 128, and the feature dimension of each text token is 512 to 1024.

[0102] S42, Cross-modal deep fusion: The visual token and text token sequences are input into the fusion module of the large multimodal model; the fusion module enables the visual token and text token to perform multiple rounds of bidirectional interaction and feature updates through its internal cross-modal attention layer;

[0103] S43. Generate interpretable results: Based on the features after deep fusion, perform anomaly judgment and simultaneously generate the structured evidence chain; the structured evidence chain includes at least anomaly level, anomaly time window, a list of key visual token identifiers, and natural language reasoning text; the anomaly level includes critical level, important level, and general level.

[0104] Visual tokens and scene context text information are converted into token sequences and input into a large-scale multimodal model, providing a foundation for cross-modal interaction. Through multi-round bidirectional interaction via a cross-modal attention layer within the model, deep alignment and fusion of visual evidence and textual semantics are achieved. This process ensures that the final judgment is a comprehensive reasoning result that fully integrates what is seen (visual evidence) and what is known (scene rules). Simultaneously, the requirement to generate a structured evidence chain ensures that the reasoning process itself is recorded in a structured manner, achieving a leap from "black-box judgment" to "white-box analysis," greatly improving the system's credibility and usability.

[0105] Furthermore, in step S42, the cross-modal attention layer adopts an alternating update strategy, specifically: in odd-numbered layers, visual tokens are used as queries, and text tokens are used as keys and values; in even-numbered layers, text tokens are used as queries, and visual tokens are used as keys and values.

[0106] By specifically defining an "alternating update strategy for the cross-modal attention layer" (with odd and even layers using different modalities as queries), an efficient cross-modal fusion mechanism is defined. This principle forces visual tokens and text tokens to repeatedly and bidirectionally "question" and "respond" to each other across multiple layers of the model. The effect is that, during deep fusion, visual evidence is interpreted by textual semantics (e.g., a visual token is associated with "phone"), while textual semantics are also anchored to specific visual evidence (e.g., the text "exam" is associated with the visual evidence "hand hiding an object"), thus achieving true semantic-level alignment and making the model's joint inference more accurate and reliable. Specifically, the number of cross-modal attention layers is 2 to 6, and an alternating update strategy is adopted: odd-numbered layers use visual tokens as queries and text tokens as keys and values; even-numbered layers use text tokens as queries and visual tokens as keys and values; each cross-modal attention layer uses 8 to 16 multi-head attention, and before attention calculation, the query, key, and value are linearly mapped, and after calculation, they are added to the input features through residual connections and normalized using LayerNorm.

[0107] Furthermore, step S5 specifically includes:

[0108] S51. Warning Judgment and Generation: Analyze the anomaly judgment conclusion and structured evidence chain in the analysis results, and generate a warning message of the corresponding level according to the anomaly level; the warning message integrates the identified behavior, target location, anomaly time window and evidence summary, and the evidence summary is extracted and generated based on the natural language reasoning text and key visual token information in the structured evidence chain.

[0109] S52. Warning Push: The warning message is pushed to one or more designated management terminals in real time through a preset communication interface; the communication interface includes one or more of the following: WeChat interface, DingTalk interface, and monitoring system interface, and an audible and visual alarm is triggered simultaneously when necessary.

[0110] The structured evidence chain in the analysis results is used to generate alert messages, ensuring that the pushed information is rich, specific, and verifiable (including behavior, location, time, and evidence summary). Complex cloud-based analysis results are transformed into formatted information that managers can quickly understand and make decisions about. This significantly improves the practical value and response efficiency of alert information, enabling managers to take swift intervention measures based on clear clues.

[0111] Example 2

[0112] This invention also discloses a behavior recognition and anomaly warning system, comprising: a video acquisition and preprocessing module for acquiring monitoring video streams of a monitored area and performing frame extraction and image quality enhancement processing; the monitored area is a lecture hall; a behavior recognition module connected to the video acquisition and preprocessing module for extracting skeleton features and visual features in parallel from the processed video frame sequence, using the skeleton features to generate spatial attention guidance information to modulate the extraction and analysis process of the visual features, and fusing the modulated visual features and the skeleton features to identify the target's behavior category and output structured behavior information; and a visual token. The processing and communication module, connected to the behavior recognition module, is used to extract and compress visual tokens based on the structured behavior information, and send the compressed visual token data and scene context text information to the cloud; the cloud multimodal analysis module, deployed on the cloud server, is used to decode and reconstruct the received visual token data and scene context text information, and perform cross-modal deep fusion reasoning through a built-in large-scale multimodal model, outputting analysis results including anomaly judgment conclusions and structured evidence chains; the early warning service module, connected to the cloud multimodal analysis module, is used to generate and push early warning information based on the analysis results.

[0113] The system of this invention can solve the problems of insufficient semantic understanding in complex scenarios, weak anti-interference ability in complex environments, lack of interpretability of early warning results, and poor system architecture adaptability.

[0114] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for behavior recognition and anomaly early warning, characterized in that, Includes the following steps: S1. Video Acquisition and Preprocessing: Acquire the monitoring video stream of the monitored area, and perform frame extraction and image quality enhancement processing to obtain a preprocessed video frame sequence; S2. Multimodal behavior recognition: For the video frame sequence, extract in parallel skeleton features to represent human posture and motion patterns, as well as visual features to represent objects and scene context in the scene. Spatial attention guidance information is generated using the skeleton features to modulate the extraction and analysis process of the visual features; the modulated visual features and the skeleton features are fused, and the behavior category of the target is identified based on the fused features to obtain the identified behavior, and structured behavior information containing the identified behavior is output. S3. Visual Evidence Compression and Transmission: Based on the structured behavioral information, locate the time window in which the identified behavior occurs. From the visual features of the video frame sequence segment corresponding to the time window, select M of the most representative feature vectors as visual tokens to represent the core visual content of the identified behavior, where M is a preset positive integer. Compress the M visual tokens using a pre-trained vector quantization codebook to obtain the corresponding codeword index sequence. Send the codeword index sequence and the scene context text information describing the current scene state to the cloud server. S4. Cloud-based Explainable Anomaly Reasoning: On the cloud server, the data from step S3 is received and parsed. The same vector quantization codebook is used to reconstruct the visual token based on the codeword index sequence. At the same time, the scene context text information is converted into a text token sequence. The visual token and text token sequence are input into the fusion module of the large multimodal model. The fusion module enables visual tokens and text tokens to perform multi-round bidirectional interactions and feature updates through its internal cross-modal attention layer, thereby achieving deep fusion of visual tokens and text tokens and obtaining deeply fused features. Anomaly detection is performed based on the deeply fused features, and an analysis result containing anomaly detection conclusion and a structured evidence chain is generated simultaneously. The structured evidence chain includes at least anomaly level, anomaly time window, a list of key visual token identifiers, and natural language inference text. The structured evidence chain is used to explain the basis of the detection conclusion. The cross-modal attention layer adopts an alternating update strategy. In odd-numbered layers, visual tokens are used for queries, and text tokens are used for keys and values; In even-numbered layers, text tokens are used for queries, and visual tokens are used for keys and values; S5. Warning Generation and Push: Based on the analysis results, if an anomaly is determined, a warning message will be automatically generated and pushed.

2. The method according to claim 1, characterized in that, Step S2 specifically includes: S21. Skeleton Feature Extraction: Human skeleton key point detection is performed on the input video frame sequence to obtain a human skeleton key point sequence; the human skeleton key point sequence is constructed into a spatiotemporal graph and processed using a graph convolutional network to extract the skeleton features. S22. Visual feature extraction: The video frame sequence is processed using a temporal visual transformer network to extract the visual features; S23. Skeleton-guided attention fusion: A spatial attention mask is generated using the extracted skeleton features; the spatial attention mask is integrated into the visual feature extraction process to guide computational resources to focus on the region of interest defined by the key points of the human skeleton. S24. Behavior classification: The attention-guided visual features are fused with the skeleton features, and the fused features are classified to obtain the identified behaviors. The structured behavior information containing the identified behaviors is then output.

3. The method according to claim 2, characterized in that, In step S23, the step of generating a spatial attention mask using the extracted skeleton features specifically involves: generating a Gaussian response map centered on each human skeleton key point in the visual feature map space based on the image coordinates of the key points of the human skeleton; and normalizing the Gaussian response maps of multiple people after fusing and performing morphological operations to obtain a spatial attention mask matrix.

4. The method according to claim 2, characterized in that, In step S23, the spatial attention mask is incorporated into the visual feature extraction process, which is specifically achieved by modulating the self-attention calculation of visual features. The modulation causes the computing resources to be tilted toward the human skeleton key point association region indicated by the spatial attention mask.

5. The method according to claim 2, characterized in that, In step S24, before fusing the attention-guided visual features with the skeleton features, a step of temporal attention enhancement of the visual features is included, wherein the temporal attention enhancement adopts a second-order temporal attention mechanism.

6. The method according to claim 1, characterized in that, Step S5 specifically includes: S51. Warning Judgment and Generation: Parse the anomaly judgment conclusion and structured evidence chain in the analysis results, and generate a warning message of the corresponding level according to the anomaly level. The warning message integrates the identified behavior, target location, anomaly time window and evidence summary. S52. Warning push: The warning message is pushed to one or more designated management terminals in real time through a preset communication interface.

Images