Abnormal behavior detection method, device and medium executed on an end-side device

Abstract

The application provides an abnormal behavior detection method, device and medium executed on an end-side device, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring visual data and audio data; processing the visual data of the original resolution by using a visual coding scheme based on packing and filling to extract visual features, the scheme converting the visual data into a fixed-length sequence stream without scaling the visual data; extracting audio features; aligning the visual features and the audio features by using a two-stage mechanism combining time axis coarse alignment and semantic fine alignment; and fusing and inputting the aligned features into a large language model decoding module to generate an abnormal behavior detection result. By using the original resolution processing mechanism, the loss of pixel details is avoided, and the recognition accuracy is improved. By using the accurate audio-visual bimodal alignment and fusion, the reliability of the detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to the field of computer vision and multi-modal information processing, and specifically relates to an abnormal behavior detection method and device executed on an edge device and a medium. BACKGROUND

[0002] With the development of artificial intelligence technology, automatic abnormal behavior detection using multi-modal information such as video and audio has become an important direction to improve security efficiency.

[0003] At present, a common technical solution is to upload collected data to a cloud server for centralized processing. Although this solution can utilize the powerful computing resources of the cloud to perform complex analysis, it faces many challenges in real-time monitoring scenarios, for example, high-definition video streams consume huge network bandwidth, and data transmission inevitably causes delays, affecting the real-time performance of detection, and uploading data related to the privacy of citizens to the cloud also poses a risk of data leakage.

[0004] Therefore, edge computing mode, which performs calculations on edge devices close to data sources, becomes a better choice. However, edge devices usually have limited computing power, which poses high requirements on the efficiency and lightweight design of algorithm models. Existing technologies still have deficiencies in edge abnormal detection. For example, some solutions often need to perform scaling or cropping and other preprocessing operations on video images from different cameras and different resolutions to adapt to the input size of the model, which inevitably loses pixel information in the original image and adversely affects the detection of subtle abnormal behaviors such as item left behind and small attack actions by personnel. In addition, how to efficiently and accurately fuse visual and audio data streams that may have deviations in time to make reliable joint judgments is also a technical problem that needs to be solved in the field of edge multi-modal abnormal detection. SUMMARY

[0005] The present application aims to provide an edge abnormal detection method and system based on native resolution processing and multi-modal alignment, which aims to solve the problems of information loss caused by image preprocessing, difficulty in accurate alignment of multi-modal data, and difficulty in achieving high-precision real-time detection under the condition of limited computing power of edge devices in the prior art.

[0006] To achieve the above object, the application provides an abnormal behavior detection method executed on an end-side device, comprising: acquiring visual data and audio data; processing the visual data by using a visual coding scheme based on packing and padding to extract visual features, the visual coding scheme converting the visual data into a fixed-length sequence stream without scaling the original resolution of the visual data; processing the audio data to extract audio features; before fusing the visual features and the audio features, aligning the visual features and the audio features by using a two-stage mechanism combining time axis coarse alignment and semantic fine alignment; fusing the aligned visual features and audio features, and inputting the fused features into a large language model decoding module to generate an abnormal behavior detection result.

[0007] Further, the step of converting the visual data into a fixed-length sequence stream comprises: dividing the input image into multiple image blocks; splicing the sequence of the image blocks from multiple images in one processing batch into a single sequence stream in the sequence dimension; and truncating or padding the single sequence stream according to a preset maximum block length to obtain the fixed-length sequence stream.

[0008] Further, the time axis coarse alignment is realized by a dynamic window aggregation strategy, which aggregates multiple continuous audio feature frames into an audio feature matching the time granularity of the visual feature.

[0009] Further, the dynamic window aggregation strategy comprises: judging whether an audio period is an abnormal candidate period according to audio energy variance; when the audio period is a stable period, performing aggregation by using maximum value pooling; and when the audio period is an abnormal candidate period, performing aggregation by using weighted average pooling.

[0010] Optionally, the semantic fine alignment is realized by combining a lightweight cross-modal attention mechanism and a fast dynamic time warping algorithm to match the visual features and the audio features in the semantic level.

[0011] Further, before processing by the lightweight cross-modal attention mechanism, the method further comprises: screening the audio features extracted from the audio signals corresponding to the target orientation according to the target orientation recognized in the visual data.

[0012] Further, the method further comprises: judging whether the alignment similarity of the visual features and the audio features is lower than a preset similarity threshold; if yes, triggering a single-modal degradation mode to use only the visual features for abnormal behavior detection.

[0013] Further, the visual features are extracted by a visual coding module based on a visual Transformer architecture, and the total parameter quantity of the model implementing the method is less than 700M.

[0014] To achieve the above object, the application further provides an end-side abnormal behavior detection device, comprising: a data acquisition module for acquiring visual data and audio data; a processing module configured to: process the visual data by using a visual coding scheme based on packing and padding to extract visual features, the visual coding scheme converting the visual data into a sequence stream of fixed length without scaling the original resolution of the visual data; process the audio data to extract audio features; before fusing the visual features and the audio features, align the visual features and the audio features by using a two-stage mechanism combining time axis coarse alignment and semantic fine alignment; fuse the aligned visual features and audio features, and input the fused features into a large language model decoding module to generate and output an abnormal behavior detection result.

[0015] The application further provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described above.

[0016] Compared with the prior art, the technical scheme provided by the application has the following beneficial effects:

[0017] Firstly, by using the native resolution processing mechanism, the application avoids the loss of pixel details caused by traditional image scaling operations, retains complete visual information, and significantly improves the recognition accuracy of small targets and subtle actions; at the same time, through precise audio-visual dual modal alignment and fusion, the detection blind area of single modal in complex situations such as occlusion and light change is effectively overcome, the reliability of detection is improved, and high-precision and high-reliability detection effect is realized.

[0018] Secondly, by using a lightweight model architecture and optimizing the inference process, a millisecond-level response speed is achieved, meeting the stringent requirements of real-time monitoring and analysis, and enabling timely detection and early warning of abnormal behavior, with the advantages of low delay and high real-time performance.

[0019] Thirdly, the lightweight design enables the model to be directly deployed on edge computing devices, with data processing completed locally without the need for uploading to the cloud, thereby fundamentally solving the problems of data transmission delay, bandwidth occupation and potential privacy leakage, facilitating end-side deployment and effectively protecting user privacy.

[0020] Finally, the fault-tolerant degradation mechanism included in the scheme ensures that the system can still operate stably when the quality of part of the sensor data is poor, improving the overall robustness. BRIEF DESCRIPTION OF DRAWINGS

[0021] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0022] Figure 1 A flowchart illustrating the overall architecture of an end-side anomaly detection system provided in this application embodiment;

[0023] Figure 2 A schematic diagram of a multimodal model structure provided in an embodiment of this application;

[0024] Figure 3 This is a flowchart of the native resolution visual processing provided in the embodiments of this application;

[0025] Figure 4 A flowchart of an end-side anomaly detection method provided in an embodiment of this application;

[0026] Figure 5 This is a timing diagram of the audiovisual feature alignment signaling interaction provided in the embodiments of this application.

[0027] Figure labeling: 10 - Edge device; 20 - Visual data acquisition and preprocessing module; 30 - Audio data acquisition and preprocessing module; 40 - Time axis coarse alignment module; 50 - Semantic fine alignment module; 60 - Fault tolerance processing module; 70 - Multimodal fusion and inference module; 100 - Native resolution processing module; 110 - Visual encoding module; 120 - Text encoding module; 130 - Visual feature to text feature mapping module; 140 - Visual-text sequence fusion module; 150 - Large language model decoding module. Detailed Implementation

[0028] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0029] For the core scenario of concurrent monitoring of multiple cameras in a government service hall, requiring real-time capture of abnormal behavior, a multimodal model structure was designed around the requirements of fast response and low latency. This was achieved by optimizing the lightweight network architecture and building a parallel inference mechanism for multi-camera data, reducing redundant computational steps and effectively improving inference speed. Combined with a visual-audio alignment strategy, this ensures real-time processing of multiple video streams and rapid response to abnormal signals, avoiding missed opportunities for handling anomalies due to delays. On the other hand, to ensure recognition accuracy, image scaling and cropping, which would damage the pixel distribution of the original image or video, were eliminated during the model data preprocessing stage. This maximizes the preservation of pixel information in the original image or video, ensuring the model can acquire scene details and local features, avoiding false positives or false negatives caused by the loss of key information, and achieving a synergistic guarantee of real-time performance and recognition accuracy.

[0030] Figure 1 This document presents a flowchart illustrating the overall architecture of an edge-side anomaly detection system provided in an embodiment of this application. The system is deployed on an edge device 10, such as an edge computing unit located in a public place like a government service hall, transportation hub, or bank branch. The edge device 10 integrates or connects to data acquisition sensors such as high-definition cameras and microphones, and has a built-in processor with a certain computing power (e.g., including a neural network processing unit). The entire anomaly detection process is executed locally on the edge device 10, eliminating the need to upload large amounts of raw data to the cloud.

[0031] Reference Figure 4 The abnormal behavior detection method provided in this embodiment mainly includes the following steps:

[0032] Step S10: Acquire visual and audio data. Specifically, the camera on the edge device 10 continuously captures video streams as visual data, while the microphone on the device captures ambient sound as audio data. These two data streams are sent in parallel to the subsequent processing module.

[0033] The cameras deployed in the government service hall vary in model and resolution, with common resolutions including "1920×1080", "1280×720", and "800×600". Traditionally, a uniform size (such as scaling to a fixed size) is used to ensure consistent input, but this results in significant information loss.

[0034] To address this issue, this solution proposes a visual encoding mechanism based on native resolution, which preserves the original pixel information while achieving efficient adaptation to the Transformer encoder. Since the input sequence length of the Transformer must be fixed, and images at different resolutions yield different numbers of tokens after block segmentation, a sliding window and padding mechanism is designed to achieve a fixed-length input at native resolution.

[0035] Each block of an image is treated as a visual token, arranged from left to right and top to bottom to form a visual sequence input Transformer. For input images of different sizes (e.g., "1920×1080" and "800×600"), the same block size will produce different numbers of tokens. To ensure consistency during multi-batch training and inference, instead of simply concatenating these sequences along the batch dimension, a packing and padding approach is used along the sequence dimension. That is, all image blocks within a batch are concatenated into a unified sequence stream, so that the Transformer only needs to process a unified sequence stream input, without having to forcibly align the spatial dimensions of all images.

[0036] Step S20: Extract native resolution visual features using a "packaging and padding" scheme. This step is... Figure 1 The visual data acquisition and preprocessing module 20 and Figure 2 The native resolution processing module 100 and the visual encoding module 110 work together to achieve this. The core of this scheme is to process the input visual data without any scaling or cropping, while maintaining the original resolution, thereby preserving all pixel information in the image. This is crucial for detecting subtle abnormal behaviors (such as lost small objects or subtle aggressive gestures).

[0037] Combination Figure 3 As shown, the specific implementation process of this step is as follows: First, in the input image step S210, each frame of the image acquired from the video stream is input to the native resolution processing module 100. Then, in the sliding window segmentation step S220, this module uses a fixed-size (e.g., 16×16 pixels) sliding window to divide the input single high-resolution image (1920×1080) into multiple image blocks in a non-overlapping manner. For a 1920×1080 image, it will be divided into (1920 / 16) × (1080 / 16), which, after rounding up, is approximately 120 × 68, or 8160 image blocks. Next, in the block serialization step S230, these two-dimensionally arranged image blocks are flattened in order from left to right and from top to bottom to form a one-dimensional image block sequence.

[0038] To improve processing efficiency, the system typically processes multiple images in batches. Therefore, in the sequence stitching step S240, image block sequences from multiple images (e.g., four images) within the same processing batch are stitched together along the sequence dimension to form a longer single sequence stream. Then, in the length determination step S250, the system determines whether the total length of the stitched single sequence stream exceeds a preset maximum block length. This maximum block length is determined by the maximum input sequence length that the subsequent visual encoding module 110 can accept, for example, it can be set to 4096 image blocks. Finally, in the truncation / padding step S260, if the length of the single sequence stream exceeds the maximum block length, the excess portion is truncated from the end; if the length does not reach the maximum block length, specific padding values ​​(e.g., zero vectors) are used to pad the end of the sequence until its length reaches the maximum block length.

[0039] For the task of detecting abnormal behavior in government service halls, this solution first ensures information integrity by eliminating the pixel detail loss caused by traditional image scaling operations. It guarantees the complete preservation of key information such as facial features and subtle details of objects, laying a data foundation for subsequent accurate analysis. Simultaneously, the solution boasts excellent structural flexibility, supporting video or image data of any input resolution without requiring additional resampling or alignment preprocessing. This simplifies the data flow and further avoids information loss in intermediate steps. Furthermore, to balance processing efficiency, the solution employs a standardized sequence length design, enabling the Transformer encoder to operate efficiently in a fixed computation graph, reducing computational waste caused by dynamic input sizes. Ultimately, it achieves a synergistic unity of information integrity, flexible adaptation, and efficient inference.

[0040] Through the above steps, regardless of the change in input image resolution, the system can always generate a fixed-length sequence stream. This fixed-length sequence stream is then fed into the visual encoding module 110 in the input Transformer step S270. In this embodiment, the visual encoding module 110 adopts a lightweight model based on the visual Transformer architecture, specifically adapted and optimized based on the DINOv3-ViT-B / 16 architecture, to perform deep processing on the input image patch sequence to extract high-dimensional visual features that can represent the image content. Adaptation is performed for the native resolution input; simultaneously, the number of visual tokens is controlled to 384, thereby significantly reducing the computational load; subsequently, the initialization parameters reuse the unmodified parts of the DINOv3 pre-trained weights, making the total number of parameters only 86M; finally, while maintaining the model's original visual semantic transfer capabilities, extremely high inference speed is achieved.

[0041] Step S30: Extract audio features. This step is performed by... Figure 1The audio data acquisition and preprocessing module 30 completes the process. Specifically, the system first divides the acquired 16kHz raw audio stream into continuous audio frames, each lasting 10 milliseconds. For each audio frame, the system calculates its 13-dimensional Mel-frequency cepstral coefficients, a commonly used feature in speech and audio processing that effectively characterizes sound perception. To match the dimensionality with visual features and further compress the information, the extracted Mel-frequency cepstral coefficient sequence is then passed through a multilayer perceptron consisting of two fully connected layers, ultimately being compressed into a 512-dimensional audio feature vector. Accordingly, the audio data is converted into an audio feature sequence A=[a1,a2,…,a], with each unit containing a 512-dimensional feature vector in 10-millisecond time units. S (S=100 / second).

[0042] To address the issues of single visual modalities being susceptible to light and occlusion, and single audio modalities being susceptible to environmental noise interference in government service hall scenarios, a precise visual-audio dual-modal alignment mechanism is constructed. While ensuring efficient collaboration with the visual encoding process and multimodal model architecture, this mechanism resolves the problems of temporal misalignment and semantic disconnect in multimodal features, improving the accuracy of abnormal behavior detection. Simultaneously, it adapts to low-computing-power environments on the edge without adding additional inference latency. Combining the characteristics of visual encoding output features (fixed temporal granularity, unified dimension) with the real-time requirements of audio features, a two-stage mechanism of "coarse temporal alignment + fine semantic alignment" is designed to ensure a balance between alignment accuracy and efficiency.

[0043] Step S40: Perform a two-stage alignment process: "coarse timeline alignment + fine semantic alignment". Because visual features (one frame every 40 milliseconds) and audio features (one frame every 10 milliseconds) have inconsistent temporal granularity and may have content delays or misalignments, direct fusion is ineffective. Therefore, this embodiment employs an innovative two-stage mechanism to precisely align the two before fusion. The signaling interaction of this process can be found in [reference needed]. Figure 5 .

[0044] The first stage is coarse alignment of the timeline, which is... Figure 1 The timeline coarse alignment module 40 is executed, and its goal is to align the temporal granularity of the audio feature sequence with that of the visual feature sequence.

[0045] A "dynamic window aggregation" strategy is adopted, aggregating every four consecutive audio feature frames (total duration 40 milliseconds, consistent with the visual feature time interval). The aggregation method is as follows: for stable time periods without abnormal candidates (judged by audio energy variance, variance ≤ 0.2), max pooling is used to retain key features; for abnormal candidate time periods (variance > 0.2), weighted average pooling (center frame weight 0.4, adjacent frames 0.3 each) is used to retain feature details. This generates an audio feature subsequence A'=[a'1,a'2,…,a'] that matches the temporal granularity of the visual features. T (T=25 / second) to achieve coarse alignment of the time axis.

[0046] Specifically, the system employs a dynamic window aggregation strategy, aggregating every four consecutive 10-millisecond audio feature frames (a total of 40 milliseconds) into a single 512-dimensional audio feature that temporally corresponds to a visual feature frame. To enhance the intelligence of the aggregation process, this strategy adaptively selects the pooling method based on the stationarity of the audio segment. The system calculates the energy variance of the audio signal within this 40-millisecond window and presets a variance threshold (e.g., 0.2). If the calculated variance is less than or equal to the threshold, the segment is considered a stationary segment (e.g., background white noise), and max pooling is used for aggregation to extract the most significant stable sound signal within that segment. If the variance is greater than the threshold, the segment is considered an abnormal candidate segment (e.g., a sudden scream, the sound of breaking glass, or an argument), and weighted average pooling is used for aggregation. For example, higher weights (e.g., 0.3 each) can be assigned to the two audio frames at the center of the window, while lower weights (e.g., 0.2 each) can be assigned to the two audio frames at the edge of the window, thereby highlighting the audio features at the core moment of the abnormal event. After this stage of processing, we obtain the aggregated audio feature sequence A', which corresponds 1:1 in time to the visual feature sequence.

[0047] The second stage is semantic fine alignment, which is... Figure 1 The semantic fine alignment module 50 is executed. Building upon timeline alignment, this stage aims to further eliminate discrepancies between audiovisual features at both the content and semantic levels.

[0048] Based on the coarsely aligned feature sequences, and combining the spatial location information of visual features with the semantic attributes of audio features, accurate cross-modal semantic matching is achieved:

[0049] As a preferred implementation, the system can perform a spatial semantic filtering step before performing core alignment calculations. For example, when the vision module detects a suspicious target in the left-hand area of ​​the image, the system can prioritize or increase the weight of audio features collected by the microphone in the left-hand area based on the pre-configured physical location correspondence between the camera and microphone. This step can effectively filter noise interference from irrelevant directions, reducing computational complexity and ambiguity for subsequent precise alignment. Combining the physical location correspondence between the microphone and camera in the government service hall (e.g., matching the camera in the east area with the microphone in the east area), audio features a' from the same area are filtered. same This eliminates invalid audio features across regions, reducing computational load (by approximately 40% in alignment calculations).

[0050] Next, the semantic fine-alignment module 50 internally calculates the semantic similarity between visual feature V and aggregated audio feature A' using a lightweight cross-modal attention mechanism. The similarity calculation follows the formula:

[0051]

[0052] in, In time step Visual feature vectors, In time step The aggregated audio feature vector. It is a visual feature The corresponding physical coordinate encoding, which is derived from the image patch of which region of the image the visual feature was originally generated from, introduces prior knowledge of spatial location into the model. It is an adjustable spatial location weight hyperparameter (e.g., set to 0.3) used to balance the importance of spatial location information and visual content information. This is the dimension of the feature vector (512 in this example). Using this formula, the system can calculate the semantic similarity score between any visual feature frame and any audio feature frame, and these scores form a similarity matrix.

[0053] The attention weight matrix W is obtained by normalizing the exponential function, and the audio feature subsequence A' is reweighted to generate the semantically associated audio feature sequence A''.

[0054] Finally, to find the globally optimal alignment relationship between audiovisual feature sequences, the system employs a fast dynamic time warping algorithm with path constraints.

[0055] Path constraints: The search range is limited to ±2 time steps on both sides of the diagonal to avoid path jumps; the cost matrix is ​​calculated using Manhattan distance to reduce computational complexity (40% less computation than Euclidean distance).

[0056] The algorithm uses the similarity matrix as the cost matrix to find a path from the bottom left to the top right corner of the matrix that maximizes the cumulative cost (i.e., the sum of similarities). This path is the optimal alignment path, indicating which frame in the audio feature sequence each frame in the visual feature sequence should correspond to. To improve computational efficiency and meet the real-time requirements of the endpoint, the search path of this algorithm is restricted to a narrow strip region around the diagonal (e.g., ±2 time steps on either side of the diagonal), because in real-time streaming, audiovisual delays are typically not significant.

[0057] Find the optimal alignment path P=[(t1¹,t2¹),(t1²,t2²),…,(t1 K ,t2 K Adjust the correspondence between the visual feature sequence V and the audio feature sequence A'' to obtain the final aligned feature pair {(v t1 k ,a'' t2 k )}.

[0058] At the level of collaboration with the visual encoding process, the solution achieves efficient integration in three aspects: feature interface adaptation, preprocessing synchronization, and masking of invalid regions. The visual encoding module first compresses the block features to 512 dimensions using 1×1 convolutions, maintaining consistency with the audio feature dimensions and avoiding the overhead of dimension transformation before alignment. Preprocessing synchronization uses the same source timestamps to collect visual and audio data. If the device lacks synchronization capabilities, it compensates with 10-second calibration data to ensure a synchronization error ≤5 milliseconds, reducing the impact of time offset on the fusion effect from the source. After visual encoding is completed, a mask vector (Mask) is generated for the filled regions. v In the feature alignment stage, the similarity calculation result is multiplied by the mask. v' The method automatically masks the features in this region, effectively avoiding invalid features from interfering with the accuracy of cross-modal fusion.

[0059] In terms of collaboration with the multimodal model architecture, the solution constructs a connection mechanism from three aspects: fusion layer adaptation, lightweight optimization, and fault-tolerant degradation collaboration. The aligned visual-audio feature pair {(v,a'')} is first processed by a lightweight feature interaction layer (1×1 convolution + ReLU activation function) to generate the interactive feature Inter_feat=Conv (v+a''), which is then input into the multimodal sequence fusion layer and interleaved with the text features, effectively reducing the computational pressure of the fusion layer. The alignment module itself has a parameter size controlled within 15M and also uses INT8 quantization compression. On the edge device, it can be computed in parallel with visual encoding and text encoding using independent NPU computing units. Moreover, the alignment process is hidden within the visual encoding time window of ≤20 milliseconds, without adding extra end-to-end latency. At the same time, it has fault-tolerant degradation collaboration capability. When the alignment similarity is <0.5 for three consecutive time steps (determined as alignment failure), the model will automatically trigger "single-modal degradation", which will block audio features and use only visual features for inference, while outputting an alignment anomaly warning to ensure the overall stability of the model.

[0060] Through this step, the system obtains feature pairs (V, which are finally precisely aligned in both time and semantics). ).

[0061] Step S50: Fuse the aligned features and input them into the large language model. This step is... Figure 1 The multimodal fusion and inference module 70 is complete; its internal structure can be found in [reference]. Figure 2 Aligned visual features V and audio features The data is fed into the visual-text sequence fusion module 140. One feasible fusion method is to concatenate the two feature vectors along their feature dimensions to form a longer fused feature vector. Then, this fused feature sequence is input into the large language model decoding module 150.

[0062] To enable edge deployment, the entire multimodal model used in this embodiment is designed to be lightweight. Specifically, the visual encoding module 110 is based on DINOv3-ViT-B / 16 (approximately 86M parameters), the text encoding module 120 (for supporting optional question-answering functionality) adopts the MobileClip2-B architecture (approximately 63.4M parameters), and the core large language model decoding module 150 adopts the Qwen2.5-0.5B architecture (approximately 500M parameters). The total number of parameters in the entire model is approximately 649.4M, which is much smaller than that of conventional large multimodal models, thus ensuring that it can perform low-latency, high-efficiency inference on edge devices 10 with limited computing power.

[0063] Step S60: Generate abnormal behavior detection results. After receiving the fused features, the large language model decoding module 150 performs autoregressive decoding to generate natural language text describing the current scene or detection result. For example, when fighting is detected, the system outputs: "Abnormal behavior detected: violent behavior, confidence level: 96%". This text result can then be used to trigger alarms, send notifications to security personnel, or be logged in the log system.

[0064] This embodiment, through the complete technical process described above, achieves low-latency, high-precision abnormal behavior detection of multimodal data at native resolution on the edge device. Because the native resolution is preserved, the system can accurately identify aggressive actions at the edges of the screen or with slight occlusion. Simultaneously, by fusing features such as "arguing sounds" in the audio, it effectively eliminates purely visual misjudgments that may occur due to crowding, significantly improving the accuracy and reliability of the detection.

[0065] Example 2

[0066] This embodiment is a variation of Embodiment 1, intended to illustrate the good flexibility and scalability of the two-stage alignment framework proposed in this application. In particular, the aggregation strategy in the coarse alignment stage of the time axis can be simplified to adapt to application scenarios with more demanding computing resources or higher real-time requirements.

[0067] The system configuration, hardware environment, and all processing steps except for coarse timeline alignment in this embodiment are consistent with those in Embodiment 1. This includes data acquisition (step S10), native resolution visual processing (step S20), audio feature extraction (step S30), semantic fine alignment (the second stage of step S40), and subsequent fusion decoding (steps S50 and S60).

[0068] Compared with Embodiment 1, the main difference in this embodiment lies in the first stage of step S40, namely the specific implementation method of coarse timeline alignment. In Embodiment 1, the coarse timeline alignment module 40 adopts a dynamic window aggregation strategy, which dynamically selects the pooling method based on the audio energy variance.

[0069] In this embodiment, to further reduce computational overhead, the dynamic strategy is simplified to a fixed, non-adaptive aggregation method. Specifically, regardless of the audio energy variance within the 40-millisecond window, the time axis coarse alignment module 40 always uses average pooling to aggregate every four consecutive 10-millisecond audio feature frames. That is, the average of these four 512-dimensional audio feature vectors is calculated by adding them element-wise, resulting in a single 512-dimensional aggregated audio feature.

[0070] The advantage of this simplified processing lies in its simple computation process. Compared to dynamic strategies that require calculating variance first, then making conditional judgments and potentially involving weighted calculations, average pooling has a significantly smaller computational load and faster processing speed, making it particularly advantageous for deployment on embedded devices with extremely limited computing power.

[0071] Of course, this simplification may also lead to performance trade-offs. Average pooling treats all audio features within a window equally, which may smooth out some brief but crucial peaks in anomalous sound signals. It is not as effective as weighted average pooling in highlighting the feature details of anomalous moments, nor as effective as max pooling in preserving the strongest signals during stable periods. However, in many scenarios, anomalous events (such as arguments) usually last for a period of time. Even with average pooling, the aggregated audio features can still retain enough anomalous information to be recognized by subsequent semantic fine-alignment and decoding modules.

[0072] The verification results of this embodiment demonstrate that the proposed two-stage alignment framework is modular, and the coarse time axis alignment strategy can be flexibly configured according to actual hardware resources and performance requirements. It can employ a refined dynamic strategy as described in Embodiment 1 to pursue higher detection accuracy, or a simplified strategy as described in this embodiment to achieve higher processing efficiency. This provides broader possibilities for the adaptation and application of this technical solution on edge devices of different levels.

[0073] Example 3

[0074] This embodiment is another variation of Embodiment 1, intended to illustrate that the implementation of the second stage in the two-stage alignment framework, namely semantic fine alignment, is also flexible and pluggable. In certain application scenarios with good data synchronization, the semantic fine alignment algorithm can be simplified to further reduce computational complexity and improve end-to-end processing speed.

[0075] The system configuration, hardware environment, and most of the processing steps in this embodiment, including data acquisition (step S10), native resolution visual processing (step S20), audio feature extraction (step S30), and timeline coarse alignment (the first stage of step S40), are consistent with those in embodiment 1.

[0076] The main difference between this embodiment and Embodiment 1 lies in the second stage of step S40, namely the specific workflow of the semantic fine alignment module 50. In Embodiment 1, the semantic fine alignment module 50 combines a lightweight cross-modal attention mechanism and a fast dynamic time warping algorithm to find the globally optimal audiovisual alignment relationship through complex path search.

[0077] In this embodiment, the semantic fine alignment process is simplified. The specific process is as follows: First, the semantic fine alignment module 50, as described in Embodiment 1, uses a cross-modal attention mechanism to calculate the similarity matrix between the visual feature sequence V and the aggregated audio feature sequence A', where each element... Represents the first The visual frame and the first Semantic correlations between audio frames.

[0078] However, this embodiment no longer performs a computationally expensive fast dynamic time warping algorithm for global path search, but instead adopts a more direct alignment method. This method is based on the assumption that after coarse alignment of the timeline, the audiovisual features are basically aligned in time. The visual feature is mainly related to the first Each audio feature is related. Therefore, the system can directly reweight the audio feature sequence using attention weights. One feasible implementation is to, for each time step... The system extracts the values ​​on the diagonal of the similarity matrix. As attention weights, or for calculating the first The normalized results of the attention scores of each visual feature and all audio features are then used to adjust the audio features. .

[0079] More specifically, an attention-weighted audio feature can be constructed that only considers the current time step. For example, you can directly use the diagonal elements of the attention matrix as weights to scale audio features, or directly use the same timestamp. Visual features below and aggregated audio features Pair them up, assuming they are already aligned.

[0080] This simplified approach is applicable only if the hardware at the data acquisition end (camera and microphone) has excellent time synchronization and the latency jitter in data transmission to the processing module is minimal; for example, the timestamp deviation between audiovisual data streams can be stabilized within 5 milliseconds. Under such ideal conditions, after coarse alignment of the timeline, the audiovisual features are highly correlated in the sequence index, eliminating the need for complex time-series distortion correction through dynamic time warping.

[0081] A significant advantage of this simplified approach is the substantial reduction in computational complexity during the semantic fine-alignment stage. Even the fast version of the dynamic temporal warping algorithm has a time complexity exceeding linear time; whereas the direct pairing or simple weighting operation in this embodiment has a near-linear time complexity, thus significantly improving the end-to-end speed of the entire detection process.

[0082] This embodiment demonstrates that the cross-modal attention and dynamic time warping algorithm in the semantic fine alignment mechanism proposed in this application are two decoupled components. In scenarios with extremely high real-time requirements and guaranteed data synchronization, the dynamic time warping step can be omitted, and effective alignment can still be achieved by using only the attention mechanism for semantic association and weighting. This flexibility allows the technical solution of this application to better adapt to diverse application needs and deployment environments.

[0083] Example 4

[0084] This embodiment specifically illustrates an enhanced fault-tolerant processing mechanism in the technical solution of this application. This mechanism is used to deal with special cases where the audiovisual modal data quality is poor or the alignment fails, thereby significantly improving the robustness of the entire anomaly detection system and its reliability in practical applications.

[0085] The system configuration in this embodiment is basically the same as in Embodiment 1, deployed on the edge device 10, and loaded with a lightweight multimodal model including a visual encoding module 110, an audio processing module, a two-stage alignment module (a coarse timeline alignment module 40 and a fine semantic alignment module 50), and a large language model decoding module 150. Furthermore, this embodiment also specifically implements and activates [the following functionality / feature]. Figure 1 The fault-tolerant processing module 60 shown is configured with a network linkage interface with the background manual monitoring platform.

[0086] The working process is described as follows: During normal operation, in addition to executing the complete multimodal detection process described in Example 1, the fault-tolerant processing module 60 continuously monitors the alignment quality output by the semantic fine alignment module 50. The alignment quality can be evaluated by a quantified alignment similarity. This similarity can be calculated based on the cumulative cost (i.e., cumulative similarity) of the optimal path found by the fast dynamic time warping algorithm, and normalized to obtain a value between 0 and 1; or, in a simplified scheme without using dynamic time warping, the average value of the diagonal elements in the attention matrix can be directly used as the similarity measure.

[0087] The system presets a similarity threshold, for example, 0.5. Under normal circumstances, because the audiovisual content is related (for example, someone is speaking in the video, and there is corresponding speech in the audio), the alignment similarity should be much higher than this threshold.

[0088] In a specific scenario, suppose a microphone in a certain area of ​​a government service hall is accidentally and completely blocked, resulting in extremely poor audio signal quality from the microphone, filled with noise that is completely unrelated to the clear image content captured by the camera (such as people moving normally).

[0089] When this extremely low-quality audio data is processed and fed into the semantic fine alignment module 50, the calculated semantic similarity between the audiovisual features will be very low. The fault-tolerant processing module 60 detects during monitoring that the alignment similarity is below the preset similarity threshold of 0.5 for three consecutive time steps (e.g., 120 milliseconds). At this point, the system determines that a "multimodal alignment failure" event has occurred.

[0090] Once alignment failure is determined, the fault tolerance module 60 immediately triggers an enhanced fault tolerance degradation mechanism, which includes two parallel actions:

[0091] 1. Single-modal degraded inference: The system immediately switches to single-modal (pure visual) detection mode. Specifically, the system bypasses the audio processing flow and the entire two-stage alignment module, directly feeding the visual features extracted by the visual encoding module 110 into the large language model decoding module 150 in the multimodal fusion and inference module 70 (which at this time only processes single-modal input). The decoding module 150 performs abnormal behavior detection based solely on visual information and outputs a single-modal-based detection result. For example, even without audio, if an abnormal rapid gathering of a crowd is visually observed, the system can still output the result: "Abnormal crowd gathering detected." This process ensures that the system's basic detection function is not interrupted even if the audio modality fails.

[0092] 2. Collaborative Early Warning and Manual Review: To compensate for potential blind spots in single-modal detection, the system marks the event as "low confidence - alignment failure" and automatically packages related multimedia evidence. For example, it packages 5 seconds of original high-definition video footage before and after the alignment failure determination with low-quality original audio footage into a single event file. This file is sent to the backend manual monitoring platform via a pre-defined network interface, highlighting or displaying a pop-up message on the monitoring interface to alert on-duty security personnel for manual review. Security personnel can replay the video and audio, combining them with other on-site information to make a final, accurate judgment.

[0093] Through the fault-tolerant degradation and human-machine collaboration mechanism demonstrated in this embodiment, the technical solution of this application ensures extremely high system robustness. It can still provide uninterrupted basic detection services even under harsh conditions such as partial sensor failure or severe data quality degradation, avoiding the risk of the entire system collapsing due to a single modality data problem. Simultaneously, by intelligently pushing ambiguous events that are difficult for machines to determine to human decision-making, an effective combination of human and machine intelligence is achieved. This ensures the efficiency of the automated system while utilizing human experience and wisdom to handle complex and unexpected situations, thereby greatly enhancing the overall reliability and practical value of the entire security system.

[0094] This model supports two task modes: no-prompt reasoning (pre-set task) and multimodal question answering. The anomaly detection task pre-sets various anomaly categories related to government service halls, including but not limited to violent behavior detection, fall detection, medical assistance detection, and stampede detection. The reasoning process does not require any text prompts and can automatically complete the image analysis and output the anomaly category and its corresponding confidence level. The question answering task allows users to input questions in natural language such as "How many people are in the picture?" or "Has anyone fallen?" The model will automatically parse the text and combine it with visual features to generate accurate answers, and it can adapt to question answering reasoning scenarios of real-time video frames or static images.

[0095] On the one hand, by optimizing the model inference process, low-latency response is achieved, and the output speed of the first token is significantly accelerated, meeting the needs of real-time analysis of monitoring videos in government service halls. On the other hand, a lightweight architecture design is adopted, keeping the overall parameters within 665M, eliminating the need for complex computing equipment and allowing direct deployment on GPUs, NPUs, or high-performance ARM chips, thus lowering the threshold for edge deployment. At the same time, the model achieves efficient alignment of visual and text encoding at the multimodal fusion level, ensuring not only cross-modal analysis accuracy but also providing flexible support for subsequent task expansion. In addition, the solution is highly versatile, capable of stably running pre-built related analysis tasks and supporting user-defined task configurations, comprehensively covering various intelligent analysis needs in government service scenarios, improving the level of intelligence, and reducing labor costs.

[0096] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0097] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. An abnormal behavior detection method executed on an edge device, characterized in that, include: Acquire visual and audio data; The visual data is processed using a packing and padding-based visual encoding scheme to extract visual features. The visual encoding scheme converts the visual data into a fixed-length sequence stream without scaling the original resolution of the visual data. The audio data is processed to extract audio features; Before fusing the visual features and the audio features, a two-stage mechanism combining coarse temporal alignment and fine semantic alignment is used to align the visual features and the audio features. The aligned visual and audio features are fused, and the fused features are input into the large language model decoding module to generate abnormal behavior detection results. The semantic fine alignment is achieved by combining a lightweight cross-modal attention mechanism and a fast dynamic time warping algorithm to match the visual features and the audio features at the semantic level.

2. The method according to claim 1, characterized in that, The step of converting the visual data into a fixed-length sequence stream includes: The input image is divided into multiple image blocks; The sequence of image patches from multiple images within a processing batch is concatenated into a single sequence stream along the sequence dimension. The single sequence stream is truncated or padded according to the preset maximum block length to obtain the fixed-length sequence stream.

3. The method according to claim 1, characterized in that, The coarse alignment of the timeline is achieved through a dynamic window aggregation strategy, which aggregates multiple consecutive audio feature frames into an audio feature that matches the temporal granularity of the visual feature frame.

4. The method according to claim 3, characterized in that, The dynamic window aggregation strategy includes: Determine whether an audio segment is an abnormal candidate segment based on the audio energy variance; When the audio period is a stable period, maximum pooling is used for aggregation; When the audio period is an abnormal candidate period, weighted average pooling is used for aggregation.

5. The method according to claim 1, characterized in that, Before processing via the lightweight cross-modal attention mechanism, the following is also included: Based on the target location identified in the visual data, the audio features extracted from the audio signal corresponding to the target location are selected.

6. The method according to claim 1, characterized in that, The method further includes: Determine whether the alignment similarity between the visual feature and the audio feature is lower than a preset similarity threshold; If so, a single-modal degradation mode is triggered, using only the visual features for abnormal behavior detection.

7. The method according to claim 1, characterized in that, The visual features are extracted through a visual encoding module based on a visual Transformer architecture, and the total number of parameters in the model implementing the method is less than 700M.

8. An end-side abnormal behavior detection device, characterized in that, include: The data acquisition module is used to acquire visual and audio data; The processing module is configured as follows: The visual data is processed using a packing and padding-based visual encoding scheme to extract visual features. The visual encoding scheme converts the visual data into a fixed-length sequence stream without scaling the original resolution of the visual data. The audio data is processed to extract audio features; Before fusing the visual features and the audio features, a two-stage mechanism combining coarse temporal alignment and fine semantic alignment is used to align the visual features and the audio features. The aligned visual and audio features are fused, and the fused features are input into the large language model decoding module to generate and output abnormal behavior detection results. The semantic fine alignment is achieved by combining a lightweight cross-modal attention mechanism and a fast dynamic time warping algorithm to match the visual features and the audio features at the semantic level.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in claim 1.

Images