Intelligent perception and explanation method for abnormal behavior based on visual-linguistic-temporal fusion

Abstract

The application provides a multi-modal abnormal behavior intelligent perception and explanation method based on visual-language-time sequence fusion. The method generates a time sequence visual behavior segment to be perceived by acquiring a real-time behavior monitoring video stream, and generates a language prior constraint vector by performing semantic embedding coding on a domain expert knowledge text; generates a abnormal behavior logic anchor point to be explained based on the above segment and vector; performs causal chain reconstruction on the anchor point to generate an abnormal behavior time sequence evolution graph with causal logic constraints; and finally performs evidence chain credibility quantitative evaluation to generate a perception and explanation description matching a preset business logic format. Through deep fusion and causal reconstruction of visual, language and time sequence features, the application solves the technical defects of traditional schemes in single perception dimension and lack of logical explanation for complex abnormal behaviors, and improves the credibility and business adaptability of the perception result.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer vision technology, and more specifically, to a multimodal intelligent perception and interpretation method for abnormal behavior based on vision-language-temporal fusion. Background Technology

[0002] With the widespread application of intelligent monitoring technology in public safety, smart transportation, and industrial inspection, real-time perception and automated interpretation of abnormal behavior within a given scenario have become crucial for improving regulatory efficiency. Abnormal behavior perception not only requires accurate identification of behavior types but also the extraction of key evidence within complex spatiotemporal contexts, and the provision of causal explanations consistent with business logic, incorporating domain knowledge to assist managers in decision-making.

[0003] Existing abnormal behavior monitoring solutions typically employ a deep learning-based target detection and behavior recognition architecture, using pre-trained models to extract features and classify video frames. This approach first captures live video streams via cameras and then uses convolutional neural networks to identify human postures or motion trajectories within the video. The identified features are then compared to pre-defined abnormal behavior templates, triggering an alarm when the match exceeds a threshold. Finally, the system generates a simple alarm log, indicating the time and area where the anomaly occurred.

[0004] However, this traditional monitoring solution has significant technical shortcomings. Existing recognition methods rely excessively on single visual pixel features and lack effective integration of prior domain knowledge, leading to false alarms in complex lighting or occlusion scenarios. Furthermore, current technologies can only provide classification results, lacking logical analysis of the abnormal behavior from its cause to its evolution, failing to generate explanatory evidence chains with causal relationships, and exhibiting low correlation between alarm information and the logical fields of the underlying business system. This results in insufficient business interpretability and credibility of the perception results, making it difficult to meet the intelligent perception requirements of high-security scenarios. Summary of the Invention

[0005] This application provides a multimodal intelligent perception and interpretation method for abnormal behavior based on vision-language-temporal fusion, so as to at least alleviate the above-mentioned technical problems.

[0006] A multimodal intelligent perception and interpretation method for abnormal behavior based on vision-language-temporal fusion includes: Step 1: Based on the real-time behavior monitoring video stream of the scene to be monitored, generate the temporal visual behavior segment to be perceived, and perform semantic embedding encoding on the domain expert knowledge text that matches the scene to be monitored to generate a language prior constraint vector including the abnormal behavior judgment criteria. Step 2: Based on the visual behavior segments to be perceived and the language prior constraint vectors, generate logical anchor points for the abnormal behavior to be explained; Step 3: Reconstruct the causal chain of the abnormal behavior logic anchor point to be explained, so as to generate an abnormal behavior temporal evolution map with causal logic constraints; Step 4: Based on the time-series evolution map of the abnormal behavior, perform quantitative evaluation of the credibility of the evidence chain to generate intelligent perception and explanation descriptions of abnormal behavior that match the preset business logic format.

[0007] Optionally, in step 1, based on the real-time behavior monitoring video stream of the scene to be monitored, a temporal visual behavior segment to be perceived is generated, including: The real-time behavior monitoring video stream is subjected to spatiotemporal dimension feature decoupling processing to generate a visual semantic representation sequence containing target motion trajectory and action form evolution information; The visual semantic representation sequence is synchronously segmented along the time axis based on a preset sampling step size to generate a visual behavior segment to be perceived with uniform time-stamp weights.

[0008] Optionally, step 1 further includes: Analyze the temporal visual behavior segment to be perceived to determine its included feature channels; Determine the feature association topology matrix between the different semantic dimension features in the language prior constraint vector and the feature channels included in the temporal visual behavior segment to be perceived; Based on the feature association topology matrix, the language prior constraint vector is reprojected into the feature space to obtain the channel adaptation constraint vector. The channel adaptation constraint vector is mapped to the feature channel corresponding to the temporal visual behavior segment to be perceived, so as to form the constraint for generating the logical anchor point of the abnormal behavior to be explained.

[0009] Optionally, step 2: Based on the perceived temporal visual behavior segment and the language prior constraint vector, generate logical anchor points for the abnormal behavior to be explained, including: Extract multidimensional motion vector features from the temporal visual behavior segments to be perceived; Dynamic imaging contrast analysis is performed based on the multidimensional motion vector features to generate visually significant anomaly representation quantities that characterize the degree of behavioral shift in different feature channels. Based on the linguistic prior constraint vector and the visual saliency anomaly representation quantity, generate logical anchor points for the abnormal behavior to be explained.

[0010] Optionally, logical anchor points for the anomalous behavior to be explained are generated based on the linguistic prior constraint vector and the visual saliency anomaly representation quantity, including: The visual saliency anomaly representation is matched with the language prior constraint vector through cross-modal semantic mapping to determine the anomalous attribute label of the visual saliency anomaly representation at the language logic level. Key action frames are extracted from the temporal visual behavior segment to be perceived based on the abnormal attribute labels. The extracted key action frames are spatiotemporally locked to determine the logical anchor point of the abnormal behavior to be explained.

[0011] Optionally, step 3: reconstruct the behavioral causal chain of the logical anchor point of the abnormal behavior to be explained, so as to generate a temporal evolution map of abnormal behavior with causal logical constraints, including: The logical anchor point of the abnormal behavior to be explained is reconstructed into a behavioral causal chain to generate behavioral explanation semantic features, which include the abnormal triggering dimension, the abnormal evolution dimension, and the abnormal impact dimension. Based on the semantic features of the behavior interpretation, a temporal evolution map of abnormal behavior with causal logic constraints is generated.

[0012] Optionally, based on the semantic features of the behavior explanation, a temporal evolution map of abnormal behavior with causal logic constraints is generated, including: The semantic features of the behavior interpretation are mapped to the spatiotemporal coordinate system corresponding to the temporal visual behavior segment to be perceived, so as to generate spatiotemporal projection interpretation features; Spatiotemporal alignment is performed on the spatiotemporal projection interpretation features to generate spatiotemporally complete interpretation features; Feature association is performed on the spatiotemporally complete explanatory features to generate an abnormal behavior temporal evolution graph with causal logic constraints. The nodes in the abnormal behavior temporal evolution graph represent abnormal state semantic nodes, and the edges represent temporal evolution causal logic edges.

[0013] Optionally, step 4: Based on the anomalous behavior temporal evolution graph, perform a quantitative evaluation of the evidence chain credibility to generate an intelligent perception and explanation description of the anomalous behavior that matches a preset business logic format, including: Based on the time-series evolution map of the abnormal behavior, a quantitative assessment of the credibility of the evidence chain is performed to generate a credibility score for multi-source heterogeneous evidence. Based on the credibility score of the multi-source heterogeneous evidence, modal behavioral evidence is obtained by parsing the temporal evolution map of the abnormal behavior to form a preliminary weighted evidence set; Based on the aforementioned preliminary weighted evidence set, conflict resolution is performed to obtain a temporally consistent evidence set. Generate an abnormal behavior perception evidence set based on the time-consistent evidence set; Based on the abnormal behavior benchmark feature library, the abnormal behavior perception evidence set is mapped to business fields to obtain an associated field mapping table. The associated field mapping table is parsed to obtain the evidence feature fields; Based on the evidence feature fields, generate intelligent perception and explanation descriptions of abnormal behaviors that match the preset business logic format.

[0014] Optionally, based on the evidence feature fields, an intelligent perception and explanation description of abnormal behavior matching a preset business logic format is generated, including: The evidence feature fields are processed by feature attribute retrieval to extract the anomaly level parameters and anomaly trigger time corresponding to the scene to be monitored. The anomaly level parameters and the anomaly trigger time are encapsulated to generate a data packet to be orchestrated; The data packets to be orchestrated are orchestrated to generate intelligent perception and explanation descriptions of abnormal behaviors that match a preset business logic format.

[0015] Optionally, step 4 also includes: Obtain the standard anomaly judgment logic corresponding to the scenario to be monitored from the preset business knowledge base; Based on the aforementioned standard anomaly determination logic, anomaly determination benchmark features are generated; Based on the aforementioned anomaly determination benchmark features, an abnormal behavior benchmark feature library is constructed.

[0016] Technical advantages of the technical solution provided in this application This application presents a multimodal intelligent perception and interpretation method for abnormal behavior based on vision-language-temporal fusion. Addressing the technical shortcomings of traditional monitoring solutions, such as lack of domain knowledge integration, single perception dimension, and lack of causal explanation capability, this method deeply fuses visual semantic representations generated by decoupling real-time video streams with language prior constraint vectors encoded by domain expert knowledge. This solves the problems of visual recognition being susceptible to interference and lacking semantic guidance in traditional solutions. Compared to traditional methods that rely solely on visual pixel comparison, this application constructs a feature association topology matrix and performs feature space reprojection, enabling language-level judgment criteria to directly guide the analysis of visual features, significantly improving the accuracy and robustness of abnormal behavior localization in complex contexts.

[0017] Based on visual-language fusion-generated logical anchor points for unexplained abnormal behaviors, this application reconstructs a temporal evolution graph of abnormal behaviors with causal logic constraints through behavioral causal chain reconstruction. This overcomes the technical limitations of traditional solutions, which can only output isolated alarms and cannot trace the causes of behaviors. Traditional solutions only provide time-point classification labels, while this application extracts multi-dimensional semantic features of abnormal triggering, evolution, and impact, aligns and correlates them in a spatiotemporal coordinate system, and constructs a graph structure containing semantic nodes and causal logic edges. Compared to traditional alarm logs, this structured graph can clearly present the occurrence of abnormal behaviors, providing an evidentiary basis for anomaly explanation that conforms to human cognitive logic.

[0018] By performing quantitative evaluation of the credibility of the evidence chain on the time-series evolution graph and introducing conflict resolution and business field mapping mechanisms, this application solves the problems of disconnect between perception results and business logic, and the lack of evidence credibility assessment. Traditional solutions lack dynamic handling of data conflicts when issuing alarms, while this application eliminates logical contradictions between multi-source data through progressive processing of the initial weighted evidence set and the time-series consistent evidence set. Simultaneously, an abnormal behavior benchmark feature library is constructed based on a business knowledge base, automatically mapping perception evidence to business fields and extracting the anomaly level and trigger time. Compared to the single information output of traditional solutions, the explanations and descriptions generated by this application are closely integrated with the business system, and the degree of information structuring and business adaptability are significantly improved.

[0019] Finally, the perception solution formed through a closed-loop process of language prior guidance, spatiotemporal feature fusion, causal logic graph reconstruction, and quantitative interpretation of the evidence chain exhibits higher logical completeness when facing complex dynamic scenarios. Traditional solutions struggle to explain why something is judged as abnormal, while this application, through attribute retrieval and data encapsulation of evidence feature fields, can generate descriptive reports with high credibility and causal explanatory power. This allows the monitoring system to evolve from simply capturing phenomena to understanding logic, better meeting the practical needs of public safety and other fields for closed-loop perception and accurate handling of abnormal behavior. Attached Figure Description

[0020] Figure 1 This is a flowchart of a multimodal abnormal behavior intelligent perception and interpretation method based on vision-language-temporal fusion, according to an embodiment of this application.

[0021] Figure 2 This application provides an embodiment of a multimodal intelligent perception and interpretation device for abnormal behavior based on vision-language-temporal fusion.

[0022] Figure 3 This is an electronic device according to an embodiment of the present application. Detailed Implementation

[0023] like Figure 1 As shown in the figure, this application provides an embodiment of a multimodal intelligent perception and interpretation method for abnormal behavior based on vision-language-temporal fusion, which includes: Step 1: Based on the real-time behavior monitoring video stream of the scene to be monitored, generate the temporal visual behavior segment to be perceived, and perform semantic embedding encoding on the domain expert knowledge text that matches the scene to be monitored to generate a language prior constraint vector including the abnormal behavior judgment criteria. Step 2: Based on the visual behavior segments to be perceived and the language prior constraint vectors, generate logical anchor points for the abnormal behavior to be explained; Step 3: Reconstruct the causal chain of the abnormal behavior logic anchor point to be explained, so as to generate an abnormal behavior temporal evolution map with causal logic constraints; Step 4: Based on the time-series evolution map of the abnormal behavior, perform quantitative evaluation of the credibility of the evidence chain to generate intelligent perception and explanation descriptions of abnormal behavior that match the preset business logic format.

[0024] Optionally, in step 1, based on the real-time behavior monitoring video stream of the scene to be monitored, a temporal visual behavior segment to be perceived is generated, including: The real-time behavior monitoring video stream is subjected to spatiotemporal dimension feature decoupling processing to generate a visual semantic representation sequence containing target motion trajectory and action form evolution information; The visual semantic representation sequence is synchronously segmented along the time axis based on a preset sampling step size to generate a visual behavior segment to be perceived with uniform time-stamp weights.

[0025] Preferably, the specific implementation process of generating the perceived temporal visual behavior segment based on the real-time behavior monitoring video stream of the scene to be monitored in step 1 of this application is as follows: First, for the real-time behavior monitoring video stream collected by monitoring cameras deployed in public spaces such as parks and industrial parks, this application will perform spatiotemporal feature decoupling processing on the real-time behavior monitoring video stream. This processing is not simply extracting single-frame image features, but separating and analyzing the real-time behavior monitoring video stream in both spatial and temporal dimensions: In the spatial dimension, the motion vector between adjacent video frames is calculated pixel by pixel to generate an instantaneous motion vector field that can characterize the motion trend of each spatial position in the scene. Each element in the instantaneous motion vector field contains the horizontal displacement component and the vertical displacement component of the pixel point at that position on the image plane; In the temporal dimension, the behavior evolution rate field is constructed by calculating the rate of change of pixel intensity at the same spatial position point in the video frame sequence over time. This behavior evolution rate field reflects the degree of dynamic change of the object or human body action form in the scene on the time axis. By fusing the instantaneous motion vector field and the behavior evolution rate field element by element, a composite spatiotemporal motion feature tensor is constructed. Each element of this spatiotemporal motion feature tensor simultaneously encodes information on spatial motion direction and temporal evolution rate.

[0026] Preferably, after generating the aforementioned spatiotemporal motion feature tensor, in the specific technical implementation of step 1, this application further extracts visual features with semantic expressive capabilities from the spatiotemporal motion feature tensor. Specifically, a pre-trained behavior recognition neural network model is used to perform multi-layer convolution and pooling operations on the spatiotemporal motion feature tensor to extract a high-dimensional visual semantic representation vector. This visual semantic representation vector is not an isolated feature value, but a highly abstract description of the target behavior in the real-time behavior monitoring video stream. Part of the visual semantic representation vector encodes the target's centroid coordinate change trajectory within a continuous time window, constituting the target's motion trajectory information. This target motion trajectory information reflects the target's movement path, speed, and direction changes in the scene. Another part of the visual semantic representation vector encodes parameters such as the target's limb joint angles and contour deformation, constituting action morphology evolution information. This action morphology evolution information reflects the target's posture change sequence when performing different actions. The aforementioned target motion trajectory information and action morphology evolution information together form a structured dataset, namely, a visual semantic representation sequence. Each unit in this visual semantic representation sequence corresponds to a behavioral segment within a tiny time window (such as 0.5 seconds) and contains the complete motion and morphological features within that time window.

[0027] Preferably, to ensure that subsequent processing steps obtain uniform and aligned behavioral segments in the time dimension, the implementation method of time-axis synchronous segmentation of the visual semantic representation sequence based on a preset sampling step size in step 1 is as follows: First, this application defines a sliding segmentation window on the time axis of the real-time behavior monitoring video stream according to a preset sampling step size (e.g., a 2-second segmentation unit and a 1-second sliding step size). Then, this application maps each representation unit in the aforementioned generated visual semantic representation sequence to these segmentation windows based on its corresponding timestamp. When the timestamp of a visual semantic representation unit falls entirely within a segmentation window, that visual semantic representation unit is assigned to that window. In this way, the originally continuous and unstructured visual semantic representation sequence is discretized and reorganized into a series of temporally adjacent, non-overlapping (or partially overlapping) segments.

[0028] Preferably, to assign uniform time-stamped weights to these segmented segments for subsequent computation, this application performs further data encapsulation processing. For each segment corresponding to a segmentation window, this application calculates its temporal position parameter and duration parameter within the entire visual semantic representation sequence, and normalizes and encodes these parameters to form a time-stamped weight vector. For example, this time-stamped weight vector may include information such as the ratio of the segment's start time to the total duration of the real-time behavior monitoring video stream, and the ratio of the segment's center time to its total length. This time-stamped weight vector is then embedded into the segmented visual semantic representation sequence and concatenated with the original target motion trajectory information and action morphology evolution information to generate a completely new data structure, namely, a perceptible temporal visual behavior segment with uniform time-stamped weights.

[0029] Preferably, the generated temporal visual behavior segment with unified time-stamped weights is a structured data packet, providing standardized, semantically meaningful input for abnormal behavior analysis in subsequent steps of this application. Within this data packet, target motion trajectory information records the spatial position changes of the target within a specific time period, for example, reflecting a tourist moving from point A to point B; action morphology evolution information records the changes in the tourist's limb posture during movement, such as the frequency and amplitude of arm waving; and time-stamped weight information clarifies the specific time position (e.g., from the 10th to the 12th second) of this behavior within the entire real-time behavior monitoring video stream. Through this spatiotemporal feature decoupling and temporal synchronization segmentation, this application transforms the original, unstructured pixel stream into a structured temporal segment containing three layers of information: target motion trajectory information, action morphology evolution information, and time-stamped weight information, laying a data foundation for subsequent fusion analysis using language prior constraint vectors.

[0030] Preferably, unlike traditional monitoring schemes that only perform simple frame difference or optical flow calculations on real-time behavior monitoring video streams, this application generates a visual semantic representation sequence through spatiotemporal dimension feature decoupling. Its technical advantage lies in its ability to explicitly distinguish between the spatial information of where the target is moving and the morphological information of how the target is moving. This explicit decoupling makes the subsequent feature fusion process more targeted: when a language prior constraint vector (e.g., containing the criterion of "rapid arm swing") is involved, this application can prioritize matching it with the action morphological evolution information in the visual semantic representation sequence, rather than comparing it with the target's motion trajectory information, thereby significantly improving the accuracy and efficiency of cross-modal feature fusion. Simultaneously, the synchronous segmentation of the visual behavior segments to be perceived based on time-stamp weights ensures that behavior segments from different sources and with different time starting points can be compared and analyzed within a unified time reference system. This is beneficial for analyzing complex abnormal behaviors with temporal causal relationships, such as falling before climbing.

[0031] Optionally, step 1 further includes: Analyze the temporal visual behavior segment to be perceived to determine its included feature channels; Determine the feature association topology matrix between the different semantic dimension features in the language prior constraint vector and the feature channels included in the temporal visual behavior segment to be perceived; Based on the feature association topology matrix, the language prior constraint vector is reprojected into the feature space to obtain the channel adaptation constraint vector. The channel adaptation constraint vector is mapped to the feature channel corresponding to the temporal visual behavior segment to be perceived, so as to form the constraint for generating the logical anchor point of the abnormal behavior to be explained.

[0032] Preferably, the specific implementation process of parsing the perceptible temporal visual behavior segment with unified time-stamped weights to determine its included feature channels in step 1 is as follows: First, this application obtains the perceptible temporal visual behavior segment with unified time-stamped weights generated by the aforementioned processing steps. This perceptible temporal visual behavior segment with unified time-stamped weights is a structured multidimensional data organization form, which is internally organized into multiple mutually independent feature channels according to functional semantics. This application performs channel dimension separation operation on the perceptible temporal visual behavior segment with unified time-stamped weights, and identifies and extracts the different predefined feature channels one by one: for example, the target motion trajectory channel is used to store the centroid coordinate sequence of the target within a continuous time window, the action morphology evolution channel is used to store the sequence of limb joint angles and contour deformation parameters, and the time-stamped weight channel is used to store the normalized time position and duration encoding. Through this channel separation and parsing, this application clearly knows which specific feature channels the perceptible temporal visual behavior segment with unified time-stamped weights includes and the data dimension and physical meaning of each feature channel, establishing a clear mapping benchmark for subsequent cross-modal feature fusion.

[0033] Preferably, after identifying the feature channels contained in the perceived temporal visual behavior segment with unified time-stamp weights, step 1 further performs the operation of determining the feature association topology matrix between the different semantic dimension features in the language prior constraint vector and the feature channels included in the perceived temporal visual behavior segment with unified time-stamp weights. This application obtains a language prior constraint vector generated by semantic embedding encoding of domain expert knowledge text. This language prior constraint vector is an embedding representation in a high-dimensional semantic space, where each semantic dimension feature corresponds to a specific abnormal behavior judgment criterion (e.g., the semantic dimension feature corresponding to "rapid arm swing" and the semantic dimension feature corresponding to "long-term stillness after falling"). The system calculates the association strength between each semantic dimension feature in the language prior constraint vector and each feature channel (e.g., target motion trajectory channel, action form evolution channel, time-stamp weight channel) in the perceived temporal visual behavior segment with unified time-stamp weights. Specifically, it uses cosine similarity or mutual information measurement methods to calculate the correlation value between the feature vector of each semantic dimension feature and the feature vector of each feature channel. All calculated correlation values ​​are arranged according to semantic dimension features as rows and feature channels as columns, forming a two-dimensional matrix, which is the feature association topology matrix. The first feature association topology matrix contains the first feature channel. Line 1 The elements of the column represent the first... The semantic dimension feature and the first The degree of correlation between feature channels is indicated by the larger the value, the better the semantic judgment criterion matches the information type contained in the feature channel. i and j are both integers.

[0034] Preferably, after obtaining the feature association topology matrix, step 1 then performs a feature space reprojection based on the feature association topology matrix to obtain the channel adaptation constraint vector. This application treats the language prior constraint vector as a semantic point in the source space and the feature association topology matrix as a linear transformation mapping matrix from the semantic space to the feature channel space. Specifically, this application performs matrix-vector multiplication on the feature association topology matrix and the language prior constraint vector: each row of the feature association topology matrix (corresponding to a feature channel) is multiplied by the language prior constraint vector to obtain a new vector. The dimension of this new vector is equal to the number of feature channels, and each element represents the language constraint strength that the corresponding feature channel should obtain after weighted fusion. This newly generated vector is called the channel adaptation constraint vector. Through this feature space reprojection process, the abnormal behavior judgment criteria originally expressed in the pure semantic space are transformed into a set of numerical constraints directly corresponding to each feature channel of the visual behavior segment to be perceived with unified time-scale weights, enabling language-level knowledge to be injected into the visual feature parsing process in a quantitative form.

[0035] Preferably, after obtaining the channel adaptation constraint vector, step 1 finally maps the channel adaptation constraint vector to the feature channel corresponding to the perceptible temporal visual behavior segment with unified time-scale weights, so as to form constraints for generating the logical anchor point of the abnormal behavior to be explained. This application applies each element in the channel adaptation constraint vector to the corresponding feature channel of the perceptible temporal visual behavior segment with unified time-scale weights, according to its corresponding feature channel index. The mapping method adopts feature modulation: for the target motion trajectory channel, the corresponding constraint strength value is multiplied element-wise with the motion vector of each time step in the target motion trajectory channel, so that the response of motion features is enhanced in spatial locations or motion directions that are important in language prior judgments; for the action form evolution channel, the corresponding constraint strength value is weighted with the joint angle change rate in the action form evolution channel, so that the posture changes related to the abnormal behavior criteria are highlighted; for the time-scale weight channel, the original value in its time-scale weight channel remains unchanged, and it is only used as a reference for temporal localization. After this feature-by-feature channel constraint mapping, the originally neutral visual features are injected with linguistic prior guidance information, forming a temporal visual behavior segment to be perceived with semantic constraint enhancement and unified time-stamp weight. This segment is the constrained input data for generating logical anchor points for the abnormal behavior to be explained.

[0036] Preferably, in a specific application scenario (e.g., detecting "elderly falls" in park surveillance), the above-mentioned technical processing demonstrates a clear functional support relationship between the links. The linguistic prior constraint vector includes the semantic dimension feature of "the human body's center of mass rapidly descends and then remains in a low position for a long time." The calculation results of its feature association topology matrix show that this semantic dimension feature has a high correlation strength with the action morphology evolution channel (e.g., reaching 0.85), while its correlation strength with the target motion trajectory channel is low (e.g., only 0.12). Therefore, after reprojection in the feature space, the element values ​​corresponding to the action morphology evolution channel in the channel adaptation constraint vector are larger, while the element values ​​corresponding to the target motion trajectory channel are smaller. After mapping this channel adaptation constraint vector to the perceived temporal visual behavior segment with uniform time-scale weights, the joint angle change features within the action morphology evolution channel are significantly enhanced, while the displacement features within the target motion trajectory channel are almost unaffected. Therefore, in the subsequent step 2, when generating the logical anchor point of the abnormal behavior to be explained, the abnormality will be mainly determined based on the characteristics such as the rapid descent of the center of mass and the maintenance of low angles of limb joints in the action form evolution channel, while ignoring irrelevant motion trajectory fluctuations. This achieves accurate guidance of visual feature analysis by language priors and improves the robustness of abnormal behavior localization in complex backgrounds.

[0037] Preferably, the feature association topology matrix designed in this application is not fixed, but dynamically calculated and generated based on each input segment of the visual behavior to be perceived with uniform time-stamped weights. This dynamic adaptability is reflected in the fact that the statistical distribution of feature channels may change for real-time behavior monitoring video streams collected in different scenes and at different time periods (for example, the difference in illumination between day and night will lead to different noise distributions in the motion vector field). The dynamically calculated feature association topology matrix can capture in real time the actual correlation between visual features and linguistic semantics in the current segment of the visual behavior to be perceived with uniform time-stamped weights, so that the channel adaptation constraint vector can adaptively adjust the constraint strength of each feature channel, avoiding overfitting or mismatching problems that may be caused by using a fixed mapping matrix. Through the complete processing chain of parsing feature channels—constructing feature association topology matrix—feature space reprojection—mapping constraints, this application integrates domain expert knowledge into the underlying visual feature parsing process in a quantifiable, adaptive, and interpretable manner, providing semantically guided input for the subsequent generation of abnormal behavior logic anchors, which is different from the technical architecture of separating visual and language processing in traditional solutions.

[0038] Optionally, step 2: Based on the perceived temporal visual behavior segment and the language prior constraint vector, generate logical anchor points for the abnormal behavior to be explained, including: Extract multidimensional motion vector features from the temporal visual behavior segments to be perceived; Dynamic imaging contrast analysis is performed based on the multidimensional motion vector features to generate visually significant anomaly representation quantities that characterize the degree of behavioral shift in different feature channels. Based on the linguistic prior constraint vector and the visual saliency anomaly representation quantity, generate logical anchor points for the abnormal behavior to be explained.

[0039] Preferably, the specific implementation process of extracting multidimensional motion vector features from the perceived temporal visual behavior segment with unified time-scale weights in step 2 is as follows: This application first obtains the perceived temporal visual behavior segment with unified time-scale weights generated in the aforementioned steps. The perceived temporal visual behavior segment with unified time-scale weights is internally organized into a target motion trajectory channel, an action form evolution channel, and a time-scale weight channel according to feature channels. This application extracts the horizontal and vertical displacement components of the centroid coordinates of the target at each time step on the image plane from the target motion trajectory channel to form a two-dimensional planar motion vector sequence; simultaneously, it extracts the three-dimensional spatial coordinate changes of limb joints from the action form evolution channel to form a joint motion vector sequence. The above two-dimensional planar motion vector sequence and the joint motion vector sequence are spliced ​​on the time axis according to corresponding frames to form a joint feature vector set containing multiple motion dimensions, which is the multidimensional motion vector feature. For example, for a human walking in park surveillance, the multidimensional motion vector feature includes not only the direction and speed of the human centroid movement in the image, but also fine-grained limb movement information such as the arm swing angle and the degree of knee flexion.

[0040] Preferably, after extracting the multidimensional motion vector features, step 2 then performs dynamic imaging contrast analysis based on the multidimensional motion vector features to generate visually significant anomaly representations of behavioral deviation in different feature channels. This application compares the multidimensional motion vector features with a pre-built normal behavior benchmark motion pattern library on a time-window basis. The normal behavior benchmark motion pattern library stores the motion vector statistical distribution parameters of each feature channel obtained from a large number of normal behavior samples under the same monitoring scenario (e.g., the mean and standard deviation range of the horizontal movement speed of the center of mass during normal walking, common intervals of arm swing amplitude, etc.). For each feature channel in the multidimensional motion vector features (e.g., the horizontal velocity channel of the center of mass, the arm swing amplitude channel), this application calculates the deviation between the motion vector value in the current frame or the current time window and the normal benchmark value, and normalizes this deviation by dividing it by the standard deviation of the normal benchmark to obtain a dimensionless deviation score. The deviation scores of all feature channels are organized into a two-dimensional tensor according to time order and channel order; this two-dimensional tensor is the visually significant anomaly representation. Each element in the visual saliency anomaly representation quantifies the degree of deviation of the current behavior from normal behavior at the corresponding time point and in the corresponding motion dimension. The higher the value, the more abnormal the behavior in that dimension appears.

[0041] Preferably, to improve the robustness of dynamic imaging contrast analysis, the contrast process designed in this application does not simply use a fixed global benchmark, but rather employs an adaptive benchmark update mechanism within a sliding time window. Specifically, for each current time window, this application selects multidimensional motion vector features within several adjacent time windows (e.g., three windows before and after), calculates the local mean and local standard deviation of the motion vectors of each feature channel within these neighboring windows, and forms a dynamic benchmark. Then, the multidimensional motion vector features of the current window are compared with the dynamic benchmark to calculate the degree of offset. This design can adapt to non-sudden changes in the scene, such as gradual changes in lighting and slow camera angle drift, avoiding misjudging normal scene adaptation processes as abnormalities. For example, as the light gradually dims in the evening, the noise in pedestrian centroid detection may increase. The dynamic benchmark can follow this change, keeping the offset score for normal walking at a low level, while sudden falls will produce a higher offset score, thereby reducing the false alarm rate.

[0042] Preferably, after generating the visual saliency anomaly representation, step 2 finally generates logical anchor points for the anomalous behavior to be explained based on the linguistic prior constraint vector and the visual saliency anomaly representation. This application first obtains the linguistic prior constraint vector generated in the previous steps. Each semantic dimension feature in the linguistic prior constraint vector corresponds to an anomalous behavior judgment criterion, and the linguistic prior constraint vector has been processed through feature association topology matrix and feature space reprojection to obtain channel adaptation constraint vectors adapted to each feature channel. This application performs a channel-by-channel weighted fusion of the visual saliency anomaly representation (reflecting the degree of deviation of the actually observed behavior) and the channel adaptation constraint vector (reflecting the attention weight of the anomaly judgment criterion emphasized by the linguistic prior to each feature channel): for each feature channel, the visual saliency anomaly representation of the feature channel is multiplied by the element value of the corresponding channel adaptation constraint vector to obtain a weighted anomaly response intensity. Then, the weighted anomaly response intensities of all feature channels are summed to obtain a comprehensive anomaly response value. When the comprehensive anomaly response value exceeds a preset judgment threshold, this application determines that there is an anomalous behavior within the current time window and triggers anchor point generation.

[0043] Preferably, after triggering anchor point generation, this application further locates several consecutive time frames with the highest abnormal response intensity and their corresponding feature channel combinations from the visual saliency anomaly representation quantity. This application extracts the time coordinates of these several consecutive time frames within a visual behavior segment to be perceived with uniform time-scale weights, and combines them with the motion attributes corresponding to the feature channel with the highest abnormal response intensity (e.g., if the velocity channel in the centroid's vertical direction has the highest response intensity, it is determined to be a rapid descent behavior), encapsulating them into a structured data object. The structured data object includes the time interval of the anomaly occurrence, the motion dimension involved in the anomaly, and the quantized intensity value of the abnormal response. This structured data object serves as the logical anchor point for the abnormal behavior to be explained. The logical anchor point for the abnormal behavior to be explained provides an accurate spatiotemporal starting point and a preliminary basis for anomaly type judgment for the subsequent reconstruction of the behavioral causal chain in step 3.

[0044] Preferably, in a specific application scenario (e.g., detecting "elderly falls" in park surveillance), the above processing demonstrates a clear functional support relationship between the links. This application extracts multi-dimensional motion vector features from a visual behavior segment to be perceived with uniform time-scale weights, including a centroid vertical velocity channel and a limb joint angle change rate channel. Dynamic imaging comparison analysis reveals that within a certain 2-second window, the centroid vertical velocity channel's offset score reaches a high value (e.g., 8.5, while the normal baseline average is 0), while the limb joint angle change rate channel also shows a large offset (e.g., 6.2). At this time, in the channel adaptation constraint vector obtained after preprocessing of the language prior constraint vector, the dimension corresponding to the fall semantics assigns a high weight (e.g., 0.9) to the centroid vertical velocity channel and a medium weight (e.g., 0.6) to the limb joint angle change rate channel. After weighted fusion, the comprehensive abnormal response value exceeds a threshold. This application generates an unexplained abnormal behavior logic anchor point, which marks the start and end times of the 2-second window and indicates that the abnormal type tends to be a severe vertical displacement accompanied by a sudden change in joint angle. In this way, subsequent reconstruction of the causal chain can be based on this accurate spatiotemporal anchor point, tracing back the gait instability characteristics before the fall and analyzing the long period of stillness after the fall, thereby generating a complete explanatory map of abnormal behavior.

[0045] Optionally, logical anchor points for the anomalous behavior to be explained are generated based on the linguistic prior constraint vector and the visual saliency anomaly representation quantity, including: The visual saliency anomaly representation is matched with the language prior constraint vector through cross-modal semantic mapping to determine the anomalous attribute label of the visual saliency anomaly representation at the language logic level. Key action frames are extracted from the temporal visual behavior segment to be perceived based on the abnormal attribute labels. The extracted key action frames are spatiotemporally locked to determine the logical anchor point of the abnormal behavior to be explained.

[0046] Preferably, the specific implementation process of performing cross-modal semantic mapping matching between the visual saliency anomaly representation and the language prior constraint vector in step 2 to determine the anomalous attribute label of the visual saliency anomaly representation at the linguistic logic level is as follows: This application first obtains the visual saliency anomaly representation generated by the aforementioned dynamic imaging contrast analysis, and the language prior constraint vector generated by domain expert knowledge text encoding. The visual saliency anomaly representation is a two-dimensional tensor, each element of which quantifies the degree of deviation of behavior from the normal baseline at a specific time point and a specific motion dimension; the language prior constraint vector is a high-dimensional semantic embedding vector, each semantic dimension feature of which corresponds to a specific abnormal behavior judgment criterion (e.g., rapid descent of the human body's center of mass or abnormal arm waving). This application expands the visual saliency anomaly representation into a one-dimensional vector along the feature channel dimension, and then projects this one-dimensional vector and the language prior constraint vector into a shared cross-modal semantic space, the dimension of which is consistent with the semantic dimension of the language prior constraint vector. The projection process employs a pre-trained cross-modal mapping matrix, which is learned through a large number of paired visual anomaly representations and semantic label samples. This matrix can convert feature vectors in the visual domain into representation vectors in the semantic domain. After projecting the visual saliency anomaly representation, a visual semantic projection vector is obtained. Each dimension of this vector represents the response strength of the current visual anomaly representation on the corresponding semantic dimension feature.

[0047] Preferably, after completing cross-modal projection, this application performs semantic matching to determine the anomalous attribute labels corresponding to the visual saliency anomaly representation. Specifically, this application calculates the dimension-wise similarity between the visual semantic projection vector and the linguistic prior constraint vector, for example, using cosine similarity or dot product, to obtain the matching score for each semantic dimension feature. Since each semantic dimension feature in the linguistic prior constraint vector has been pre-associated with a readable anomalous attribute label (such as "falling," "climbing," "fighting," etc.), this application selects one or more semantic dimension features with the highest matching scores and uses the anomalous attribute labels corresponding to these features as the judgment result of the visual saliency anomaly representation at the linguistic logic level. For example, if the matching score of the semantic dimension feature "rapid descent of the human center of mass" in the visual semantic projection vector is the highest (e.g., reaching 0.92), then the system marks the visual saliency anomaly representation as the anomalous attribute label "falling." Through this cross-modal semantic mapping matching, visual features that originally only contained numerical offset degrees are given anomalous categories with clear semantic meanings, providing semantic guidance for the subsequent extraction of key action frames.

[0048] Preferably, after obtaining the abnormal attribute labels, step 2 then extracts key action frames from the perceptible temporal visual behavior segment with uniform time-stamped weights based on the abnormal attribute labels. This application uses the abnormal attribute labels as retrieval conditions, backtracking to the original perceptible temporal visual behavior segment with uniform time-stamped weights. This application first determines the approximate time interval (e.g., from the 10th to the 12th second) of the abnormal behavior based on the time window position where the highest matching score is generated in the visual saliency anomaly representation quantity. Then, within this time interval, this application analyzes the multidimensional motion vector features corresponding to each video frame in chronological order from the perceptible temporal visual behavior segment with uniform time-stamped weights, calculating the offset degree of the feature channel most relevant to the abnormal attribute label in each video frame. For example, if the abnormal attribute label is "falling," then the focus is on the vertical velocity channel of the centroid and the limb joint angle change rate channel; the system selects one or more video frames where the offset degree of these channels reaches a local maximum as candidate key frames. Furthermore, this application performs temporal clustering on these candidate keyframes, retains the frame sequence that appears consecutively and has the highest degree of offset, and identifies the intermediate frame or offset peak frame in the frame sequence as the key action frame. This key action frame represents the most discriminative instantaneous posture or motion state during the occurrence of abnormal behavior.

[0049] Preferably, to improve the accuracy of key action frame extraction, this application designs an attention filtering mechanism guided by anomaly attribute labels. This application transforms the anomaly attribute labels into an attention weight vector, the weight allocation of which for each feature channel is consistent with the aforementioned channel adaptation constraint vector. Then, this application slides a short window (e.g., 0.5 seconds) on the timeline of the visual behavior segment to be perceived, which has uniform time-scale weights, and calculates the sum of the weighted offsets of all frames within each window to obtain a time-attention response curve. The peak position of this curve is the moment when the anomalous behavior is most significant. This application extends several frames forward and backward (e.g., two frames each) from this peak moment, extracting these frames together as a set of key action frames. This design ensures that the extracted key action frames not only include the instantaneous state of the anomalous outbreak but also cover the transition frames before and after the anomalous occurrence, preserving contextual information for subsequent causal chain reconstruction. For example, for a fall, the set of key action frames may include frames of the body beginning to tilt, frames of the center of mass accelerating downwards, and frames of contact with the ground.

[0050] Preferably, after extracting the key action frames, step 2 finally performs spatiotemporal coordinate locking on the extracted key action frames to determine the logical anchor point of the abnormal behavior to be explained. For each key action frame, this application reads the time-encoded information stored in the time-weight channel of the corresponding perceptible temporal visual behavior segment with unified time-weight, and restores this time-encoded information to the absolute timestamp in the original video stream (e.g., the number of milliseconds calculated from the start of the video), forming time coordinates. Simultaneously, this application reads the two-dimensional coordinates (e.g., pixel coordinates) of the centroid on the image plane from the target motion trajectory channel corresponding to the key action frame, forming spatial coordinates. This application associates and encapsulates the time coordinates and spatial coordinates to generate a spatiotemporal coordinate tuple. For all extracted key action frames, this application arranges their spatiotemporal coordinate tuples in chronological order to form a spatiotemporal coordinate sequence. This spatiotemporal coordinate sequence, together with the previously determined abnormal attribute labels and the weighted offset values ​​corresponding to each key action frame, constitutes a structured data object, which is the logical anchor point of the abnormal behavior to be explained.

[0051] Preferably, in a specific application scenario (e.g., detecting "tourist climbing over fences" behavior in park surveillance), the above processing demonstrates a clear functional support relationship between the links. This application first matches the visual saliency anomaly representation quantity to the anomalous attribute label "climbing" through cross-modal semantic mapping matching, achieving a high matching score (e.g., 0.88). Then, based on this anomalous attribute label, this application extracts three key action frames from the visual behavior segment to be perceived, which has uniform time-scale weights: the first frame shows the tourist's hands on the top edge of the fence, the second frame shows the tourist's center of gravity rising to the top of the fence, and the third frame shows the tourist's legs stepping over the fence. This application locks the spatiotemporal coordinates of these three key action frames: the time coordinate of the first frame is 15.2 seconds, and the spatial coordinate is the left side of the fence; the time coordinate of the second frame is 15.8 seconds, and the spatial coordinate is the top of the fence; the time coordinate of the third frame is 16.3 seconds, and the spatial coordinate is the right side of the fence. After encapsulating these spatiotemporal coordinates with the abnormal attribute labels, the generated logical anchor points for the unexplained abnormal behavior clearly record the time and location information of the start, transition, and completion stages of the abnormal behavior. These logical anchor points provide accurate spatiotemporal anchoring and semantic categories for the subsequent reconstruction of the causal chain of behavior in step 3, enabling the causal graph to analyze the antecedents (such as tourists loitering near the fence) and consequences (such as the subsequent movement trajectory after entering the restricted area) of the "climbing" behavior.

[0052] Optionally, step 3: reconstruct the behavioral causal chain of the logical anchor point of the abnormal behavior to be explained, so as to generate a temporal evolution map of abnormal behavior with causal logical constraints, including: The logical anchor point of the abnormal behavior to be explained is reconstructed into a behavioral causal chain to generate behavioral explanation semantic features, which include the abnormal triggering dimension, the abnormal evolution dimension, and the abnormal impact dimension. Based on the semantic features of the behavior interpretation, a temporal evolution map of abnormal behavior with causal logic constraints is generated.

[0053] Preferably, the specific implementation process of reconstructing the behavioral causal chain of the logical anchor point of the abnormal behavior to be explained in step 3 to generate behavioral explanation semantic features is as follows: This application first obtains the logical anchor point of the abnormal behavior to be explained generated in step 2. The logical anchor point of the abnormal behavior to be explained is a structured data object, which contains abnormal attribute labels (e.g., "falling" or "climbing"), key action frame sequences and their corresponding spatiotemporal coordinate tuples, and weighted offset values ​​of each key action frame. This application uses the key action frame sequence in the logical anchor point of the abnormal behavior to be explained as the core anchor point of the causal chain, defines the time corresponding to the earliest key action frame in the sequence as the abnormal trigger start time, and defines the time corresponding to the latest key action frame in the sequence as the abnormal completion time. Based on these two moments, this application backtracks a causal tracing window of a preset length (e.g., 5 seconds before the abnormal behavior occurs) and extends a consequence observation window of a preset length (e.g., 10 seconds after the abnormal behavior occurs) in the visual behavior segment to be perceived with uniform time scale weights, thereby defining a complete time range for behavioral causal analysis and laying the time boundary for the subsequent extraction of semantic features in the three dimensions of abnormal triggering, evolution and impact.

[0054] Preferably, after defining the time frame for causal analysis, this application performs semantic feature extraction for the anomaly trigger dimension. This application arranges all video frames within the causal tracing window in chronological order and extracts the motion vector change rate of the feature channel with the highest correlation to the anomaly attribute label from each frame. For example, if the anomaly attribute label is "fall," this application focuses on the change trend of the vertical velocity channel of the centroid within the tracing window, calculating the acceleration characteristics of this channel and the critical time point when the velocity moves from the normal fluctuation range to the abnormal deviation range. This application uses the behavioral state before the critical time point (such as the gait cycle characteristics during normal walking) as the pre-trigger steady-state feature, and the behavioral state after the critical time point until the start of the anomaly trigger (such as the body beginning to tilt, gait disorder) as the trigger transition feature. This application correlates and encodes the pre-trigger steady-state feature and the trigger transition feature to form a trigger dimension semantic feature vector, which contains information about the cause of the abnormal behavior, such as sudden loss of balance during walking.

[0055] Preferably, this application then performs semantic feature extraction operations for the anomaly evolution dimension. Using a sequence of key action frames as the timeline, this application analyzes the continuous change in behavioral state from the initial triggering moment to the completion moment of the anomaly. This application performs differential analysis on adjacent frames in the key action frame sequence, calculating the change in motion vectors and the sequence of posture angle changes between every two adjacent key action frames to obtain an intermediate state description of the behavioral evolution. For example, for the behavior of "falling," the evolution dimension features include: the first stage where the body tilt angle increases from 0 degrees to 30 degrees; the second stage where the vertical velocity of the center of mass accelerates from negative to touching the ground; and the third stage where the limb joint angle changes from extension to flexion. This application concatenates these stages in chronological order to form an evolution dimension semantic feature vector. Each element in this vector corresponds to an evolution stage and records the duration, amplitude of motion, and response intensity of the feature channels for that stage. In this way, this application expands the originally discrete key action frames into a continuous behavioral evolution trajectory, revealing the intrinsic dynamic mechanism of the abnormal behavior from occurrence to completion.

[0056] Preferably, this application performs semantic feature extraction operations on the abnormal impact dimension at the end. This application uses all video frames within the consequence observation window as the analysis object, extracting the target's state changes and environmental response information after the abnormal behavior is completed from these frames. Specifically, this application monitors the numerical changes of feature channels related to the abnormal attribute label after the completion time: for example, for the "fall" behavior, the system tracks whether the vertical velocity of the center of mass approaches zero, whether the limb joint angle remains unchanged for a long time (i.e., prolonged stillness), and whether the movement trajectories of other surrounding targets show avoidance or aggregation. This application also quantifies the degree of impact by combining the contextual information of the monitored scene (such as ground material, fence height, etc.), generating an impact dimension semantic feature vector. This vector contains the consequence information of the abnormal behavior, such as "remaining still for more than 10 seconds after falling, with two people approaching to check." Thus, through the extraction operations of the above three dimensions, this application generates behavioral explanation semantic features, which is a three-part joint data structure describing the triggering cause, evolution process, and impact consequences of the abnormal behavior.

[0057] Preferably, step 3 then proceeds to generate a temporal evolution map of abnormal behavior with causal logic constraints based on the behavioral interpretation semantic features. This application first concatenates the trigger dimension semantic feature vector, evolution dimension semantic feature vector, and influence dimension semantic feature vector in the behavioral interpretation semantic features in chronological order to form a unified temporal semantic feature sequence. Then, this application defines each semantic unit in the temporal semantic feature sequence (e.g., "pre-trigger steady state," "trigger transition," "evolutionary first stage," "evolutionary second stage," "influence consequences," etc.) as an abnormal state semantic node, and assigns a unique identifier to each abnormal state semantic node. Next, this application establishes temporal evolution causal logic edges between adjacent abnormal state semantic nodes based on the chronological order and the transmission relationship between feature channels. The direction of these temporal evolution causal logic edges points from earlier nodes to later nodes, and each temporal evolution causal logic edge is assigned a causal logic label describing how the previous state leads to the occurrence of the next state (e.g., an abnormal increase in the center of mass velocity leads to an increase in the body tilt angle).

[0058] Preferably, after generating the anomalous state semantic nodes and temporal evolution causal logic edges, this application further organizes these nodes and temporal evolution causal logic edges in a structured manner to form an anomalous behavior temporal evolution graph with causal logic constraints. Specifically, this application uses the node corresponding to the start time of the anomalous trigger as the root node of the graph, the node corresponding to the completion time of the anomalous trigger as the convergence node of the graph, and arranges all nodes from left to right along the time axis. This application also adds attribute information to each node, including the timestamp, spatial coordinates, response intensity of the feature channel, and the associated original video frame index. A causal confidence score is added to each temporal evolution causal logic edge, which is calculated based on the continuity and consistency of feature changes between two consecutive nodes. The final generated anomalous behavior temporal evolution graph is a directed acyclic graph structure, in which the anomalous state semantic nodes represent the key states in the development process of anomalous behavior, and the temporal evolution causal logic edges represent the causal transformation relationships between states. This graph provides a structured causal reasoning basis for the quantitative assessment of the evidence chain credibility in the subsequent step 4, enabling the monitoring system to clearly trace back the complete logical chain "from normal to abnormal, and from abnormal to consequence".

[0059] Optionally, based on the semantic features of the behavior explanation, a temporal evolution map of abnormal behavior with causal logic constraints is generated, including: The semantic features of the behavior interpretation are mapped to the spatiotemporal coordinate system corresponding to the temporal visual behavior segment to be perceived, so as to generate spatiotemporal projection interpretation features; Spatiotemporal alignment is performed on the spatiotemporal projection interpretation features to generate spatiotemporally complete interpretation features; Feature association is performed on the spatiotemporally complete explanatory features to generate an abnormal behavior temporal evolution graph with causal logic constraints. The nodes in the abnormal behavior temporal evolution graph represent abnormal state semantic nodes, and the edges represent temporal evolution causal logic edges.

[0060] Preferably, the specific implementation process of mapping the behavioral interpretation semantic features to the spatiotemporal coordinate system corresponding to the visual behavior segment to be perceived with unified time-stamp weights in step 3 to generate spatiotemporal projection interpretation features is as follows: This application first obtains the behavioral interpretation semantic features generated by the aforementioned behavioral causal chain reconstruction process. This behavioral interpretation semantic feature is a three-part joint data structure, which includes a trigger dimension semantic feature vector, an evolution dimension semantic feature vector, and an influence dimension semantic feature vector. Each dimension semantic feature vector is composed of multiple semantic units arranged in chronological order (e.g., "steady state before triggering", "trigger transition", "evolutionary first stage", "evolutionary second stage", "influence consequences", etc.). This application also obtains the spatiotemporal coordinate system corresponding to the visual behavior segment to be perceived with unified time-stamp weights. This spatiotemporal coordinate system uses the pixel coordinates of the video frame as the spatial axis (horizontal and vertical directions) and the frame number or absolute timestamp as the time axis. For each semantic unit in the behavioral interpretation semantic features, this application maps it to a point in the spatiotemporal coordinate system according to the center time of the time window associated with the semantic unit in the original generation process and the centroid coordinates of the spatial region. Specifically, this application converts the temporal attributes of semantic units into coordinate values ​​on the time axis, and the spatial attributes of semantic units (such as the pixel coordinates of the centroid of the target in a key action frame) into coordinate values ​​on the spatial axis, thereby generating a projection point with a clear temporal and spatial location in the spatiotemporal coordinate system. All the projection points corresponding to the semantic units together constitute a discrete point set, which is the spatiotemporal projection interpretation feature. This spatiotemporal projection interpretation feature anchors abstract semantic information to a specific spatiotemporal location in the video, providing a spatial-temporal reference for subsequent alignment and association operations.

[0061] Preferably, after generating the spatiotemporal projection interpretation features, step 3 then performs spatiotemporal alignment on the spatiotemporal projection interpretation features to generate spatiotemporally complete interpretation features. Since different semantic units in the behavioral interpretation semantic features may originate from different feature channels or different temporal resolutions (e.g., the temporal granularity of the trigger dimension semantic feature vector is coarser, while the temporal granularity of the evolution dimension semantic feature vector is finer), the projection points in the directly mapped spatiotemporal projection interpretation features may have uneven distribution or timestamp misalignment on the time axis. To solve this problem, this application performs spatiotemporal alignment processing on the spatiotemporal projection interpretation features: In the time dimension, this application uses a linear interpolation method, taking the minimum time interval among all projection points as the reference time step, to interpolate and fill missing or sparse regions on the time axis, generating equally spaced time series, and associating each interpolation point with the semantic labels of its neighboring projection points; in the spatial dimension, this application uses a coordinate normalization method to unify the spatial coordinates of different projection points to the same image resolution (e.g., dividing all coordinates by the width and height of the image), eliminating spatial scale differences caused by varying target distances. After spatiotemporal alignment, the originally discrete and uneven set of projected points is transformed into a dense set of points that are regularly arranged in both time and space. This set of points is the spatiotemporally complete explanatory feature. This spatiotemporally complete explanatory feature ensures that each time step has a corresponding spatial location and semantic label, providing a complete and consistent data foundation for subsequent feature association operations.

[0062] Preferably, after completing spatiotemporal alignment, step 3 finally performs feature association on the spatiotemporally complete explanatory features to generate a temporal evolution map of abnormal behavior with causal logic constraints. This application treats each spatiotemporal point in the spatiotemporally complete explanatory features as a candidate node, arranged in chronological order. For each candidate node, this application defines its associated semantic label (e.g., "steady state before triggering," "first stage of evolution," etc.) as the type of the node and assigns it a unique identifier, forming an abnormal state semantic node. Then, based on the inherent causal dependencies in the behavioral explanatory semantic features, this application establishes connections between adjacent abnormal state semantic nodes. Specifically, for two temporally adjacent nodes, this application analyzes the feature channel changes traversed from the former to the latter: if there is a continuous change in the feature channel between the semantic unit corresponding to the former node and the semantic unit corresponding to the latter node (e.g., the centroid vertical velocity transitions from the normal range to the abnormal range), this application establishes a temporal evolution causal logic edge between these two nodes, with the edge pointing from the earlier node to the later node. For nodes that are not adjacent but have a direct causal relationship (e.g., the pre-triggered steady state directly leads to the first stage of evolution), this application can also skip intermediate nodes and establish shortcut edges to simplify the graph structure.

[0063] Preferably, in the process of establishing the temporal evolution causal logic edge, this application also attaches a causal confidence score and a causal logic label to each edge. The calculation of the causal confidence score is based on the consistency and continuity of feature channel changes within the time window corresponding to the semantic nodes of the two abnormal states: this application extracts the feature vector at the end of the time window of the previous node and the feature vector at the beginning of the time window of the next node, calculates the cosine similarity between the two and the degree of consistency of the change direction, and uses the weighted sum of the similarity and the degree of consistency as the causal confidence score (e.g., the score ranges from 0 to 1, with a higher value indicating a more certain causal relationship). The causal logic label is generated through predefined semantic mapping rules: this application matches the corresponding descriptive text (e.g., "abnormal increase in the centroid velocity leads to an increase in the body tilt angle") from a preset causal logic template library according to the common transformation relationship between the semantic labels of the previous node and the semantic labels of the next node, and attaches the descriptive text to the edge. Each temporal evolution causal logic edge simultaneously includes direction, confidence score, and semantic description, enabling the graph to not only express the temporal relationship of state transitions but also quantify causal strength and provide readable interpretations.

[0064] Preferably, after generating all anomalous state semantic nodes and temporal evolution causal logic edges, this application structures these nodes and edges to form the final graph. This application sets the earliest anomalous state semantic node (usually corresponding to the "pre-trigger steady state") as the root node and the latest node (usually corresponding to the "impact consequences") as the leaf node, and arranges all nodes hierarchically from left to right along the timeline. This application also attaches detailed attribute information to each anomalous state semantic node, including the timestamp range (start and end times), spatial coordinate region (pixel coordinate range of the target centroid), feature channel response intensity vector (average offset score of each feature channel within the node's time window), and the associated list of original video frame indices. The previously calculated causal confidence score and causal logic label are attached to each temporal evolution causal logic edge. The final anomalous behavior temporal evolution graph with causal logic constraints is a directed acyclic graph structure, where the anomalous state semantic nodes represent the key states in the entire process of anomalous behavior from pre-trigger to evolution to impact, and the temporal evolution causal logic edges represent the causal transition relationships between states. This graph differs from the isolated time point markers of traditional alarm logs. It can fully present the cause-effect logic chain, enabling the monitoring system to trace the causes of abnormal behavior, track the development process, and predict the possible impact. It provides a structured causal reasoning basis for the quantitative evaluation of the evidence chain in the subsequent step 4.

[0065] Preferably, in a specific application scenario (e.g., detecting "tourist climbing over fences" behavior in park surveillance), the above processing demonstrates a clear functional support relationship between the links. This application first maps semantic units from the behavioral interpretation semantic features, such as "lingering near the fence" (steady state before triggering), "hands on the fence" (trigger transition), "body center of gravity rising" (first stage of evolution), "legs crossing the fence" (second stage of evolution), and "entering the restricted area and remaining there" (impact consequences), onto a spatiotemporal coordinate system, generating corresponding spatiotemporal projection points. After spatiotemporal alignment, these points are evenly distributed on the time axis (e.g., one point every 0.5 seconds), and after spatial coordinate normalization, they form a smooth trajectory. Then, this application transforms these spatiotemporal points into anomalous state semantic nodes and sequentially establishes causal logic edges for temporal evolution: the edge from "loitering" to "building the fence" is labeled with the causal logic tag "approaching the fence and generating contact intention," with a high confidence score (e.g., 0.92); the edge from "building the fence" to "raising the center of gravity" is labeled with the tag "using upper limbs to pull the body up," with a medium confidence score (e.g., 0.85); the edge from "raising the center of gravity" to "crossing the fence" is labeled with the tag "lower limbs crossing to complete the climb," with a high confidence score (e.g., 0.90); and the edge from "crossing the fence" to "entering the restricted area" is labeled with the tag "entering the restricted area after successfully climbing over," with a high confidence score (e.g., 0.88). The resulting anomalous behavior temporal evolution graph clearly shows the complete causal chain of the "climbing the fence" behavior, enabling security personnel or subsequent analysis modules to accurately understand the occurrence logic of the anomalous behavior, rather than just seeing the isolated alarm of "climbing over."

[0066] Optionally, step 4: Based on the anomalous behavior temporal evolution graph, perform a quantitative evaluation of the evidence chain credibility to generate an intelligent perception and explanation description of the anomalous behavior that matches a preset business logic format, including: Based on the time-series evolution map of the abnormal behavior, a quantitative assessment of the credibility of the evidence chain is performed to generate a credibility score for multi-source heterogeneous evidence. Based on the credibility score of the multi-source heterogeneous evidence, modal behavioral evidence is obtained by parsing the temporal evolution map of the abnormal behavior to form a preliminary weighted evidence set; Based on the aforementioned preliminary weighted evidence set, conflict resolution is performed to obtain a temporally consistent evidence set. Generate an abnormal behavior perception evidence set based on the time-consistent evidence set; Based on the abnormal behavior benchmark feature library, the abnormal behavior perception evidence set is mapped to business fields to obtain an associated field mapping table. The associated field mapping table is parsed to obtain the evidence feature fields; Based on the evidence feature fields, generate intelligent perception and explanation descriptions of abnormal behaviors that match the preset business logic format.

[0067] Preferably, the specific implementation process of performing a quantitative evaluation of the evidence chain credibility based on the abnormal behavior temporal evolution graph in step 4 to generate a multi-source heterogeneous evidence credibility score is as follows: This application first obtains the abnormal behavior temporal evolution graph with causal logic constraints generated in step 3. The abnormal behavior temporal evolution graph with causal logic constraints contains multiple abnormal state semantic nodes and temporal evolution causal logic edges connecting these nodes. Each abnormal state semantic node corresponds to a key state in the development process of abnormal behavior (such as "steady state before triggering", "first stage of evolution", etc.), and each temporal evolution causal logic edge represents the causal transformation relationship between states. This application performs a quantitative evaluation of the credibility of the evidence chain on the temporal evolution graph of abnormal behavior with causal logic constraints. Specifically, for each abnormal state semantic node in the temporal evolution graph with causal logic constraints, this application extracts the spatiotemporal coordinate region associated with the abnormal state semantic node, the feature channel response intensity vector, and the causal confidence score. For each temporal evolution causal logic edge, this application extracts its causal logic label and causal confidence score. This application organizes the attribute information of these abnormal state semantic nodes and temporal evolution causal logic edges into a multidimensional evidence vector set in chronological order. Each evidence vector in this multidimensional evidence vector set corresponds to a specific behavioral state or state transition and includes the source type of the evidence (e.g., offset score from visual feature channels, semantic matching score from language priors, causal transmission strength from the graph structure, etc.). Then, this application employs a predefined evidence credibility assessment function to perform a weighted summation of the values ​​in each dimension of each evidence vector, resulting in a score between 0 and 1. This score is the multi-source heterogeneous evidence credibility score. This multi-source heterogeneous evidence credibility score quantifies the reliability of each evidence fragment in the overall abnormal behavior determination, providing a numerical basis for subsequent evidence screening and conflict resolution.

[0068] Preferably, after generating the credibility score of multi-source heterogeneous evidence, step 4 then performs the operation of parsing the modal behavioral evidence obtained from the temporal evolution graph of abnormal behavior based on the credibility score of multi-source heterogeneous evidence to form a preliminary weighted evidence set. This application traverses all abnormal state semantic nodes and temporal evolution causal logic edges in the temporal evolution graph of abnormal behavior with causal logic constraints, associating the evidence fragment corresponding to each abnormal state semantic node and each temporal evolution causal logic edge with the corresponding multi-source heterogeneous evidence credibility score. Specifically, this application assigns a weight value to each evidence fragment, which is equal to its multi-source heterogeneous evidence credibility score. Then, this application organizes all weighted evidence fragments into an ordered list according to the temporal order of the evidence fragments; this ordered list is the preliminary weighted evidence set. For example, in fall behavior detection, the initial weighted evidence set might include the following elements: Abnormal state semantic node A (steady state before triggering) weight 0.85, temporal evolution causal logic edge AB (centroid velocity transitions from normal to abnormal) weight 0.92, abnormal state semantic node B (body tilting) weight 0.78, temporal evolution causal logic edge BC (increased tilt angle) weight 0.88, and abnormal state semantic node C (centroid accelerating downward) weight 0.95, etc. This initial weighted evidence set preserves the original temporal position and credibility weight of each evidence fragment, providing basic data for subsequent conflict resolution processing.

[0069] Preferably, after forming a preliminary weighted evidence set, step 4 then performs conflict resolution processing based on the preliminary weighted evidence set to obtain a time-consistent evidence set. This application performs consistency checks on adjacent evidence segments in the preliminary weighted evidence set to identify and resolve potential logical conflicts. Specifically, this application checks whether the semantic labels of two adjacent evidence segments are logically compatible: for example, if the label of the preceding evidence segment is "centroid vertical velocity decreases" and the label of the following evidence segment is "centroid vertical velocity increases," then a directional conflict exists. For adjacent evidence pairs with conflicts, this application compares the multi-source heterogeneous evidence confidence scores of the two, retains the evidence segment with the higher score, and removes the evidence segment with the lower score. Simultaneously, the weight of the removed evidence segment is proportionally added to the subsequent propagation of the retained evidence segment. Furthermore, this application also checks the evidence coverage integrity on the timeline: if there is a blank area with a time interval greater than a preset threshold (e.g., 0.5 seconds) between two adjacent evidence segments, this application uses linear interpolation to generate a transitional evidence segment based on the characteristics of the preceding and following evidence segments, and sets its confidence score to the minimum of the preceding and following scores. After the aforementioned conflict resolution and interpolation completion processes, the logical contradictions in the initial weighted evidence set are eliminated, and the evidence coverage along the timeline becomes continuous and consistent, forming a new evidence set known as the temporally consistent evidence set. This temporally consistent evidence set achieves temporal coherence and logical compatibility among the evidence, providing a reliable data source for the subsequent generation of evidence for detecting abnormal behavior.

[0070] Preferably, after obtaining the temporally consistent evidence set, step 4 then performs the operation of generating an abnormal behavior perception evidence set based on the temporally consistent evidence set. This application clusters each evidence fragment in the temporally consistent evidence set according to chronological order, merging multiple temporally continuous and semantically related evidence fragments into one evidence group. For example, for fall behavior, the three evidence fragments—the semantic node of the pre-trigger steady-state abnormal state, the causal logic edge of the temporal evolution from steady state to tilt, and the semantic node of the tilt abnormal state—can be merged into one evidence group, which collectively describes the triggering stage of the abnormal behavior. Each evidence group corresponds to a complete stage of the abnormal behavior (triggering stage, evolution stage, and impact stage). This application extracts a representative feature vector for each evidence group, which is composed of the average of the response intensities of the main feature channels of all evidence fragments within the group, and calculates the comprehensive confidence score of the evidence group (e.g., a weighted average of the confidence scores of all evidence fragments within the group). This application arranges all evidence groups in chronological order to form a hierarchical evidence structure, which is the abnormal behavior perception evidence set. This abnormal behavior perception evidence set organizes originally discrete evidence fragments into evidence groups with semantic integrity. Each evidence group corresponds to a logical stage of abnormal behavior, which facilitates subsequent business field mapping.

[0071] Preferably, after generating the abnormal behavior perception evidence set, step 4 then performs the operation of mapping the abnormal behavior perception evidence set to business fields based on the abnormal behavior benchmark feature library to obtain an associated field mapping table. This application pre-constructs an abnormal behavior benchmark feature library, which stores the standard judgment logic and business field templates corresponding to various abnormal behaviors (such as falls, climbing, fighting, etc.) in the monitored scenario. For example, for the behavior of "falling," the business field template includes fields such as "abnormal type," "abnormal level," "trigger time," "duration," "impact range," and "handling suggestions." This application matches the representative feature vector of each evidence group in the abnormal behavior perception evidence set with the standard feature template in the abnormal behavior benchmark feature library, calculates the matching degree using cosine similarity, and selects the abnormal type with the highest matching degree as the labeling type for that evidence group. Then, based on the matched business field template, this application extracts the corresponding attribute values ​​from the evidence group: for example, extracting the abnormal trigger time from the trigger stage evidence group, extracting the abnormal duration from the evolution stage evidence group, and extracting the impact range parameters (such as the time of stillness after falling, the number of surrounding people, etc.) from the impact stage evidence group. This application fills the extracted attribute values ​​into the corresponding fields of the template, generating a key-value pair set, which is the associated field mapping table. This associated field mapping table transforms the underlying evidence features into structured fields that the business system can recognize, realizing the connection between the perception results and business logic.

[0072] Preferably, after obtaining the associated field mapping table, step 4 then performs feature field parsing on the associated field mapping table to obtain the evidence feature fields. This application parses each key-value pair in the associated field mapping table, using the key name as the field identifier and the key value as the field content, and checks the completeness and legality of the field content. For example, for the anomaly level field, this application checks whether its value is within a preset level range (e.g., level 1 to 5). If it exceeds the range, it is truncated or assigned to the nearest valid level. For the trigger time field, this application checks whether its time format conforms to a preset standard (e.g., ISO 8601 format). If it does not conform, it performs format conversion. Furthermore, this application obtains the default value for each field from the abnormal behavior benchmark feature library (e.g., filling in "unknown" when a field is missing) to complete the parsed field content. After parsing and completion, each field in the associated field mapping table is standardized into a unified data format; these standardized fields are the evidence feature fields. These evidence feature fields are the basic data units that can be directly used to generate the final interpretation description.

[0073] Preferably, after obtaining the evidence feature fields, step 4 finally executes the operation of generating an intelligent perception and interpretation description of abnormal behavior that matches a preset business logic format based on the evidence feature fields. This application arranges the evidence feature fields according to a preset business logic format template. This template defines the structure of the final output description, for example, using JSON format or natural language description format. In JSON format, this application organizes the evidence feature fields into a set of key-value pairs and adds metadata (such as generation time and algorithm version number). In natural language description format, this application uses a predefined sentence template, filling the values ​​from the evidence feature fields into the corresponding placeholder positions in the template to generate a readable text description. For example, for the "fall" behavior, the template is: "[Trigger Time] [Spatial Coordinates] [Abnormal Type] behavior occurred near [Spatial Coordinates], lasting [Duration] seconds, with an abnormality level of [Abnormality Level], and [Handling Recommendation] is suggested." This application fills the values ​​of "Trigger Time," "Spatial Coordinates," "Abnormal Type," "Duration," "Abnormality Level," and "Handling Recommendation" from the evidence feature fields into the template to generate a complete explanation description. The final generated intelligent perception and explanation description of abnormal behavior contains both structured field data and a readable report in natural language, which can be directly output to monitoring personnel or downstream business systems, realizing a complete closed loop from low-level visual perception to high-level semantic interpretation. This intelligent perception and explanation description of abnormal behavior differs from the isolated labels of traditional alarm logs. It contains a complete chain of evidence including the triggering cause, evolution process, and impact consequences of the abnormal behavior. Furthermore, all evidence has undergone credible quantification evaluation and conflict resolution processing, exhibiting high reliability and interpretability.

[0074] Optionally, based on the evidence feature fields, an intelligent perception and explanation description of abnormal behavior matching a preset business logic format is generated, including: The evidence feature fields are processed by feature attribute retrieval to extract the anomaly level parameters and anomaly trigger time corresponding to the scene to be monitored. The anomaly level parameters and the anomaly trigger time are encapsulated to generate a data packet to be orchestrated; The data packets to be orchestrated are orchestrated to generate intelligent perception and explanation descriptions of abnormal behaviors that match a preset business logic format.

[0075] Preferably, the specific implementation process of performing feature attribute retrieval processing on the evidence feature field in step 4 to extract the anomaly level parameter and anomaly trigger time corresponding to the monitored scenario is as follows: This application first obtains the evidence feature field obtained by the aforementioned feature field parsing processing. This evidence feature field is a set of key-value pairs after standardization processing, where the key name of each key-value pair is a field identifier (e.g., "anomaly type", "anomaly level", "trigger time", "duration", "impact range", "disposal suggestion", etc.), and the key value is the corresponding field content. This application performs feature attribute retrieval processing on the evidence feature field. Specifically, according to the preset retrieval rules, the key-value pair corresponding to the anomaly level parameter is located and extracted from the evidence feature field. For example, this application searches for the entry with the key name "anomaly level" in the evidence feature field, reads its key value (e.g., "level 3" or "high"), and uses this key value as the anomaly level parameter. At the same time, this application searches for the entry with the key name "trigger time", reads its key value (e.g., "2024-10-15 14:23:05"), and uses this key value as the anomaly trigger time. If no directly corresponding key name exists in the evidence feature field, this application employs fuzzy matching or rule-based reasoning to infer the anomaly level parameter based on other relevant fields (such as anomaly type combined with duration). Through this retrieval process, this application accurately extracts the core parameters required for subsequent encapsulation from the structured evidence feature fields, ensuring data integrity and usability.

[0076] Preferably, after extracting the anomaly level parameters and the anomaly trigger time, step 4 then encapsulates the anomaly level parameters and the anomaly trigger time to generate a data package to be orchestrated. This application uses the extracted anomaly level parameters and anomaly trigger time as basic data, and simultaneously extracts other relevant auxiliary fields (such as anomaly type, spatial coordinates, duration, confidence score, etc.) from the evidence feature fields to form a temporary data record. This application encapsulates this data record: First, according to a preset data encapsulation format (such as a JSON object or XML structure), each field is organized into key-value pairs; second, metadata headers are added to the encapsulated data, including data generation timestamp, algorithm version number, and the identifier of the scene to be monitored; finally, the encapsulated data undergoes integrity verification to ensure that all necessary fields (at least including the anomaly level parameters and the anomaly trigger time) exist and are in the correct format. The data object generated after the above encapsulation process is the data package to be orchestrated. This data package to be orchestrated is an intermediate data structure that retains the key information extracted from the evidence feature fields and adds metadata to facilitate subsequent orchestration processing, providing standardized input for generating the final interpretation description.

[0077] Preferably, after generating the data packet to be arranged, step 4 finally performs the arrangement of the data packet to generate an intelligent perception and interpretation description of abnormal behavior that matches a preset business logic format. This application obtains a preset business logic format template, which defines the specific structure of the final output description and may include structured formats (such as JSON, XML) or natural language formats (such as sentence templates). This application arranges the data packet to be arranged according to the template type: if the template is a JSON structured format, this application rearranges the key-value pairs in the data packet to be arranged according to the field order specified in the template, adds a root node and necessary nesting levels, and generates a string that conforms to JSON syntax; if the template is a natural language format, this application loads a predefined sentence template (e.g., [trigger time] occurs at [spatial coordinates] [abnormal type] behavior, abnormal level is [abnormal level], [handling suggestion]), and then sequentially fills the values ​​of the corresponding fields in the data packet to be arranged into the placeholder positions in the template to generate a fluent Chinese description. During the compilation process, this application also performs unit conversion or formatting of numerical values ​​according to business needs (for example, converting the anomaly level parameter "3" into "Level 3" or "Medium Risk") to make the output more in line with human reading habits.

[0078] Preferably, in a specific application scenario (e.g., detecting an elderly person falling in park surveillance), the above processing demonstrates a clear functional support relationship between the links. This application extracts the anomaly level parameter as "high" (corresponding to a value of level 4) from the evidence feature field, the anomaly trigger time as "2024-10-15 14:23:05", and simultaneously extracts the anomaly type as "fall", the actual location corresponding to the spatial coordinates "(342, 568)" as "lakeside path", the duration as "15 seconds", and the handling suggestion as "immediately dispatch security personnel to check". The system encapsulates these parameters into a data packet to be orchestrated, which is stored in JSON format as follows: {"Anomaly Type": "Fall", "Anomaly Level": "High", "Trigger Time": "2024-10-15 14:23:05", "Spatial Coordinates": "Lakeside Path", "Duration": "15 seconds", "Handling Suggestion": "Immediately dispatch security personnel to check"}. Then, based on a preset business logic format template (requiring simultaneous output of JSON and natural language description), this application generates a JSON string for downstream systems to parse, and simultaneously generates a natural language description: "At 14:23:05 on October 15, 2024, a fall occurred on the lakeside trail, with a high anomaly level and a duration of 15 seconds. It is recommended to immediately dispatch security personnel to investigate." This description is directly output to the monitoring center's large screen or pushed to the security personnel's mobile terminals, achieving a rapid closed loop from underlying visual perception to high-level business interpretation.

[0079] Preferably, to improve the adaptability and scalability of the orchestration results, the orchestration module designed in this application supports dynamically loading business logic format templates for different scenarios. This application pre-configures an independent template library for each monitored scenario (such as parks, schools, factories, and transportation hubs), with each library containing multiple output formats (such as emergency alarm templates, daily report templates, and data archive templates). Based on the identifier of the current monitored scenario and the output type specified by the caller, this application dynamically loads the corresponding business logic format template from the template library. During the orchestration process, this application also supports conditional filling: for example, when the anomaly level parameter is high, an emergency prefix is ​​automatically added to the output description, and the polling interval is shortened; when the anomaly level parameter is low, a regular report format is used. This dynamic template mechanism enables the intelligent perception and interpretation description of abnormal behavior generated by this application to flexibly adapt to the needs of different business scenarios and users, enhancing the versatility and practicality of the solution.

[0080] Preferably, after generating the final intelligent perception and interpretation description of abnormal behavior, this application also performs an output distribution operation. This application encapsulates the orchestrated description into a message according to a preset transmission protocol (such as HTTP, MQTT) and sends it to a designated receiving end (e.g., a monitoring center server, a security personnel's mobile terminal, a data storage system). Simultaneously, this application records the generated description content, generation timestamp, and corresponding data packet summary in a local log for subsequent auditing and traceability. Through the complete processing chain of retrieval and extraction—parameter encapsulation—template orchestration—dynamic adaptation—output distribution described above, this application transforms the evidence feature fields generated by the underlying multimodal perception and causal reasoning into a high-level semantic description that can be directly consumed by the business system and directly understood by security personnel. This solves the problem of low business processing efficiency caused by isolated alarm tags output by traditional monitoring solutions, achieving an intelligent leap from data perception to business interpretation.

[0081] Optionally, step 4 also includes: Obtain the standard anomaly judgment logic corresponding to the scenario to be monitored from the preset business knowledge base; Based on the aforementioned standard anomaly determination logic, anomaly determination benchmark features are generated; Based on the aforementioned anomaly determination benchmark features, an abnormal behavior benchmark feature library is constructed.

[0082] Preferably, the specific implementation process of obtaining the standard anomaly judgment logic corresponding to the monitored scenario from the preset business knowledge base in step 4 is as follows: This application first determines the type identifier of the current monitored scenario. For example, different scenarios such as parks, schools, factories, and transportation hubs have different abnormal behavior focus and judgment criteria. Based on the scenario type identifier, this application initiates a query request to the preset business knowledge base. The preset business knowledge base is a structured storage system, which stores the corresponding standard anomaly judgment logic according to scenario type. Each standard anomaly judgment logic contains a set of judgment rules for specific abnormal behaviors (such as falling, climbing, fighting, lingering, etc.). The judgment rules are described in a condition-conclusion form. For example, for the behavior of falling, the rule can be expressed as: if the rate of change of the vertical velocity of the center of mass exceeds a threshold and the angle of the limb joint changes significantly in a short period of time and then remains still for a long time, then it is judged as a fall. This application retrieves and reads the corresponding standard anomaly judgment logic set from the preset business knowledge base by accurately matching the scenario type identifier. This set is returned to the caller in a parsable data format (such as JSON or XML). This acquisition operation provides a standardized rule basis for the subsequent generation of anomaly judgment benchmark features.

[0083] Preferably, after obtaining the standard anomaly judgment logic, step 4 then executes the operation of generating anomaly judgment benchmark features based on the standard anomaly judgment logic. This application parses each standard anomaly judgment logic and extracts its key judgment elements. Specifically, for a judgment rule, this application identifies the feature channel type (e.g., vertical velocity of the center of mass, limb joint angle, rest time, etc.), feature threshold parameters (e.g., velocity change rate greater than 5 m / s², rest time exceeding 10 seconds, etc.), and logical combination relationships (e.g., "AND" or "OR") involved in the condition part of the rule. This application transforms these elements into a numerical feature vector, where each dimension of the feature vector corresponds to a benchmark value or benchmark range for a feature channel, and the element values ​​in the vector are the threshold values ​​for distinguishing between normal and abnormal behavior for the corresponding feature channel. For example, for "falling" behavior, the anomaly judgment benchmark features may include the following dimensions: vertical velocity change rate threshold (greater than 5 m / s²), limb joint angle change amplitude threshold (greater than 30 degrees), rest duration threshold (greater than 10 seconds), etc. The feature vectors corresponding to all standard anomaly judgment logics are collected to form the anomaly judgment benchmark features. This anomaly detection benchmark feature quantifies the judgment boundaries of various abnormal behaviors, providing numerical benchmark data for the subsequent construction of an abnormal behavior benchmark feature library.

[0084] Preferably, after generating the anomaly judgment benchmark features, step 4 finally executes the operation of constructing an anomaly behavior benchmark feature library based on the anomaly judgment benchmark features. This application organizes the anomaly judgment benchmark features generated in the previous step into an indexed manner according to scenario type and anomaly type. Specifically, this application uses scenario type + anomaly type as a composite primary key to store the corresponding anomaly judgment benchmark features in a high-performance key-value storage system or vector database. Simultaneously, this application adds metadata to each anomaly judgment benchmark feature, including the preset severity level of the anomaly (e.g., high, medium, low), recommended handling suggestion templates (e.g., "immediately dispatch security personnel"), and the weight coefficient of the feature channel (used for weighted matching during subsequent business field mapping). Furthermore, this application also extracts natural language description text corresponding to the standard anomaly judgment logic from a preset business knowledge base, storing it as an explanation of the anomaly type in the anomaly behavior benchmark feature library. After the above organization, the formed anomaly behavior benchmark feature library is a database that can be quickly queried and matched, where each record corresponds to a specific anomaly behavior in a specific scenario, recording the numerical judgment benchmark, business field template, and semantic explanation of the anomaly behavior. The abnormal behavior baseline feature library is invoked in the aforementioned business field mapping step 4 to match the abnormal behavior perception evidence set to the corresponding abnormal type and extract the business field.

[0085] Preferably, in a specific application scenario (e.g., the initial deployment of a park monitoring system), the above construction process demonstrates a clear functional support relationship between the links. This application first obtains the standard anomaly judgment logic corresponding to the park scenario from a pre-set business knowledge base, including fall judgment logic: if the rate of change of the human body's center of gravity in the vertical direction is greater than 5 meters per second squared, and the angle change of the limb joints is greater than 30 degrees, and the subsequent stillness time exceeds 10 seconds, then it is judged as a fall. This application parses this logic into anomaly judgment benchmark features: a threshold of 5 meters per second squared for the rate of change of the center of gravity's vertical velocity, a threshold of 30 degrees for the change of the joint angle, and a threshold of 10 seconds for the stillness time. Then, this application uses "park + fall" as the primary key to store the anomaly judgment benchmark feature in the abnormal behavior benchmark feature library, and adds metadata: the anomaly level is "high", and the handling suggestion is "immediately dispatch security personnel to investigate". At the same time, this application also processes other anomaly types such as "climbing" and "fighting", ultimately forming an abnormal behavior benchmark feature library covering a variety of abnormal behaviors in park scenarios. In subsequent operation, when the system extracts anomaly perception evidence sets from real-time behavior monitoring video streams, it can quickly identify the anomaly type and generate corresponding business fields by matching the evidence features with the anomaly judgment benchmark features in the library.

[0086] Preferably, the abnormal behavior benchmark feature library designed in this application supports dynamic updates and incremental expansion. During operation, when new abnormal behavior patterns are discovered or feedback from manual review is received, new abnormal judgment benchmark features can be added to the abnormal behavior benchmark feature library, or the threshold parameters of existing abnormal judgment benchmark features can be adjusted. Specifically, this application provides a management interface that allows authorized users to upload new standard abnormal judgment logic or modify existing logic. Upon receiving an update request, this application re-executes the parsing and abnormal judgment benchmark feature generation operations, and overwrites the original records with the updated records or inserts new records. This dynamic update mechanism enables the abnormal behavior benchmark feature library to adapt to changes in business needs and the evolution of abnormal behavior in different scenarios, avoiding the limitations of a fixed rule base and enhancing the long-term applicability and self-evolution capability of the solution in this application.

[0087] Preferably, to ensure the quality of the abnormal behavior benchmark feature library construction, this application introduces a multi-source verification mechanism when generating abnormal judgment benchmark features. This application not only obtains standard abnormal judgment logic from a pre-set business knowledge base but also performs statistical verification using historical abnormal behavior data samples. For example, this application extracts a large number of labeled normal and abnormal behavior samples from historical monitoring data, calculates the distribution difference of each feature channel across the two types of samples, and uses statistical hypothesis testing methods (such as t-tests) to evaluate the effectiveness of the judgment threshold. If the judgment threshold generated according to the standard abnormal judgment logic leads to a high false positive or false negative rate on historical samples, this application automatically adjusts the threshold parameters (e.g., adjusting the static time threshold from 10 seconds to 8 seconds) so that the adjusted abnormal judgment benchmark features achieve better classification performance on historical samples. Through this rule-driven + data verification dual mechanism, the abnormal behavior benchmark feature library constructed by this application integrates domain expert knowledge and adapts to the data distribution of actual scenarios, exhibiting high reliability and robustness. This abnormal behavior benchmark feature library provides an accurate and quantifiable reference standard for business field mapping and abnormal type matching in step 4, and is a key bridge connecting the underlying perception results and the high-level business interpretation.

[0088] like Figure 2 As shown in the figure, this application provides an embodiment of a multimodal abnormal behavior intelligent perception and interpretation device based on vision-language-temporal fusion, which includes: First program unit: used to generate a temporal visual behavior segment to be perceived based on the real-time behavior monitoring video stream of the scene to be monitored, and to perform semantic embedding encoding processing on the domain expert knowledge text that matches the scene to be monitored, so as to generate a language prior constraint vector including abnormal behavior judgment criteria. The second program unit is used to generate logical anchor points for the abnormal behavior to be explained based on the visual behavior segments to be perceived in time sequence and the language prior constraint vector. The third program unit is used to reconstruct the causal chain of the abnormal behavior logic anchor point to be explained, so as to generate an abnormal behavior temporal evolution map with causal logic constraints. The fourth program unit is used to perform quantitative evaluation of the credibility of the evidence chain based on the time-series evolution graph of the abnormal behavior, so as to generate an intelligent perception and explanation description of the abnormal behavior that matches the preset business logic format.

[0089] like Figure 3 The diagram shown is a schematic of an electronic device according to an embodiment of this application. It includes a memory and a processor. The memory stores a computer-executable program, and the processor is used to run the computer-executable program to implement the steps in the above method.

[0090] Figures 2-3 For an exemplary description, please refer to the above. Figure 1 This will not be elaborated upon here.

Claims

1. A multimodal intelligent perception and interpretation method for abnormal behavior based on vision-language-temporal fusion, characterized in that, include: Step 1: Based on the real-time behavior monitoring video stream of the scene to be monitored, generate the temporal visual behavior segment to be perceived, and perform semantic embedding encoding on the domain expert knowledge text that matches the scene to be monitored to generate a language prior constraint vector including the abnormal behavior judgment criteria. Step 2: Based on the visual behavior segments to be perceived and the language prior constraint vectors, generate logical anchor points for the abnormal behavior to be explained; Step 3: Reconstruct the causal chain of the abnormal behavior logic anchor point to be explained, so as to generate an abnormal behavior temporal evolution map with causal logic constraints; Step 4: Based on the time-series evolution graph of the abnormal behavior, perform quantitative evaluation of the credibility of the evidence chain to generate intelligent perception and explanation description of abnormal behavior that matches the preset business logic format; Step 3 includes: The logical anchor point of the abnormal behavior to be explained is reconstructed into a behavioral causal chain to generate behavioral explanation semantic features, which include the abnormal triggering dimension, the abnormal evolution dimension, and the abnormal impact dimension. Specifically, a sequence of key action frames is obtained from the logical anchor point of the abnormal behavior to be explained. The start time of the abnormal trigger is determined based on the earliest key action frame in the sequence, and the completion time of the abnormal is determined based on the latest key action frame. A preset time range before the start time of the abnormal trigger is used as a causal tracing window, and a preset time range after the completion time of the abnormal is used as a consequence observation window. The video frames within the causal tracing window are arranged in chronological order. The motion vector change rate of the feature channel with the highest correlation to the abnormal attribute label in the logical anchor point of the abnormal behavior to be explained is extracted. The critical time point when the feature channel enters the abnormal deviation range from the normal fluctuation range is determined. The behavior state before the critical time point is encoded as the pre-trigger steady-state feature, and the behavior state from the critical time point to the start time of the abnormal trigger is encoded as the trigger transition feature to generate the semantic feature of the abnormal trigger dimension. Differential analysis is performed on adjacent key action frames in the key action frame sequence to calculate the change in motion vector and attitude angle between adjacent key action frames, and multiple behavior evolution stages are organized in chronological order to generate the semantic features of the abnormal evolution dimension. Based on the behavioral explanation semantic features, the abnormal trigger dimension semantic features, the abnormal evolution dimension semantic features, and the abnormal impact dimension semantic features are concatenated in chronological order to form a temporal semantic feature sequence. Each semantic unit in the temporal semantic feature sequence is determined as an abnormal state semantic node, and a temporal evolution causal logic edge is established based on the temporal sequence relationship and feature channel transmission relationship between adjacent abnormal state semantic nodes. The abnormal state semantic nodes and the temporal evolution causal logic edges are structured to generate an abnormal behavior temporal evolution graph with causal logic constraints, wherein the abnormal state semantic nodes represent the behavioral states in the abnormal behavior evolution process, and the temporal evolution causal logic edges represent the causal transformation relationship between adjacent behavioral states.

2. The method according to claim 1, characterized in that, Step 1 generates a temporal visual behavior segment to be perceived based on the real-time behavior monitoring video stream of the scene to be monitored, including: The real-time behavior monitoring video stream is subjected to spatiotemporal dimension feature decoupling processing to generate a visual semantic representation sequence containing target motion trajectory and action form evolution information; The visual semantic representation sequence is synchronously segmented along the time axis based on a preset sampling step size to generate a visual behavior segment to be perceived with uniform time-stamp weights.

3. The method according to claim 1, characterized in that, Step 1 also includes: Analyze the temporal visual behavior segment to be perceived to determine its included feature channels; Determine the feature association topology matrix between the different semantic dimension features in the language prior constraint vector and the feature channels included in the temporal visual behavior segment to be perceived; Based on the feature association topology matrix, the language prior constraint vector is reprojected into the feature space to obtain the channel adaptation constraint vector. The channel adaptation constraint vector is mapped to the feature channel corresponding to the temporal visual behavior segment to be perceived, so as to form a constraint for generating the logical anchor point of the abnormal behavior to be explained.

4. The method according to claim 1, characterized in that, Step 2: Based on the visual behavior segments to be perceived in time sequence and the language prior constraint vector, generate logical anchor points for the abnormal behavior to be explained, including: Extract multidimensional motion vector features from the temporal visual behavior segments to be perceived; Dynamic imaging contrast analysis is performed based on the multidimensional motion vector features to generate visually significant anomaly representation quantities that characterize the degree of behavioral shift in different feature channels. Based on the linguistic prior constraint vector and the visual saliency anomaly representation quantity, generate logical anchor points for the abnormal behavior to be explained.

5. The method according to claim 4, characterized in that, Based on the linguistic prior constraint vector and the visual saliency anomaly representation quantity, logical anchor points for the anomalous behavior to be explained are generated, including: The visual saliency anomaly representation is matched with the language prior constraint vector through cross-modal semantic mapping to determine the anomaly attribute label of the visual saliency anomaly representation at the language logic level. Key action frames are extracted from the temporal visual behavior segment to be perceived based on the abnormal attribute labels. The extracted key action frames are spatiotemporally locked to determine the logical anchor point of the abnormal behavior to be explained.

6. The method according to claim 1, characterized in that, Based on the semantic features of the behavior explanation, an abnormal behavior temporal evolution map with causal logic constraints is generated, including: The semantic features of the behavior interpretation are mapped to the spatiotemporal coordinate system corresponding to the temporal visual behavior segment to be perceived, so as to generate spatiotemporal projection interpretation features; Spatiotemporal alignment is performed on the spatiotemporal projection interpretation features to generate spatiotemporally complete interpretation features; Feature association is performed on the spatiotemporally complete explanatory features to generate an abnormal behavior temporal evolution graph with causal logic constraints. The nodes in the abnormal behavior temporal evolution graph represent abnormal state semantic nodes, and the edges represent temporal evolution causal logic edges.

7. The method according to claim 1, characterized in that, Step 4: Based on the aforementioned abnormal behavior time-series evolution graph, perform quantitative evaluation of the evidence chain credibility to generate an intelligent perception and explanation description of abnormal behavior that matches a preset business logic format, including: Based on the time-series evolution map of the abnormal behavior, a quantitative assessment of the credibility of the evidence chain is performed to generate a credibility score for multi-source heterogeneous evidence. Based on the credibility score of the multi-source heterogeneous evidence, modal behavioral evidence is obtained by parsing the temporal evolution map of the abnormal behavior to form a preliminary weighted evidence set; Based on the aforementioned preliminary weighted evidence set, conflict resolution is performed to obtain a temporally consistent evidence set. Generate an abnormal behavior perception evidence set based on the time-consistent evidence set; Based on the abnormal behavior benchmark feature library, the abnormal behavior perception evidence set is mapped to business fields to obtain an associated field mapping table. The associated field mapping table is parsed to obtain the evidence feature fields; Based on the evidence feature fields, generate intelligent perception and explanation descriptions of abnormal behaviors that match the preset business logic format.

8. The method according to claim 7, characterized in that, Based on the evidence feature fields, generate an intelligent perception and explanation description of abnormal behavior that matches a preset business logic format, including: The evidence feature fields are processed by feature attribute retrieval to extract the anomaly level parameters and anomaly trigger time corresponding to the scene to be monitored. The anomaly level parameters and the anomaly trigger time are encapsulated to generate a data packet to be orchestrated; The data packets to be orchestrated are orchestrated to generate intelligent perception and explanation descriptions of abnormal behaviors that match a preset business logic format.

9. The method according to claim 8, characterized in that, Step 4 also includes: Obtain the standard anomaly judgment logic corresponding to the scenario to be monitored from the preset business knowledge base; Based on the aforementioned standard anomaly determination logic, anomaly determination benchmark features are generated; Based on the aforementioned anomaly determination benchmark features, an abnormal behavior benchmark feature library is constructed.

Images