Multi-source alarm information collaborative verification method and system based on cross-modal causal attention

Abstract

The invention discloses a multi-source alarm information collaborative verification method and system based on cross-modal causal attention. The method comprises the following steps: firstly, collecting building model data, video data and sensor data and completing standardization processing; respectively extracting an alarm feature vector, a video key / value vector and a sensor time sequence key / value vector through a special encoder; according to the method, a cross-modal bidirectional causal attention mechanism is introduced, and optimization calculation is carried out based on building space prior information: world coordinates of a sensor and a camera are obtained by analyzing a building model, a visual area of the camera and a scope of action of the sensor are calculated, a spatial relationship library and bipartite graph topological constraints are constructed, and an attention calculation range is limited; and finally, the attention output feature is mapped into a [0, 1] interval risk score, alarm authenticity and grade determination is realized, and associated video and sensing data are pushed. According to the method, cross-modal information interaction is enhanced through causal logic, the calculation efficiency and feature correlation are improved through spatial constraints, the false alarm rate is effectively reduced, and an accurate and efficient alarm verification scheme is provided for intelligent security and protection.

Description

Technical Field

[0001] This invention relates to the fields of intelligent fire protection and multimodal information technology, and in particular to a method and system for collaborative verification of multi-source alarm information based on cross-modal causal attention. Background Technology

[0002] Existing fire alarm systems primarily rely on single sensors such as smoke detectors and heat detectors for alarm activation. While these sensors are highly sensitive, they are susceptible to interference from dust, steam, and heat in complex environments, leading to false alarms. Meanwhile, fire video surveillance systems are typically deployed independently of the alarm system, lacking an effective linkage mechanism, resulting in significant information silos.

[0003] In existing technologies, when a sensor alarms, central control personnel often need to manually retrieve footage from nearby cameras for verification. However, sensor alarms are not semantically correlated with camera video, and the retrieved footage may not be directly related to the actual fire situation. This increases the burden of manual identification, causes delays in emergency response, and may even lead to a waste of fire-fighting resources.

[0004] Therefore, there is an urgent need for a new method that can overcome the differences between sensor alarms and video signals to achieve collaborative alarm verification and visual inspection. Summary of the Invention

[0005] This invention aims to provide a multi-source alarm information collaborative verification method and system based on cross-modal causal attention. Its core idea is to use the causal attention mechanism to intelligently associate and verify alarm information with video information, simulating the logical reasoning process of humans actively seeking relevant visual evidence after hearing an alarm.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, this invention proposes a collaborative verification method for multi-source alarm information based on cross-modal causal attention, comprising: S1: Collect and standardize multimodal data; the multimodal data includes building model data, video data, and sensor data; S2: The multimodal data is processed using an alarm feature encoder, a video feature encoder, and a sensor time-series feature encoder to obtain the alarm feature vector q, and the key vector is output. Sum value vector Output key vector Sum value vector ; S3: Output the key vector based on the alarm feature vector q. Sum value vector Output key vector Sum value vector It adopts a cross-modal bidirectional causal attention mechanism to conduct cross-modal information interaction based on causal relationships; S4: Transform the feature vector output by the cross-modal bidirectional causal attention mechanism into a quantifiable risk score to form an alarm decision and push relevant video and sensor data. Specifically, a spatial relationship database is constructed by calculating the spatial correspondence between sensor locations and the camera's visible area, and the calculation range of the cross-modal bidirectional causal attention mechanism is constrained by prior information about the building space.

[0007] This invention integrates multimodal data from building models, videos, and sensors, extracts differentiated features using a dedicated encoder, and achieves causal interaction of multi-source information based on a cross-modal bidirectional causal attention mechanism. Simultaneously, it utilizes prior information about the building space to construct a spatial relationship database to constrain the scope of attention calculation. This effectively solves the problems of one-sidedness in single-modal alarm information and loose correlation between cross-modal data. It not only significantly improves the accuracy and reliability of alarm verification and reduces false alarms and missed alarms, but also rapidly forms accurate alarm decisions through quantitative risk scoring, simultaneously pushing related video and sensor data to provide comprehensive data support for alarm handling and greatly improve emergency response efficiency.

[0008] The standardization process in S1 specifically includes: By combining alarm task requirements with spatial relevance, a four-dimensional quantitative evaluation system is established, including basic quality dimension, task relevance dimension, scenario dimension and historical adaptation dimension. Based on the aforementioned basic quality dimension, task association dimension, scenario dimension, and historical adaptation dimension, the weight configuration is optimized according to the dynamic weighting criterion. The dynamic weighting criterion includes: w1: The more complex the scenario and the higher the historical adaptability, the higher the weight of the basic quality. w2: The more complex the scenario and the lower the historical adaptability, the higher the task association weight; w3: The weight of the scene dimension increases linearly with complexity; w4: The weight of historical adaptation dimensions increases linearly with accuracy; Calculate data scores: Qtotal=ω1×Q1+ω2×Q2+ω3×Q3+ω4×Q4; Confirm the data level of the data score Qtotal, and perform corresponding level data augmentation processing on the multimodal data according to the data level; Where w1+w2+w3+w4=1; and the weights are updated and optimized according to a preset period.

[0009] The alarm feature encoder in S2 specifically includes: A knowledge graph for building safety alarms is constructed, defining alarm type nodes, risk level nodes, impact range nodes, and handling priority nodes as graph nodes. Based on historical alarm data and expert experience, the graph edge weights are set according to the correlation strength between nodes. A domain-pre-trained BERT model is used to convert alarm types into a 256-dimensional initial semantic vector Etype, and risk level and impact range labels into 64-dimensional vectors through a semantic mapping layer. The 64-dimensional vectors are then concatenated with the initial semantic vector Etype to form a 320-dimensional semantic feature vector Esem. Esem=Concat(Etype,Wr*Er+Ws*Es); Where Wr and Ws are learnable weight matrices, Er is the risk level vector, and Es is the influence range vector; The semantic feature vector Esem, the dynamically normalized intensity feature Xintensity, and the duration feature are fused through an attention layer; the fusion result is then converted into a fixed-dimensional alarm feature vector q through a linear mapping layer.

[0010] The video feature encoder in S2 specifically includes: Using a 3D convolutional neural network (3D-CNN) or a spatiotemporal Transformer, the spatial-temporal features of the video stream are extracted, and the key vector is output. Sum value vector .

[0011] The sensor timing feature encoder in S2 specifically includes: The sensor temporal feature encoding uses a Long Short-Term Memory (LSTM) network or a gated recurrent unit (GRU) to encode the sensor's historical data sequence, outputting a key vector. Sum value vector .

[0012] S3 includes: outputting a key vector from the alarm feature vector q. Sum value vector Output key vector Sum value vector Perform a second normalization; Match the time windows of the output key vector Kv and value vector Vv with the time windows (30s) of the output key vector Ks and value vector Vs, take the sensor data segments within 30s corresponding to the output key vector Kv and value vector Vv, filter irrelevant modal features through the semantic label of q, and retain semantically relevant features; The split vectors are mapped to the causal feature space through a linear projection layer to enhance causal correlation. The Xavier normal distribution is used for initialization. The bidirectional attention weights of video and sensor, and sensor and video are calculated separately. The average of the forward and reverse weights is taken to obtain the final bidirectional causal attention weights. Based on bidirectional causal weights, the key vector Kv and value vector Vv are output respectively, and the key vector Ks and value vector Vs are output respectively, preserving the original semantics and injecting causal information; Final output: Vector-enhanced video key vector, enhanced video value vector, enhanced sensor key vector, and enhanced sensor value vector.

[0013] The process involves calculating the spatial correspondence between sensor locations and the camera's field of view to construct a spatial relationship database. Prior information about the architectural space is then used to constrain the calculation range of the cross-modal bidirectional causal attention mechanism. Specifically, this includes: Extract world coordinate information from sensors and cameras; Calculate the visible area polygon Dc for each camera; calculate the scope Ds for each sensor; Based on spatial domain and distance rules, a candidate camera set for sensors and a candidate sensor set for cameras are defined; an undirected bipartite graph is formed with the sensor set and the camera set as two vertex sets and the relationships in the candidate sets as edges. A multidimensional binary matrix is ​​constructed. Based on the multidimensional binary matrix, sensor or camera basic information, visible area polygon Dc, sensor scope Ds, and candidate set, a topological relation library is formed. The sensor and camera topological constraints in the topological relation library are used as the computational boundary of the cross-modal bidirectional causal attention mechanism.

[0014] Based on the same inventive concept, this invention also provides a multi-source alarm information collaborative verification system based on cross-modal causal attention, comprising: Multimodal input module: collects and standardizes multimodal data; the multimodal data includes building model data, video data, and sensor data; Feature encoding module: This module performs feature calculations on the multimodal data using an alarm feature encoder, a video feature encoder, and a sensor time-series feature encoder to obtain an alarm feature vector q and output a key vector. Sum value vector Output key vector Sum value vector ; Cross-modal causal attention module: Outputs a key vector based on the alarm feature vector q. Sum value vector Output key vector Sum value vector It adopts a cross-modal bidirectional causal attention mechanism to conduct cross-modal information interaction based on causal relationships.

[0015] Fusion Decision Module: Transforms the feature vectors output by the cross-modal bidirectional causal attention mechanism into quantifiable risk scores, forms alarm decisions, and pushes relevant video and sensor data.

[0016] Spatial topology mapping module: By calculating the spatial correspondence between sensor points and the camera's visible area, a spatial relationship database is constructed, and the calculation range of the cross-modal bidirectional causal attention mechanism is constrained by prior information about the architectural space.

[0017] Based on the same inventive concept, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a multi-source alarm information collaborative verification method based on cross-modal causal attention as described above.

[0018] Based on the same inventive concept, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements a multi-source alarm information collaborative verification method based on cross-modal causal attention as described above.

[0019] Compared with the prior art, the technical advantages of the present invention are as follows: This invention effectively addresses the key issues of high false alarm rates and isolated multi-source information in traditional fire alarm systems by introducing a cross-modal causal attention mechanism and spatial topological constraints. Specifically, firstly, by simulating the causal reasoning logic of alarm-evidence, cross-modal interaction is elevated from shallow feature alignment to deep semantic association verification, significantly reducing false alarms caused by environmental interference and improving the accuracy of alarm judgment. Secondly, by utilizing spatial prior information provided by BIM or digital twins to construct sensor-camera topological relationships, attention calculation is constrained to a limited range related to physical space, avoiding computational redundancy caused by global search and greatly improving system response efficiency. Thirdly, it supports bidirectional verification processes, such as sensor alarms being verified by video and video alarms being verified by sensors, realizing cross-complementarity and collaborative decision-making of multimodal information and enhancing the robustness of the system in complex scenarios. Attached Figure Description

[0020] Figure 1 This application presents a system architecture diagram of a multi-source alarm information collaborative verification method based on cross-modal causal attention in an embodiment of the present application.

[0021] Figure 2 This application embodiment presents a schematic diagram of a multi-source alarm information collaborative verification method based on cross-modal causal attention. Detailed Implementation

[0022] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Example

[0023] To address the aforementioned technical problems, this application provides a collaborative verification method for multi-source alarm information based on cross-modal causal attention, such as... Figure 1-2 As shown, it specifically includes: S1: Collect and standardize multimodal data; the multimodal data includes building model data, video data, and sensor data; Preprocessing of collected multimodal data: The collected multimodal data is formatted and redundancy is cleaned, and missing values ​​are handled in a targeted manner.

[0024] The preprocessing results are standardized, specifically including: establishing a four-dimensional quantitative evaluation system based on alarm task requirements and spatial relevance, including basic quality, task relevance, scenario, and historical adaptation dimensions. Basic quality and task relevance are the core dimensions, while scenario and historical adaptation are auxiliary dimensions. Evaluation Dimensions Key Indicators Quantization method (Q∈[0,1]) Basic quality (Q1) Architectural models: deviation rate, annotation completeness; Video: PSNR, frame rate stability; Sensors: sampling rate, missing rate, signal-to-noise ratio. A new quality level mapping table has been added, such as PSNR=28dB corresponding to Q1=0.8, and PSNR=25dB corresponding to Q1=0.6. Task association (Q2) Spatial correlation (distance), temporal correlation (overlap of peak periods), and content correlation (matching degree of abnormal scenarios). Optimize distance weighting: For critical areas (such as power distribution rooms), add 0.5 for distances ≤3m, 0.3 for distances 3-5m, and 0.1 for distances >5m. Scenario Dimension (Q3, New) Alarm scenario complexity: such as alarms in densely populated areas (high complexity), alarms in empty server rooms (low complexity); Environmental interference: such as video in rainy weather (high interference), video in constant temperature indoor environments (low interference). Construct a scene complexity-interference matrix. High complexity + high interference corresponds to Q3 = 0.3 (requires stronger enhancement); low complexity + low interference corresponds to Q3 = 0.9 (light enhancement is sufficient). History adaptation (Q4, added) The accuracy of this modal data in historical alarm scenarios: for example, the historical accuracy of video data in a rainy day scenario involving intruders is 85%. Q4 = historical accuracy / 100. For example, if the accuracy is 85%, Q4 = 0.85. In this embodiment, to provide high-quality and highly relevant data for the subsequent cross-modal causal attention module, the weight ratio of the two core dimensions of basic quality and task relevance is prioritized, and then dynamically adjusted through the two auxiliary dimensions of scene dimension and historical adaptation.

[0025] In this embodiment, the weight configuration is optimized according to the dynamic weighting criteria based on the basic quality dimension, task association dimension, scene dimension, and historical adaptation dimension.

[0026] The dynamic weighting criterion includes: w1: The more complex the scenario and the higher the historical adaptability, the higher the weight of the basic quality. w2: The more complex the scenario and the lower the historical adaptability, the higher the task association weight; w3: The weight of the scene dimension increases linearly with complexity; w4: The weight of historical adaptation dimensions increases linearly with accuracy; Calculate data scores: Qtotal=ω1×Q1+ω2×Q2+ω3×Q3+ω4×Q4; Confirm the data level of the data score Qtotal, and perform corresponding level data augmentation processing on the multimodal data according to the data level; Where w1+w2+w3+w4=1.

[0027] To avoid the weights failing to adapt to changes in the scenario, it is preferable to update and optimize the weights according to a preset cycle.

[0028] Core dimensions (Q1+Q2): Total basic weight ratio = 0.3 (Q1 scenario adjustment coefficient) + 0.2 (Q1 historical adjustment coefficient) + 0.3 (Q2 scenario adjustment coefficient) + 0.2 (Q2 historical adjustment coefficient) = 0.8 in 1.0, ensuring that the core dimensions always occupy a dominant position.

[0029] Auxiliary dimensions (Q3+Q4): Their effects are embedded into the weight calculation of the core dimensions through coefficients of 0.3 / 0.2, so as to avoid excessive interference of the core objectives by the auxiliary dimensions. For example, Q3 affects the weights of Q1 and Q2 through a coefficient of 0.3, which achieves scenario adaptation while ensuring the importance of the core dimensions.

[0030] Preferably, when the scenario is highly complex, such as an alarm in a densely populated area during rainy weather, when Q3=0.3, 0.3×Q3=0.09, the weight of task association Q2 will increase by 0.09, while the weight of basic quality Q1 will decrease by 0.3×(1-Q3)=0.3×0.7=0.21. At this time, the weight of Q2 is relatively improved, which meets the actual need of ensuring the correlation between data and tasks in complex scenarios.

[0031] Preferably, when the historical adaptability is high, such as when the video data has a historical accuracy of 90% in a rainy scene and Q4=0.9, 0.2×Q4=0.18, which will increase the weight of the basic quality Q1 by 0.18, while decreasing the weight of the task association Q2 by 0.2×(1-Q4)=0.02. At this time, the weight of Q1 is relatively improved, which is consistent with the basic quality of historical reliable data being more credible, and there is no need to overly rely on the logic of association verification.

[0032] In this embodiment, the data levels include: Grade A (Q_total≥0.9): High-quality adapted data, with only lightweight noise reduction; Grade B (0.7 ≤ Q_total < 0.9): Well-adapted to the data, with targeted local enhancements; Level C (0.5 ≤ Q_total < 0.7): Generally adaptable data, enhanced across all dimensions; Level D (Q_total < 0.5): Low-fit data, cross-modal collaborative enhancement + quality repair.

[0033] The multimodal data is augmented according to its data level, specifically including: For Level A data, Qtotal ≥ 0.9: Lightweight augmentation is used for data augmentation. Architectural model data: Only key area annotations are verified; no coordinate correction is performed. Video data: Fast noise reduction is employed, with Gaussian filtering being the preferred method for noise reduction. Super-resolution processing is avoided to prevent excessive enhancement that could lead to distortion. Sensor data: Only obvious outliers, such as data that exceeds the measurement range, are removed, while the original time series trend is preserved.

[0034] For Level B data (0.7 ≤ Qtotal < 0.9): Data augmentation is performed using localized, targeted augmentation. When the deviation rate of building model data is 5%-8%, local laser correction is adopted. Laser scanning correction is only performed on areas with deviations exceeding 5%, such as the positioning of equipment in the computer room, to reduce the amount of calculation for global correction. For video data, such as PSNR=25-30dB, in rainy scenes: optimize the priority enhancement of abnormal areas, add environmental interference compensation, and in rainy videos, the DerainNet deraining algorithm is preferred to remove rain streaks, and then perform super-resolution enhancement on abnormal areas to avoid rain streaks affecting the recognition of abnormal areas. Sensor data with a missing rate of 3%-5% fluctuates significantly: upgrade the temporal attention interpolation, combine the concurrent data of adjacent sensors in the same area (such as the temperature sensor referencing the fluctuation trend of the humidity sensor), and adjust the attention weight to make the interpolated data more consistent with the actual environmental changes.

[0035] For C-level data (0.5 ≤ Qtotal < 0.7): Data augmentation is performed across all dimensions. When the deviation rate of building model data is 8%-10%, laser point cloud and BIM model fusion correction is adopted. The laser-scanned point cloud data is matched with the BIM model point by point to generate a deviation heat map. The coordinates of the areas with deviation >8% in the heat map are reconstructed, and the deviation rate after correction is ≤5%. For video data, if PSNR < 25dB, in densely populated scenes: a new cross-frame semantic fusion is added, which not only aligns the 10 adjacent video frames, but also extracts semantic labels such as people and equipment in each frame through semantic segmentation models such as MaskR-CNN, and fuses the features of the same semantic target in different frames to improve the clarity of abnormal targets, such as people carrying suspicious items. When sensor data has a missing rate of 5%-8% and a signal-to-noise ratio of 10-15dB: optimize multi-sensor collaborative repair and build a correlation model between sensors and the environment. For example, when temperature sensor data is missing, combine the air conditioner operating status and adjacent temperature sensor data, and use the GRU model for prediction, which can reduce the prediction error to ≤3%.

[0036] For Level D data (Q_total < 0.5): cross-modal collaborative augmentation is employed for data augmentation. When the quality of single-modal data is extremely low, the complementarity of multimodal data can be used for collaborative repair to overcome the bottleneck of single-modal enhancement. Preferably, when video data has a PSNR < 20dB (Level D) due to heavy fog, the sensor data is considered high quality (Level A).

[0037] Enhancement logic: ① Extract personnel movement signals from sensor data (e.g., infrared sensors detect personnel trajectories); ② Map personnel trajectory coordinates to pixel coordinates of video frames; ③ Guided by personnel trajectories, perform semantically guided super-resolution on corresponding areas in the video (e.g., focus on enhancing only the areas covered by the trajectory, and appropriately defogging other areas) to avoid noise amplification caused by global enhancement.

[0038] Preferably, when there is a sensor data missing rate >8% (Grade D), the video data is good (Grade B).

[0039] Enhanced logic: ① Extract environmental state features from video data (e.g., identify the color of the device's indicator light through video frames to determine if the device is functioning properly); ② Use environmental state features as constraints and input them into the multi-sensor collaborative repair model; ③ When predicting missing data, ensure that the predicted value is consistent with the environmental state in the video (e.g., if the video shows a red light alarm on the device, the predicted temperature data must meet the alarm threshold), thereby improving the rationality of the repaired data.

[0040] S2: The multimodal data is processed using an alarm feature encoder, a video feature encoder, and a sensor time-series feature encoder to obtain the alarm feature vector q, and the key vector is output. Sum value vector Output key vector Sum value vector .

[0041] Preferably, the alarm feature encoder in S2 specifically includes: A knowledge graph for building safety alarms is constructed, defining alarm type nodes, risk level nodes, impact range nodes, and handling priority nodes as graph nodes; based on historical alarm data and expert experience, the graph edge weights are set according to the correlation strength between nodes. Four types of core graph nodes are defined, as shown in the table below: Node type Node instance Node attributes Alarm type node Smoke alarm, flame alarm, electrical overload alarm, intrusion alarm, abnormal temperature alarm, access control malfunction alarm Type ID, trigger threshold (e.g., smoke alarm threshold: concentration ≥ 0.1 mg / m³) Risk level nodes High risk, medium risk, low risk Risk coefficient (high risk = 1.2, medium risk = 1.0, low risk = 0.8) Nodes affected Single-point impact (e.g., a sensor in a specific room), area impact (e.g., a specific floor), global impact (e.g., the entire building). Impact coefficients (global impact = 1.0, regional impact = 0.6, single-point impact = 0.3) Priority nodes to handle Emergency (≤5min response), Normal (≤30min response), Low-progression (≤2h response) Priority weights (urgent = 0.9, normal = 0.6, low risk = 0.3) In this embodiment, based on the historical alarm data of a building over the past three years, totaling 1260 valid alarm records, and the evaluation of two security experts, a weighted approach combining data statistics and expert scoring was adopted, with a preferred weight ratio of 7:3 to determine the correlation strength between nodes. Some core edge weights are as follows: Alarm type and corresponding risk level: Smoke alarm: High risk, weight 0.9; Temperature anomaly alarm: Medium risk, weight 0.7; Access control anomaly alarm: Low risk, weight 0.5; Alarm type and corresponding impact range: Flame alarm: Global impact, weight 0.9; Electrical overload alarm: regional influence, weight 0.8; Intrusion alarm: Single point of impact, weight 0.6; Alarm type and corresponding handling priority: Electrical overload alarm: Emergency, weight 1.0; Smoke alarm: Emergency, weight 0.9; Temperature anomaly alarm: Normal, weight 0.7.

[0042] A domain-pretrained BERT model is used to convert alarm types into a 256-dimensional initial semantic vector Etype. Risk level and impact range labels are converted into 64-dimensional vectors through a semantic mapping layer and concatenated with the initial semantic vector Etype to form a 320-dimensional semantic feature vector Esem. Esem=Concat(Etype,Wr*Er+Ws*Es); Where Wr and Ws are learnable weight matrices, Er is the risk level vector, and Es is the influence range vector; The semantic feature vector Esem, the dynamically normalized intensity feature Xintensity, and the duration feature are fused through an attention layer; the fusion result is then converted into a fixed-dimensional alarm feature vector q through a linear mapping layer.

[0043] Preferably, the dynamically normalized intensity feature Xintensity has the following characteristics: the original intensity is level 0-10, which is then dynamically normalized to a scalar in the range [0,1] after scene adaptation, and then converted into a 320-dimensional vector through a linear expansion layer, consistent with the Esem dimension. For example, the original value of the gas leak alarm intensity in a high-risk area is level 9, and after normalization, Xintensity is 1.0 (after truncation), and the expanded vector norm value is 0.88.

[0044] Dynamically normalized duration feature Xduration: The original duration is 0-3600s, and after dynamic normalization, it becomes a scalar in the interval [0,1]. It is also expanded into a 320-dimensional vector. For example, the duration of the abnormal temperature alarm in the storage area is 300s, and after normalization, Xduration = 0.21, and after expansion, the vector norm value is 0.75.

[0045] In this embodiment, based on the publicly available BERT-base model with a 12-layer Transformer and 768 hidden layer dimensions, a fine-tuning dataset is composed of 1260 historical building alarm logs and 5000 publicly available building safety alarm datasets to adapt the model to the relevant domain. Input format: [CLS] Alarm type: Smoke alarm [SEP] Trigger area: 15th floor computer room [SEP]; Fine-tuning task: Semantic classification of alarm types: 6 alarm types, 30 training epochs, learning rate 2e-5, batch size=32, after training, the model achieved a classification accuracy of 98.2% on the test set; Semantic vector extraction: The output vector of the BERT model at the [CLS] position is 768-dimensional. It is converted into an initial semantic vector Etype of 256 dimensions through a linear dimensionality reduction layer: the weight matrix dimension is 768×256. This ensures that the cosine similarity of Etype for different alarm types is consistent with the edge weights of the graph. For example, the Etype similarity between smoke alarm and flame alarm is 0.9, and the similarity between smoke alarm and access control anomaly alarm is 0.4.

[0046] Preferably, the generation of the risk level and impact range vector specifically includes: Vector initialization: Initialize a 64-dimensional vector Er_raw for each risk level (high / medium / low) (high risk: [1,0,0,...], medium risk: [0,1,0,...], low risk: [0,0,1,...]); initialize a 64-dimensional vector Es_raw for each impact range (global / regional / single point) (global impact: [1,0,0,...], regional impact: [0,1,0,...], single point impact: [0,0,1,...]). Semantic mapping layer training: Construct a 2-layer fully connected semantic mapping layer, including a 64-dimensional input, a 128-dimensional hidden layer, and a 64-dimensional output. Use the matching degree between vector and graph edge weights as the loss function to train the mapping layer parameters and obtain the learnable weight matrices Wr (64×64) and Ws (64×64). Final vector calculation: The 64-dimensional risk level vector Er and the influence range vector Es are obtained by Er=Wr×Er_raw and Es=Ws×Es_raw. For example, the norm value of the Er (high risk) vector of smoke alarm is 0.92, and the norm value of the Es (area influence) vector is 0.85.

[0047] In this embodiment, the multi-feature attention fusion and query vector generation specifically include: setting attention weights based on alarm handling priorities; for emergency priority alarms, such as electrical overload and flame alarms, the intensity feature weight β=0.4, the semantic feature weight α=0.3, and the duration weight γ=0.3; for regular priority alarms, such as smoke and abnormal temperature alarms, the semantic feature weight α=0.4, the intensity feature weight β=0.3, and the duration weight γ=0.3; and for low-priority alarms, such as access control anomaly alarms, the duration weight γ=0.4, the semantic feature weight α=0.3, and the intensity feature weight β=0.3.

[0048] Preferably, the fusion of the semantic feature vector Esem, the dynamically normalized intensity feature Xintensity, and the duration feature through an attention layer specifically includes: Based on the priority nodes of the building safety alarm knowledge graph and combined with the park's safety management regulations, basic priority weights wbase (value range [0,1]) are assigned to various alarm types. The wbase of the core alarm types is shown in the table below: Alarm Types Priority of handling wbase Priority level Gas leak alarm in production workshop urgent 0.95 High priority Electrical overload alarm in R&D center urgent 0.90 High priority Smoke alarm in warehouse area conventional 0.75 Medium priority Office building temperature abnormality alarm conventional 0.65 Medium priority Park access control anomaly alarm slow 0.55 low priority Green belt irrigation overflow alarm slow 0.40 low priority In this embodiment, the following weight allocation logic is applied, but it is not limited to this: High-priority alarms with wbase ≥ 0.8: trigger weight rules b = 0.4, a = 0.3, c = 0.3. For example, a gas leak alarm in a production workshop with wbase = 0.95 has the highest intensity feature weight, highlighting the severity of the anomaly.

[0049] Medium priority alarms 0.6≤wbase<0.8: trigger weight rules a=0.4, b=0.3, c=0.3. For example, for a smoke alarm in a warehouse area, wbase=0.75, the semantic feature has the highest weight, strengthening the association between alarm type and scene. Low-priority alarms with wbase < 0.6: trigger weight rules c = 0.4, a = 0.3, b = 0.3. For example, for an abnormal alarm at a park access control system with wbase = 0.55, the duration has the highest weight, so pay attention to the continuous impact of the abnormality.

[0050] In this embodiment, the fusion calculation process includes: taking a medium-priority smoke alarm in a warehouse area as an example, determining the weight: wbase=0.75, which belongs to medium priority, and assigning weights a=0.4, beta=0.3, and c=0.3, satisfying a+b+c=1.

[0051] Semantic feature weighting: a×Esem=0.4×Esem (320 dimensions, highlighting semantic associations); Intensity feature weighting: b×Xintensity=0.3×Xintensity (320 dimensions, preserving intensity information); Duration feature weighting: c×Xduration=0.3×Xduration (320 dimensions, including duration effect); Fusion output: The three weighted vectors are added element by element to obtain a 320-dimensional fusion feature vector Ffusion. The formula execution result is: Ffusion=0.4×Esem+0.3×Xintensity+0.3×Xduration.

[0052] Furthermore, the 320-dimensional fusion feature vector Ffusion is converted into a fixed-dimensional alarm feature vector q through a linear mapping layer.

[0053] The video feature encoder in S2 specifically includes: It employs a 3D convolutional neural network (3D-CNN) or a spatiotemporal Transformer to extract the spatial-temporal features of the video stream and output the key vector Kv and the value vector Vv.

[0054] In this embodiment, a three-dimensional convolutional neural network (3D-CNN) is used, specifically including: Preferably, a lightweight 3D-CNN structure using convolution, pooling, bottleneck layers, and fully connected layers is adopted, as shown in the table below: Input layer video clip: 32 frames × 256 × 256 × 3 (frame length × height × width × number of channels) 32 × 256 × 256 × 3 receives standardized video input.

[0055] 3D Convolutional Layer 1 Convolutional Kernel: 3×3×3 (temporal×spatial×spatial), Number=64, Stride=1, Padding=1, Activation Function=ReLU32×256×256×64 Initially extracts spatiotemporal local features.

[0056] 3D max pooling layer 1 pooling kernel: 2×2×2, step size = 216×128×128×64 dimensionality reduction and parameter reduction, retaining key features.

[0057] 3D Convolutional Layer 2 Convolutional Kernel: 3×3×3, Number=128, Stride=1, Padding=1, Activation Function=GELU16×128×128×128 Enhances Spatiotemporal Correlation Feature Extraction.

[0058] 3D max-pooling layer 2 pooling kernel: 2×2×2, step size = 28×64×64×128 further reduces dimensionality and improves computational efficiency.

[0059] 3D convolutional layer with 3 convolutional kernels: 3×3×3, number of kernels = 256, stride = 1, padding = 1, activation function = GELU. 8×64×64×256 depth extraction of complex spatiotemporal features.

[0060] Global average pooling layer pooling kernel: 8×64×64, outputting 1×1×1×256256-dimensional compressed spatiotemporal dimension to obtain global features.

[0061] The bottleneck layer is a fully connected layer with 256-dimensional input and 512-dimensional output. The activation function is Swish, which combines 512-dimensional feature dimension expansion and nonlinear transformation.

[0062] Output layer (dual branch) branch 1-Kv: fully connected layer, 512-dimensional → 512-dimensional, no activation function, 512-dimensional output video key vector.

[0063] Branch 2-V_v: Fully connected layer, 512-dimensional → 512-dimensional, with no activation function, outputting a 512-dimensional video value vector.

[0064] Further, video preprocessing is performed: Video stream framing: Preferably, the input video stream is segmented into 32 frames per segment using a sliding window. The selectable window step size is 16 frames, and the overlap rate is 50%. Each frame image is normalized to the [0,1] interval, the mean of the dataset is subtracted, and the result is divided by the standard deviation. The resolution is uniformly adjusted to 256×256. During training, the resolution is randomly cropped to 224×224, and during inference, the resolution is centered and cropped to 224×224.

[0065] We used a publicly available security video dataset, including a subset of the UCF101 behavior recognition dataset and a self-built park alarm video dataset, totaling 12,000 video clips, containing 10 types of abnormal behaviors such as intrusion, loitering, climbing, and fire / smoke. The optimal optimizer was AdamW, the learning rate was 2e-5, the training epochs were 50, the batch size was 16, and the loss function was cross-entropy loss (for classification tasks) + contrastive loss (for feature discrimination optimization). Preferably, Dropout is added to the convolutional layers with a probability of 0.2, and L2 regularization (λ=1e-4) is added to the fully connected layers to avoid overfitting. After global average pooling and bottleneck layer processing, a 512-dimensional key vector Kv for attention weight calculation and a value vector Vv for feature fusion are generated through a dual-branch output layer. Before output, the vectors are L2 normalized to ensure scale consistency with the alarm feature vector q and the sensor feature vector.

[0066] In another embodiment, for complex behaviors, such as multi-person collaboration anomalies and dynamic occlusion scenarios, spatiotemporal Transformer video feature encoding is employed, specifically including: The continuous video stream is converted into a discrete spatiotemporal block sequence, and a low-dimensional embedding vector is generated through linear projection to inject spatiotemporal location information.

[0067] Preferably, spatial segmentation: the single frame 224×224 image is divided into blocks of 16×16 pixels, and the number of blocks in a single frame = (224 / 16)×(224 / 16)=196 blocks; Time Blocking: Divide the 16-frame video into 2-frame time blocks. The number of time blocks = 16 / 2 = 8 blocks.

[0068] Through the linear projection layer, the weight matrix dimension is 3072×768, which compresses the dimension of a single spatiotemporal block into a 768-dimensional embedding vector. The linear projection layer uses the Xavier normal distribution to initialize the weights. The normalized embedding vector is split into 12 attention heads, and the spatiotemporal dependencies are computed in parallel. The steps are as follows: Xnorm is transformed into query vector Q, key vector K, and value vector V through three linear projection layers (Wq, Wk, and Wv, each with a dimension of 768×768); Q, K, and V are split according to the number of attention heads, and the spatiotemporal attention score is calculated for each head vector. The Score_head is normalized using the Softmax function to obtain the attention weight of each spatiotemporal block on other blocks. The normalized weights are multiplied by V_head to obtain 12 784×64-dimensional context vectors. The 12 context vectors C are concatenated along the dimension axis to obtain a 784×768-dimensional multi-head attention vector. The attention output is added to the original input of the current layer to complete the residual connection, and a nonlinear transformation is performed to obtain the feedforward network output. The feedforward network output is added to the residual connection result to obtain the final output of the current spatiotemporal attention layer. Based on the above process, six spatiotemporal attention layers are stacked, and the parameters of each layer are trained independently to deeply capture the complex spatiotemporal dependencies in the video stream. After processing by the six spatiotemporal attention layers, a 784×768-dimensional spatiotemporal block feature sequence is obtained. The feature vectors of the 784 spatiotemporal blocks are averaged to compress the spatiotemporal block dimension. The 768-dimensional global features are further converted to 512-dimensional through a linear mapping layer. A dual-branch output layer is designed to generate the key vector Kv and the value vector Vv respectively.

[0069] In this embodiment, the sensor timing feature encoder in S2 specifically includes: the sensor timing feature encoding uses a Long Short-Term Memory (LSTM) network or a gated recurrent unit (GRU) to encode the sensor's historical data sequence and output a key vector Ks and a value vector Vs.

[0070] Preferably, when using a Long Short-Term Memory (LSTM) network, the specific steps include: performing forward LSTM computation and backward LSTM computation sequentially, followed by bidirectional concatenation; performing global temporal pooling; reducing the feature dimension through fully connected layers; and preferably, adding the GELU activation function and dropout (p=0.2) to enhance feature generalization ability.

[0071] Preferably, a gated recurrent unit (GRU) is used to encode the sensor's historical data sequence. Specifically, this involves employing a two-layer bidirectional GRU structure to deeply capture the forward and backward dependencies of the time-series data, with each layer's output undergoing dropout regularization. The GRU output is a 300-step time-series feature sequence, which needs to be compressed into fixed-dimensional global features through global pooling. The global features are then converted to 512 dimensions, and an independent dual-branch structure is designed to generate a key vector Ks and a value vector Vs, respectively.

[0072] In this embodiment, S3: Based on the alarm feature vector q, output key vector Kv and value vector Vv, output key vector Ks and value vector Vs, and adopt a cross-modal bidirectional causal attention mechanism to perform cross-modal information interaction based on causal relationships.

[0073] Preferably, the alarm feature vector q, output key vector Kv and value vector Vv, and output key vector Ks and value vector Vs are normalized twice. The time window (16 frames, approximately 0.67s) of the output key vector Kv and value vector Vv is matched with the time window (30s) of the output key vector Ks and value vector Vs. The sensor data segments within 30s corresponding to the output key vector Kv and value vector Vv are taken, and irrelevant modal features (such as vibration sensor features filtered by fire alarm) are filtered out by the semantic label of q (e.g., fire), while semantically relevant features are retained. q, Kv, and Ks are split into 8 heads, each with a dimension of 64. The split vectors are mapped to the causal feature space through a linear projection layer to enhance causal correlation. Xavier normal distribution is used for initialization to ensure the stability of the feature distribution after projection.

[0074] Furthermore, the bidirectional attention weights for video and sensor are calculated separately, and causal coefficient modulation is incorporated: Positive causal flow weights: Calculate the similarity between q and the output key vector Kv, and use it as the priority weight for video features; Based on the similarity between q and the output key vector Kv, calculate the causal association score between the output key vector Kv and the output key vector Ks; Based on the similarity between q and the output key vector Kv, calculate the causal association score between the output key vector Kv and the output key vector Ks; The three results above are then weighted and normalized.

[0075] Similarly, reverse causal flow weights: Calculate the similarity between q and the output key vector Ks; based on the similarity between q and Ks, calculate the causal association score between Ks and Kv; and normalize the weights of these two results.

[0076] The final bidirectional causal attention weights are obtained by averaging the positive and negative weights.

[0077] Based on the bidirectional causal weights, enhance the output key vectors Kv and Ks, and the value vectors Vv and Vs respectively, preserving the original semantics and injecting causal information.

[0078] Preferably, introduce a causal verification gate to filter out abnormal features without causal correlation and ensure the effectiveness of the output vectors.

[0079] Final output: Vector-enhanced video key vector: Kv

[0083] , , , , (512 dimensions); Enhanced video value vector: Vv final (512 dimensions); Enhanced sensor key vector: Ks final (512 dimensions); Enhanced sensor value vector: Vs final (512 dimensions).

[0080] S4: Convert the feature vectors output by the cross-modal bidirectional causal attention mechanism into quantifiable risk scores, form an alarm decision, and push relevant video and sensing data; Preferably, adopt a 2-layer fully connected and Sigmoid output structure to realize the mapping from high-dimensional features to [0,1] risk scores. Input the enhanced features into the discriminant function above, and after being processed by two fully connected layers and the GELU activation function, obtain the intermediate feature vectors, and the intermediate feature vectors output the risk scores through the Sigmoid activation function.

[0081] Furthermore, based on the risk scores and preset thresholds, implement a dual determination of alarm authenticity and risk level: Confirm the alarm: When the risk score is greater than or equal to 0.5, it is determined as a real alarm, triggering the data push and response process; Reject the alarm: Otherwise, it is determined as a false alarm or a low-value alarm, and only the system records it without triggering the push.

[0082] Preferably, the risk level determination rule: Medium risk: 0.5 < r < 0.7, response priority level 3; High risk: 0.7 < r < 0.9, response priority level 2; Extremely high risk: r > 0.9, response priority level 1 (highest); Generate a standardized decision information packet, including core information such as alarm ID, timestamp, risk score, authenticity status, risk level, causal coefficient C, etc., for easy parsing and display by the terminal.

[0083] The process involves calculating the spatial correspondence between sensor locations and the camera's field of view to construct a spatial relationship database. Prior information about the architectural space is then used to constrain the calculation range of the cross-modal bidirectional causal attention mechanism. Specifically, this includes: Extract world coordinate information from sensors and cameras; Calculate the visible area polygon Dc for each camera; calculate the scope Ds for each sensor; Based on spatial domain and distance rules, a candidate camera set for sensors and a candidate sensor set for cameras are defined; an undirected bipartite graph is formed with the sensor set and the camera set as two vertex sets and the relationships in the candidate sets as edges. A multidimensional binary matrix is ​​constructed. Based on the multidimensional binary matrix, sensor or camera basic information, visible area polygon Dc, sensor scope Ds, and candidate set, a topological relation library is formed. The sensor and camera topological constraints in the topological relation library are used as the computational boundary of the cross-modal bidirectional causal attention mechanism.

[0084] Preferably, the calculation of the visible area polygon Dc for each camera specifically includes: obtaining the field of view (FOV), horizontal rotation angle (yaw), pitch angle (pitch), installation height (hc), z0 (building ground / reference plane height), and farthest effective visible distance (Rc) based on the camera device attributes; polygon generation logic: coordinate projection: projecting the camera's world coordinates Pcj onto the reference plane to obtain the projection point Pcj; Angle range determination: With the projection point Pcj as the center, expand FOV / 2 to both sides by the horizontal rotation angle yaw to form the horizontal visible angle range [yaw-FOV / 2, yaw+FOV / 2].

[0085] Vertex sampling and polygon construction: Eight vertices are sampled evenly within the above angle range. The coordinates of each vertex are calculated by converting polar coordinates to rectangular coordinates. The vertices are connected in clockwise order to form the initial visible area polygon. Preferably, Boolean operations are used to remove the occluded areas in the initial polygon to obtain the final unoccluded visible area polygon.

[0086] Preferably, the calculation of the effective range Ds of each sensor can be either circular or sector-shaped.

[0087] When the area of ​​effect is circular, the sensor reference plane projection point is taken as the center and the detection radius is taken as the radius, and the discretization is a regular 12-sided polygon.

[0088] When the scope of action is fan-shaped, the sensor reference plane projection point is used as the center, and a preset detection angle is used. The detection radius R is used as the parameter, and 8 vertices are sampled according to the angle interval to form a fan-shaped polygon.

[0089] Example 2: The present invention also provides a multi-source alarm information collaborative verification system based on cross-modal causal attention, comprising: Multimodal input module: collects and standardizes multimodal data; the multimodal data includes building model data, video data, and sensor data; Feature encoding module: This module performs feature calculations on the multimodal data using an alarm feature encoder, a video feature encoder, and a sensor time-series feature encoder to obtain an alarm feature vector q and output a key vector. Sum value vector Output key vector Sum value vector ; Cross-modal causal attention module: Outputs a key vector based on the alarm feature vector q. Sum value vector Output key vector Sum value vector It adopts a cross-modal bidirectional causal attention mechanism to conduct cross-modal information interaction based on causal relationships; Fusion Decision Module: Transforms the feature vectors output by the cross-modal bidirectional causal attention mechanism into quantifiable risk scores, forms alarm decisions, and pushes relevant video and sensor data; Spatial topology mapping module: By calculating the spatial correspondence between sensor points and the camera's visible area, a spatial relationship database is constructed, and the calculation range of the cross-modal bidirectional causal attention mechanism is constrained by prior information about the architectural space.

[0090] Example 3: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a multi-source alarm information collaborative verification method based on cross-modal causal attention as described above.

[0091] Example 4: A readable storage medium storing a computer program, which, when executed by a processor, implements a multi-source alarm information collaborative verification method based on cross-modal causal attention as described above.

[0092] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-source alarm information collaborative verification method based on cross-modal causal attention, characterized by: S1: Collect and standardize multi-modal data; the multi-modal data includes building model data, video data and sensor data; S2: performing feature calculation on the multi-modal data by the alarm feature encoder, the video feature encoder and the sensor time sequence feature encoder to obtain an alarm feature vector q, output a key vector and a value vector , output a key vector and a value vector ; S3: output the key vector according to the alarm feature vector q and the value vector , output the key vector and the value vector , and cross-modal information interaction is performed based on the cross-modal bidirectional causal attention mechanism based on the causal relationship; S4: Convert the feature vector output by the cross-modal bidirectional causal attention mechanism into a quantifiable risk score, form an alarm decision, and push related video and sensor data; Wherein, by calculating the spatial correspondence between sensor points and camera visual area, a spatial relationship library is constructed, and the cross-modal bidirectional causal attention mechanism calculation range is constrained by building space prior information.

2. The multi-source alarm information collaborative verification method based on cross-modal causal attention according to claim 1, characterized in that: The standardization processing in S1 specifically includes: A four-dimensional quantitative evaluation system is established by combining alarm task requirements and spatial correlation, including basic quality dimension, task correlation dimension, scene dimension and historical adaptation dimension; Based on the basic quality dimension, task correlation dimension, scene dimension and historical adaptation dimension, the dynamic weight criterion is used for optimization weight configuration; The dynamic weight criterion includes: w1: the more complex the scene, the higher the historical adaptation, the higher the basic quality weight; w2: the more complex the scene, the lower the historical adaptation, the higher the task correlation weight; w3: the scene dimension weight increases linearly with complexity; w4: the historical adaptation dimension weight increases linearly with accuracy; Calculate the data score: Qtotal=ω1×Q1+ω2×Q2+ω3×Q3+ω4×Q4; Confirm the data level of data score Qtotal, and perform corresponding level data enhancement processing on the multi-modal data according to the data level; Wherein, w1+w2+w3+w4=1; and the weight is updated and optimized according to a preset period.

3. The multi-source alarm information collaborative verification method based on cross-modal causal attention according to claim 2, characterized in that: The alarm feature encoder in S2 specifically includes: An architectural safety alarm knowledge graph is constructed, and alarm type nodes, risk level nodes, influence range nodes, and disposal priority nodes are defined as graph nodes; based on historical alarm data and expert experience, the association strength between nodes is defined to set the edge weight of the graph; a domain pre-training BERT model is used to convert alarm types into 256-dimensional initial semantic vectors Etype, and risk levels and influence range labels are converted into 64-dimensional vectors through a semantic mapping layer, and the 64-dimensional vectors are concatenated with the initial semantic vectors Etype to form a 320-dimensional semantic feature vector Esem; Esem=Concat(Etype,Wr*Er+Ws*Es); Wherein Wr, Ws are learnable weight matrices, Er is a risk level vector, and Es is an influence range vector; The semantic feature vector Esem, the intensity feature Xintensity after dynamic normalization, and the duration feature are fused through an attention layer; the fusion result is converted into a fixed-dimensional alarm feature vector q through a linear mapping layer.

4. The multi-source alarm information collaborative verification method based on cross-modal causal attention according to claim 3, characterized in that: The video feature encoder in S2 specifically includes: Adopt three-dimensional convolution neural network 3D-CNN or space-time Transformer, extract the space-time feature of video stream, output key vector And value vector .

5. The multi-source alarm information collaborative verification method based on cross-modal causal attention according to claim 4, characterized in that: The sensor time series feature encoder in S2 specifically includes: The sensor timing feature encoding adopts a long short-term memory network (LSTM) or a gated recurrent unit (GRU) to encode a sensor historical data sequence, and outputs a key vector and a value vector .

6. The multi-source alarm information collaborative verification method based on cross-modal causal attention according to claim 5, characterized in that, S3 includes: For the alarm feature vector q, output key vector and value vector , output key vector and value vector perform secondary normalization; The output key vector and the value vector of the time window are matched with the output key vector and the value vector of the time window, the output key vector and the value vector corresponding to the time period of the sensor data segment are filtered by the semantic label of the alarm feature vector q, and the semantically related features are retained. The split vectors are mapped to the causal feature space through a linear projection layer to enhance the causal correlation, and Xavier normal distribution is used for initialization; the bidirectional attention weight of the video and the sensor and the sensor and the video is calculated respectively, the forward and reverse weights are averaged to obtain the final bidirectional causal attention weight; Based on the bidirectional causal weights, the output key vectors are respectively enhanced , value vectors , output key vectors and value vectors , the original semantics are preserved and causal information is injected; Final output: vector enhanced video key vector, enhanced video value vector, enhanced sensor key vector, and enhanced sensor value vector.

7. The multi-source alarm information collaborative verification method based on cross-modal causal attention according to claim 6, characterized in that, The spatial relationship library is constructed by calculating the spatial correspondence between the sensor point and the camera visual area, and the cross-modal bidirectional causal attention mechanism calculation range is constrained by using the prior information of building space, and specifically includes: Extracting the world coordinate information of the sensor and the camera; Calculating the visual area polygon Dc of each camera; calculating the scope Ds of each sensor; Based on the spatial scope and distance rules, the candidate camera set of the sensor and the candidate sensor set of the camera are defined; the sensor set and the camera set are taken as two vertex sets, and the associated relationship in the candidate set is taken as an edge to form an undirected bipartite graph; A multi-dimensional binary matrix is constructed, and based on the multi-dimensional binary matrix, the sensor or camera basic information, the visual area polygon Dc, the scope Ds of the sensor, and the candidate set, a topological relationship library is formed, and the topological constraints of the sensors and cameras in the topological relationship library are used as the calculation boundary of the cross-modal bidirectional causal attention mechanism.

8. The system of claim 1-7, wherein the system is a multi-source alarm information collaborative verification method based on cross-modal causal attention. It includes: A multi-modal input module: collecting and standardizing processing multi-modal data; the multi-modal data includes building model data, video data and sensor data; The feature coding module: through the alarm feature encoder, the video feature encoder and the sensor time sequence feature encoder, the feature calculation is carried out on the multi-modal data, the alarm feature vector q is obtained, and the key vector is output And the value vector , the key vector And the value vector ; Cross-modal causal attention module: output key vector and value vector , output key vector and value vector , cross-modal information interaction is carried out based on the cross-modal bidirectional causal attention mechanism based on the causal relationship; A fusion decision module: converting the feature vector output by the cross-modal bidirectional causal attention mechanism into a quantifiable risk score to form an alarm decision and related video and sensor data push; A spatial topological mapping module: constructing a spatial relationship library by calculating the spatial correspondence between the sensor point and the camera visual area, and constraining the calculation range of the cross-modal bidirectional causal attention mechanism by using the prior information of building space.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the multi-source alarm information collaborative verification method based on cross-modal causal attention as claimed in any one of claims 1-7.

10. A readable storage medium, characterized by, The readable storage medium stores a computer program, and the computer program is executed by the processor to realize the multi-source alarm information collaborative verification method based on cross-modal causal attention as claimed in any one of claims 1-7.

Images