A method for early warning of abnormal behavior in industrial parks based on spatiotemporal graph neural network data fusion
By analyzing multi-source data through spatiotemporal graph neural networks and constructing a time-series evidence graph, the problems of inconsistency in multi-source data and false alarms caused by equipment failure in the early warning of abnormal behavior in the park are solved, and efficient identification and differentiated early warning of abnormal behavior are achieved.
Patent Information
- Application Number
- CN202611360202.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2026-08-06
- Filing Date
- 2026-09-03
- Publication Date
- 2026-11-10
AI Technical Summary
Existing methods for early warning of abnormal behavior in industrial parks are unable to effectively handle inconsistencies between multiple data sources and false alarms or missed alarms caused by faults. Furthermore, they are difficult to distinguish between abnormal behavior and equipment failures, resulting in frequent false alarms or missed alarms.
A data fusion method based on spatiotemporal graph neural networks is adopted to generate standard event records through time correction and entity association, construct a time-series evidence graph, analyze evidence relationships using spatiotemporal graph neural networks, determine the risk level of abnormal behavior, and output early warnings.
It improves the ability to identify abnormal behavior, reduces false alarms caused by equipment failure, provides a more comprehensive data foundation and differentiated early warning, and improves the efficiency of park security management.
Smart Images

Figure CN123070489A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent sensing, and in particular to a method for early warning of abnormal behavior in a park based on spatiotemporal graph neural network data fusion. Background Technology
[0002] With the continuous advancement of smart park construction, parks typically include access control systems, video surveillance systems, personnel positioning systems, and equipment operation management systems to collect information such as personnel passage, area movement, equipment operation, and environmental status. Existing methods for early warning of abnormal behavior in parks mostly rely on identifying personnel actions from single video data sources, or on stitching or weighted fusion of features from multiple sources such as access control, video, and positioning data, and then using preset rules or neural networks to determine whether personnel are engaging in unauthorized entry, tailgating, abnormal loitering, or other violations. Due to differences in sampling frequency, recognition accuracy, coverage, and time reference among different monitoring systems, inconsistencies can easily arise between access control records and video trajectories, deviations between positioning results and actual locations, and discrepancies between equipment operation records and on-site personnel.
[0003] Existing multi-source data fusion methods typically focus on the comprehensive analysis of already collected observation results. For observations where an event from one data source should appear in another but did not, they are usually treated as missing data, failing to transform them into anomalous evidence with clear business implications. Furthermore, when expected and actual observation information are inconsistent, existing methods struggle to distinguish whether the inconsistency is caused by abnormal behavior such as tailgating, identity theft, or unauthorized entry, or by malfunctions such as camera obstruction, location drift, communication delays, or data source offlineness, easily leading to false alarms or missed alarms. Moreover, the relationships of support, conflict, missing data, and latency between different data sources evolve continuously with personnel movement and equipment status changes; static rules or ordinary feature fusion models are insufficient to effectively represent the spatial correlations and temporal changes between multi-source evidence.
[0004] To address this, a method for early warning of abnormal behavior in industrial parks based on spatiotemporal graph neural network data fusion is proposed. Summary of the Invention
[0005] The main objective of this invention is to provide a method for early warning of abnormal behavior in industrial parks based on spatiotemporal graph neural network data fusion, which can effectively solve the problems in the background technology.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] A method for early warning of abnormal behavior in industrial parks based on spatiotemporal graph neural network data fusion includes the following steps:
[0008] S1. Obtain multi-source monitoring data and basic park data within the park. The multi-source monitoring data includes at least access control event data, video monitoring data, personnel positioning data, equipment operation data, and data source operation status data. The basic park data includes at least personnel access information and park spatial topology information.
[0009] S2. Perform time correction and entity association on the multi-source monitoring data, and convert data from different sources belonging to the same person, area or equipment into standard event records;
[0010] S3. Based on the standard event records corresponding to one or more triggering data sources and the preset cross-data source observation rules, generate expected observation information corresponding to one or more verification data sources. The verification data source is different from the triggering data source that generates the corresponding standard event records. The expected observation information includes at least the expected observation subject, expected event type, expected occurrence area, and expected time window.
[0011] S4. Within the expected time window and the delay time window after the expected time window, query the actual observation information of the verification data source, generate supporting evidence, conflicting evidence, missing evidence or delay evidence according to the correspondence between the actual observation information and the expected observation information, and determine the credibility of each piece of evidence according to the data quality of the triggering data source and the verification data source, the rule reliability of the cross-data source observation rule and the correspondence.
[0012] S5. Using the personnel, regions, equipment, and data sources corresponding to the standard event records as graph nodes, and the supporting evidence, conflicting evidence, missing evidence, and delay evidence as evidence relationship edges with different edge types, construct multiple time-series evidence graphs arranged in chronological order.
[0013] S6. Input the multiple time-series evidence graphs into the spatiotemporal graph neural network, extract the spatiotemporal evolution features of the evidence relationship over time according to the edge type and evidence credibility of each evidence relationship edge, and obtain the personnel abnormality probability, data source failure probability and evidence attribution results.
[0014] S7. The evidence attribution results are used to identify conflicting evidence, missing evidence, or delayed evidence related to personnel abnormality as abnormal evidence. Based on the personnel abnormality probability, data source failure probability, evidence credibility of abnormal evidence, and duration of abnormal evidence, the abnormal behavior risk level is determined, and the park abnormal behavior early warning information corresponding to the abnormal behavior risk level is output.
[0015] Furthermore, in step S2: a correspondence is established between different data sources and personnel entities based on access card identifiers, video personnel characteristics, personnel positioning tags, and device operation accounts; for multi-source monitoring data where a direct correspondence cannot be established, candidate correspondences are determined based on event occurrence time, event occurrence location, personnel movement direction, and park spatial topology information, and the standard event record is generated based on the candidate correspondences.
[0016] Furthermore, in step S3, the cross-data source observation rules include at least one of the following: generating corresponding video personnel trajectories and expected observation information on changes in personnel positioning areas based on access control events; generating corresponding access control records and expected observation information on changes in personnel positioning areas based on video personnel access events; generating corresponding access control records and expected observation information on video personnel trajectories based on changes in personnel positioning areas; and generating expected observation information on video personnel trajectories or personnel positioning information corresponding to personnel located in the area where the device is located, based on device operation events and the association results between device operation accounts and personnel identities.
[0017] Furthermore, in step S4: when the actual observation information matches the expected observation information, the supporting evidence is generated; when the actual observation information exists but is inconsistent with the expected observation information, the conflict evidence is generated; when no actual observation information corresponding to the expected observation information is obtained within the expected time window, the missing evidence is generated; when the actual observation information corresponding to the expected observation information appears within a delayed time window after the expected time window, the time delay evidence is generated to replace the previously generated corresponding missing evidence.
[0018] Furthermore, the time is determined according to the following formula. Connect graph nodes Graph Nodes The Credibility of evidence related to the evidence relationship:
[0019]
[0020] in, Indicates time Connect graph nodes Graph Nodes The The credibility of evidence related to the evidence relationship; and These represent the triggering data sources. and verify data source At any moment Data quality coefficient; The evidence relation edge is based on the first... Rule reliability for cross-data source observation rules; Indicates the first Evidence correspondence coefficient of evidence-related edges;
[0021] The data quality coefficient is determined based on the online status, data integrity, and historical observation accuracy of the corresponding data source, and the rule reliability is determined based on the historical verification results or preset calibration values of the corresponding cross-data source observation rules.
[0022] For supporting evidence, conflicting evidence, and time-delayed evidence, the evidence correspondence coefficient is determined based on the degree of correspondence and difference between the actual observation information and the expected observation information in terms of the observation subject, the time of occurrence, and the area of occurrence; for missing evidence, the evidence correspondence coefficient is determined based on the effective coverage status of the verification data source and the degree of persistence in not obtaining corresponding actual observation information within the expected time window.
[0023] The data quality coefficient, rule reliability, and evidence correspondence coefficient are all normalized to the interval [0,1].
[0024] Furthermore, in step S5: the plurality of time-series evidence graphs are constructed according to the preset graph snapshot interval; the node features of the graph nodes include at least personnel permission status, personnel location area, personnel movement status, equipment operation status, area security level, and data source operation status; the edge features of the evidence relationship edges include at least edge type, evidence credibility, generation time, and duration; the edge type includes at least supporting evidence edges, conflicting evidence edges, missing evidence edges, and delayed evidence edges.
[0025] Furthermore, in step S6, the spatiotemporal graph neural network determines the graph nodes according to the following formula. At any moment Spatial aggregation characteristics and spatiotemporal state characteristics:
[0026]
[0027] in, Represents graph nodes At any moment Spatial aggregation characteristics; Represents graph nodes At any moment The spatiotemporal state characteristics; Represents graph nodes The spatiotemporal state characteristics of the previous moment; and Representing graph nodes and adjacent graph nodes At any moment Node characteristics; The set of edge types representing evidence relationship edges; Indicates passing through the first Evidence-related edges and graph nodes The set of adjacent graph nodes connected; This indicates the credibility of the evidence corresponding to the evidence relationship edge; and Indicates network parameters; Represents a nonlinear activation function; Indicates a gated loop unit;
[0028] For the target personnel to be assessed for abnormal behavior and the target data sources associated with the target personnel, a first aggregated feature is obtained based on the spatiotemporal state characteristics of the graph nodes associated with the target personnel. The second aggregation feature is obtained based on the spatiotemporal state features of the graph nodes associated with the target data source. The abnormal probability of the personnel and the failure probability of the data source are generated according to the following formula:
[0029]
[0030] in, This indicates the probability of the person being abnormal. This indicates the probability of failure of the data source. , , and This represents the output layer parameters of the spatiotemporal graph neural network. This represents a logical mapping function.
[0031] Furthermore, in step S7: an initial anomaly risk is determined based on the probability of personnel anomalies, the credibility of the anomaly evidence, the duration of the anomaly evidence, and the security level of the area or equipment involved in the anomaly; the initial anomaly risk is corrected based on the probability of data source failure, wherein, when the probability of data source failure increases, the initial anomaly risk caused by a single missing piece of evidence from the corresponding data source is reduced; when multiple adjacent data sources form mutually supporting conflicting evidence or missing evidence, the corresponding initial anomaly risk is increased; the risk level and type of the anomaly behavior are determined based on the corrected initial anomaly risk, wherein the type of anomaly behavior includes at least one of unauthorized entry, tailgating, identity theft, abnormal loitering, and unauthorized equipment operation.
[0032] A spatiotemporal graph neural network-based data fusion-based early warning system for abnormal behavior in industrial parks includes a data interface, a memory, and a processor. The memory stores computer programs and cross-data source observation rules, and the processor is configured with:
[0033] The data acquisition module is used to acquire multi-source monitoring data and basic data of the park through the data interface;
[0034] The event standardization module is used to perform time correction and entity association on the multi-source monitoring data and generate standard event records.
[0035] The expected observation generation module is used to generate expected observation information for a verification data source that is different from the triggering data source, based on the standard event records corresponding to one or more triggering data sources and the cross-data source observation rules.
[0036] The evidence relationship generation module is used to generate supporting evidence, conflicting evidence, missing evidence, or time-delayed evidence based on the correspondence between the actual observation information of the verification data source and the expected observation information, and to determine the credibility of each piece of evidence.
[0037] The temporal evidence graph construction module is used to construct multiple temporal evidence graphs by using personnel, regions, equipment and data source status as graph nodes and various pieces of evidence as evidence relationship edges with different edge types.
[0038] The spatiotemporal analysis module is used to process the multiple time-series evidence graphs through a spatiotemporal graph neural network to obtain the probability of personnel anomalies, the probability of data source failure, and the evidence attribution results.
[0039] The early warning output module is used to identify conflicting evidence, missing evidence, or delayed evidence related to personnel abnormality as abnormal evidence based on the evidence attribution results, and to determine the risk level of abnormal behavior and output early warning information for abnormal behavior in the park based on the probability of personnel abnormality, the probability of data source failure, and the credibility of the abnormal evidence.
[0040] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method steps.
[0041] Compared with the prior art, the present invention has the following beneficial effects:
[0042] 1. This invention generates expected observation information corresponding to different verification data sources based on standard event records in the triggering data source, and converts the correspondence between expected observation information and actual observation information into supporting evidence, conflicting evidence, missing evidence, or time-delay evidence, so that data missing and data inconsistency that were originally not effectively utilized can form evidence relationships with clear semantics, providing a more comprehensive data foundation for judging abnormal behavior in the park.
[0043] 2. This invention uses personnel, areas, equipment, and data sources as graph nodes and different types of evidence as evidence relationship edges to construct a continuous temporal evidence graph. It also extracts the spatial correlation features and temporal evolution features of evidence relationships through a spatiotemporal graph neural network. This enables comprehensive analysis of the dynamic correspondence between access control, video, positioning, and equipment operation data, thereby improving the ability to identify abnormal behaviors such as tailgating, identity theft, unauthorized entry, abnormal loitering, and unauthorized equipment operation.
[0044] 3. This invention simultaneously obtains the probability of personnel anomalies and the probability of data source failure, and distinguishes between evidence related to personnel anomalies and abnormal observations caused by data source failures based on the evidence attribution results. When there is camera obstruction, positioning drift, communication delay, or data source offline, it can reduce the impact of the corresponding missing evidence on the risk of personnel anomalies, thereby reducing false alarms caused by monitoring equipment failures.
[0045] 4. This invention determines the risk level of abnormal behavior based on the credibility of evidence, the duration of evidence, the probability of data source failure, and the security level of the area or equipment involved in the anomaly. It can provide differentiated early warnings for abnormal behaviors of different sources, durations, and impact ranges, and provide early warning information such as anomaly type, location, and risk level to park security management personnel, thereby improving the efficiency of handling abnormal events in the park. Attached Figure Description
[0046] Figure 1 This is a schematic diagram of the method flow of the present invention;
[0047] Figure 2 This is a schematic diagram of the system modules of the present invention. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0049] Example 1
[0050] like Figure 1-2 As shown, a method for early warning of abnormal behavior in a park based on spatiotemporal graph neural network data fusion can be deployed on the park security management server, or it can be executed collaboratively by edge computing nodes set up in various areas of the park and the park management platform.
[0051] The park is equipped with access control devices, video cameras, personnel positioning base stations, personnel positioning tags, and production equipment or control terminals. The park's security management server communicates with the access control system, video surveillance system, personnel positioning system, equipment management system, and personnel access control system to acquire multi-source monitoring data and basic park data.
[0052] S1. Acquire multi-source monitoring data and basic park data.
[0053] Acquire multi-source monitoring data within the park, including access control event data, video monitoring data, personnel positioning data, equipment operation data, and data source operation status data.
[0054] The access control event data may include access control device identifier, access card identifier, authenticator identifier, authentication time, authentication result, passage direction, door opening time, door closing time, and door opening duration.
[0055] Video monitoring data can include camera identification, video capture time, personnel target identification, personnel identification results, personnel trajectory, personnel movement direction, the time when personnel enter or leave the monitored area, the number of people passing through the access control area at the same time, image clarity, and the proportion of personnel obstruction.
[0056] Personnel location data can include personnel location tag identification, location time, location coordinates, personnel location area, movement speed, movement direction, location reliability, and location signal strength.
[0057] Equipment operation data may include equipment identifier, operator account, operation type, operation time, equipment operating status, parameter modification content, and operation result. The operation type may include equipment start-up, equipment stop-up, operating parameter modification, control mode switching, and safety protection deactivation.
[0058] Data source operational status data can include data source online status, data upload latency, data missing rate, historical recognition accuracy, camera obstruction status, positioning signal quality, and access control device communication status.
[0059] The basic data for the park includes at least personnel access information and park spatial topology information. Personnel access information may include personnel identity, department, job type, access control permissions, equipment operation permissions, validity period of permissions, shift information, and work orders. Park spatial topology information may include the adjacency relationships between areas, the correspondence between access control equipment and areas, camera coverage areas, positioning base station coverage areas, equipment location areas, and passable paths between different areas.
[0060] For example, the park includes general office areas, production areas, equipment control areas, and high-risk restricted areas. Different personnel have different access permissions and equipment operation permissions based on their job positions and tasks.
[0061] S2, Time Correction and Entity Association
[0062] Since access control systems, video surveillance systems, personnel positioning systems, and equipment management systems may use different local times, the time of the park's unified time server is used as the benchmark to correct the timestamps of different data sources.
[0063] For data source The time deviation can be determined based on its most recent time synchronization record. and according to the corrected time The corresponding monitoring data will be mapped to the park's unified timeline.
[0064] Subsequently, entity associations were established for people, regions, and devices from different data sources.
[0065] For personnel entities, access card identifiers, video facial features, personnel location tags, and device operation accounts can be associated with the same personnel identifier based on a pre-established identity mapping table. For example, personnel A corresponds to access card number A01, video identity number F01, location tag number U01, and device operation account E01.
[0066] When a person's identity cannot be directly identified in a video, a candidate correspondence can be determined based on the time the target appears in the video, the time of access control authentication, the person's location, and the direction of the person's movement.
[0067] Specifically, when a person swipes their card and appears in the camera's view near the corresponding access control point within a preset time range, and their movement direction is consistent with the access control's passage direction, a candidate association can be established between the video target and the person who swipes the card. If the person's location tag simultaneously moves from the outside area to the inside area, the reliability of this candidate association is increased.
[0068] After completing time correction and entity association, data from different sources are converted into standard event records. Each standard event record includes at least the event subject, event type, event time, event location, data source, event content, and data quality information.
[0069] For example, a card swipe event for access control can generate the following standard event record: the event subject is person A, the event type is access control authentication passed, the event time is 10:15:20, the event area is the entrance to the production area, and the data source is access control device M1.
[0070] S3. Generate expected observation information across data sources.
[0071] Based on the standard event records corresponding to the triggering data source and the preset cross-data source observation rules, expected observation information corresponding to the verification data source that is different from the triggering data source is generated.
[0072] The cross-data source observation rules can be pre-stored in a rule base. Each rule includes at least the triggering event type, triggering data source type, verification data source type, expected observation subject, expected event type, expected occurrence area, expected time window, and rule reliability.
[0073] For example, when access control device M1 generates an access control event indicating that person A has passed authentication, the access control data source serves as the triggering data source, while the video surveillance data source and the personnel positioning data source serve as the verification data source.
[0074] Based on the cross-data source observation rules from access control to video, the following expected observation information is generated: the expected observation subject is person A, the expected event type is person passing through the access control area, the expected occurrence area is the inner area of the access control covered by camera C1, and the expected time window is 0 to 5 seconds after the access control authentication time.
[0075] Based on the cross-data source observation rules from access control to location, another expected observation information is generated: the expected observation subject is person A, the expected event type is that the area where the person is located changes from the outside area to the inside area, and the expected time window is 0 to 8 seconds after the access control authentication time.
[0076] For example, when device D1 generates a parameter modification event, the device operation data source serves as the triggering data source. Based on the association result between the device operation account and the personnel identity, the corresponding operator is identified, and expected observation information from the video surveillance data source or personnel location data source is generated to verify whether the corresponding personnel are located in the area where device D1 is located at the time of device operation.
[0077] The expected time window can be set based on the data source sampling frequency, the area travel distance, and historical event statistics. For the observation rules from access control to video, the expected time window can be set to 3 to 8 seconds; for the observation rules from access control to positioning, it can be set to 5 to 15 seconds; and for the observation rules from device operation to personnel presence verification, it can be set to 10 seconds before and after the operation time.
[0078] S4. Generate evidence and determine its credibility.
[0079] Within the expected time window and the subsequent delayed time window, query and verify the actual observation information of the data source.
[0080] Supporting evidence is generated when the actual observation information matches the expected observation information. For example, after person A swipes their card, the video surveillance system detects that person A passes through the access control area in the corresponding direction, and the person's location changes from the area outside the door to the area inside the door. In this case, the video monitoring result and the location result constitute supporting evidence.
[0081] When actual observation information exists but is inconsistent with expected observation information, conflict evidence is generated. For example, if access control records show that person A swipes their card, but video recognition results show that the person who actually passed through is person B, then conflict evidence of identity is generated; if access control only records one authorization, but video detects that two people pass through consecutively, then conflict evidence of people is generated.
[0082] Missing evidence is generated when no corresponding actual observation information is obtained within the expected time window. For example, if access control records show that person A swipes their card to enter, but the camera inside the door does not detect the corresponding person's trajectory, then missing video evidence is generated.
[0083] When the corresponding actual observation information appears after the expected time window ends but within the delayed time window, time-delayed evidence is generated to replace the previously generated missing evidence. For example, if the video trajectory does not appear within the expected 5-second time window but appears within the subsequent 3-second delayed time window, the original missing evidence is replaced with time-delayed evidence.
[0084] For time Connect graph nodes Graph Nodes The The credibility of evidence for a class of evidentiary relationships is determined by the following formula:
[0085]
[0086] in, Indicates the credibility of the evidence; Indicates triggering the data source At any moment Data quality coefficient; Indicates verification of data source At any moment Data quality coefficient; The evidence relation edge is based on the first... Rule reliability for cross-data source observation rules; Indicates the first Evidence correspondence coefficient of evidence-related edges.
[0087] The data quality coefficient is determined based on the online status of the data source, data completeness, and historical observation accuracy. In one embodiment, the data quality coefficient can be expressed as:
[0088]
[0089] in, This represents the online status coefficient of the data source, which is 1 when online, 0 when offline, and a value between 0 and 1 when there is intermittent communication. This indicates the data completeness for the current statistical period. This indicates the historical observation accuracy of the data source; , , The weights are preset, and the sum of the three is 1.
[0090] For video surveillance data sources, the data quality coefficient can also be adjusted based on image clarity and occlusion ratio. For example, when the occlusion ratio of people is high or the image brightness is below the normal range, the data quality coefficient of the video surveillance data source can be reduced.
[0091] Rule reliability The reliability can be determined based on the verification accuracy of the corresponding cross-data source observation rules in historical samples. For example, if 95% of normal access events can be detected by the camera inside the door within 5 seconds in historical access control card swipe events, then the initial rule reliability of the access control to video rule can be set to 0.95. For new rules lacking historical samples, a preset calibration value can be used and updated based on actual verification results during subsequent operation.
[0092] For supporting evidence, the evidence correspondence coefficient is determined based on the degree of matching of the observation subject, the time of occurrence, the area of occurrence, and the direction of movement. The higher the degree of matching, the larger the evidence correspondence coefficient.
[0093] For conflicting evidence, the evidence correspondence coefficient is determined based on the degree of difference between the actual observation and the expected observation. For example, when the access control identity and the video identity are completely different, the degree of identity conflict is high; the greater the difference between the number of people authorized by the access control and the number of people detected by the video, the greater the evidence correspondence coefficient for the conflicting evidence.
[0094] For delay evidence, the evidence correspondence coefficient can be determined based on the degree to which the actual observed information exceeds the expected time window. The shorter the delay, the more likely it is to be a normal data transmission delay; the longer the delay, the higher the degree of delay anomaly.
[0095] For missing evidence, the evidence correspondence coefficient is determined based on the effective coverage status of the verification data source and the degree of persistence in not obtaining corresponding actual observation information. When the verification data source is online, has effective coverage, and high data integrity, but the expected observation is not detected for a sustained period, the evidence correspondence coefficient for missing evidence is high; when the verification data source is offline or the camera is severely obstructed, the evidence correspondence coefficient for missing evidence is low.
[0096] The data quality coefficient, rule reliability, and evidence correspondence coefficient are all normalized to the interval [0,1].
[0097] S5. Constructing a temporal evidence diagram
[0098] Multiple time-series evidence graphs are constructed according to a preset snapshot interval. The snapshot interval can be set to 1 second, 2 seconds, or other time lengths suitable for the park's real-time early warning requirements.
[0099] The graph uses personnel, areas, equipment, and data sources as nodes. Personnel nodes represent monitored personnel; area nodes represent different functional areas within the park; equipment nodes represent production equipment, control terminals, or other controlled facilities; and data source nodes represent access control devices, cameras, positioning base stations, or equipment log systems.
[0100] The node characteristics of a personnel node can include personnel identity type, personnel permission status, current location, movement speed, movement direction, and number of historical anomalies.
[0101] The node characteristics of a regional node can include the regional security level, the regional open status, the current number of personnel, and the types of personnel allowed to enter.
[0102] The node characteristics of a device node can include the device's operating status, the device's hazard level, the current operation type, and the operation permission requirements.
[0103] The node characteristics of a data source node can include online status, data integrity, historical accuracy, current latency, and effective coverage status.
[0104] Supporting evidence, conflicting evidence, missing evidence, and delayed evidence are treated as different types of evidence relationships. Each evidence relationship edge must include at least the edge type, evidence credibility, generation time, duration, triggering data source, and verification data source.
[0105] For example, an access control authentication relationship can be established between person A and access control device M1; a video supporting evidence relationship or a video missing evidence relationship can be established between person A and camera C1; an entry relationship relationship can be established between person A and area Z1; and an operation relationship relationship can be established between person A and device D1.
[0106] Evidence diagrams formed at consecutive moments together constitute a time-series evidence diagram sequence, used to represent the formation, continuation, disappearance, or type change process of evidentiary relationships.
[0107] S6. Spatiotemporal Graph Neural Network Analysis
[0108] Multiple temporal evidence graphs are input into a pre-trained spatiotemporal graph neural network. The spatiotemporal graph neural network includes a spatial relationship aggregation part, a temporal state update part, and an output part.
[0109] The spatial relationship aggregation part performs weighted aggregation of the features of adjacent nodes based on the edge type and credibility of different evidence relationship edges.
[0110] Graph Node At any moment Spatial aggregation characteristics and spatiotemporal state characteristics are determined according to the following formula:
[0111]
[0112]
[0113] in, Represents graph nodes At any moment Spatial aggregation characteristics; Represents graph nodes At any moment The spatiotemporal state characteristics; It represents the spatiotemporal state characteristics of the previous moment; and Representing graph nodes and adjacent graph nodes Node characteristics; The set of edge types representing evidence relationship edges; Indicates passing through the first Evidence-related edges and graph nodes The set of connected adjacent nodes; This indicates the credibility of the evidence corresponding to the evidence relationship edge; and This represents the network parameters obtained from model training; Represents a nonlinear activation function; This indicates a gated loop unit.
[0114] For the target personnel to be assessed for abnormal behavior, the spatiotemporal state features of the target personnel node, as well as the associated regional nodes, device nodes, and data source nodes, are aggregated to obtain the first aggregated feature. .
[0115] The spatiotemporal state features of the data source nodes associated with the evidence relationship edges corresponding to the target personnel are aggregated to obtain the second aggregated feature. .
[0116] The aggregation can employ weighted average aggregation, max pooling, or attention aggregation. In this embodiment, weighted average aggregation, where weights are determined based on the credibility of the evidence, is used.
[0117] The probabilities of personnel abnormality and data source failure are determined by the following formula:
[0118]
[0119]
[0120] in, Indicates the probability of abnormal personnel; Indicates the probability of data source failure; , , and This represents the output layer parameters of the spatiotemporal graph neural network; This represents a logical mapping function.
[0121] During model training, labeled historical monitoring data from the park can be used to construct training samples. Personnel anomaly tags include normal access, unauthorized entry, tailgating, identity theft, abnormal loitering, and unauthorized device operation. Data source fault tags include normal, camera obstruction, missed video detection, location drift, location tag disconnection, access control data delay, and abnormal data communication.
[0122] During training, the loss for classifying personnel anomalies and the loss for classifying data source failures are calculated separately, and then weighted and summed to obtain the total model loss. Through joint training, the model is able to distinguish between anomalous evidence caused by personnel behavior and data inconsistencies caused by monitoring equipment failures.
[0123] The attribution results of evidence can be determined based on the contribution of each evidence relationship edge to the probability of personnel abnormality and the probability of data source failure.
[0124] In one embodiment, target evidence relationship edges can be sequentially masked, and the changes in the probability of personnel abnormality and the probability of data source failure before and after masking can be compared. When the probability of personnel abnormality decreases significantly after masking a certain evidence relationship edge, while the probability of data source failure changes only slightly, the evidence relationship edge is attributed to evidence related to personnel abnormality; when the probability of data source failure decreases significantly after masking a certain evidence relationship edge, the evidence relationship edge is attributed to evidence related to data source failure.
[0125] In other embodiments, the evidence attribution result can also be determined based on graph attention weights, gradient contribution values, or intermediate feature contributions corresponding to each evidence edge.
[0126] S7. Determine the level of abnormal risk and issue an early warning.
[0127] Based on the evidence attribution results, conflicting evidence, missing evidence, or delayed evidence related to personnel anomalies are identified as anomalous evidence.
[0128] The risk value of abnormal behavior is determined based on the probability of abnormal personnel, the probability of data source failure, the credibility of abnormal evidence, the duration of abnormal evidence, and the security level of the area or equipment involved in the abnormality.
[0129] In one embodiment, the risk value for abnormal behavior can be determined according to the following formula:
[0130]
[0131] in, Indicates the risk value of abnormal behavior; Indicates the probability of abnormal personnel; Indicates the overall credibility of unusual evidence; Indicates the degree of persistence corresponding to the duration of abnormal evidence; Indicates the security level of the area or equipment involved in the anomaly; Indicates the probability of data source failure; to Preset weights.
[0132] The overall credibility of anomalous evidence can be determined by weighted averaging or taking the maximum value of the credibility of multiple anomalous pieces of evidence.
[0133] When the probability of data source failure is high, and the anomaly assessment mainly stems from missing evidence generated by a single data source, the risk value for the corresponding abnormal behavior should be reduced. For example, if a camera goes offline and fails to detect the expected personnel trajectory, it should not be directly classified as an anomaly.
[0134] When multiple adjacent data sources generate conflicting or missing evidence pointing to the same person's abnormal behavior, the risk value of the corresponding abnormal behavior is increased. For example, if video surveillance, location systems, and access control systems all generate evidence pointing to person B's unauthorized entry, the risk level of that abnormal behavior is increased.
[0135] Multiple warning levels can be set based on the risk value of abnormal behavior. For example, no warning is issued when the risk value is below the first risk threshold; a "pay attention" message is issued when the risk value reaches the first risk threshold but is below the second risk threshold; a general alarm is issued when the risk value reaches the second risk threshold but is below the third risk threshold; and an emergency alarm is issued when the risk value reaches the third risk threshold.
[0136] Early warning information may include the identity of the abnormal person, the type of abnormal behavior, the area where the abnormality occurred, the associated equipment, the type of abnormal evidence, the credibility of the abnormal evidence, the probability of data source failure, the risk level of the abnormal behavior, and the recommended handling measures.
[0137] Typical Application Example 1: Tail-Following Warning
[0138] Personnel A has access to the production area and completes card authentication at access control device M1. The access control system only generates one authentication success event, but camera C1 detects that personnel A and personnel B pass through the access control consecutively.
[0139] The system generates expected observation information of video personnel passage based on access control events, with the expected number of people passing through being one. However, since the actual video observation results show two people, conflicting evidence is generated indicating a discrepancy between the number of authorized personnel and the number of people passing through the video.
[0140] The system further queried the access control records and location data of person B, but found no access control authorization record for person B. It also detected that person B's location area changed from the outside area to the inside area, thus forming evidence of missing access control records and evidence of location entry.
[0141] The spatiotemporal graph neural network outputs a higher probability of abnormal personnel and a lower probability of data source failure based on the evolution of multiple pieces of evidence over time. It also attributes evidence of conflicting numbers and missing access control records for personnel B to evidence of abnormal personnel, thereby outputting a tailgating warning.
[0142] Typical Application Example 2: Video Missed Detection Judgment
[0143] After person A swipes their card on access control device M1, the personnel positioning system detects that person A has entered the area inside the door from the outside area, but camera C1 does not detect the entry trajectory of person A.
[0144] The system generated evidence of missing video footage. Simultaneously, based on the operating status data from camera C1's data source, it was determined that its image was severely obstructed, resulting in a low data quality coefficient.
[0145] Based on access control supporting evidence, location supporting evidence, and low-quality camera status, the spatiotemporal graph neural network outputs a low probability of abnormal personnel and a high probability of data source failure. It also attributes the missing video evidence to the camera's failure to detect related evidence. Instead of outputting a high-level warning for abnormal personnel, it outputs a camera anomaly prompt to the equipment maintenance personnel.
[0146] Typical Application Example 3: Warning of Unauthorized Device Operation
[0147] Device D1 generates an operation parameter modification event, and the corresponding operation account is associated with person A. However, the person's location data indicates that person A is located in another area, and a camera near device D1 detects person B.
[0148] The system generates video or expected location observation information based on the equipment operation events, indicating that the corresponding personnel should be located in the equipment area. Since personnel A is not in the equipment area, evidence of a location conflict is generated; since on-site personnel B does not have operating permissions for equipment D1, evidence of a discrepancy between the on-site personnel's identity and their operating account is generated.
[0149] The spatiotemporal graph neural network combines the device's operating status, personnel's permission status, and continuous evidence relationships to output a high probability of personnel anomalies. It also attributes relevant conflicting evidence to evidence of unauthorized device operation, thereby outputting corresponding levels of abnormal behavior warnings.
[0150] This invention is not limited to the aforementioned access control, video, personnel positioning, and equipment operation data; it can also integrate vehicle identification data, visitor appointment data, fire monitoring data, and environmental sensor data. The technical concept of this invention can be adopted as long as expected observation information from other verification data sources is generated based on standard events from one or more triggering data sources, and a time-series evidence graph is constructed based on the correspondence between actual and expected observations.
[0151] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
Claims
1. A method for early warning of abnormal behavior in a park based on spatiotemporal graph neural network data fusion, characterized in that: Includes the following steps: S1. Obtain multi-source monitoring data and basic park data within the park. The multi-source monitoring data includes at least access control event data, video monitoring data, personnel positioning data, equipment operation data, and data source operation status data. The basic park data includes at least personnel access information and park spatial topology information. S2. Perform time correction and entity association on the multi-source monitoring data, and convert data from different sources belonging to the same person, area or equipment into standard event records; S3. Based on the standard event records corresponding to one or more triggering data sources and the preset cross-data source observation rules, generate expected observation information corresponding to one or more verification data sources. The verification data source is different from the triggering data source that generates the corresponding standard event records. The expected observation information includes at least the expected observation subject, expected event type, expected occurrence area, and expected time window. S4. Within the expected time window and the delay time window after the expected time window, query the actual observation information of the verification data source, generate supporting evidence, conflicting evidence, missing evidence or delay evidence according to the correspondence between the actual observation information and the expected observation information, and determine the credibility of each piece of evidence according to the data quality of the triggering data source and the verification data source, the rule reliability of the cross-data source observation rule and the correspondence. S5. Using the personnel, regions, equipment, and data sources corresponding to the standard event records as graph nodes, and the supporting evidence, conflicting evidence, missing evidence, and delay evidence as evidence relationship edges with different edge types, construct multiple time-series evidence graphs arranged in chronological order. S6. Input the multiple time-series evidence graphs into the spatiotemporal graph neural network, extract the spatiotemporal evolution features of the evidence relationship over time according to the edge type and evidence credibility of each evidence relationship edge, and obtain the personnel abnormality probability, data source failure probability and evidence attribution results. S7. The evidence attribution results are used to identify conflicting evidence, missing evidence, or delayed evidence related to personnel abnormality as abnormal evidence. Based on the personnel abnormality probability, data source failure probability, evidence credibility of abnormal evidence, and duration of abnormal evidence, the abnormal behavior risk level is determined, and the park abnormal behavior early warning information corresponding to the abnormal behavior risk level is output.
2. The method for early warning of abnormal behavior in a park based on spatiotemporal graph neural network data fusion according to claim 1, characterized in that, In step S2: Based on access card identification, video personnel characteristics, personnel positioning tags, and device operation accounts, establish a correspondence between different data sources and personnel entities; for multi-source monitoring data that cannot be directly correlated, determine candidate correspondences based on event occurrence time, event location, personnel movement direction, and park spatial topology information, and generate the standard event record based on the candidate correspondences.
3. The method for early warning of abnormal behavior in a park based on spatiotemporal graph neural network data fusion according to claim 1, characterized in that, In step S3, the cross-data source observation rules include at least one of the following: generating corresponding video personnel trajectories and expected observation information on changes in personnel positioning areas based on access control events; generating corresponding access control records and expected observation information on changes in personnel positioning areas based on video personnel access events; generating corresponding access control records and expected observation information on video personnel trajectories based on changes in personnel positioning areas; and generating expected observation information on video personnel trajectories or personnel positioning information corresponding to personnel located in the area where the device is located, based on device operation events and the association results between device operation accounts and personnel identities.
4. The method for early warning of abnormal behavior in a park based on spatiotemporal graph neural network data fusion according to claim 1, characterized in that, In step S4: when the actual observation information matches the expected observation information, the supporting evidence is generated; when the actual observation information exists but is inconsistent with the expected observation information, the conflict evidence is generated; when no actual observation information corresponding to the expected observation information is obtained within the expected time window, the missing evidence is generated; when the actual observation information corresponding to the expected observation information appears within a delayed time window after the expected time window, the time delay evidence is generated to replace the previously generated corresponding missing evidence.
5. The method for early warning of abnormal behavior in a park based on spatiotemporal graph neural network data fusion according to claim 4, characterized in that, Determine the time according to the following formula. Connect graph nodes Graph Nodes The Credibility of evidence related to the evidence relationship: in, Indicates time Connect graph nodes Graph Nodes The The credibility of evidence related to the evidence relationship; and These represent the triggering data sources. and verify data source At any moment Data quality coefficient; The first evidence relation edge is the basis for generating the evidence relation edge. Rule reliability for cross-data source observation rules; Indicates the first Evidence correspondence coefficient of evidence-related edges; The data quality coefficient is determined based on the online status, data integrity, and historical observation accuracy of the corresponding data source, and the rule reliability is determined based on the historical verification results or preset calibration values of the corresponding cross-data source observation rules. For supporting evidence, conflicting evidence, and time-delayed evidence, the evidence correspondence coefficient is determined based on the degree of correspondence and difference between the actual observation information and the expected observation information in terms of the observation subject, the time of occurrence, and the area of occurrence; for missing evidence, the evidence correspondence coefficient is determined based on the effective coverage status of the verification data source and the degree of persistence in not obtaining corresponding actual observation information within the expected time window. The data quality coefficient, rule reliability, and evidence correspondence coefficient are all normalized to the interval [0,1].
6. The method for early warning of abnormal behavior in a park based on spatiotemporal graph neural network data fusion according to claim 5, characterized in that, In step S5: Construct the multiple time-series evidence graphs according to the preset graph snapshot interval; the node features of the graph nodes include at least personnel permission status, personnel location area, personnel movement status, equipment operation status, area security level, and data source operation status; the edge features of the evidence relationship edges include at least edge type, evidence credibility, generation time, and duration; the edge types include at least supporting evidence edges, conflicting evidence edges, missing evidence edges, and delayed evidence edges.
7. The method for early warning of abnormal behavior in a park based on spatiotemporal graph neural network data fusion according to claim 6, characterized in that, In step S6, the spatiotemporal graph neural network determines the graph nodes according to the following formula. At any moment Spatial aggregation characteristics and spatiotemporal state characteristics: in, Represents graph nodes At any moment Spatial aggregation characteristics; Represents graph nodes At any moment The spatiotemporal state characteristics; Represents graph nodes The spatiotemporal state characteristics of the previous moment; and Representing graph nodes and adjacent graph nodes At any moment Node characteristics; The set of edge types representing evidence relationship edges; Indicates passing through the first Evidence-related edges and graph nodes The set of adjacent graph nodes connected; This indicates the credibility of the evidence corresponding to the evidence relationship edge; and Indicates network parameters; Represents a nonlinear activation function; Indicates a gated loop unit; For the target personnel to be assessed for abnormal behavior and the target data sources associated with the target personnel, a first aggregated feature is obtained based on the spatiotemporal state characteristics of the graph nodes associated with the target personnel. The second aggregation feature is obtained based on the spatiotemporal state features of the graph nodes associated with the target data source. The abnormal probability of the personnel and the failure probability of the data source are generated according to the following formula: in, This indicates the probability of the person being abnormal. This indicates the probability of failure of the data source. , , and This represents the output layer parameters of the spatiotemporal graph neural network. This represents a logical mapping function.
8. The method for early warning of abnormal behavior in a park based on spatiotemporal graph neural network data fusion according to claim 7, characterized in that, In step S7: An initial anomaly risk is determined based on the probability of personnel anomalies, the credibility of the anomaly evidence, the duration of the anomaly evidence, and the security level of the area or equipment involved in the anomaly. The initial anomaly risk is then corrected based on the probability of data source failure. Specifically, when the probability of data source failure increases, the initial anomaly risk caused by a single missing piece of evidence from the corresponding data source is reduced; when multiple adjacent data sources form mutually supporting conflicting or missing evidence, the corresponding initial anomaly risk is increased. The anomaly behavior risk level and anomaly behavior type are determined based on the corrected initial anomaly risk. The anomaly behavior type includes at least one of the following: unauthorized entry, tailgating, identity theft, abnormal loitering, and unauthorized equipment operation.
9. A park abnormal behavior early warning system based on spatiotemporal graph neural network data fusion, characterized in that: It includes a data interface, a memory, and a processor. The memory stores computer programs and cross-data source observation rules. The processor is configured with: The data acquisition module is used to acquire multi-source monitoring data and basic data of the park through the data interface; The event standardization module is used to perform time correction and entity association on the multi-source monitoring data and generate standard event records. The expected observation generation module is used to generate expected observation information for a verification data source that is different from the triggering data source, based on the standard event records corresponding to one or more triggering data sources and the cross-data source observation rules. The evidence relationship generation module is used to generate supporting evidence, conflicting evidence, missing evidence, or time-delayed evidence based on the correspondence between the actual observation information of the verification data source and the expected observation information, and to determine the credibility of each piece of evidence. The temporal evidence graph construction module is used to construct multiple temporal evidence graphs by using personnel, regions, equipment and data source status as graph nodes and various pieces of evidence as evidence relationship edges with different edge types. The spatiotemporal analysis module is used to process the multiple time-series evidence graphs through a spatiotemporal graph neural network to obtain the probability of personnel anomalies, the probability of data source failure, and the evidence attribution results. The early warning output module is used to identify conflicting evidence, missing evidence, or delayed evidence related to personnel abnormality as abnormal evidence based on the evidence attribution results, and to determine the risk level of abnormal behavior and output early warning information for abnormal behavior in the park based on the probability of personnel abnormality, the probability of data source failure, and the credibility of the abnormal evidence.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 8.
