A conference room abnormal behavior detection method and system based on picture recognition

Abstract

The application relates to the technical field of image recognition, and discloses a conference room abnormal behavior detection method and system based on picture recognition, which comprises the following steps: performing structural analysis on video frames in a conference room video frame sequence to obtain a conference room scene graph; performing graph structure disturbance on the scene graph, and performing causal attention allocation during the disturbance process, so that the attention weight borne by a disturbance edge is transferred to an adjacent relationship edge of the disturbance edge, and an anti-fact intervention scene graph of the conference room is obtained; performing deviation measurement on a semantic vector of the anti-fact intervention scene graph and a rule semantic vector of a conference room safety rule text to obtain an anti-fact score of the conference room; when the anti-fact score exceeds a preset abnormal threshold value or an aggregated value of a compensation effect score of the conference room exceeds a preset coordination threshold value, it is determined that the video frame has a rule violation behavior, and an abnormal explanation is output. The application can improve the accuracy and interpretability of conference room abnormal behavior detection.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and system for detecting abnormal behavior in a conference room based on image recognition. Background Technology

[0002] In existing technologies, the abnormal behavior detection methods for conference rooms based on scene graphs lack a semantic verification mechanism for the visual relationship discrimination results during the scene graph construction stage. It is difficult to systematically filter out erroneous relationship edges that contradict common sense about conference room behavior after relationship extraction, resulting in the inclusion of physically impossible relationship triples in the scene graph. Such noisy edges are treated as equals in subsequent reasoning, reducing the accuracy and robustness of violation judgment.

[0003] In existing technologies, interpretability detection methods based on graph structure perturbation, when performing edge removal operations, use random perturbation or fixed rules to select perturbation targets. This makes it difficult to quantify the contribution of a single relation edge to the violation conclusion from a causal inference perspective. Furthermore, the causal information from perturbation removal is directly discarded, resulting in the inability to discover collaborative violation factors exposed by the removal of the main edge. Therefore, there is an urgent need to develop a conference room abnormal behavior detection method with common sense verification capabilities and counterfactual causal reasoning mechanisms to address the problems of scene graph noise interference and insufficient interpretability in existing technologies, thereby improving the accuracy of abnormal behavior detection and the completeness of attribution explanation. Summary of the Invention

[0004] This invention provides a method and system for detecting abnormal behavior in a conference room based on image recognition, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a method for detecting abnormal behavior in a conference room based on image recognition, comprising:

[0006] S01. Perform structured parsing on the video frames in the video frame sequence of the conference room to obtain a scene diagram of the conference room;

[0007] S02. Perturb the graph structure of the scene graph and perform causal attention allocation during the perturbation process. Transfer the attention weight carried by the perturbation edge to the adjacent relationship edge of the perturbation edge to obtain the counterfactual intervention scene graph of the conference room.

[0008] S03. The semantic vector of the counterfactual intervention scenario diagram and the rule semantic vector of the conference room security rule text are used to measure the deviation and obtain the counterfactual score of the conference room. When the counterfactual score exceeds a preset anomaly threshold, or the aggregate value of the compensation effect score of the conference room exceeds a preset coordination threshold, it is determined that the video frame has a violation and an anomaly explanation is output.

[0009] In a preferred embodiment, the step of performing structured parsing of the video frames in the conference room video frame sequence to obtain a scene diagram of the conference room includes:

[0010] Visual perception is performed on video frames in the video frame sequence of the conference room to obtain candidate entities of the conference room;

[0011] Based on the spatial positional relationships and interaction postures of the candidate entities, the initial relationship edges of the candidate entities are obtained;

[0012] Perform a semantic consistency check on the initial relation edges, filter out the initial relation edges that contradict the common sense of the meeting room behavior, and obtain the reliable relation edges of the meeting room;

[0013] Using the entities associated with the reliable relationship edges as nodes and the visual relationships they represent as edges, construct a scene graph of the conference room.

[0014] In a preferred embodiment, the step of performing a semantic consistency check on the initial relation edges to filter out initial relation edges that contradict common sense about meeting room behavior, thereby obtaining reliable relation edges for the meeting room, includes:

[0015] Using pre-defined legal entity relationships as a priori constraints, construct the legal relationship tuple for the meeting room;

[0016] The head entity attributes, relation predicates, and tail entity attributes associated with the initial relation edges are extracted into triples to obtain the relation triples to be tested for the meeting room.

[0017] The triplet of the relation to be tested is compared with the legal relation tuple item by item, and the initial relation edges that fail to match are marked as semantic conflict edges.

[0018] The semantically conflicting edges are removed from the initial relation edges to obtain the reliable relation edges of the conference room.

[0019] In a preferred embodiment, the step of perturbing the scene graph and performing causal attention allocation during the perturbation process, transferring the attention weight carried by the perturbed edge to the adjacent edge of the perturbed edge, to obtain the counterfactual intervention scene graph of the conference room, includes:

[0020] The semantic correlation between each reliable relation edge in the scene graph and the conference room security rule text is determined to obtain candidate perturbation edges of the scene graph;

[0021] The scene graph is graph encoded to obtain the attention weights of the candidate perturbation edges, and the candidate perturbation edge with the largest attention weight is selected as the target perturbation edge.

[0022] Based on the adjacency topology of the target perturbation edge and the target perturbation edge sharing an endpoint with it in the scenario graph, the adjacency relationship edge of the target perturbation edge is determined;

[0023] The adjacency edges are subjected to liveness verification, and semantically inactive adjacency edges are filtered out to obtain candidate receiving edges of the scene graph;

[0024] Based on the attention reception coefficient of the candidate receiving edge and the semantic affinity between the candidate receiving edge and the target perturbation edge, the attention weight of the target perturbation edge is redistributed to the candidate receiving edge, and the target perturbation edge is removed from the scene graph to obtain the counterfactual intervention scene graph of the conference room.

[0025] In a preferred embodiment, the step of performing a liveness check on the adjacency edges to filter out semantically inactive adjacency edges and obtain candidate receiving edges for the scene graph includes:

[0026] Based on the relational predicate of the target perturbation edge, determine the interaction type of the target perturbation edge;

[0027] Based on the preset visual relation dependency prior, the preconditions of the adjacent relation edges are traced to obtain the preconditions on which the semantics of the adjacent relation edges depend.

[0028] The interaction type of the target perturbation edge is determined to be subordinate to the premise relationship. If the interaction type belongs to the premise relationship, the adjacent relationship edge is marked as a semantically inactive edge.

[0029] Remove the semantically inactive edges from the adjacency edges to obtain the candidate receiving edges of the scene graph.

[0030] In a preferred embodiment, the step of tracing the preconditions of the adjacent edges based on a preset visual relation dependency prior to obtain the preconditions upon which the semantics of the adjacent edges depend includes:

[0031] Based on the physical dependencies and logical temporal relationships between interactive behaviors in the conference room scenario, a visual relation dependency prior library is constructed, which stores the mapping relationship between each interaction type and its prerequisite interaction type.

[0032] Perform relation predicate parsing on the adjacency edges to determine the interaction type of the adjacency edges;

[0033] Based on the visual relation dependency prior library, the prerequisite conditions are retrieved by using the interaction type of the adjacent relation edge as the index, and the prerequisite interaction type on which the semantics of the adjacent relation edge depends is obtained.

[0034] The aforementioned prerequisite interaction type is used as the prerequisite relation upon which the semantics of the adjacency relation edge are determined.

[0035] In a preferred embodiment, the step of reallocating the attention weight of the target perturbation edge to the candidate receiving edge based on the semantic affinity between the candidate receiving edge and the target perturbation edge includes:

[0036] The formula for calculating the attention weight transfer amount of the candidate receiving edge is as follows: ;

[0037] In the formula, Indicates the candidate receiving edge Attention weight transfer amount, This represents the target perturbation edge. Indicates the first Candidate receiving edge, This represents the attention weight of the target perturbation edge. (*) indicates the semantic affinity. This indicates the preset temperature coefficient. This represents the attention reception coefficient. This represents the attention reception coefficient for traversing the indexed edges. Represents the liveness indicator function. This represents the candidate receiving edge.

[0038] In a preferred embodiment, the step of measuring the deviation between the semantic vector of the counterfactual intervention scenario diagram and the rule semantic vector of the conference room security rule text to obtain the counterfactual score of the conference room includes:

[0039] The original degree of violation of the meeting room is obtained by measuring the deviation between the semantic vector of the scene graph and the semantic vector of the rule. The degree of intervention violation in the meeting room is obtained by measuring the deviation between the semantic vector of the counterfactual intervention scenario diagram and the semantic vector of the rule.

[0040] The difference between the original violation level and the intervention violation level is used as the counterfactual score for the meeting room;

[0041] The amount of attention weight shift is used as the compensation effect score for the meeting room.

[0042] In a preferred embodiment, when the counterfactual score exceeds a preset anomaly threshold, or the aggregated value of the conference room's compensation effect score exceeds a preset synergy threshold, determining that the video frame contains a violation and outputting an anomaly explanation includes:

[0043] When the counterfactual score exceeds a preset anomaly threshold, the video frame is determined to have a violation. The entity identifier associated with the target perturbation edge in the scene graph and the counterfactual score are output as the primary cause explanation. At the same time, the entity identifier and the corresponding compensation effect score of the candidate receiving edge compensation effect score in the counterfactual intervention scene graph that exceed a preset collaborative threshold are output as the collaborative cause explanation.

[0044] When the aggregated value of the compensation effect score exceeds a preset collaborative threshold, the video frame is determined to have a violation. The entity identifier and corresponding compensation effect score of the candidate receiving edge whose compensation effect score exceeds the preset collaborative threshold are output as a collaborative cause explanation. At the same time, the entity identifier associated with the target perturbation edge and the counterfactual score are output as the primary cause explanation.

[0045] To address the aforementioned problems, the present invention also provides a conference room abnormal behavior detection system based on image recognition, the system comprising:

[0046] The visual structured parsing module is used to perform structured parsing on the video frames in the video frame sequence of the conference room to obtain a scene diagram of the conference room;

[0047] The counterfactual graph intervention generation module is used to perturb the graph structure of the scene graph and perform causal attention allocation during the perturbation process, transferring the attention weight carried by the perturbation edge to the adjacent relationship edge of the perturbation edge to obtain the counterfactual intervention scene graph of the conference room;

[0048] The anomaly determination module is used to measure the deviation between the semantic vector of the counterfactual intervention scenario diagram and the rule semantic vector of the conference room security rule text to obtain the counterfactual score of the conference room. When the counterfactual score exceeds a preset anomaly threshold, or the aggregated value of the compensation effect score of the conference room exceeds a preset coordination threshold, the video frame is determined to have a violation, and an anomaly explanation is output.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. This invention significantly improves the reliability of conference room scene graph construction and the accuracy of subsequent reasoning by introducing a semantic consistency verification mechanism. Its advantages are: after visual relationship discrimination, common sense verification is performed on each initial relationship edge using preset legitimate entity relationships as prior constraints. This systematically identifies and filters out erroneous relationship triples that violate physical common sense, avoiding interference from noisy edges in violation judgment. This ensures the semantic quality of the scene graph data from the source, thereby improving the accuracy and robustness of abnormal behavior detection.

[0051] 2. This invention combines graph structure perturbation with causal attention redistribution to achieve hierarchical causal explanation and automatic discovery of synergistic factors in violation behavior. Its advantages are: using counterfactual intervention as the reasoning framework, it quantifies the causal contribution of the principal cause by removing key relation edges and observing changes in the degree of violation; simultaneously, it redistributes the attention weights of the removed edges to adjacent edges that pass the survival check based on semantic affinity, thus exposing synergistic violation factors that were originally masked by the principal cause. The final output is a hierarchical explanation of anomalies containing both the principal and synergistic causes, significantly enhancing the interpretability and credibility of the detection results. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a method for detecting abnormal behavior in a conference room based on image recognition, provided in an embodiment of the present invention.

[0053] Figure 2 A functional module diagram of a conference room abnormal behavior detection system based on image recognition provided in an embodiment of the present invention;

[0054] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0056] This application provides a method for detecting abnormal behavior in a conference room based on image recognition. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0057] Reference Figure 1 The diagram shown is a flowchart illustrating a method for detecting abnormal behavior in a conference room based on image recognition, according to an embodiment of the present invention. In this embodiment, the method for detecting abnormal behavior in a conference room based on image recognition includes:

[0058] S01. Perform structured parsing on the video frames in the video frame sequence of the conference room to obtain a scene diagram of the conference room;

[0059] In this embodiment of the invention, the step of performing structured parsing on the video frames in the conference room video frame sequence to obtain a scene diagram of the conference room includes:

[0060] Visual perception is performed on video frames in the video frame sequence of the conference room to obtain candidate entities of the conference room;

[0061] Based on the spatial positional relationships and interaction postures of the candidate entities, the initial relationship edges of the candidate entities are obtained;

[0062] Perform a semantic consistency check on the initial relation edges, filter out the initial relation edges that contradict the common sense of the meeting room behavior, and obtain the reliable relation edges of the meeting room;

[0063] Using the entities associated with the reliable relationship edges as nodes and the visual relationships they represent as edges, construct a scene graph of the conference room.

[0064] The step of performing a semantic consistency check on the initial relation edges to filter out initial relation edges that contradict common sense about meeting room behavior, thereby obtaining reliable relation edges for the meeting room, includes:

[0065] Using pre-defined legal entity relationships as a priori constraints, construct the legal relationship tuple for the meeting room;

[0066] The head entity attributes, relation predicates, and tail entity attributes associated with the initial relation edges are extracted into triples to obtain the relation triples to be tested for the meeting room.

[0067] The triplet of the relation to be tested is compared with the legal relation tuple item by item, and the initial relation edges that fail to match are marked as semantic conflict edges.

[0068] The semantically conflicting edges are removed from the initial relation edges to obtain the reliable relation edges of the conference room.

[0069] Visual perception is performed on the video frames in the video frame sequence of the conference room. All target objects appearing in the video frames are detected frame by frame. The semantic category of each target object is identified and its spatial position in the picture is determined. Each detected target object is treated as a candidate entity, and the candidate entities of the conference room are obtained by summarizing them.

[0070] Based on the spatial positional relationship and interaction posture of candidate entities, the spatial proximity and limb orientation between any two candidate entities are analyzed. When two candidate entities meet the preset proximity conditions in space and have a clear interaction direction, it is determined that there is a visual relationship between the two candidate entities. This visual relationship is recorded as an initial relationship edge. After traversing all candidate entity pairs, the initial relationship edges of the candidate entities are obtained.

[0071] Based on the natural attributes of people and objects in the conference room scene and the possible combinations of their interactions, a set of legal entity relationships is pre-determined as a priori constraints. Legal combinations of head entity category, relationship type and tail entity category are listed one by one to construct legal relationship tuples for the conference room.

[0072] For each initial relation edge, the semantic category of the starting entity connected to the initial relation edge is extracted as the head entity attribute, the visual relation type of the initial relation edge itself is extracted as the relation predicate, and the semantic category of the ending entity connected to the initial relation edge is extracted as the tail entity attribute. The above three elements are combined into a relation triplet to be tested. This extraction operation is performed on all initial relation edges one by one to obtain the relation triplet to be tested for the meeting room.

[0073] Perform a full match comparison between each triplet to be tested and the entries in the valid triplet. Check whether the head entity attribute, relation predicate, and tail entity attribute of the triplet to be tested have a completely identical record in the valid triplet. Mark the initial relation edge corresponding to the triplet to be tested that has no matching record as a semantic conflict edge.

[0074] Remove all initial relation edges marked as semantically conflicting from all initial relation edges. The remaining initial relation edges are the reliable relation edges that conform to common sense about meeting room behavior.

[0075] Using the entities associated with reliable relationship edges as nodes, the starting and ending entities of each reliable relationship edge are each treated as independent nodes in the graph structure. Using the visual relationship represented by the reliable relationship edge as an edge, a connection edge with a relationship type label is established between the corresponding two nodes. All reliable relationship edges are added to the graph structure one by one to construct the scene graph of the conference room.

[0076] The beneficial effect is that by adding a semantic consistency check based on prior knowledge after visual relationship discrimination, the erroneous relation edges that violate physical common sense are filtered out by comparing each legitimate relation tuple, thus eliminating semantic conflict noise in the scene graph from the source. This ensures that the entity relationship information carried by the scene graph is consistent with the common sense of behavior in the conference room, providing a high-quality structured input foundation for subsequent counterfactual reasoning and significantly improving the accuracy of abnormal behavior detection.

[0077] S02. Perturb the graph structure of the scene graph and perform causal attention allocation during the perturbation process. Transfer the attention weight carried by the perturbation edge to the adjacent relationship edge of the perturbation edge to obtain the counterfactual intervention scene graph of the conference room.

[0078] In this embodiment of the invention, the step of perturbing the scene graph and performing causal attention allocation during the perturbation process, transferring the attention weight carried by the perturbed edge to the adjacent edge of the perturbed edge, to obtain the counterfactual intervention scene graph of the conference room, includes:

[0079] The semantic correlation between each reliable relation edge in the scene graph and the conference room security rule text is determined to obtain candidate perturbation edges of the scene graph;

[0080] The scene graph is graph encoded to obtain the attention weights of the candidate perturbation edges, and the candidate perturbation edge with the largest attention weight is selected as the target perturbation edge.

[0081] Based on the adjacency topology of the target perturbation edge and the target perturbation edge sharing an endpoint with it in the scenario graph, the adjacency relationship edge of the target perturbation edge is determined;

[0082] The adjacency edges are subjected to liveness verification, and semantically inactive adjacency edges are filtered out to obtain candidate receiving edges of the scene graph;

[0083] Based on the attention reception coefficient of the candidate receiving edge and the semantic affinity between the candidate receiving edge and the target perturbation edge, the attention weight of the target perturbation edge is redistributed to the candidate receiving edge, and the target perturbation edge is removed from the scene graph to obtain the counterfactual intervention scene graph of the conference room.

[0084] The step of performing a liveness check on the adjacency edges, filtering out semantically inactive adjacency edges, and obtaining candidate receiving edges for the scene graph includes:

[0085] Based on the relational predicate of the target perturbation edge, determine the interaction type of the target perturbation edge;

[0086] Based on the preset visual relation dependency prior, the preconditions of the adjacent relation edges are traced to obtain the preconditions on which the semantics of the adjacent relation edges depend.

[0087] The interaction type of the target perturbation edge is determined to be subordinate to the premise relationship. If the interaction type belongs to the premise relationship, the adjacent relationship edge is marked as a semantically inactive edge.

[0088] Remove the semantically inactive edges from the adjacency edges to obtain the candidate receiving edges of the scene graph.

[0089] The method of tracing the preconditions of the adjacent edges based on the preset visual relation dependency prior to obtain the preconditions upon which the semantics of the adjacent edges depend includes:

[0090] Based on the physical dependencies and logical temporal relationships between interactive behaviors in the conference room scenario, a visual relation dependency prior library is constructed, which stores the mapping relationship between each interaction type and its prerequisite interaction type.

[0091] Perform relation predicate parsing on the adjacency edges to determine the interaction type of the adjacency edges;

[0092] Based on the visual relation dependency prior library, the prerequisite conditions are retrieved by using the interaction type of the adjacent relation edge as the index, and the prerequisite interaction type on which the semantics of the adjacent relation edge depends is obtained.

[0093] The aforementioned prerequisite interaction type is used as the prerequisite relation upon which the semantics of the adjacency relation edge are determined.

[0094] The step of redistributing the attention weights of the target perturbation edge to the candidate receiving edge based on the semantic affinity between the candidate receiving edge and the target perturbation edge includes:

[0095] The formula for calculating the attention weight transfer amount of the candidate receiving edge is as follows:

[0096] ;

[0097] In the formula, Indicates the candidate receiving edge Attention weight transfer amount, This represents the target perturbation edge. Indicates the first Candidate receiving edge, This represents the attention weight of the target perturbation edge. (*) indicates the semantic affinity. This indicates the preset temperature coefficient. This represents the attention reception coefficient. This represents the attention reception coefficient for traversing the indexed edges. Represents the liveness indicator function. This represents the candidate receiving edge.

[0098] The relation predicate of each reliable relation edge in the scene graph is compared one by one with the behavioral predicate described in the conference room safety rules text to determine the degree of semantic association between the two. Reliable relation edges with an association degree exceeding the preset judgment standard are selected as candidate perturbation edges. After all the filtering is completed, the candidate perturbation edges of the scene graph are summarized.

[0099] The scene graph is input into the graph encoder for graph encoding. The graph encoder aggregates information from each node and edge in the scene graph and assigns an attention weight to each candidate perturbation edge. This attention weight represents the contribution strength of the corresponding candidate perturbation edge to the overall semantic representation of the current scene graph. The attention weight values ​​of all candidate perturbation edges are compared, and the candidate perturbation edge with the largest attention weight is selected as the target perturbation edge.

[0100] In the scene graph, find all relation edges that share at least one end node with the target perturbation edge. The two end nodes of the target perturbation edge itself are the starting entity and the ending entity, respectively. Identify other relation edges in the scene graph that have the starting entity as the endpoint and other relation edges that have the ending entity as the endpoint, and summarize them to obtain the adjacency relation edges of the target perturbation edge.

[0101] Extract the relation predicate of the target perturbation edge, and determine the interaction type of the target perturbation edge based on the behavior category described by the relation predicate.

[0102] Based on the physical dependencies and logical temporal relationships between interactive behaviors in a conference room setting, we pre-sort out the correspondence between various interactive behaviors and their necessary prerequisite interactive behaviors, and store each type of interaction and its dependent prerequisite interactive types in the form of mapping entries to build a visual relational dependency prior library.

[0103] For each adjacency edge, perform relation predicate parsing to extract the behavior category described by the relation predicate of that adjacency edge and determine the interaction type of that adjacency edge.

[0104] Using the interaction type of the adjacent edge as the retrieval condition, search for a mapping entry that matches the interaction type in the visual relation dependency prior library, read all the prerequisite interaction types recorded in the mapping entry, and use the read prerequisite interaction types as the prerequisite relations on which the semantics of the adjacent edge depend.

[0105] The interaction type of the target perturbation edge is determined to be subordinate to the premise relation of the adjacent edge. The interaction type of the target perturbation edge is checked one by one to see if it appears in the list of premise interaction types contained in the premise relation. If the interaction type of the target perturbation edge does indeed belong to an item in the list, the adjacent edge is marked as a semantically inactive edge.

[0106] Remove all adjacent edges marked as semantically inactive from all adjacent edges. The remaining adjacent edges are the candidate receiving edges that pass the liveness check.

[0107] In the scene graph, count the total number of relational edges associated with each of the two end nodes connected to each candidate receiving edge. Use the sum of these two values ​​as the topology degree of the candidate receiving edge. Calculate the ratio of the topology degree of each candidate receiving edge to the total topology degree of all candidate receiving edges. Use this ratio as the attention receiving coefficient of the candidate receiving edge.

[0108] Based on the co-occurrence frequency of each interaction type in the meeting room scenario during violations, a semantic affinity matrix is ​​pre-constructed. This matrix stores the semantic affinity values ​​between any two interaction types. The higher the semantic affinity value, the closer the association between the two interaction types in the violation.

[0109] Using the interaction type of the candidate receiving edge and the interaction type of the target perturbation edge as row and column indices, the semantic affinity value between the candidate receiving edge and the target perturbation edge is read from the preset semantic affinity matrix.

[0110] Using a preset temperature coefficient as the amplification factor for the exponential operation, the semantic affinity value is multiplied by the temperature coefficient and used as the exponent. The natural constant is then raised to the power of this exponent to obtain the affinity index of the candidate receiving edge. The affinity index is then multiplied by the attention receiving coefficient of the candidate receiving edge to obtain the allocation numerator of the candidate receiving edge.

[0111] Perform the same calculation process on each candidate receiving edge to obtain the allocation numerator for each candidate receiving edge. Sum the allocation numerators of all candidate receiving edges one by one and use the sum as the allocation denominator.

[0112] Divide the allocation numerator of the candidate receiving edge by the allocation denominator to obtain the allocation ratio of the candidate receiving edge. Multiply the allocation ratio by the attention weight of the target perturbation edge, and then multiply by the survival indicator function value of the candidate receiving edge. The survival indicator function value is one when the candidate receiving edge passes the survival check and zero when it fails. The final product is the amount of attention weight transfer received by the candidate receiving edge.

[0113] Calculate the attention weight transfer amount for each candidate receiving edge, and redistribute the attention weight of the target perturbation edge to each candidate receiving edge. Delete the target perturbation edge from the scene graph. The graph structure that remains after deletion is the counterfactual intervention scene graph of the conference room.

[0114] The beneficial effect is that by using causal attention allocation, the attention weight of the target perturbation edge is directionally transferred to the adjacent relationship edge that has passed the survival check based on semantic affinity. While removing the main cause relationship edge, the previously masked collaborative violation factors are exposed due to the compensation weight, realizing a hierarchical causal explanation of the main cause and collaborative cause of the violation. This solves the problem of incomplete attribution caused by the direct discarding of perturbation information in traditional methods, and significantly enhances the interpretability and logical rigor of the anomaly detection results.

[0115] S03. The semantic vector of the counterfactual intervention scenario diagram and the rule semantic vector of the conference room security rule text are used to measure the deviation and obtain the counterfactual score of the conference room. When the counterfactual score exceeds a preset anomaly threshold, or the aggregate value of the compensation effect score of the conference room exceeds a preset coordination threshold, it is determined that the video frame has a violation and an anomaly explanation is output.

[0116] In this embodiment of the invention, the step of measuring the deviation between the semantic vector of the counterfactual intervention scenario diagram and the rule semantic vector of the conference room security rule text to obtain the counterfactual score of the conference room includes:

[0117] The original degree of violation of the meeting room is obtained by measuring the deviation between the semantic vector of the scene graph and the semantic vector of the rule.

[0118] The degree of intervention violation in the meeting room is obtained by measuring the deviation between the semantic vector of the counterfactual intervention scenario diagram and the semantic vector of the rule.

[0119] The difference between the original violation level and the intervention violation level is used as the counterfactual score for the meeting room;

[0120] The amount of attention weight shift is used as the compensation effect score for the meeting room.

[0121] When the counterfactual score exceeds a preset anomaly threshold, or the aggregated value of the compensation effect score of the meeting room exceeds a preset synergy threshold, the video frame is determined to have a violation, and an anomaly explanation is output, including:

[0122] When the counterfactual score exceeds a preset anomaly threshold, the video frame is determined to have a violation. The entity identifier associated with the target perturbation edge in the scene graph and the counterfactual score are output as the primary cause explanation. At the same time, the entity identifier and the corresponding compensation effect score of the candidate receiving edge compensation effect score in the counterfactual intervention scene graph that exceed a preset collaborative threshold are output as the collaborative cause explanation.

[0123] When the aggregated value of the compensation effect score exceeds a preset collaborative threshold, the video frame is determined to have a violation. The entity identifier and corresponding compensation effect score of the candidate receiving edge whose compensation effect score exceeds the preset collaborative threshold are output as a collaborative cause explanation. At the same time, the entity identifier associated with the target perturbation edge and the counterfactual score are output as the primary cause explanation.

[0124] The scene graph is input into a graph encoder for encoding. The graph encoder aggregates information from each node and relational edge in the scene graph, compressing the aggregated graph structure information into a fixed-length numerical sequence, which is the semantic vector of the scene graph. The conference room safety rule text is input into a text encoder for encoding. The text encoder extracts semantic elements such as subjects, behaviors, objects, and constraints described in the safety rule text, converting these semantic elements into a numerical sequence with the same dimension as the semantic vector of the scene graph. This numerical sequence is the rule semantic vector.

[0125] In the shared semantic space, the distance between the scene graph semantic vector and the rule semantic vector is calculated. Cosine distance is used as the metric, and the cosine value of the angle between the two vectors is used as the deviation metric. The deviation metric is the original violation degree of the meeting room. The larger the original violation degree value, the smaller the deviation between the scene graph and the safety rules, and the higher the violation degree.

[0126] The counterfactual intervention scenario graph is input into the same graph encoder for encoding, resulting in a semantic vector of the counterfactual intervention scenario graph. The cosine distance between the semantic vector of the counterfactual intervention scenario graph and the regular semantic vector is calculated in the shared semantic space. This cosine distance is used as the deviation metric, which represents the degree of intervention violation in the meeting room. A higher deviation metric indicates a smaller deviation from the security rules and a lower degree of violation after removing the target perturbation edge and redistributing attention weights.

[0127] The difference between the original violation level and the intervention violation level is the counterfactual score of the meeting room. The larger the counterfactual score, the more significant the causal contribution of the target perturbation edge to the violation.

[0128] The amount of attention weight transfer received by each candidate receiving edge during the redistribution process is directly used as the compensation effect score of that candidate receiving edge. The larger the compensation effect score, the stronger the cooperative violation signal exposed by the candidate receiving edge after the target perturbation edge is removed.

[0129] The compensation effect scores of all candidate receiving edges are summed one by one, and the sum is used as the aggregate value of the compensation effect scores of the meeting room.

[0130] The counterfactual score is compared with a preset anomaly threshold, and the aggregated value of the compensation effect score is compared with a preset coordination threshold. When the counterfactual score exceeds the preset anomaly threshold, a violation is determined in the current video frame. If the determination is successful, the head entity identifier and tail entity identifier connected to the target perturbation edge in the scene graph are obtained, and these two entity identifiers, along with the counterfactual score, are output as the primary cause explanation. Simultaneously, the compensation effect score of each candidate receiving edge is checked one by one to see if it exceeds the preset coordination threshold. The head entity identifier and tail entity identifier connected to the candidate receiving edge that exceeds the threshold, along with the compensation effect score of that edge, are output as the coordination cause explanation.

[0131] When the aggregated compensation effect score exceeds a preset coordination threshold, a violation is determined in the current video frame. Once determined, the compensation effect score of each candidate receiving edge is checked to see if it exceeds the preset coordination threshold. The head and tail entity identifiers connected to the candidate receiving edges exceeding the threshold, along with the compensation effect score of that edge, are output as a coordination causal explanation. Simultaneously, the head and tail entity identifiers connected to the target perturbation edge in the scene graph are acquired, and these two entity identifiers, along with the counterfactual score, are output as the primary causal explanation.

[0132] The beneficial effect is that by comparing the difference in the degree of violation between the original scene graph and the counterfactual intervention scene graph in the shared semantic space, the causal contribution of the target perturbation edge is quantified as a counterfactual score. At the same time, the amount of attention weight transfer obtained by the candidate receiving edge after redistribution is used as the compensation effect score, realizing the independent quantitative evaluation of the main cause and co-cause of the violation. Furthermore, the dual threshold judgment mechanism triggers the output of the main cause explanation and the co-cause explanation respectively, providing a hierarchical and quantifiable causal attribution chain for abnormal behavior detection in the conference room.

[0133] like Figure 2 The diagram shown is a functional block diagram of a conference room abnormal behavior detection system based on image recognition, provided in an embodiment of the present invention.

[0134] The conference room abnormal behavior detection system based on image recognition described in this invention can be installed in an electronic device. Depending on the functions implemented, the system may include a visual structured analysis module, a counterfactual graph intervention generation module, and an anomaly determination module. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0135] In this embodiment, the functions of each module / unit are as follows:

[0136] The visual structured parsing module is used to perform structured parsing on the video frames in the video frame sequence of the conference room to obtain a scene diagram of the conference room.

[0137] The counterfactual graph intervention generation module is used to perturb the graph structure of the scene graph and perform causal attention allocation during the perturbation process, transferring the attention weight carried by the perturbation edge to the adjacent relationship edge of the perturbation edge, thereby obtaining the counterfactual intervention scene graph of the conference room.

[0138] The anomaly determination module is used to measure the deviation between the semantic vector of the counterfactual intervention scenario diagram and the rule semantic vector of the conference room security rule text to obtain the counterfactual score of the conference room. When the counterfactual score exceeds a preset anomaly threshold, or the aggregated value of the compensation effect score of the conference room exceeds a preset coordination threshold, the video frame is determined to have a violation, and an anomaly explanation is output.

[0139] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0140] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0141] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0142] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0143] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for detecting abnormal behavior in a conference room based on image recognition, characterized in that, The method includes: S01. Perform structured parsing on the video frames in the video frame sequence of the conference room to obtain a scene diagram of the conference room; S02. Perturb the graph structure of the scene graph and perform causal attention allocation during the perturbation process, transferring the attention weight carried by the perturbation edge to the adjacent edge of the perturbation edge, to obtain the counterfactual intervention scene graph of the conference room, including: The semantic correlation between each reliable relation edge in the scene graph and the conference room security rule text is determined to obtain candidate perturbation edges of the scene graph; The scene graph is graph encoded to obtain the attention weights of the candidate perturbation edges, and the candidate perturbation edge with the largest attention weight is selected as the target perturbation edge. Based on the adjacency topology of the target perturbation edge and the target perturbation edge sharing an endpoint with it in the scenario graph, the adjacency relationship edge of the target perturbation edge is determined; The adjacency edges are subjected to liveness verification, and semantically inactive adjacency edges are filtered out to obtain candidate receiving edges of the scene graph; Based on the attention reception coefficient of the candidate receiving edge and the semantic affinity between the candidate receiving edge and the target perturbation edge, the attention weight of the target perturbation edge is redistributed to the candidate receiving edge, and the target perturbation edge is removed from the scene graph to obtain the counterfactual intervention scene graph of the conference room. The formula for calculating the attention weight transfer amount of the candidate receiving edge is as follows: ； In the formula, Indicates the candidate receiving edge Attention weight transfer amount, This represents the target perturbation edge. Indicates the first Candidate receiving edge, This represents the attention weight of the target perturbation edge. (*) indicates the semantic affinity. This indicates the preset temperature coefficient. This represents the attention reception coefficient. This represents the attention reception coefficient for traversing the indexed edges. Represents the liveness indicator function. Indicates the candidate receiving edge; The amount of attention weight shift is used as the compensation effect score of the meeting room; The compensation effect scores of all candidate receiving edges are summed one by one, and the sum is used as the aggregate value of the compensation effect scores of the meeting room. S03. The semantic vector of the counterfactual intervention scenario diagram and the rule semantic vector of the conference room security rule text are used to measure the deviation and obtain the counterfactual score of the conference room. When the counterfactual score exceeds a preset anomaly threshold, or the aggregate value of the compensation effect score of the conference room exceeds a preset coordination threshold, it is determined that the video frame has a violation and an anomaly explanation is output.

2. The method for detecting abnormal behavior in a conference room based on image recognition as described in claim 1, characterized in that, The step of performing structured parsing on the video frames in the conference room video frame sequence to obtain a scene diagram of the conference room includes: Visual perception is performed on video frames in the video frame sequence of the conference room to obtain candidate entities of the conference room; Based on the spatial positional relationships and interaction postures of the candidate entities, the initial relationship edges of the candidate entities are obtained; Perform a semantic consistency check on the initial relation edges, filter out the initial relation edges that contradict the common sense of the meeting room behavior, and obtain the reliable relation edges of the meeting room; Using the entities associated with the reliable relationship edges as nodes and the visual relationships they represent as edges, construct a scene graph of the conference room.

3. The method for detecting abnormal behavior in a conference room based on image recognition as described in claim 2, characterized in that, The step of performing a semantic consistency check on the initial relation edges to filter out initial relation edges that contradict common sense about meeting room behavior, thereby obtaining reliable relation edges for the meeting room, includes: Using pre-defined legal entity relationships as a priori constraints, construct the legal relationship tuple for the meeting room; The head entity attributes, relation predicates, and tail entity attributes associated with the initial relation edges are extracted into triples to obtain the relation triples to be tested for the meeting room. The triplet of the relation to be tested is compared with the legal relation tuple item by item, and the initial relation edges that fail to match are marked as semantic conflict edges. The semantically conflicting edges are removed from the initial relation edges to obtain the reliable relation edges of the conference room.

4. The method for detecting abnormal behavior in a conference room based on image recognition as described in claim 1, characterized in that, The step of performing a liveness check on the adjacency edges, filtering out semantically inactive adjacency edges, and obtaining candidate receiving edges for the scene graph includes: Based on the relational predicate of the target perturbation edge, determine the interaction type of the target perturbation edge; Based on the preset visual relation dependency prior, the preconditions of the adjacent relation edges are traced to obtain the preconditions on which the semantics of the adjacent relation edges depend. The interaction type of the target perturbation edge is determined to be subordinate to the premise relationship. If the interaction type belongs to the premise relationship, the adjacent relationship edge is marked as a semantically inactive edge. Remove the semantically inactive edges from the adjacency edges to obtain the candidate receiving edges of the scene graph.

5. The method for detecting abnormal behavior in a conference room based on image recognition as described in claim 4, characterized in that, The method of tracing the preconditions of the adjacent edges based on the preset visual relation dependency prior to obtain the preconditions upon which the semantics of the adjacent edges depend includes: Based on the physical dependencies and logical temporal relationships between interactive behaviors in the conference room scenario, a visual relation dependency prior library is constructed, which stores the mapping relationship between each interaction type and its prerequisite interaction type. Perform relation predicate parsing on the adjacency edges to determine the interaction type of the adjacency edges; Based on the visual relation dependency prior library, the prerequisite conditions are retrieved by using the interaction type of the adjacent relation edge as the index, and the prerequisite interaction type on which the semantics of the adjacent relation edge depends is obtained. The aforementioned prerequisite interaction type is used as the prerequisite relation upon which the semantics of the adjacency relation edge are determined.

6. The method for detecting abnormal behavior in a conference room based on image recognition as described in claim 1, characterized in that, The step of measuring the deviation between the semantic vector of the counterfactual intervention scenario diagram and the rule semantic vector of the conference room security rule text to obtain the counterfactual score of the conference room includes: The original degree of violation of the meeting room is obtained by measuring the deviation between the semantic vector of the scene graph and the semantic vector of the rule. The degree of intervention violation in the meeting room is obtained by measuring the deviation between the semantic vector of the counterfactual intervention scenario diagram and the semantic vector of the rule. The difference between the original violation level and the intervention violation level is used as the counterfactual score of the meeting room.

7. The method for detecting abnormal behavior in a conference room based on image recognition as described in claim 1, characterized in that, When the counterfactual score exceeds a preset anomaly threshold, or the aggregated value of the compensation effect score of the meeting room exceeds a preset synergy threshold, the video frame is determined to have a violation, and an anomaly explanation is output, including: When the counterfactual score exceeds a preset anomaly threshold, the video frame is determined to have a violation. The entity identifier associated with the target perturbation edge in the scene graph and the counterfactual score are output as the primary cause explanation. At the same time, the entity identifier and the corresponding compensation effect score of the candidate receiving edge compensation effect score in the counterfactual intervention scene graph that exceed a preset collaborative threshold are output as the collaborative cause explanation. When the aggregated value of the compensation effect score exceeds a preset collaborative threshold, the video frame is determined to have a violation. The entity identifier and corresponding compensation effect score of the candidate receiving edge whose compensation effect score exceeds the preset collaborative threshold are output as a collaborative cause explanation. At the same time, the entity identifier associated with the target perturbation edge and the counterfactual score are output as the primary cause explanation.

8. A conference room abnormal behavior detection system based on image recognition, characterized in that, To implement the conference room abnormal behavior detection method based on image recognition as described in claim 1, the system comprises: The visual structured parsing module is used to perform structured parsing on the video frames in the video frame sequence of the conference room to obtain a scene diagram of the conference room; The counterfactual graph intervention generation module is used to perturb the graph structure of the scene graph and perform causal attention allocation during the perturbation process, transferring the attention weight carried by the perturbation edge to the adjacent relationship edge of the perturbation edge, thereby obtaining the counterfactual intervention scene graph of the conference room, including: The semantic correlation between each reliable relation edge in the scene graph and the conference room security rule text is determined to obtain candidate perturbation edges of the scene graph; The scene graph is graph encoded to obtain the attention weights of the candidate perturbation edges, and the candidate perturbation edge with the largest attention weight is selected as the target perturbation edge. Based on the adjacency topology of the target perturbation edge and the target perturbation edge sharing an endpoint with it in the scenario graph, the adjacency relationship edge of the target perturbation edge is determined; The adjacency edges are subjected to liveness verification, and semantically inactive adjacency edges are filtered out to obtain candidate receiving edges of the scene graph; Based on the attention reception coefficient of the candidate receiving edge and the semantic affinity between the candidate receiving edge and the target perturbation edge, the attention weight of the target perturbation edge is redistributed to the candidate receiving edge, and the target perturbation edge is removed from the scene graph to obtain the counterfactual intervention scene graph of the conference room. The formula for calculating the attention weight transfer amount of the candidate receiving edge is as follows: ； In the formula, Indicates the candidate receiving edge Attention weight transfer amount, This represents the target perturbation edge. Indicates the first Candidate receiving edge, This represents the attention weight of the target perturbation edge. (*) indicates the semantic affinity. This indicates the preset temperature coefficient. This represents the attention reception coefficient. This represents the attention reception coefficient for traversing the indexed edges. Represents the liveness indicator function. Indicates the candidate receiving edge; The amount of attention weight shift is used as the compensation effect score of the meeting room; The compensation effect scores of all candidate receiving edges are summed one by one, and the sum is used as the aggregate value of the compensation effect scores of the meeting room. The anomaly determination module is used to measure the deviation between the semantic vector of the counterfactual intervention scenario diagram and the rule semantic vector of the conference room security rule text to obtain the counterfactual score of the conference room. When the counterfactual score exceeds a preset anomaly threshold, or the aggregated value of the compensation effect score of the conference room exceeds a preset coordination threshold, the video frame is determined to have a violation, and an anomaly explanation is output.

Images