Event processing method and apparatus, electronic device, and storage medium

Abstract

This application relates to the field of data processing technology and provides an event processing method, comprising: acquiring multimodal data to be processed; performing event detection processing on the multimodal data using a preset multimodal large model to obtain a target event and a first evaluation result of the target event; querying the verification link of the target event in a preset knowledge graph based on the target event and the first evaluation result; determining a combination of small models for verifying the target event in a preset small model pool based on the verification link; performing verification processing on the target event based on the combination of small models to obtain a second evaluation result of the target event; and determining an early warning strategy for the target event based on the second evaluation result. This invention solves the problems of existing methods being limited by lighting, angle, and quality issues, relying on manual intervention and active judgment, lacking a deep understanding and intelligent response to complex scenarios and public safety events, resulting in poor real-time risk identification and early warning capabilities.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to an event processing method, apparatus, electronic device and storage medium. Background Technology

[0002] As urban public safety becomes increasingly complex, the number of public emergencies, mass incidents, and dangerous events continues to rise. Public safety departments typically rely on video surveillance systems, manual patrols, and some intelligent identification algorithms for risk detection and response. However, in practical applications, existing technological solutions still have significant limitations and struggle to meet the real-time early warning needs in multi-dimensional scenarios. Currently, traditional monitoring and identity verification technologies are limited by lighting, angle, and quality issues, making them susceptible to deception and unable to reliably identify high-risk individuals; traditional monitoring systems are primarily used for event recording, lacking advanced analysis and intelligent identification capabilities, and thus unable to provide real-time early warnings; manual patrols and security personnel may experience fatigue and subjective judgment, making them ineffective in handling large-scale venues and high-risk events; traditional data analysis and machine learning methods require large amounts of labeled data and cannot adapt to the complex relationships in multi-dimensional data; sensors and rule-based systems are limited by specific technological constraints, such as sensor accuracy and the rigidity of rule-based systems.

[0003] Therefore, there is an urgent need for an event handling method that can accurately identify high-risk behaviors, predict event risks, and automatically coordinate responses. This is to address the problems of existing methods being limited by lighting, angle, and quality issues, relying on manual intervention and proactive judgment, lacking a deep understanding of complex scenarios and public safety events, and lacking intelligent response capabilities, resulting in poor real-time risk identification and early warning capabilities. Summary of the Invention

[0004] This application provides an event processing method that addresses the shortcomings of existing methods, such as limitations imposed by lighting, angle, and quality issues, reliance on manual intervention and proactive judgment, and a lack of analysis and identification of multidimensional data, resulting in poor proactive identification and early warning capabilities for real-time risks. The method utilizes a pre-set multimodal large model to perform event detection processing on the multimodal data to be processed, obtaining a target event and a first evaluation result. Based on the target event and the first evaluation result, a verification link for the target event is retrieved from a pre-set knowledge graph. Based on the verification link, a combination of small models for verifying the target event is determined from a pre-set small model pool. This combination of small models is then used to verify the target event, obtaining a second evaluation result. Based on the second evaluation result, an early warning strategy for the target event is determined. This method solves the problems of existing methods being limited by lighting, angle, and quality issues, relying on manual intervention and proactive judgment, lacking a deep understanding and intelligent response to complex scenarios and public safety events, and resulting in poor real-time risk identification and early warning capabilities.

[0005] In a first aspect, embodiments of this application provide an event handling method, the method comprising the following steps:

[0006] Acquire the multimodal data to be processed;

[0007] The multimodal data is processed by a pre-defined multimodal large model to obtain the target event and the first evaluation result of the target event.

[0008] Based on the target event and the first evaluation result, the verification link of the target event is retrieved from the preset knowledge graph;

[0009] Based on the verification link, a combination of small models for verifying the target event is determined from a preset small model pool;

[0010] Based on the combination of small models, the target event is validated to obtain a second evaluation result of the target event;

[0011] Based on the second evaluation results, an early warning strategy for the target event is determined.

[0012] Optionally, the step of performing event detection processing on the multimodal data using a preset multimodal large model to obtain the target event and a first evaluation result of the target event includes:

[0013] Spatiotemporal alignment and fusion archiving are performed on the multimodal data from different sources to obtain fused multimodal data.

[0014] Semantic extraction is performed on the fused data using a pre-defined multimodal large model to obtain semantically structured data corresponding to the multimodal data;

[0015] Based on the semantically structured data, the target event and its first evaluation result are obtained.

[0016] Optionally, the step of querying the verification link of the target event in a preset knowledge graph based on the target event and the first evaluation result includes:

[0017] According to a preset mapping strategy, the target event and the first evaluation result are mapped as graph query conditions;

[0018] Using the graph query conditions, the verification link of the target event can be retrieved from the preset knowledge graph.

[0019] Optionally, before retrieving the verification link of the target event from the preset knowledge graph, the method further includes:

[0020] Acquire sample event data, and extract knowledge nodes based on the sample event data. The knowledge nodes include event nodes, behavioral feature nodes, scene nodes, small model nodes, and risk nodes.

[0021] Calculate the dynamic association weights between various knowledge nodes;

[0022] By using the dynamic association weights, the various knowledge nodes are connected to obtain a knowledge graph.

[0023] Optionally, calculating the dynamic association weights between each knowledge node includes:

[0024] Calculate the co-occurrence weight between two knowledge nodes based on their occurrence frequency and co-occurrence frequency;

[0025] Determine the prior knowledge weight between two knowledge nodes based on the expert scores of the two knowledge nodes.

[0026] Based on the hit rate of small model nodes for other types of knowledge nodes, determine the model hit rate weights of small model nodes and other types of knowledge nodes.

[0027] Determine the context weight between two knowledge nodes based on their contextual relationship.

[0028] The co-occurrence weight, the prior knowledge weight, the model hit rate weight, and the context weight are weighted and fused to obtain the dynamic association weight between each knowledge node.

[0029] Optionally, the step of performing verification processing on the target event based on the small model combination to obtain a second evaluation result of the target event includes:

[0030] Each small model in the small model combination is loaded according to the combination strategy;

[0031] The multimodal data corresponding to the target event is processed through various small models to obtain the recognition results of the small models;

[0032] Based on the first evaluation result, the verification link, and the small model identification result, a second evaluation result for the target event is calculated.

[0033] Optionally, after determining the early warning strategy for the target event based on the second evaluation result, the method further includes:

[0034] Obtain the processing result of the target event;

[0035] Based on the processing results, the multimodal large model and the knowledge graph are optimized and adjusted.

[0036] Secondly, embodiments of this application provide an event processing apparatus, the event processing apparatus comprising:

[0037] The acquisition module is used to acquire the multimodal data to be processed.

[0038] The first processing module is used to perform event detection processing on the multimodal data through a preset multimodal large model to obtain the target event and the first evaluation result of the target event;

[0039] The query module is used to query the verification link of the target event in a preset knowledge graph based on the target event and the first evaluation result;

[0040] The first determining module is used to determine, based on the verification link, a combination of small models for verifying the target event from a preset small model pool;

[0041] The second processing module is used to perform verification processing on the target event based on the small model combination to obtain a second evaluation result of the target event;

[0042] The second determining module is used to determine the early warning strategy for the target event based on the second evaluation result.

[0043] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the event handling method provided in embodiments of the present invention.

[0044] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the event handling method provided in the embodiments of the invention.

[0045] The above-mentioned solution of this application has the following beneficial effects: acquiring multimodal data to be processed; performing event detection processing on the multimodal data through a preset multimodal large model to obtain the target event and the first evaluation result of the target event; based on the target event and the first evaluation result, querying the verification link of the target event in a preset knowledge graph; based on the verification link, determining the combination of small models for verifying the target event in a preset small model pool; based on the combination of small models, performing verification processing on the target event to obtain the second evaluation result of the target event; based on the second evaluation result, determining the early warning strategy for the target event. This invention solves the problems of existing methods being limited by lighting, angle, and quality issues, relying on manual intervention and active judgment, lacking a deep understanding and intelligent response to complex scenarios and public safety events, resulting in poor real-time risk identification and early warning capabilities.

[0046] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating an event handling method provided in one embodiment of this application;

[0049] Figure 2 This is a schematic diagram of the structure of an event processing device provided in one embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0052] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0053] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0054] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0055] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0056] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0057] like Figure 1 As shown, Figure 1 This is a flowchart of an event handling method provided by an embodiment of the present invention. The event handling method includes the following steps:

[0058] 101. Obtain the multimodal data to be processed.

[0059] In this embodiment of the invention, the event processing method described above can be applied to an event processing platform, which can be built on a server-based or distributed platform. The event processing platform includes a data interface (for sensors or users to upload data), a knowledge database, and a knowledge database construction program. The data interface can be used to acquire multimodal data to be processed, and the knowledge database construction program can be used to construct the knowledge database. The knowledge database is specifically used to provide additional association information for identified data entities, thereby improving the depth of the data recognition system's understanding of the content.

[0060] The aforementioned multimodal data to be processed can be multimodal data that needs to be analyzed and processed.

[0061] The aforementioned multimodal data can originate from multimodal data from urban monitoring systems, checkpoint cameras, public surveillance cameras, security terminal videos, text alarm information, etc.

[0062] The aforementioned multimodal data can be video data, image data, text data, checkpoint data, etc.

[0063] 102. Perform event detection processing on multimodal data using a pre-set multimodal large model to obtain the target event and the first evaluation result of the target event.

[0064] In this embodiment of the invention, the aforementioned pre-defined multimodal large model can be a multimodal large model built based on deep learning or machine learning, such as LLM, MLLMs, etc. The aforementioned LLM (Large Multimodal Model) is a large-scale artificial intelligence model capable of simultaneously processing and understanding multiple data types (such as text, images, audio, video, etc.). By integrating information from different modalities, LLMs achieve more comprehensive and in-depth understanding and reasoning capabilities, breaking through the limitations of traditional single-modal models. The aforementioned MLLMs combine the language understanding and reasoning capabilities of large language models (LLMs) with the ability to process multi-sensory inputs (such as images, audio, video), aiming to simulate the way humans integrate information through multiple senses, thereby achieving a comprehensive understanding of complex scenes. The aforementioned pre-defined multimodal large model can process multimodal data such as video, images, and text, supporting cross-modal understanding and reasoning.

[0065] The above event detection processing can be a process of performing event detection on multimodal data using a preset multimodal large model.

[0066] The aforementioned target event can be a structured and semantic event description result output after multimodal data has been processed by a multimodal large model for event detection.

[0067] The aforementioned first assessment result can be an assessment result corresponding to the target event. It can be risk assessment data for the target event. Risk assessment data can represent the characteristics of the target event's potential risk through quantitative or categorical methods. Risk assessment data can include related target events, the confidence level of the target events, etc. The aforementioned related target events can be the correlation or dependence between other information related to the target event. The aforementioned confidence level can be the degree of credibility of the target event.

[0068] It should be noted that this can be achieved by performing holistic semantic analysis on multimodal data using a pre-defined multimodal large-scale model. This analysis identifies scene types, character states, behavioral characteristics, object categories, and interaction relationships, generating a semantically structured result of scene description, event tags, key entities, and risk factors. A preliminary assessment of the risk factors based on this semantically structured result is then performed to obtain the first assessment result. This holistic semantic analysis can be a comprehensive analysis process involving a pre-defined multimodal large-scale model, providing a global, relational, and in-depth understanding of the multimodal data. The scene types mentioned above could be shopping malls, campuses, squares, etc. The character states mentioned above could be a series of implicit, persistent attributes and contextual information about people in the scene. The behavioral characteristics mentioned above could be directly observable, instantaneous external actions between people, objects, or between people and objects, such as running, punching, or gathering. The object categories mentioned above could be object types that have direct or potential significance to public safety, such as attack weapons, dangerous goods, or abnormal vehicles. The aforementioned interactive relationships can be dynamic associations with clear semantic orientations that occur between different entities in a scene (especially between people and between people and objects). For example, it could be, "Person A is chasing Person B"; "Person C is moving towards Person D"; etc.

[0069] In one possible implementation, for example, a pre-set multimodal big model is used to perform overall semantic parsing on the multimodal data to parse out the overall situation of "a student running in the corridor with a suspected dangerous object during the school lunch break, while other students around him are in a state of panic and avoidance", and output the target event such as "dangerous event" or "safety accident" and the first evaluation result corresponding to the target event.

[0070] 103. Based on the target event and the first evaluation result, query the verification link of the target event in the preset knowledge graph.

[0071] In this embodiment of the invention, the aforementioned preset knowledge graph can be a knowledge graph pre-set by the system, and the aforementioned knowledge graph can be a structured semantic relationship network constructed based on knowledge in the public safety domain. A knowledge graph (KG) is a graph structure that constructs semantic relationships such as "people-objects-events-places-time" and is used for event reasoning.

[0072] A knowledge graph is a semantic network that describes the objective world using a graph structure. Essentially, it is a knowledge base that reveals the relationships between entities (such as people, places, and events). The core of a knowledge graph is its graph structure, which consists of nodes and edges. Nodes represent entities or concepts, and edges represent the relationships between entities. This structure organizes data through triples (entity-relationship-entity).

[0073] The aforementioned pre-defined knowledge graph contains different types of nodes and edges connecting them. Nodes include event nodes, behavioral feature nodes, scene nodes, model nodes, and risk nodes. Event nodes can represent events such as fighting, tailing, gathering, or waving dangerous objects, indicating the type of outcome requiring a warning. Behavioral feature nodes can represent actions such as pushing, running, confrontation, or raising a hand, indicating the atomic actions constituting the event (i.e., behavioral features extracted by the multimodal large model). Scene nodes can represent environments such as campuses, hospitals, subway stations, shopping malls, or late-night barbecue stalls, indicating the environmental context in which the event occurs. Model nodes can represent clustering analysis models or hazardous material detection models. Risk nodes can be categorized as low, medium, high, or extremely high, indicating the severity label of the event. These nodes can be connected by edges with dynamic association weights, which integrate co-occurrence weights, prior knowledge weights, model hit rate weights, and context weights. For example, pushing and shoving (high-frequency co-occurrence) → fighting; fighting (optimal verification tool) → fight recognition model; late-night barbecue stalls (contextual weighting) → fighting, etc.

[0074] The aforementioned verification chain can be based on the target event and the first evaluation result, finding verification methods or processes related to the target event and the first evaluation result in a pre-defined knowledge graph, thereby confirming the authenticity and accuracy of the target event.

[0075] It should be noted that the verification chain can be a structured and executable collaborative verification scheme generated from a pre-defined knowledge graph through graph reasoning and multi-factor weight calculation, based on the target event and the first evaluation result. The verification chain can be an instruction that explicitly lists one or more small models to be invoked to verify the hypothesis of the target event, the execution order and combination logic of each model, the corresponding input data specifications, and the preliminary result fusion rules.

[0076] 104. Based on the verification link, determine the combination of small models used to verify the target event from the preset small model pool.

[0077] In this embodiment of the invention, the aforementioned preset small model pool can be a pre-set small model pool of the system, containing a large number of long-tail small models for specific scenarios. Long-tail small models (Specialized Small Models, SSMs) can be high-precision, lightweight models designed for specific public safety scenarios. The small model pool includes, but is not limited to, face detection models, clustering analysis models, dangerous item detection models, dangerous event recognition models, abnormal running behavior recognition models, vehicle recognition and feature extraction models, area intrusion models, and tailgating detection models.

[0078] The aforementioned combination of small models can be a set of collaboratively executed small model tasks determined from a pre-defined pool of small models based on the verification link. Specifically, it can be one or more small models organized according to a specific collaboration logic, selected from a pre-defined pool of small model resources through resource matching, data binding, and task orchestration, using the verification link as input.

[0079] 105. Based on the combination of small models, the target event is validated to obtain the second evaluation result of the target event.

[0080] In this embodiment of the invention, the above-mentioned verification process can be a process of performing multi-dimensional and cross-dimensional verification analysis on the target event according to the arrangement logic within the combination of small models, and generating a second evaluation result of the target event by collecting the outputs of each small model, performing evidence fusion and conflict resolution according to preset rules. The preset rules can be rules pre-set by the system, and the preset rules can be a set of logical criteria and algorithms used to guide and constrain the collaborative verification and decision-making process of small models.

[0081] The aforementioned second assessment result can be a risk assessment report for the target event generated after validating the target event based on a combination of small models. The second assessment result includes a comprehensive conclusion and a precise risk level for the target event.

[0082] It should be noted that by combining small models to verify the target event, the problems of false positives and false negatives caused by the limitations of single model recognition can be significantly reduced.

[0083] 106. Based on the results of the second assessment, determine the early warning strategy for the target event.

[0084] In this embodiment of the invention, an early warning strategy for the target event can be determined based on the second evaluation result.

[0085] The aforementioned early warning strategy can be a set of executable, coordinated response instructions corresponding to the target event, determined based on the results of the second assessment. The early warning strategy specifies different control instructions for intervening in the target event, such as camera pan-tilt tracking the target, broadcasting intervention to the scene, pushing alarms to security terminals, automatically assigning nearby security personnel, and triggering the emergency response system.

[0086] It should be noted that the early warning strategy can be a target event determined in the early warning strategy library based on the results of the second assessment. The aforementioned early warning database can be a pre-set early warning database of the system, which contains pre-set response templates for different risk types, levels, and scenarios.

[0087] Furthermore, key elements in the second assessment results (such as event type, final risk level, and location of occurrence) can be used as query conditions to match the corresponding early warning strategies from the early warning strategy library.

[0088] In this embodiment of the invention, a multimodal large model is used to process multimodal data for event detection. This enables the identification of correlations and evolutionary trends between events, effectively predicting potential conflict behaviors and abnormal events in advance, and significantly improving the prediction capability of high-risk events. Through collaborative analysis using a combination of small models, target events are cross-validated using multiple models, significantly reducing false alarms and missed alarms caused by the limitations of single-model recognition. This invention addresses the problems of existing methods being limited by lighting, angle, and quality issues, relying on manual intervention and proactive judgment, lacking a deep understanding and intelligent response to complex scenarios and public safety events, resulting in poor real-time risk identification and early warning capabilities.

[0089] In this embodiment of the invention, multimodal data to be processed is acquired; event detection processing is performed on the multimodal data using a preset multimodal large model to obtain a target event and a first evaluation result of the target event; based on the target event and the first evaluation result, the verification link of the target event is queried from a preset knowledge graph; based on the verification link, a combination of small models for verifying the target event is determined from a preset small model pool; based on the combination of small models, the target event is verified to obtain a second evaluation result of the target event; based on the second evaluation result, an early warning strategy for the target event is determined. This invention solves the problems of existing methods being limited by lighting, angle, and quality issues, relying on manual intervention and active judgment, lacking a deep understanding and intelligent response to complex scenarios and public safety events, resulting in poor real-time risk identification and early warning capabilities.

[0090] It is understood that in the specific implementation of this application, multimodal data, time data, knowledge data, graph data and other related data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. The collection, use and processing of related data, as well as the training, deployment and invocation of algorithm models, must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0091] Optionally, in the step of performing event detection processing on multimodal data through a preset multimodal large model to obtain the target event and the first evaluation result of the target event, spatiotemporal alignment and fusion archiving processing can be performed on multimodal data from different sources to obtain fused multimodal data; semantic extraction is performed on the fused data through the preset multimodal large model to obtain semantic structured data corresponding to the multimodal data; and the target event and the first evaluation result of the target event are obtained based on the semantic structured data.

[0092] In this embodiment of the invention, the aforementioned multimodal data may be multimodal data from urban monitoring systems, checkpoint cameras, public security cameras, security terminal videos, text alarm information, etc.

[0093] The aforementioned multimodal data can be video data, image data, text data, checkpoint data, etc.

[0094] The aforementioned spatiotemporal alignment can be achieved through timestamp synchronization and spatial coordinate alignment. Timestamp synchronization can unify the timestamps of data from different sources to the same baseline time axis. For example, associating the reception time of an alarm text message (14:05:00) with a video clip from 14:04:55 to 14:05:05 in a monitoring system. Spatial coordinate alignment can map the image coordinates of multimodal data from different cameras to unified two-dimensional or three-dimensional geographic information.

[0095] The aforementioned fusion and archiving process can be a combination of fusion and archiving. Fusion can be a process of integrating spatiotemporally aligned multimodal data; for example, video clips, captured images, and related alarm texts from the same time period and area can be packaged into a single data packet. The aforementioned archiving process can be a process of establishing a multimodal data base library and performing tag management, data structuring, and format conversion on the data packets according to public safety industry standards (such as the GA / T series).

[0096] The fused data of the aforementioned multimodal data can be fused data that is spatiotemporally consistent, multi-source related, and in a unified format after spatiotemporal alignment and aggregation archiving of multimodal data.

[0097] The aforementioned pre-defined multimodal large models can be multimodal large models built based on deep learning or machine learning, such as LLMs, MLLMs, etc. These pre-defined multimodal large models can process multimodal data such as video, images, and text, and support cross-modal understanding and reasoning.

[0098] The aforementioned semantic extraction can be a process of extracting semantics from fused data using pre-defined multimodal data. It can be a process of identifying scene types, person states, behavioral characteristics, object categories, and interaction relationships within the multimodal data. Scene types can include shopping malls, campuses, squares, etc. Person states can be a series of implicit, persistent attributes and contexts concerning people in the scene. Behavioral characteristics can be directly observable, instantaneous external actions between people, objects, or between people and objects, such as running, punching, or gathering. Object categories can be object types that have direct or potential significance for public safety, such as attack weapons, dangerous goods, or abnormal vehicles. Interaction relationships can be dynamic associations with clear semantic directions occurring between different entities in the scene (especially between people and between people and objects), such as "Person A is chasing Person B"; "Person C is moving towards Person D," etc.

[0099] The aforementioned semantically structured data can be a structured, semantic event description result output after semantic extraction of fused data through a multimodal large model. It can be semantically structured data consisting of scene description, event tags, key entities, and risk factors.

[0100] The target events mentioned above can be identified by analyzing semantically structured data. Examples include school bullying incidents and social group conflict incidents.

[0101] The aforementioned first assessment result can be the assessment result corresponding to the target event. It can be risk assessment data made on the probability and risk level of the target event. The first assessment result includes scene complexity and event ambiguity. Scene complexity and event ambiguity are used to determine whether further verification of the target event is needed. For example, if the event type does not belong to the event types that can be handled by the various small models in the small model pool, further verification will not be performed. If the event type belongs to the event types that can be handled by the various small models in the small model pool, and the event ambiguity is greater than a threshold, further verification will be performed. If it is less than the threshold, it means that the event assessment is clear and further verification is not needed.

[0102] It should be noted that this invention utilizes a multimodal large model to extract semantics from multimodal data such as videos, images, and text. This can overcome the limitations of traditional algorithms that can only identify single actions or isolated events, and achieve accurate understanding of interpersonal relationships, behavioral logic, and scene semantics.

[0103] Optionally, in the step of querying the verification link of the target event in a preset knowledge graph based on the target event and the first evaluation result, the target event and the first evaluation result can be mapped to graph query conditions according to a preset mapping strategy; and the verification link of the target event can be queried in the preset knowledge graph through the graph query conditions.

[0104] In this embodiment of the invention, the aforementioned preset mapping strategy can be a mapping strategy pre-set by the system. The mapping strategy can be entity and concept extraction rules, attribute and parameter binding rules, context enhancement rules, query condition assembly rules, etc. Specifically, entity and concept extraction rules can extract standard nodes existing in the knowledge graph from the natural language description of the target event. For example, from "suspected armed conflict on campus," extract event type nodes (fighting), behavioral feature nodes (armed), and scene nodes (campus). Attribute and parameter binding can transform the first evaluation result into query weights or filtering conditions. For example, if the first evaluation result is a risk event related to campus conflict, then set the query priority parameter = "URGENT" (URGENT means urgent or pressing), or directly use the confidence level of the first evaluation result as the initial confidence level parameter of the query. Context enhancement rules can use scene information extracted from the target event (such as the location "campus") as the context constraint condition for the query. For example, if the location information is campus, then automatically append "scene node = [campus]" to the query condition and activate specific inference rules related to campus in the graph. Query condition assembly can be the combination of entity and concept extraction rules, attribute and parameter binding rules, and context enhancement rules into a data structure or query statement that meets the requirements of the knowledge graph query interface.

[0105] The aforementioned graph query conditions can be used to transform the target event and the first evaluation result into a pre-defined, executable internal form of the knowledge graph. The core objective of graph query conditions is to map natural language or formal descriptions onto a graph structure.

[0106] The aforementioned preset knowledge graph can be a pre-set knowledge graph of the system, and it can be a structured semantic relationship network constructed based on knowledge in the public safety domain. A knowledge graph (KG) is a graph structure that constructs semantic relationships such as "people-objects-events-places-time" and is used for event reasoning.

[0107] The aforementioned pre-defined knowledge graph contains different types of nodes and edges connecting them. Nodes include event nodes, behavioral feature nodes, scene nodes, model nodes, and risk nodes. Event nodes can represent events such as fighting, tailing, gathering, or waving dangerous objects, indicating the type of outcome requiring a warning. Behavioral feature nodes can represent actions such as pushing, running, confrontation, or raising a hand, indicating the atomic actions constituting the event (i.e., behavioral features extracted by the multimodal large model). Scene nodes can represent environments such as campuses, hospitals, subway stations, shopping malls, or late-night barbecue stalls, indicating the environmental context in which the event occurs. Model nodes can represent clustering analysis models or hazardous material detection models. Risk nodes can be categorized as low, medium, high, or extremely high, indicating the severity label of the event. These nodes can be connected by edges with dynamic association weights, which integrate co-occurrence weights, prior knowledge weights, model hit rate weights, and context weights. For example, pushing and shoving (high-frequency co-occurrence) → fighting; fighting (optimal verification tool) → fight recognition model; late-night barbecue stalls (contextual weighting) → fighting, etc.

[0108] The aforementioned verification link can be a graph query condition, a structured and executable verification method or process generated through reasoning in a preset knowledge graph.

[0109] It should be noted that the verification chain can be a list in the form of instructions that explicitly lists one or more small models to be called to verify the target event, the execution order and combination logic of each model, the corresponding input data specifications, and the preliminary result fusion rules.

[0110] Optionally, before retrieving the verification link of the target event from the preset knowledge graph, sample event data can be obtained, knowledge nodes can be extracted from the sample event data, dynamic association weights between each knowledge node can be calculated, and the knowledge nodes can be connected through the dynamic association weights to obtain the knowledge graph.

[0111] In this embodiment of the invention, the above-mentioned sample event data may be historical real events, handling reports, video cases, etc. collected from different data sources.

[0112] The aforementioned knowledge nodes include event nodes, behavioral characteristic nodes, scene nodes, small model nodes, and risk nodes. Event nodes can be fighting, tailing, gathering, or waving dangerous objects. Behavioral characteristic nodes can be pushing, running, confrontation, or raising hands. Scene nodes can be schools, hospitals, subway stations, shopping malls, or late-night barbecue stalls. Model nodes can be cluster analysis models or hazardous material detection models. Risk nodes can be categorized as low, medium, high, or extremely high.

[0113] The aforementioned dynamic association weights can be quantitative parameters that characterize the strength of associations between nodes, and can be calculated in real time and evolve over a long period of time.

[0114] The aforementioned knowledge graph can be obtained by connecting various knowledge nodes through dynamic association weights. The knowledge graph contains different types of nodes and edges connecting the nodes, including dynamic association weights between the nodes.

[0115] It should be noted that sample event data can be obtained, knowledge nodes can be extracted from the sample event data, dynamic association weights between each knowledge node can be calculated, and the knowledge nodes can be connected through the dynamic association weights to obtain a knowledge graph. Through the event chain reasoning mechanism driven by the knowledge graph, the relationship and evolution trend between events can be identified, and potential conflict behaviors and abnormal events can be effectively predicted in advance, thereby improving the ability to predict high-risk events.

[0116] Optionally, in the step of calculating the dynamic association weights between various knowledge nodes, the co-occurrence weight between two knowledge nodes can be calculated based on their occurrence frequency and co-occurrence frequency; the prior knowledge weight between two knowledge nodes can be determined based on their expert scores; the model hit rate weight between small model nodes and other types of knowledge nodes can be determined based on the hit rate of small model nodes against other types of knowledge nodes; the context weight between two knowledge nodes can be determined based on their contextual relationship; and the co-occurrence weight, prior knowledge weight, model hit rate weight, and context weight can be weighted and fused to obtain the dynamic association weights between various knowledge nodes.

[0117] In this embodiment of the invention, the co-occurrence weight can be used to evaluate the correlation strength between two knowledge nodes based on the number of times they appear and the number of times they co-occur.

[0118] The aforementioned prior knowledge weights can be determined based on the expert scores of the two knowledge nodes, representing the prior knowledge weights between them.

[0119] The aforementioned model hit rate weights can be determined based on the hit rates of small model nodes against other types of knowledge nodes, thus establishing a model hit rate weight between small model nodes and other types of knowledge nodes. The hit rate can be the hit rate of the small model against other types of knowledge nodes when validating events or behaviors.

[0120] The aforementioned context weights can be determined by the contextual relationship between two knowledge nodes. Context weights are determined based on the degree of association between two knowledge nodes.

[0121] The aforementioned weighted fusion can be a process of weightedly superimposing co-occurrence weights, prior knowledge weights, model hit rate weights, and context weights to generate dynamic association weights. This can be achieved by assigning a corresponding weight coefficient to each of the co-occurrence weight, prior knowledge weight, model hit rate weight, and context weight, and then multiplying each weight by its corresponding weight before summing the results to obtain the dynamic association weights.

[0122] Specifically, the co-occurrence weight can be expressed as:

[0123] W c = Freq(A,B) / (Freq(A)+Freq(B))

[0124] Among them, W c The co-occurrence weight is represented by Freq(A,B); Freq(A) represents the number of times node A and node B co-occur; Freq(A) represents the number of times node A appears; Freq(B) represents the number of times node B appears.

[0125] Prior knowledge weights can be expressed as:

[0126] W p = ExpertScore(A,B)

[0127] Among them, W p The prior knowledge weight is represented by ExpertScore(A,B); ExpertScore(A,B) represents the prior knowledge weight between knowledge node A and knowledge node B determined based on the expert score.

[0128] The model hit rate weights can be expressed as:

[0129] W m = TP / (TP+FN)

[0130] Among them, W m W represents the model hit rate weights; TP represents positive instances, indicating the number of times the small model correctly alerted when used to validate events or behaviors related to node B; FN represents the number of times the small model failed to identify events that should have triggered an alert. It is understandable that W... m A higher value (maximum 1) indicates that the smaller model is more reliable and has better historical performance.

[0131] Context weights can be represented as:

[0132] W ctx = f(Scene,Time,Density)

[0133] Among them, W ctxThe context weight is represented by Scene, which indicates the type of location where the event occurs, such as a campus or a subway station; Time indicates the time when the event occurs, such as late at night, lunch break, or holidays; Density indicates the density of people at the scene, such as sparse or dense.

[0134] Dynamic association weights can be expressed as:

[0135] W = αW c +βW p +γW m +δW ctx

[0136] Where W represents the dynamic association weight; W c W represents the co-occurrence weight; p W represents the weight of prior knowledge. m W represents the model hit rate weights. ctx α represents the context weight; α, β, γ, and δ are weight coefficients, and satisfy α+β+γ+δ=1.

[0137] The aforementioned dynamic association weights can be obtained by weighting and summing co-occurrence weights, prior knowledge weights, model hit rate weights, and context weights. Dynamic association weights are quantitative parameters that characterize the strength of associations between knowledge nodes and can be calculated in real time and evolve over the long term.

[0138] Understandably, we can calculate the co-occurrence weight between two knowledge nodes based on their frequency of occurrence and co-occurrence, determine the prior knowledge weight between them based on their expert scores, determine the model hit rate weight between the small model node and other types of knowledge nodes based on the hit rate of the small model node against other types of knowledge nodes, and determine the context weight between them based on their contextual relationship. By weighting and fusing the co-occurrence weight, prior knowledge weight, model hit rate weight, and context weight, we can obtain the dynamic association weight between each knowledge node, which can more accurately reflect the degree of association between knowledge nodes.

[0139] Optionally, in the step of verifying the target event based on the combination of small models to obtain the second evaluation result of the target event, each small model in the combination of small models can be loaded according to the combination strategy; the multimodal data corresponding to the target event can be processed through each small model to obtain the small model recognition result; and the second evaluation result of the target event can be calculated based on the first evaluation result, the verification link and the small model recognition result.

[0140] In this embodiment of the invention, the aforementioned combination strategy can be a predefined set of logical rules within the combination of small models, used to guide how multiple small models collaboratively execute verification tasks. The combination strategy clarifies the collaboration mode (parallel / serial / conditionally triggered), data flow path, and process control rules between small models.

[0141] The above loading can be a process of loading each small model in the small model combination according to the combination strategy.

[0142] The above-mentioned small model recognition results can be structured, confidence-based event detection conclusions output after processing the multimodal data corresponding to the target event through various small models.

[0143] Furthermore, a second evaluation result for the target event can be calculated based on the first evaluation result, the verification link, and the small model identification result.

[0144] The aforementioned first assessment result can be risk assessment data for the target event.

[0145] The aforementioned verification chain can be based on the target event and the first evaluation result, finding verification methods or processes related to the target event and the first evaluation result in a pre-defined knowledge graph.

[0146] The aforementioned second assessment result can be structured risk assessment data obtained by fusing and resolving conflicts between the first assessment result, the verification link, and the small model identification result. The second assessment result can be a confirmatory conclusion on the target event, a precise risk level, and a risk causal chain. The risk causal chain can be a clear logical causal chain explaining "where the risk comes from," constructed and output when generating the second assessment result.

[0147] It should be noted that each small model in the small model combination can be loaded according to the combination strategy, and the multimodal data corresponding to the target event can be processed through each small model to obtain the small model recognition result. Then, the second evaluation result of the target event can be calculated by combining the first evaluation result, the verification link, and the small model recognition result, thereby improving the recognition accuracy of the target event.

[0148] Optionally, after determining the early warning strategy for the target event based on the second evaluation results, the handling results of the target event can also be obtained; based on the handling results, the multimodal large model and knowledge graph can be optimized and adjusted.

[0149] In this embodiment of the invention, the above-mentioned handling result may be the processing result corresponding to the early warning strategy taken after the target event occurs, such as successfully preventing a fight.

[0150] The aforementioned optimizations and adjustments can be based on the handling results of target events to update and improve the graph association weights in the knowledge graph, thereby enhancing the accuracy and timeliness of the knowledge graph processing.

[0151] The aforementioned optimization and adjustment can be a process of adjusting the parameters of a multimodal large model based on the handling results of the target event, in order to optimize the multimodal large model to achieve stability and reach the predetermined results.

[0152] It should be noted that after determining the early warning strategy for the target event, the handling results of the target event can be obtained. Based on the handling results, the multimodal large model and knowledge graph can be optimized and adjusted, enabling the system to continuously evolve over time and adapt to new risk patterns.

[0153] like Figure 2 As shown, an embodiment of the present invention provides an event processing device, which includes:

[0154] Acquisition module 201 is used to acquire multimodal data to be processed;

[0155] The first processing module 202 is used to perform event detection processing on the multimodal data through a preset multimodal large model to obtain a target event and a first evaluation result of the target event;

[0156] The query module 203 is used to query the verification link of the target event in a preset knowledge graph based on the target event and the first evaluation result;

[0157] The first determining module 204 is used to determine a combination of small models for verifying the target event from a preset small model pool based on the verification link.

[0158] The second processing module 205 is used to perform verification processing on the target event based on the small model combination to obtain a second evaluation result of the target event;

[0159] The second determining module 206 is used to determine the early warning strategy for the target event based on the second evaluation result.

[0160] Optionally, the first processing module 202 is further configured to perform spatiotemporal alignment and fusion archiving processing on the multimodal data from different sources to obtain fused data of the multimodal data; perform semantic extraction on the fused data through a preset multimodal large model to obtain semantic structured data corresponding to the multimodal data; and obtain the target event and the first evaluation result of the target event based on the semantic structured data.

[0161] Optionally, the query module 203 is further configured to map the target event and the first evaluation result into graph query conditions according to a preset mapping strategy; and to query the verification link of the target event in a preset knowledge graph through the graph query conditions.

[0162] Optionally, the device is further configured to acquire sample event data, extract knowledge nodes based on the sample event data, the knowledge nodes including event nodes, behavioral feature nodes, scene nodes, small model nodes, and risk nodes; calculate the dynamic association weights between each knowledge node; and connect each knowledge node through the dynamic association weights to obtain a knowledge graph.

[0163] Optionally, the device is further configured to calculate the co-occurrence weight between two knowledge nodes based on their occurrence frequency and co-occurrence frequency; determine the prior knowledge weight between two knowledge nodes based on their expert scores; determine the model hit rate weight between a small model node and other types of knowledge nodes based on the hit rate of the small model node against other types of knowledge nodes; determine the context weight between two knowledge nodes based on their contextual relationship; and perform weighted fusion of the co-occurrence weight, the prior knowledge weight, the model hit rate weight, and the context weight to obtain the dynamic association weight between each knowledge node.

[0164] Optionally, the second processing module 205 is further configured to load each small model in the small model combination according to the combination strategy; process the multimodal data corresponding to the target event through each small model to obtain the small model identification result; and calculate the second evaluation result of the target event based on the first evaluation result, the verification link and the small model identification result.

[0165] Optionally, the device is further configured to obtain the handling result of the target event; and based on the handling result, to optimize and adjust the multimodal large model and the knowledge graph.

[0166] like Figure 3 As shown, embodiments of the present invention also provide an electronic device, including a processor, which can execute any of the above-described event handling methods.

[0167] Specifically, it includes a processor 301 and a memory 302, as well as a computer program stored in the memory 302 and capable of running on the processor 301 to execute event handling methods, wherein:

[0168] The processor 301 executes the calculator program containing the event handling methods stored in memory 302, and performs the following steps:

[0169] Acquire the multimodal data to be processed;

[0170] The multimodal data is processed by a pre-defined multimodal large model to obtain the target event and the first evaluation result of the target event.

[0171] Based on the target event and the first evaluation result, the verification link of the target event is retrieved from the preset knowledge graph;

[0172] Based on the verification link, a combination of small models for verifying the target event is determined from a preset small model pool;

[0173] Based on the combination of small models, the target event is validated to obtain a second evaluation result of the target event;

[0174] Based on the second evaluation results, an early warning strategy for the target event is determined.

[0175] Optionally, the process executed by processor 301 to perform event detection processing on the multimodal data using a preset multimodal large model to obtain a target event and a first evaluation result of the target event includes:

[0176] Spatiotemporal alignment and fusion archiving are performed on the multimodal data from different sources to obtain fused multimodal data.

[0177] Semantic extraction is performed on the fused data using a pre-defined multimodal large model to obtain semantically structured data corresponding to the multimodal data;

[0178] Based on the semantically structured data, the target event and its first evaluation result are obtained.

[0179] Optionally, the step of processor 301 querying the verification link of the target event in a preset knowledge graph based on the target event and the first evaluation result includes:

[0180] According to a preset mapping strategy, the target event and the first evaluation result are mapped as graph query conditions;

[0181] Using the graph query conditions, the verification link of the target event can be retrieved from the preset knowledge graph.

[0182] Optionally, before retrieving the verification link of the target event from the preset knowledge graph, the method executed by the processor 301 further includes:

[0183] Acquire sample event data, and extract knowledge nodes based on the sample event data. The knowledge nodes include event nodes, behavioral feature nodes, scene nodes, small model nodes, and risk nodes.

[0184] Calculate the dynamic association weights between various knowledge nodes;

[0185] By using the dynamic association weights, the various knowledge nodes are connected to obtain a knowledge graph.

[0186] Optionally, the calculation of dynamic association weights between various knowledge nodes performed by processor 301 includes:

[0187] Calculate the co-occurrence weight between two knowledge nodes based on their occurrence frequency and co-occurrence frequency;

[0188] Determine the prior knowledge weight between two knowledge nodes based on the expert scores of the two knowledge nodes.

[0189] Based on the hit rate of small model nodes for other types of knowledge nodes, determine the model hit rate weights of small model nodes and other types of knowledge nodes.

[0190] Determine the context weight between two knowledge nodes based on their contextual relationship.

[0191] The co-occurrence weight, the prior knowledge weight, the model hit rate weight, and the context weight are weighted and fused to obtain the dynamic association weight between each knowledge node.

[0192] Optionally, the processor 301 performs verification processing on the target event based on the small model combination to obtain a second evaluation result of the target event, including:

[0193] Each small model in the small model combination is loaded according to the combination strategy;

[0194] The multimodal data corresponding to the target event is processed through various small models to obtain the recognition results of the small models;

[0195] Based on the first evaluation result, the verification link, and the small model identification result, a second evaluation result for the target event is calculated.

[0196] Optionally, after determining the early warning strategy for the target event based on the second evaluation result, the method executed by the processor 301 further includes:

[0197] Obtain the processing result of the target event;

[0198] Based on the processing results, the multimodal large model and the knowledge graph are optimized and adjusted.

[0199] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the event handling method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0200] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An event handling method, characterized in that, The method includes the following steps: Acquire the multimodal data to be processed; The multimodal data is processed by a pre-defined multimodal large model to obtain the target event and the first evaluation result of the target event. Based on the target event and the first evaluation result, the verification link of the target event is retrieved from the preset knowledge graph; Based on the verification link, a combination of small models for verifying the target event is determined from a preset small model pool; Based on the combination of small models, the target event is validated to obtain a second evaluation result of the target event; Based on the second evaluation results, an early warning strategy for the target event is determined.

2. The event handling method as described in claim 1, characterized in that, The step of performing event detection processing on the multimodal data using a preset multimodal large model to obtain the target event and the first evaluation result of the target event includes: Spatiotemporal alignment and fusion archiving are performed on the multimodal data from different sources to obtain fused multimodal data. Semantic extraction is performed on the fused data using a pre-defined multimodal large model to obtain semantically structured data corresponding to the multimodal data; Based on the semantically structured data, the target event and its first evaluation result are obtained.

3. The event handling method as described in claim 1, characterized in that, The step of querying the verification link of the target event in a preset knowledge graph based on the target event and the first evaluation result includes: According to a preset mapping strategy, the target event and the first evaluation result are mapped as graph query conditions; Using the graph query conditions, the verification link of the target event can be retrieved from the preset knowledge graph.

4. The event handling method as described in claim 1, characterized in that, Before retrieving the verification link of the target event from the preset knowledge graph, the method further includes: Acquire sample event data, and extract knowledge nodes based on the sample event data. The knowledge nodes include event nodes, behavioral feature nodes, scene nodes, small model nodes, and risk nodes. Calculate the dynamic association weights between various knowledge nodes; By using the dynamic association weights, the various knowledge nodes are connected to obtain a knowledge graph.

5. The event handling method as described in claim 4, characterized in that, The calculation of the dynamic association weights between each knowledge node includes: Calculate the co-occurrence weight between two knowledge nodes based on their occurrence frequency and co-occurrence frequency; Determine the prior knowledge weight between two knowledge nodes based on the expert scores of the two knowledge nodes. Based on the hit rate of small model nodes for other types of knowledge nodes, determine the model hit rate weights of small model nodes and other types of knowledge nodes. Determine the context weight between two knowledge nodes based on their contextual relationship. The co-occurrence weight, the prior knowledge weight, the model hit rate weight, and the context weight are weighted and fused to obtain the dynamic association weight between each knowledge node.

6. The event handling method as described in claim 1, characterized in that, The process of verifying the target event based on the small model combination to obtain a second evaluation result of the target event includes: Each small model in the small model combination is loaded according to the combination strategy; The multimodal data corresponding to the target event is processed through various small models to obtain the recognition results of the small models; Based on the first evaluation result, the verification link, and the small model identification result, a second evaluation result for the target event is calculated.

7. The event handling method as described in any one of claims 1 to 6, characterized in that, After determining the early warning strategy for the target event based on the second evaluation result, the method further includes: Obtain the processing result of the target event; Based on the processing results, the multimodal large model and the knowledge graph are optimized and adjusted.

8. An event processing device, characterized in that, The event handling device includes: The acquisition module is used to acquire the multimodal data to be processed. The first processing module is used to perform event detection processing on the multimodal data through a preset multimodal large model to obtain the target event and the first evaluation result of the target event; The query module is used to query the verification link of the target event in a preset knowledge graph based on the target event and the first evaluation result; The first determining module is used to determine, based on the verification link, a combination of small models for verifying the target event from a preset small model pool; The second processing module is used to perform verification processing on the target event based on the small model combination to obtain a second evaluation result of the target event; The second determining module is used to determine the early warning strategy for the target event based on the second evaluation result.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the event handling method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the event handling method as described in any one of claims 1 to 7.

Images