An event distribution method based on multimodal data and knowledge graph collaborative reasoning

Abstract

This invention discloses an event allocation method based on collaborative reasoning using multimodal data and knowledge graphs, relating to the field of urban event distribution technology. Addressing the problems of fragmented single-modal data, strong rule dependence, and insufficient dynamic response in traditional event allocation, the proposed solution includes: preprocessing historical event data and constructing a knowledge graph; training a graph neural network model and a reinforcement learning PPO policy model based on the knowledge graph; fusing and extracting features from the multimodal data of new events; combining the knowledge graph and graph neural network model to reason about new events and generating allocation suggestions in collaboration with a large language model (LLM); dynamically optimizing the allocation suggestions using the reinforcement learning PPO policy model and generating work orders to achieve accurate event allocation; and iteratively optimizing the knowledge graph, graph neural network model, and reinforcement learning PPO policy model based on feedback from event handling results, forming a closed-loop optimization mechanism. This invention can achieve intelligent allocation of urban events.

Description

Technical Field

[0001] This invention relates to the field of urban event distribution technology, specifically an event distribution method based on collaborative reasoning using multimodal data and knowledge graphs. Background Technology

[0002] With the advancement of smart city construction, the efficiency and accuracy of incident allocation in urban governance have become key challenges. Traditional incident allocation methods mainly rely on the following technologies, but they have significant limitations: Existing systems typically use structured data (such as event types and address codes) and predefined rules (such as "environmental issues → environmental protection bureau") for allocation. Some systems introduce basic NLP techniques to process text descriptions, but this approach is monomodal, relying only on text or structured fields and cannot integrate multimodal data such as images and voice, resulting in the loss of key information. Moreover, the rules are rigid, the rule base needs to be maintained manually, and it is difficult to adapt to new events or complex scenarios (such as cross-departmental collaborative handling needs), resulting in a high allocation error rate.

[0003] A few solutions utilize knowledge graphs to store event relationships (such as claimant-event-address), supporting simple association queries. However, these solutions are limited by static graphs, and the updates of entities and relationships rely on manual intervention. They cannot respond to dynamic events in real time, lack collaboration with deep learning models, and are difficult to mine implicit relationships from multimodal data.

[0004] Currently, some studies attempt to use Large Language Models (LLMs) for event classification and directly output allocation suggestions. However, this method lacks structured constraints, and the results generated by Large Language Models (LLMs) may deviate from actual business logic, ignoring departmental responsibility boundaries and exhibiting poor interpretability.

[0005] In summary, existing technologies suffer from fragmented data modalities, rigid rules, and insufficient dynamic reasoning capabilities, resulting in low efficiency in distributing complex events, difficulties in cross-departmental collaboration, and significant delays in responding to multimodal demands. Summary of the Invention

[0006] This invention addresses the needs and shortcomings of current technological development by providing an event distribution method based on collaborative reasoning using multimodal data and knowledge graphs. This method solves the problems of fragmented single-modal data, strong rule dependence, and insufficient dynamic response in traditional event distribution.

[0007] The present invention provides an event allocation method based on multimodal data and knowledge graph collaborative reasoning. The technical solution adopted to solve the above-mentioned technical problems is as follows: An event allocation method based on multimodal data and knowledge graph collaborative reasoning includes the following steps: S1. Preprocess historical event data and construct a knowledge graph to provide a data foundation and association rules for subsequent model training and decision-making; S2. Based on the knowledge graph constructed in step S1, train the graph neural network model and the reinforcement learning PPO policy model to provide a foundation for event allocation decisions. S3. Perform fusion processing on the multimodal data of new events and extract features to provide standardized input for subsequent decision-making; S4. Combining the knowledge graph from step S1 and the graph neural network model from step S2, reason about new events and generate allocation suggestions in collaboration with the large language model LLM. S5. Utilize the reinforcement learning PPO strategy model from step S2 to dynamically optimize the allocation suggestions and generate work orders, thereby achieving accurate event allocation. S6. Based on the feedback of the event handling results, iteratively optimize the knowledge graph, graph neural network model, and reinforcement learning PPO strategy model to form a closed-loop optimization mechanism.

[0008] Optionally, step S1 specifically includes the following process: S1.1 Collect multimodal data covering text, images, voice, and spatiotemporal types from the historical event database; S1.2 Clean the collected data, remove garbled characters and duplicate content, standardize the address representation format, and classify the event types according to preset standards; S1.3 Extract relevant rules from historical processing records, use Nebula Graph as a graph database to construct a knowledge graph with "event, address, department, rule" as the core entities, and define entity attributes, entity relationships and edge attributes.

[0009] Preferably, the knowledge graph uses "event, address, department, and rule" as core entities, where: The attributes of an event node include ID, type, urgency, and processing time. The urgency value ranges from 0 to 1. There is an "event-occurrence-address" relationship between event nodes and address nodes. The edge attributes of this relationship are trigger time and location accuracy. The attributes of address nodes include latitude and longitude, administrative division, number of historical events, and average response time; there is an "address-association-historical event" relationship between address nodes and historical event nodes, and the edge attributes of this relationship are association strength and most recent trigger time; The attributes of the department node include a description of responsibilities, a response success rate, and a list of collaborating departments; there is a "department-responsible-rule" relationship between the department node and the rule node, and the edge attributes of this relationship are the processing success rate and the average processing time. The attributes of a rule node include the triggering condition, effective time, and associated department; there is a "rule-trigger-event" relationship between rule nodes and event nodes, and the edge attributes of this relationship are the number of hits and the last trigger time.

[0010] Preferably, the triggering condition for the rule node is a multimodal feature combination logic expression, including at least one of time window, decibel threshold and number of complaints.

[0011] Optionally, step S2 specifically includes the following process: S2.1 Based on the knowledge graph constructed in step S1, extract features from event nodes, address nodes, and department nodes respectively; S2.2. The graph neural network model is trained using the GraphSAGE algorithm. Each node samples 3 neighbors. The MEAN-POOL method is used to aggregate the features of the neighbors. Through multiple rounds of iterative training, the graph neural network model outputs the initial probability distribution of the distribution department and supports inductive learning of new entities. S2.3 Construct a reinforcement learning PPO strategy model and design a reward function R = α・distribution correctness + β・timeliness of handling + γ・cross-departmental collaboration bonus, where α, β, and γ are dynamically adjustable weight coefficients; S2.4. Based on the knowledge graph constructed in step S1, train the reinforcement learning PPO policy model so that the reinforcement learning PPO policy model can adjust the allocation strategy according to the real-time situation.

[0012] Alternatively, step S3 may specifically include the following process: S3.1 Receive multimodal data of new events, including text, images, voice, and spatiotemporal data. Specifically: For text data, the BERT model is used to extract semantic vectors, retaining semantic feature pairs of key information; for image data, the CLIP-ViT model is used to extract visual features, verifying the time in the image (compared to the complaint time) and the authenticity of the scene (confirming whether it is a construction area); for voice data, it is first converted to mono audio with a 16kHz sampling rate, then transcribed into text using the Whisper model and voiceprint features are extracted, matching the voiceprint feature library to confirm the noise type (such as mechanical noise, human voice noise, etc.); for spatiotemporal data, it is associated with the latitude and longitude coordinates corresponding to the specific address, and the timestamp is unified to UTC format. S3.2 Integrate the multimodal features extracted in step S3.1 into a standardized feature set with unified dimensions, and use it as input data for knowledge graph reasoning and large language model LLM decision-making in step S4 to achieve standardized processing of new event data.

[0013] Alternatively, step S4 may specifically include the following process: S4.1 Based on the knowledge graph of step S1, perform entity linking and neighbor aggregation for new events. By calling the graph neural network model trained in step S2, aggregate 3-hop neighbor features and output key features. S4.2 Input the multimodal features, knowledge graph reasoning results and rule base (rule base formed by pre-extracted rules) extracted in step S3 into the large language model LLM. The large language model LLM combines the hierarchical classification strategy to generate allocation suggestions and the confidence level and specific responsibilities of each department.

[0014] Preferably, the hierarchical classification strategies involved include: Coarse-grained classification: The multimodal features extracted in step S3, the knowledge graph reasoning results, and the rule base (the rule base formed by pre-extracted rules) are input into the large language model LLM. The large language model LLM outputs event labels to filter candidate departments. Fine-grained matching: By querying the correlation of departmental responsibilities through knowledge graph paths, allocation suggestions are generated.

[0015] Alternatively, step S5 may specifically include the following process: S5.1 Collect real-time status data, including historical error allocation records, current load status of each department, and cross-departmental demand intensity; S5.2. For departments that continuously misallocate data, introduce a penalty factor to dynamically adjust the weight ratio of α, β, and γ in the reward function. S5.3 The reinforcement learning PPO policy model trained in step S2 optimizes the allocation suggestions in step S4, outputs the final allocation results, and generates a joint task work order, clarifying the handling requirements and time limits to ensure that the allocation results conform to the real-time situation.

[0016] Alternatively, step S6 may specifically include the following process: S6.1 After the incident is handled, feedback results are obtained through manual marking, including the correctness of allocation, actual handling time, and cross-departmental collaboration effectiveness; S6.2 Feedback results back to the knowledge graph of step S1, update entity association strength and department handling success rate, optimize feature weights of graph neural network model in step S2, and modify reward function coefficients of reinforcement learning PPO strategy model in step S2.

[0017] The event allocation method based on multimodal data and knowledge graph collaborative reasoning of the present invention has the following advantages compared with the prior art: 1. This invention aligns text, image, speech and spatiotemporal features through a cross-modal attention mechanism, combines graph neural networks (GNN) to dynamically enhance the associative reasoning ability of knowledge graphs, and utilizes large language models (LLM) and reinforcement learning to optimize the allocation decision logic, thereby achieving accurate event classification and cross-departmental collaborative handling driven by multimodal data. This solves the problems of fragmented single-modal data, strong rule dependence and insufficient dynamic response in traditional event allocation. 2. This invention can be applied to scenarios such as citizen complaint distribution and event early warning in smart city governance, improving the accuracy and efficiency of handling complex events, reducing manual intervention costs by more than 30%, and supporting dynamic updates of multimodal data and self-optimization of distribution strategies. 3. Based on cross-modal attention mechanism and graph association reasoning, this invention improves the allocation accuracy by 15%-20% compared with traditional rule engines, especially for complex cross-departmental events. It supports joint analysis of text, image, voice and spatiotemporal data, and the event-driven new rule injection mechanism enables the system to automatically adapt to new demand patterns, shortening the time-to-response period from several weeks to real-time effectiveness. Attached Figure Description

[0018] Appendix Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0019] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments. Example

[0020] Combined with appendix Figure 1 This embodiment proposes an event allocation method based on collaborative reasoning using multimodal data and knowledge graphs, which includes the following steps: S1. Preprocess historical event data and construct a knowledge graph to provide a data foundation and association rules for subsequent model training and decision-making.

[0021] Step S1 specifically includes the following process: S1.1 Collect multimodal data covering text, images, voice, and spatiotemporal types from the historical event database; S1.2 Clean the collected data, remove garbled characters and duplicate content, standardize the address representation format, and classify the event types according to preset standards; S1.3 Extract relevant rules from historical processing records, use Nebula Graph as a graph database to construct a knowledge graph with "event, address, department, rule" as the core entities, and define entity attributes, entity relationships and edge attributes.

[0022] The knowledge graph uses "event, address, department, and rule" as its core entities, among which: The attributes of an event node include ID, type, urgency, and processing time. The urgency value ranges from 0 to 1. There is an "event-occurrence-address" relationship between event nodes and address nodes. The edge attributes of this relationship are trigger time and location accuracy. The attributes of address nodes include latitude and longitude, administrative division, number of historical events, and average response time; there is an "address-association-historical event" relationship between address nodes and historical event nodes, and the edge attributes of this relationship are association strength and most recent trigger time; The attributes of the department node include a description of responsibilities, a response success rate, and a list of collaborating departments; there is a "department-responsible-rule" relationship between the department node and the rule node, and the edge attributes of this relationship are the processing success rate and the average processing time. The attributes of a rule node include the trigger condition, effective time, and associated department. Rule nodes and event nodes have a "rule-trigger-event" relationship, with the edge attributes being the number of hits and the last trigger time. The trigger condition of a rule node is a multimodal feature combinational logic expression, including at least one of the following: time window, decibel threshold, and number of complaints.

[0023] The specific design of entities and relationships in the knowledge graph is shown in Table 1.

[0024] Table 1 Entity and Relationship Design event ID, type, urgency level (0-1), processing time Event - Occurred at - Address Trigger time, positioning accuracy address Latitude and longitude, administrative divisions, number of historical events, average response time Address - Related - Historical Events Association strength, most recent trigger time department Job description, response success rate, list of collaborating departments Department - Responsible - Rules Success rate and average processing time rule Triggering conditions, effective time, associated departments Rules - Triggers - Events Number of hits, last trigger time S2. Based on the knowledge graph constructed in step S1, train the graph neural network model GNN and the reinforcement learning policy model PPO to provide a foundation for event allocation decisions.

[0025] Step S2 specifically includes the following process: S2.1 Based on the knowledge graph constructed in step S1, extract features from event nodes, address nodes, and department nodes respectively; S2.2. The graph neural network model is trained using the GraphSAGE algorithm. Each node samples 3 neighbors. The MEAN-POOL method is used to aggregate the features of the neighbors. Through multiple rounds of iterative training, the graph neural network model outputs the initial probability distribution of the distribution department and supports inductive learning of new entities. S2.3 Construct a reinforcement learning PPO strategy model and design a reward function R = α・distribution correctness + β・timeliness of handling + γ・cross-departmental collaboration bonus, where α, β, and γ are dynamically adjustable weight coefficients; S2.4. Based on the knowledge graph constructed in step S1, train the reinforcement learning PPO policy model so that the reinforcement learning PPO policy model can adjust the allocation strategy according to the real-time situation.

[0026] S3. Perform fusion processing on the multimodal data of new events and extract features to provide standardized input for subsequent decision-making.

[0027] Step S3 specifically includes the following process: S3.1 Receive multimodal data of new events, including text, images, voice, and spatiotemporal data. Specifically: For text data, the BERT model is used to extract semantic vectors, retaining semantic feature pairs of key information; for image data, the CLIP-ViT model is used to extract visual features, verifying the time in the image (compared to the complaint time) and the authenticity of the scene (confirming whether it is a construction area); for voice data, it is first converted to mono audio with a 16kHz sampling rate, then transcribed into text using the Whisper model and voiceprint features are extracted, matching the voiceprint feature library to confirm the noise type (such as mechanical noise, human voice noise, etc.); for spatiotemporal data, it is associated with the latitude and longitude coordinates corresponding to the specific address, and the timestamp is unified to UTC format. S3.2 Integrate the multimodal features extracted in step S3.1 into a standardized feature set with unified dimensions, and use it as input data for knowledge graph reasoning and large language model LLM decision-making in step S4 to achieve standardized processing of new event data.

[0028] S4. Combining the knowledge graph from step S1 and the graph neural network model from step S2, reason about new events and generate allocation suggestions in collaboration with the large language model LLM.

[0029] Step S4 specifically includes the following process: S4.1 Based on the knowledge graph of step S1, perform entity linking and neighbor aggregation for new events. By calling the graph neural network model trained in step S2, aggregate 3-hop neighbor features and output key features. S4.2 Input the multimodal features, knowledge graph reasoning results and rule base (rule base formed by pre-extracted rules) extracted in step S3 into the large language model LLM. The large language model LLM combines the hierarchical classification strategy to generate allocation suggestions and the confidence level and specific responsibilities of each department.

[0030] The hierarchical classification strategies involved include: Coarse-grained classification: The multimodal features extracted in step S3, the knowledge graph reasoning results, and the rule base (the rule base formed by pre-extracted rules) are input into the large language model LLM. The large language model LLM outputs event labels to filter candidate departments. Fine-grained matching: By querying the correlation of departmental responsibilities through knowledge graph paths, allocation suggestions are generated.

[0031] S5. Utilize the reinforcement learning PPO strategy model from step S2 to dynamically optimize the allocation suggestions and generate work orders, thereby achieving accurate event allocation.

[0032] Step S5 specifically includes the following process: S5.1 Collect real-time status data, including historical error allocation records, current load status of each department, and cross-departmental demand intensity; S5.2. For departments that continuously misallocate data, introduce a penalty factor to dynamically adjust the weight ratio of α, β, and γ in the reward function. S5.3 The reinforcement learning PPO policy model trained in step S2 optimizes the allocation suggestions in step S4, outputs the final allocation results, and generates a joint task work order, clarifying the handling requirements and time limits to ensure that the allocation results conform to the real-time situation.

[0033] S6. Based on the feedback of the event handling results, iteratively optimize the knowledge graph, graph neural network model, and reinforcement learning PPO strategy model to form a closed-loop optimization mechanism.

[0034] Step S6 specifically includes the following process: S6.1 After the incident is handled, feedback results are obtained through manual marking, including the correctness of allocation, actual handling time, and cross-departmental collaboration effectiveness; S6.2 Feedback results back to the knowledge graph of step S1, update entity association strength and department handling success rate, optimize feature weights of graph neural network model in step S2, and modify reward function coefficients of reinforcement learning PPO strategy model in step S2.

[0035] Regarding the above operational procedures: Suppose a resident of a commercial area submits noise complaints via the app for three consecutive nights, attaching photos, audio recordings, and location data of nighttime construction, requesting urgent handling. The first two complaints were incorrectly assigned to the "Municipal Engineering Bureau," when the actual responsible department should be the "Housing and Construction Bureau + Environmental Protection Bureau for joint handling."

[0036] The first step involves data including: 1) Text: a complaint description of "continuous concrete pouring noise at 1 a.m., two complaints have been filed but the problem remains unresolved"; 2) Images: a photo of the construction site tower crane lighting at night (with an EXIF ​​timestamp of 00:45); 3) Audio: a 30-second environmental recording (average decibel 72dB, including mechanical impact sounds); 4) Spatiotemporal data: coordinates (X1, Y1), time window [2024-09-10 23:30 - 2024-09-11 01:15].

[0037] Based on the above data, a module collaborative processing flow is initiated, performing feature extraction and alignment on the data. A pseudocode example is shown below: # Pseudocode Example text_feat = ChatGLM.encode("Concrete pouring sound") # Output vector [0.87, -0.12,...] image_feat = CLIP.encode(image) # Visual feature matching for "tower crane / nighttime construction" audio_feat = Whisper(audio) + Librosa(DB=72) # Cross-modal attention computation cross_attn = Transformer( queries=text_feat, keys=torch.cat([image_feat, audio_feat]), values=... The text "concrete" and the image "tower crane" have a correlation weight of 0.89. The key data in the data is correlated. In this embodiment, the image timestamp (00:45) overlaps with the complaint time window by 92%, and the voice spectrum analysis matches the "concrete vibrator" feature library (similarity 81%), which corroborates the authenticity of the construction behavior.

[0038] The second step is to perform knowledge graph reasoning on the acquired key data, and obtain the handling methods through entity association and knowledge graph query.

[0039] First, an address association query is performed. Historical cases and responsible departments are retrieved using a knowledge graph. The pseudocode for this query is as follows: / / Neo4j query statement MATCH (a: address {coordinates: "X1, Y1"}) - [r: HAS_HISTORY] -> (c: event) WHERE c. Type IN ["Noise Complaint", "Construction Permit"] RETURN c. handling department, r. relevance strength / / Returned result: Municipal Engineering Bureau (2 times), correlation strength 0.6 Inference is enhanced by a graph neural network model, and neighbor aggregation is performed. Centered on the current event node, three-hop neighbor nodes are aggregated (address → historical event → department → enterprise qualification). Feature generation is performed, and an enhanced vector [construction qualification: 0.78, cross-departmental requirements: 0.92] is output based on the graph neural network model to represent the urgency of the need for coordinated handling by the Housing and Construction Bureau and the Environmental Protection Bureau.

[0040] The third step involves calling the Large Language Model (LLM) interface, inputting the data results generated in the first two steps, and leveraging the LLM capabilities for collaborative decision-making to obtain allocation suggestions: Housing and Urban-Rural Development Bureau (68% confidence level), Environmental Protection Bureau (61% confidence level), and Municipal Engineering Bureau (32% confidence level). The LLM input call information is as follows: / / Large language model input { "multimodal_feat": "Municipal Engineering Bureau (2 times), correlation strength 0.6", "kg_feat": "[Construction qualification: 0.78, Cross-departmental handling requirements: 0.92]", "context_rules": "Nighttime construction requires noise testing by the Environmental Protection Bureau and qualification verification by the Housing and Urban-Rural Development Bureau" } The decision output results are as follows: Housing and Urban-Rural Development Bureau (confidence level 68%), verification of construction qualification violations; Environmental Protection Bureau (61%), penalty for exceeding noise decibel standards; Municipal Engineering Bureau (32%), downgraded due to historical failures in handling such cases.

[0041] The fourth step is to make dynamic allocation decisions based on the events.

[0042] Decision-making is conducted using a reinforcement learning-based PPO policy model. The current state characteristics are: the number of historical erroneous allocations (assumed to be 2), and the current cross-departmental demand intensity of 0.92; departmental workload: the Environmental Protection Bureau currently has 15 pending cases, and the Housing and Construction Bureau has 10. A decision path is generated based on these characteristics: 1. Initial rule weights: matching degree of departmental responsibilities (70%), historical success rate (30%); 2. New penalty factor: Penalty coefficient for consecutive complaints = -0.3 2. The reward function formula for the reinforcement learning PPO strategy model is: R = α⋅ Correctness of allocation + β⋅ Timeliness of handling + γ⋅ Cross-departmental collaboration bonus, where α, β, and γ are dynamically adjustable weight coefficients. The reward function in this case is R = 1.0 × Correctness of allocation + 0.5 × Timeliness of handling - 0.3 × Repeat complaints; 3. Final allocation: Skip the Municipal Engineering Bureau and allocate to the Housing and Urban-Rural Development Bureau (primary responsible) and the Environmental Protection Bureau (co-organizer) to generate a joint task order.

[0043] The fifth step is to provide feedback on the processing results of the distributed data after the incident has been assigned and processed. The feedback results are provided through manual marking. In this case, the incident was correctly assigned. The Housing and Construction Bureau verified that the qualifications were in violation of regulations. The Environmental Protection Bureau issued a fine. The time from complaint to resolution was 26 hours, which exceeded the 24-hour limit.

[0044] Based on this feedback, the knowledge graph, graph neural network model, and reinforcement learning PPO policy model are updated. The extended knowledge graph pseudocode is as follows: / / Adding nodes and relationships CREATE (New rule: Disposal rule { Name: "Joint Handling of Nighttime Construction" Conditions: "Decibels > 65 and number of complaints ≥ 2" Departments: [Housing and Urban-Rural Development Bureau, Environmental Protection Bureau] }) MERGE (address node) - [r: trigger rule] -> (new rule) The pseudocode for adjusting the parameters of a graph neural network model is as follows: # Input: Feedback result {Correct department, Incorrect department} feedback = {"correct":["Housing and Urban-Rural Development Bureau","Environmental Protection Bureau"], "wrong":["Municipal Engineering Bureau"]} # 1. Retrieve the subgraph involved subgraph = kg.query("MATCH (e:event)-[*3]-(n) WHERE e.id='EVT001'RETURN n") # 2. Fine-tuning GNN for dept in ["Housing and Urban-Rural Development Bureau","Environmental Protection Bureau","Municipal Engineering Bureau"]: label = 1 if dept in feedback["correct"] else 0 gnn.backprop(node=dept, label=label, lr=1e-4) # 3. Update neighbor weights kg.update_edge("EVT001", "Municipal Engineering Bureau", weight=0.1) # Demoted department due to incorrect weight # 4. Save the model gnn.save("gnn_v2.pth") The pseudocode for tuning parameters in a reinforcement learning PPO policy model is as follows: # Update reward function Reward = Base score + Time-limited penalty + (if cross-departmental collaboration then 0.5 else -0.2) # Timeliness weight increased from 0.3 to 0.45 Ultimately, by optimizing the results through initial allocation error → feedback marking → rule expansion → secondary event verification, a closed loop is formed, thus improving the intelligent allocation capability.

[0045] In summary, the event allocation method based on multimodal data and knowledge graph collaborative reasoning of this invention can improve the cross-modal event allocation accuracy by 15%-20% and shorten the allocation response time compared with traditional methods. Compared with traditional rule engines, the cost of manual intervention can be reduced by more than 20%, and the rate of repeated complaints can be reduced by more than 30%.

[0046] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An event allocation method based on collaborative reasoning using multimodal data and knowledge graphs, characterized in that, Includes the following steps: S1. Preprocess historical event data and construct a knowledge graph to provide a data foundation and association rules for subsequent model training and decision-making; specifically, this includes the following process: S1.1 Collect multimodal data covering text, images, voice, and spatiotemporal types from the historical event database; S1.2 Clean the collected data, remove garbled characters and duplicate content, standardize the address representation format, and classify the event types according to preset standards; S1.3 Extract relevant rules from historical handling records. Use Nebula Graph as the graph database to construct a knowledge graph with "event, address, department, and rule" as core entities, and define entity attributes, entity relationships, and edge attributes. Specifically: Event node attributes include ID, type, urgency, and handling time, with urgency ranging from 0 to 1; Event nodes and address nodes have a "event-occurred at-address" relationship, with edge attributes of trigger time and location accuracy; Address node attributes include latitude and longitude, administrative division, number of historical events, and average response time; Address nodes and historical event nodes have a "address-associated-historical event" relationship, with edge attributes of association strength and most recent trigger time; Department node attributes include responsibility description, response success rate, and list of collaborating departments; Department nodes and rule nodes have a "department-responsible-rule" relationship, with edge attributes of handling success rate and average time; Rule node attributes include trigger condition, effective time, and associated department; Rule nodes and event nodes have a "rule-trigger-event" relationship, with edge attributes of hit count and last trigger time. S2. Based on the knowledge graph constructed in step S1, train the graph neural network model and the reinforcement learning PPO policy model to provide a foundation for event allocation decisions; specifically, this includes the following process: S2.1 Based on the knowledge graph constructed in step S1, extract features from event nodes, address nodes, and department nodes respectively; S2.

2. The graph neural network model is trained using the GraphSAGE algorithm. Each node samples 3 neighbors. The MEAN-POOL method is used to aggregate the features of the neighbors. Through multiple rounds of iterative training, the graph neural network model outputs the initial probability distribution of the distribution department and supports inductive learning of new entities. S2.3 Construct a reinforcement learning PPO strategy model and design a reward function R = α・distribution correctness + β・timeliness of handling + γ・cross-departmental collaboration bonus, where α, β, and γ are dynamically adjustable weight coefficients; S2.

4. Based on the knowledge graph constructed in step S1, train the reinforcement learning PPO policy model so that the reinforcement learning PPO policy model can adjust the allocation strategy according to the real-time situation. S3. Perform fusion processing on the multimodal data of new events and extract features to provide standardized input for subsequent decision-making; S4. Combining the knowledge graph from step S1 and the graph neural network model from step S2, reason about new events and generate allocation suggestions in collaboration with the large language model LLM. S5. Utilize the reinforcement learning PPO strategy model from step S2 to dynamically optimize the allocation suggestions and generate work orders, thereby achieving accurate event allocation. S6. Based on the feedback of the event handling results, iteratively optimize the knowledge graph, graph neural network model, and reinforcement learning PPO strategy model to form a closed-loop optimization mechanism.

2. The event allocation method based on multimodal data and knowledge graph collaborative reasoning according to claim 1, characterized in that, The triggering condition for the rule node is a multimodal feature combination logic expression, including at least one of time window, decibel threshold and number of complaints.

3. The event allocation method based on multimodal data and knowledge graph collaborative reasoning according to claim 2, characterized in that, Step S3 specifically includes the following process: S3.1 Receive multimodal data of new events, including text, image, speech, and spatiotemporal data. Specifically: for text data, use the BERT model to extract semantic vectors, retaining semantic feature pairs of key information; for image data, use the CLIP-ViT model to extract visual features and verify the time and scene authenticity in the image; for speech data, first convert it to mono audio with a 16kHz sampling rate, then use the Whisper model to transcribe it into text and extract voiceprint features, matching them to a voiceprint feature library to confirm the noise type; for spatiotemporal data, associate it with the latitude and longitude coordinates corresponding to a specific address, and unify the timestamp to UTC format. S3.2 Integrate the multimodal features extracted in step S3.1 into a standardized feature set with unified dimensions, and use it as input data for knowledge graph reasoning and large language model LLM decision-making in step S4 to achieve standardized processing of new event data.

4. The event allocation method based on multimodal data and knowledge graph collaborative reasoning according to claim 3, characterized in that, Step S4 specifically includes the following process: S4.1 Based on the knowledge graph of step S1, perform entity linking and neighbor aggregation for new events. By calling the graph neural network model trained in step S2, aggregate 3-hop neighbor features and output key features. S4.2 Input the multimodal features extracted in step S3, the knowledge graph reasoning results, and the rule base formed by the pre-extracted rules into the large language model LLM. The large language model LLM combines the hierarchical classification strategy to generate allocation suggestions and the confidence level and specific responsibilities of each department.

5. The event allocation method based on multimodal data and knowledge graph collaborative reasoning according to claim 4, characterized in that, The hierarchical classification strategy includes: Coarse-grained classification: The multimodal features extracted in step S3, the reasoning results of the knowledge graph, and the rule base formed by the pre-extracted rules are input into the large language model LLM. The large language model LLM outputs event labels to filter candidate departments. Fine-grained matching: By querying the correlation of departmental responsibilities through knowledge graph paths, allocation suggestions are generated.

6. The event allocation method based on multimodal data and knowledge graph collaborative reasoning according to claim 4, characterized in that, Step S5 specifically includes the following process: S5.1 Collect real-time status data, including historical error allocation records, current load status of each department, and cross-departmental demand intensity; S5.

2. For departments that continuously misallocate data, introduce a penalty factor to dynamically adjust the weight ratio of α, β, and γ in the reward function. S5.3 The reinforcement learning PPO policy model trained in step S2 optimizes the allocation suggestions in step S4, outputs the final allocation results, and generates a joint task work order, clarifying the handling requirements and time limits to ensure that the allocation results conform to the real-time situation.

7. The event allocation method based on multimodal data and knowledge graph collaborative reasoning according to claim 6, characterized in that, Step S6 specifically includes the following process: S6.1 After the incident is handled, feedback results are obtained through manual marking, including the correctness of allocation, actual handling time, and cross-departmental collaboration effectiveness; S6.2 Feedback results back to the knowledge graph of step S1, update entity association strength and department handling success rate, optimize feature weights of graph neural network model in step S2, and modify reward function coefficients of reinforcement learning PPO strategy model in step S2.

Images