Internet of Things sensing command and dispatch intelligent method and system based on multi-modal large model

Abstract

The invention discloses an Internet of Things sensing command and dispatch intelligent method and system based on a multi-modal large model, which deeply fuses multi-terminal field information, combines with a dynamically updated expert knowledge base, realizes high-precision alarm level judgment and processing suggestion generation, and realizes high-precision alarm level judgment and processing suggestion generation through a multi-agent collaboration method. According to the method, a complex command and dispatch task is decomposed into a series of executable subtasks, the subtasks are executed in a division-of-labor cooperation mode according to a professional agent, the execution conditions of the subtasks are output, and in the execution process of the subtasks, through a quality detection system including logic verification, fact verification, compliance examination and effect evaluation, the execution efficiency of the subtasks is improved. The method comprises the steps that task execution is completed, reliability, authenticity and legality of task execution are ensured, after task execution is completed, sub-task execution results are integrated, an execution log and a comprehensive report are generated, a task execution plan is dynamically adjusted by means of real-time environment perception and a user feedback mechanism, and an optimization scheme is stored in a historical case knowledge base; and continuous learning ability and experience accumulation are realized.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things perception technology, and in particular to an intelligent method and system for Internet of Things perception command and dispatch based on a multimodal large model. Background Art

[0002] With the widespread application of IoT technology, various sensors and monitoring equipment are being deployed throughout cities, covering key areas such as urban infrastructure management, public safety, and municipal resource allocation. Urban infrastructure management involves road traffic, bridges and tunnels, underground utility corridors, and pipeline systems such as water, electricity, and gas supply. Public safety is widespread across transportation hubs, office spaces, and densely populated areas. Municipal resource allocation is linked to energy production and consumption. Consequently, various IoT systems exist. For example, urban security monitoring primarily focuses on personal and vehicle identification and access control; energy consumption monitoring focuses solely on data collection and basic analysis; and underground utility corridor management still relies on manual inspections combined with basic sensors for alarm generation.

[0003] Existing technologies mostly rely on single sensors or static rules to judge alarm situations, and lack the ability to integrate multi-source data and dynamically adjust. Traditional command and dispatch systems usually use manual allocation or simple automation, making it difficult to achieve multi-agent collaboration and real-time optimization. In addition, the feedback mechanism and quality inspection for task execution also lack the closed-loop optimization capabilities of classification priority management and incremental knowledge base updates, as well as comprehensive coverage of logic, compliance and effect evaluation.

[0004] Therefore, in daily urban management and emergency response, the IoT platform needs to be able to make intelligent decisions. For example, when encountering emergencies such as pipeline fires or municipal pipe network failures, it is necessary to quickly analyze multi-source data and accurately determine the severity and scope of the incident in the first place, thereby supporting rapid decision-making. Summary of the Invention

[0005] The present invention aims to overcome at least one of the defects of the above-mentioned prior art and provide an intelligent method and system for IoT perception, command and dispatch based on a multimodal large model, which is used to solve the technical problems in the prior art that alarm judgment relies on a single sensor or static rules, and the command and dispatch system fails to achieve multi-agent collaboration and real-time optimization.

[0006] The present invention provides an intelligent method for IoT perception, command and dispatch based on a multimodal large model, comprising:

[0007] S01, command and dispatch intelligent agents to monitor multi-terminal on-site information of multi-scenario equipment;

[0008] S02. Performing data preprocessing and multi-source data fusion on the multi-terminal on-site information;

[0009] S03. When the multi-terminal on-site information is detected to be abnormal, the command and dispatch agent determines the current alarm level and outputs the abnormal situation and preliminary handling suggestions;

[0010] S04. The user inputs an exception handling task based on the exception situation and preliminary handling suggestions;

[0011] S05. Decompose the exception handling task into subtasks through the command and dispatch agent, and form a subtask list;

[0012] S06, commanding and dispatching the intelligent agent to execute the subtasks according to the subtask list and outputting the subtask execution status;

[0013] S07. Perform quality inspection on the execution effect of the subtask and generate a quality inspection report;

[0014] S08. Integrate the subtask execution results to generate an execution log and a comprehensive report;

[0015] S09. Optimize the command and dispatch agent based on user feedback and update the knowledge base.

[0016] The alarm level refers to an abnormal level.

[0017] Through direct device connection and platform cloud-to-cloud docking, the IoT perception platform collects data from multiple scenarios. Taking urban integrated pipeline corridor equipment as an example, various professional information collection terminals are distributed in the underground pipeline corridor, such as water immersion sensors, smoke sensors, gas sensors, displacement sensors, high-definition cameras, and smart valves. These text, image, video, and voice information from different terminals together constitute the multi-terminal on-site information of the underground integrated pipeline corridor police scene.

[0018] Before processing the multi-terminal on-site information, the platform must first be connected to the relevant existing platforms of each department to achieve comprehensive data sharing and exchange. Underground pipeline corridor scenarios generally involve the urban infrastructure management platform of the housing and construction department, the geological information platform of the natural resources department, the emergency command platform of the emergency department, and the police reception and handling systems of various units. Through the platform access and unified object model capabilities of the IoT perception platform, the data related to underground pipeline corridors in various department platforms are integrated and aggregated. For example, the housing and construction department platform provides the design drawings and construction materials of the pipeline corridor; the natural resources department platform provides water level monitoring and other information for the pipeline corridor; the emergency department platform provides data on emergency resource reserves and emergency plans. At the same time, it connects to the police reception and handling systems of various units to obtain past police records, handling personnel, and handling situations. These multiple data are deeply integrated and processed to build the final underground pipeline corridor comprehensive management thematic database, providing comprehensive and accurate data support for subsequent command and dispatch.

[0019] The multi-terminal field data information in step S02 is real-time collection of two or more sensor data and video stream data from water immersion sensors, smoke sensors, gas sensors, displacement sensors, high-definition cameras and smart valves.

[0020] The data preprocessing in step S02 is to perform noise removal and standardization on the multi-terminal field information:

[0021]

[0022] Where x is the original sensor value, μ is the mean, σ is the standard deviation, and x norm It is the standard value after data preprocessing;

[0023] The multi-source data fusion fuses the sensor data and the video stream data through Kalman filtering:

[0024]

[0025] in, is the state estimation, K t is the Kalman gain, z t is the observation value, H is the observation matrix, and t represents the current time step.

[0026] In order to improve emergency response speed, optimize resource scheduling and reduce accident risks, it is necessary to build an intelligent knowledge system that includes an expert knowledge base, a historical police case knowledge base, an emergency plan library, and command and dispatch process documents.

[0027] The expert knowledge base brings together the professional knowledge and practical experience of senior experts in the industry, covering multiple fields such as pipeline corridor structure maintenance, equipment fault diagnosis, disaster prevention and treatment, etc.

[0028] The historical police case knowledge base records in detail the background, handling process and final results of various past police incidents, providing a reference basis for the current police incident handling;

[0029] The emergency plan library has developed detailed response plans and operating procedures based on different types and levels of emergency situations;

[0030] The command and dispatch process document clarifies the responsibilities, coordination methods and workflows of each department in handling police incidents.

[0031] The command and dispatch intelligent agent plays a key role in the comprehensive management of cities. It can realize alarm perception, command and dispatch, task decomposition, task execution and dynamic tracking optimization. When the command and dispatch intelligent agent detects relevant alarms from various systems, such as when the water immersion sensor monitors that the water level is rising at a rate of 1 cm every 10 minutes or the smoke sensor detects that the smoke concentration continues to rise and exceeds the safety threshold, the model understands the sensor values and video image data, and combines the expert knowledge base to first judge the current alarm level, and then generate relevant suggestions. For example, based on the water level rising rate and the degree of smoke concentration exceeding the standard, the alarm is comprehensively judged to be at a medium level.

[0032] The step S03 of determining the current emergency level through the command and dispatch agent specifically includes:

[0033] S31. Use a neural network model combined with an expert knowledge base to classify police incidents;

[0034] S32, let the alarm level be based on the feature vector F = [f1, f2, ..., f n ,], the current alarm level is determined by the maximum a posteriori probability MAP decision rule:

[0035] L = argmax k P(L k |F,θ)

[0036] Among them, L is the prediction result of the alarm level, argmax k It is used to find the input that makes the function reach its maximum value, θ is the model parameter, P(L k |F,θ) is the conditional probability;

[0037] S33. Generate the preliminary processing suggestion based on knowledge graph query and reasoning:

[0038] suggestion = KG_query(x1,x2,scene)

[0039] Among them, KG_query() is the knowledge graph query function, and x1 and x2 are query parameters.

[0040] The abnormal situation output in step S03 includes the alarm level, alarm type, occurrence location, real-time status and impact range.

[0041] Subsequently, the command center personnel can input their requirements into the command and dispatch intelligent agent. The intelligent agent uses the large model capability to understand and analyze the exception handling tasks input by the command personnel, and combines the user preferences and historical data in the memory module to accurately understand the task objectives. For example, the command center personnel require the intelligent agent to immediately notify the direct system and the nearest maintenance personnel to handle the problem immediately, and notify the systems and alarm receivers affected by the problematic equipment, and then find similar equipment with potential hidden dangers. Therefore, the input exception handling tasks described in step S04 specifically include:

[0042] S41. Exception handling task using the pre-trained language model BEST to parse input:

[0043] T = NLP_parse(input, θ NLP )

[0044] Among them, T is the task goal, θ NLP are model parameters;

[0045] S42. Combined with the memory module, calculate the similarity between the input and the historical task:

[0046] Sim(T,T hist )=cos(Emb(T),Emb(T hist ))

[0047] Sim(T,T hist ) represents the text T and T hist The similarity score between them, Emb(T) is the embedding vector of text T, Emb(T hist ) is the historical text T hist Embedding vector of , cos() is cosine similarity;

[0048] S43. Extract mission-critical targets using slot filling techniques:

[0049] S = {action, object, priority}

[0050] S represents the slot filling result;

[0051] S44. Output a structured task target list.

[0052] The command and dispatch agent has the ability to decompose complex tasks and can convert abstract task descriptions into specific action steps to ensure efficient execution of tasks. In step S05, the exception handling task is decomposed into subtasks, specifically including:

[0053] S51. Use the hierarchical task network HTN planning to decompose the exception handling task Y into subtasks:

[0054] Y={Y1,Y2,…,Y n}

[0055] Wherein, Y represents the exception handling task, Y i ={action, condition, resource}, i=1,2,…,n, Y i Represents a subtask;

[0056] S52. Assign subtasks to professional agents based on subtask type and resource constraints:

[0057] Agent i =argmin j C(Y i ,A j )

[0058] Among them, Agent i Represented as task Y i The selected agent, C(Y i ,A j ) is the cost function, which measures the task Y i By Agent A j The cost of execution, A j represents the jth agent, j represents the agent index;

[0059] S53. Use dynamic programming to optimize the execution order of subtasks:

[0060]

[0061] Among them, t i For subtasks, c i is the resource cost;

[0062] S54. Output the subtask list and execution plan.

[0063] After breaking down the exception handling task into several subtasks, the agent autonomously plans and executes each subtask. In step S06, the subtasks are executed, and the output of the subtask execution status specifically includes:

[0064] S61. Use the distributed multi-agent framework MADDPG to enable each agent to focus on a subtask:

[0065] π m =Policy(s m ,θ m )

[0066] Among them, π m is the agent's strategy in state s mThe output under, Policy is the policy function, s m is the local state, θ m is the parameter of the policy function, m is the moment parameter;

[0067] S62, real-time perception of environmental changes and update of status:

[0068] s m =f(s m-1 ,z m )

[0069] Among them, z m is the new observation data;

[0070] S63. Use reinforcement learning to adjust the execution plan:

[0071] Q(s,a)←Q(s,a)+α[r+γmaxQ(s′,a′)-Q(s,a)]

[0072] Where Q(s,a) represents the action value function of taking action a in state s, ← is the assignment operation, α represents the learning rate, which is used to control the step size of each update, r represents the reward value of the environment feedback, γ represents the discount factor, and s′, a′ represent the next state;

[0073] S64. Use the timeline manager to track subtask status:

[0074] Progress={(Y i ,state,t i )}

[0075] Progress is a collection, Y i represents the i-th subtask, t i Indicates Y i timestamp;

[0076] S65: Output the real-time subtask execution status.

[0077] The command and dispatch agent adopts a multi-agent system, with multiple professional agents working together to coordinate task allocation and progress management. Each agent focuses on a specific task and displays the execution status of each subtask in real time during the execution of the subtask. Finally, after all subtasks are completed, the execution status log of each subtask is output.

[0078] and reports, and continuously perceive changes in the external environment, such as the impact of changes in traffic conditions on the arrival time of maintenance personnel, and adjust the task execution plan in a timely manner.

[0079] After the agent completes each subtask, it needs to test the execution quality. The quality test includes the following sub-agents: logic verification agent, fact-checking agent, and compliance review agent. The logic verification agent will test the causal rationality of the task chain. After the subtask is executed, the fact-checking agent cross-compares multiple sources to verify the authenticity of the data. Finally, the compliance review agent ensures that the output content complies with laws and regulations. In addition to verifying the authenticity of the data and the causal rationality of the task chain, it can also increase the evaluation of the task execution effect based on the quality inspection standards and indicator system of the expert knowledge base, such as the operating stability of the equipment after maintenance, the actual effectiveness of the alarm handling, etc.

[0080] Integrating the subtask execution results specifically includes:

[0081] Merge the subtasks and output the merged result R:

[0082]

[0083] Among them, R i is the subtask result;

[0084] By timestamp t i Record the execution process and generate a log Log: Log = (t i ,Y i ,status,output);

[0085] Use dashboards or charts to display results, and users can provide feedback on the task breakdown, execution, and quality.

[0086] Based on user feedback, a feedback classification and priority management mechanism is established to respond to and handle urgent and important feedback in a timely manner. Reasonable intelligent agent execution plans are included in the historical case knowledge base of underground integrated pipeline corridors to achieve knowledge accumulation and inheritance, and continuously optimize the execution strategy and effect of the intelligent agent.

[0087] The present invention also provides an IoT perception command and dispatch intelligent system based on a multimodal large model, comprising:

[0088] Monitoring module: used for multi-terminal on-site information monitoring of multi-scenario equipment;

[0089] Processing module: performs data preprocessing and multi-source data fusion on the multi-terminal on-site information;

[0090] Abnormal judgment module: When abnormalities are detected in multiple segments of on-site information, the command and dispatch intelligent agent determines the current alarm level and outputs the abnormal situation and preliminary handling suggestions;

[0091] Input module: The user inputs the exception handling task based on the exception situation and preliminary handling suggestions;

[0092] Disassembly module: disassembles the exception handling task into subtasks through the command and dispatch agent, and forms a subtask list;

[0093] Execution module: directs the dispatching agent to execute the subtasks according to the subtask list and outputs the subtask execution status;

[0094] Quality inspection module: performs quality inspection on the execution effect of subtasks and generates quality inspection reports;

[0095] Integration module: integrates the subtask execution results and generates execution logs and comprehensive reports;

[0096] Optimization module: optimizes the command and dispatch agent based on user feedback and updates the knowledge base.

[0097] The beneficial effects of the present invention include:

[0098] (1) Alert perception mechanism based on multi-source heterogeneous data fusion:

[0099] 1. This invention achieves high-precision alarm level determination and generation of treatment recommendations by deeply fusing sensor values such as water level and smoke concentration with video image data and combining them with a dynamically updated expert knowledge base;

[0100] 2. Through the adaptive data fusion algorithm, multi-dimensional features such as the speed of water level rise and the trend of smoke concentration changes are comprehensively considered to dynamically adjust the alarm assessment model, overcoming the limitations of traditional single sensor reliance or static threshold judgment.

[0101] (2) Task decomposition and execution framework for multi-agent collaboration:

[0102] 1. This paper designs a multi-agent collaboration method that decomposes complex command and dispatch tasks into executable subtasks, which are then completed through the division of labor and collaboration of specialized agents.

[0103] 2. The command and dispatch agent can dynamically plan and optimize the execution path based on task objectives, resource constraints such as maintenance personnel skills, equipment location and real-time environmental changes to ensure efficient collaboration.

[0104] (3) Dynamic optimization and feedback-driven closed-loop system:

[0105] 1. This invention uses real-time environmental perception and user feedback mechanisms to dynamically adjust task execution plans and store optimized solutions in a historical case knowledge base after task completion, thus forming a continuous learning capability.

[0106] 2. The use of feedback classification and priority management mechanisms, combined with incremental updates to the knowledge base, significantly improves the system's adaptability to complex scenarios and long-term effectiveness.

[0107] (IV) Multi-dimensional quality inspection system:

[0108] 1. This invention proposes a quality inspection system that includes logic verification, fact checking, compliance review, and effect evaluation to ensure the reliability, authenticity, and legality of task execution;

[0109] 2. The innovation lies in the execution effect evaluation indicators based on the expert knowledge base, such as equipment stability and alarm relief rate, which provide quantitative feedback for task execution and surpass the traditional verification method that only focuses on data authenticity. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0111] Figure 1 It is a schematic diagram of the steps of an intelligent method for IoT perception, command and dispatch based on a multimodal large model.

[0112] Figure 2 It is a schematic diagram of the specific execution steps of the IoT perception command and dispatch intelligent method.

[0113] Figure 3 It is a schematic diagram of the steps for the command and dispatch intelligent agent to judge the level of the alarm situation.

[0114] Figure 4 It is a schematic diagram of the steps for the command and dispatch agent to understand the exception handling task.

[0115] Figure 5 It is a schematic diagram of the steps for the command and dispatch agent to decompose a task into subtasks.

[0116] Figure 6 It is a schematic diagram of the steps for the agent to plan and execute subtasks.

[0117] Figure 7 This is a schematic diagram of an IoT perception, command and dispatch intelligent system based on a multimodal large model. DETAILED DESCRIPTION

[0118] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings:

[0119] An embodiment of the present invention provides an embodiment of an intelligent method and system for IoT perception, command and dispatch based on a multimodal large model. It should be noted that although a logical order is shown in the flow chart, under certain data, the steps shown or described can be completed in an order different from that here.

[0120] Example 1: Reference Figure 1 and Figure 2

[0121] The present invention provides an intelligent method for IoT perception, command and dispatch based on a multimodal large model, comprising:

[0122] S01, command and dispatch intelligent agents to monitor multi-terminal on-site information of multi-scenario equipment;

[0123] S02. Performing data preprocessing and multi-source data fusion on the multi-terminal on-site information;

[0124] S03. When the multi-terminal on-site information is detected to be abnormal, the command and dispatch agent determines the current alarm level and outputs the abnormal situation and preliminary handling suggestions;

[0125] S04. The user inputs an exception handling task based on the exception situation and preliminary handling suggestions;

[0126] S05. Decompose the exception handling task into subtasks through the command and dispatch agent, and form a subtask list;

[0127] S06, commanding and dispatching the intelligent agent to execute the subtasks according to the subtask list and outputting the subtask execution status;

[0128] S07. Perform quality inspection on the execution effect of the subtask and generate a quality inspection report;

[0129] S08. Integrate the subtask execution results to generate an execution log and a comprehensive report;

[0130] S09. Optimize the command and dispatch agent based on user feedback and update the knowledge base.

[0131] Through direct device connection and platform cloud-to-cloud docking, the IoT perception platform collects data from multiple scenarios. Taking urban integrated pipeline corridor equipment as an example, various professional information collection terminals are distributed in the underground pipeline corridor, such as water immersion sensors, smoke sensors, gas sensors, displacement sensors, high-definition cameras, and smart valves. These text, image, video, and voice information from different terminals together constitute the multi-terminal on-site information of the underground integrated pipeline corridor police scene.

[0132] Before processing the multi-terminal on-site information, the platform must first be connected to the relevant existing platforms of each department to achieve comprehensive data sharing and exchange. Underground pipeline corridor scenarios generally involve the urban infrastructure management platform of the housing and construction department, the geological information platform of the natural resources department, the emergency command platform of the emergency department, and the police reception and handling systems of various units. Through the platform access and unified object model capabilities of the IoT perception platform, the data related to underground pipeline corridors in various department platforms are integrated and aggregated. For example, the housing and construction department platform provides the design drawings and construction materials of the pipeline corridor; the natural resources department platform provides water level monitoring and other information for the pipeline corridor; the emergency department platform provides data on emergency resource reserves and emergency plans. At the same time, it connects to the police reception and handling systems of various units to obtain past police records, handling personnel, and handling situations. These multiple data are deeply integrated and processed to build the final underground pipeline corridor comprehensive management thematic database, providing comprehensive and accurate data support for subsequent command and dispatch.

[0133] The multi-terminal field data information in step S02 is real-time collection of two or more sensor data and video stream data from water immersion sensors, smoke sensors, gas sensors, displacement sensors, high-definition cameras and smart valves.

[0134] The data preprocessing in step S02 is to perform noise removal and standardization on the multi-terminal field information:

[0135]

[0136] Where x is the original sensor value, μ is the mean, σ is the standard deviation, and x norm It is the standard value after data preprocessing;

[0137] The multi-source data fusion fuses the sensor data and the video stream data through Kalman filtering:

[0138]

[0139] in, is the state estimation, K t is the Kalman gain, z t is the observation value, H is the observation matrix, and t represents the current time step.

[0140] In order to improve emergency response speed, optimize resource scheduling and reduce accident risks, it is necessary to build an intelligent knowledge system that includes an expert knowledge base, a historical police case knowledge base, an emergency plan library, and command and dispatch process documents.

[0141] The expert knowledge base brings together the professional knowledge and practical experience of senior experts in the industry, covering multiple fields such as pipeline corridor structure maintenance, equipment fault diagnosis, disaster prevention and treatment, etc.

[0142] The historical police case knowledge base records in detail the background, handling process and final results of various past police incidents, providing a reference basis for the current police incident handling;

[0143] The emergency plan library has developed detailed response plans and operating procedures based on different types and levels of emergency situations;

[0144] The command and dispatch process document clarifies the responsibilities, coordination methods and workflows of each department in handling police incidents.

[0145] The command and dispatch intelligent agent plays a key role in the comprehensive management of cities. It can realize alarm perception, command and dispatch, task decomposition, task execution and dynamic tracking optimization. When the command and dispatch intelligent agent detects relevant alarms from various systems, such as when the water immersion sensor monitors that the water level is rising at a rate of 1 cm every 10 minutes or the smoke sensor detects that the smoke concentration continues to rise and exceeds the safety threshold, the model understands the sensor values and video image data, and combines the expert knowledge base to first judge the current alarm level, and then generate relevant suggestions. For example, based on the water level rising rate and the degree of smoke concentration exceeding the standard, the alarm is comprehensively judged to be at a medium level.

[0146] like Figure 2 As shown in the figure, the IoT perception command and dispatch agent of multi-agent collaboration mainly includes the following steps:

[0147] 1. Alert perception: Determine alert levels and generate recommendations through multi-source sensor data fusion and expert knowledge base.

[0148] 2. Police information transmission: Send police information to the command center and relevant personnel.

[0149] 3. Mission understanding and analysis: Analyze the commander’s instructions and clarify the mission objectives based on historical data.

[0150] 4. Task decomposition: Break down complex tasks into subtasks and develop an execution plan.

[0151] 5. Autonomous execution: Multiple agents collaborate to execute subtasks and dynamically adjust plans.

[0152] 6. Quality testing: Ensure execution quality through logic verification, fact checking, compliance review and effect evaluation.

[0153] 7. Results integration and delivery: Generate execution logs and reports.

[0154] 8. Feedback optimization: Optimize the system and update the knowledge base based on user feedback.

[0155] Police information delivery is to efficiently send police information to the command center and relevant personnel:

[0156] Information encapsulation: Alarm information is formatted into structured data: I = {type, location, status, impact range, recommendation};

[0157] Communication optimization: Use priority queue to determine the sending order, priority P i Based on the alert level and recipient role:

[0158] P i =w1·L+w2·R i

[0159] Among them, L is the alarm level, R i is the importance of the receiver, w1 and w2 are weights.

[0160] Multi-channel distribution: Send to the command center and personnel devices via API (such as WebSocket, SMS gateway), ensuring low latency and target latency T d ≤1s;

[0161] And output the successful delivery confirmation of the alarm information.

[0162] Example 2: Reference Figure 3

[0163] Compared with Example 1, the difference is:

[0164] The step S03 of determining the current emergency level through the command and dispatch agent specifically includes:

[0165] S31. Use a neural network model combined with an expert knowledge base to classify police incidents;

[0166] S32, let the alarm level be based on the feature vector F = [f1, f2, ..., f n ,], the current alarm level is determined by the maximum a posteriori probability MAP decision rule:

[0167] L = argmax k P(L k |F,θ)

[0168] Among them, L is the prediction result of the alarm level, argmax k It is used to find the input that makes the function reach its maximum value, θ is the model parameter, P(L k |F,θ) is the conditional probability;

[0169] S33. Generate the preliminary processing suggestion based on knowledge graph query and reasoning:

[0170] suggestion = KG_query(x1,x2,scene)

[0171] Among them, KG_query() is the knowledge graph query function, and x1 and x2 are query parameters.

[0172] The abnormal situation output in step S03 includes the alarm level, alarm type, occurrence location, real-time status and impact range.

[0173] For example, the water level rise rate v w ≥1cm / 10min, and smoke concentration c s ≥0.05 mg / m 3 , trigger suggestion: suggestion = KG_query(v w ,c s , scene).

[0174] The following is an example of the alarm level determination and preliminary handling suggestions for rising water levels:

[0175] The feature vector X may include:

[0176] X1: Water level sensor reading (cm)

[0177] X2: Water level rising speed (cm / min)

[0178] X3: Video analysis of flooded area (m 2 )

[0179] X4: Whether there are other flooding alarms in the adjacent area (Boolean value)

[0180] The following is a simplified example of an expert knowledge base rule:

[0181] Rule 1 (low level): IF (X1 < 5 cm AND X2 < 0.5 cm / min) OR (video recognition water immersion area < 0.1 m 2 , and no other sensor alarms) THENL = low.

[0182] Rule 2 (Intermediate): IF((5cm≤X1<20cm AND 0.5cm / min≤X2<2cm / min) OR(Video recognition water immersion area ≥0.1m 2 and <1m 2 ))AND NOT(Rule 3 condition is met)THENL=medium.

[0183] Rule 3 (High Level): IF (X1 ≥ 20 cm OR X2 ≥ 2 cm / min OR Video Identification Water Flooding Area ≥ 1 m 2 OR(X1>10cm AND there is a flood alarm in the adjacent area)) THENL=high.

[0184] Input data example: water level sensor reading 15cm, water level rise rate 1cm / min, video analysis flooding area 0.5m 2 , no proximity alarm.

[0185] Judgment result based on simplified rules: L = intermediate.

[0186] Knowledge graph query input: L = intermediate, feature X = [water level 15 cm, speed 1 cm / min, area 0.5 square meters, no proximity alarm].

[0187] The knowledge graph may contain related information:

[0188] 1. "Intermediate Water Level Alarm" → "Related Plan P-W02"

[0189] 2. "Plan P-W02" → "Action Item: Start the regional drainage pump"

[0190] 3. "Plan P-W02" → "Action Item: Notify the A District Inspection Team"

[0191] 4. "Plan P-W02" → "Checkpoint: Status of adjacent valves V101 and V102"

[0192] Generate suggestion S:

[0193] 1. Start drainage pump No. 2 in area A of the tunnel.

[0194] 2. "Notify the A District Inspection Team (Zhang San, Li Si) to go and confirm."

[0195] 3. "Check the status of valves V101 and V102 and prepare to close the upstream valve."

[0196] 4. "Continuously monitor water level changes and raise the alert level if the rate of change accelerates or the scope expands."

[0197] The following are examples of fire alarm level determination and initial handling suggestions:

[0198] The feature vector X may contain:

[0199] X1: Smoke concentration (ppm)

[0200] X2: Temperature reading (℃)

[0201] X3: Temperature rise rate (℃ / min)

[0202] X4: Video analysis flame recognition confidence (0-1)

[0203] X5: Video analysis of smoke area size (m 2 )

[0204] The following is a simplified example of an expert knowledge base rule:

[0205] Rule 1 (low level / suspect): IF (smoke concentration > threshold 1 AND temperature < threshold T1) OR (video identification shows that the smoke area is small and there is no flame) THENL = low.

[0206] Rule 2 (medium level): IF (smoke concentration > threshold 2 AND temperature > threshold T2 AND temperature rise rate > threshold R1) OR (video identifies obvious smoke and flame confidence > 0.6) THENL = medium.

[0207] Rule 3 (high level): IF (smoke concentration > threshold 3 AND temperature > threshold T3 AND temperature rise rate > threshold R2) OR (video recognition flame confidence > 0.9) THENL = high.

[0208] Input data example: smoke concentration 500ppm (assuming threshold 2 = 400ppm), temperature 60°C (assuming threshold T2 = 55°C), temperature rise rate 5°C / min (assuming threshold R1 = 3°C / min), video flame confidence 0.7.

[0209] Judgment result based on simplified rules: L = intermediate (satisfies multiple intermediate conditions).

[0210] Knowledge graph query input: L = medium, feature X = [smoke 500ppm, temperature 60℃, temperature rise 5℃min, flame confidence 0.7, smoke area medium].

[0211] The knowledge graph may contain related information:

[0212] "Intermediate fire alarm" → "Related emergency plan P-F03"

[0213] "Plan P-F03" → "Action Item: Start the regional ventilation and smoke exhaust system"

[0214] "Plan P-F03" → "Action Item: Notify the Fire Control Room"

[0215] "Plan P-F03" → "Action Item: Notify the nearest micro-fire station"

[0216] "Plan P-F03" → "Action Item: Cut off non-fire power supply in the area"

[0217] "Fire type (subdivided by sensor, such as cable fire)" → "Recommended extinguishing agent: carbon dioxide"

[0218] Generate suggestion S:

[0219] 1. Start the ventilation and smoke exhaust system for Section B's pipe gallery.

[0220] 2. "Immediately notify the fire control room and report the location and characteristics of the fire."

[0221] 3. "Notify Mini Fire Station No. 3 to investigate."

[0222] 4. "Prepare to cut off the non-fire power supply to Section B of the tunnel."

[0223] 5. "It is recommended that on-site personnel use a carbon dioxide fire extinguisher for initial control (if applicable)."

[0224] In another preferred embodiment, this technology can also be used for alert perception based on a rule engine. Sensor data can be matched one by one through the rule engine to output alert levels and suggestions.

[0225] Specifically, it replaces the machine learning model and uses a predefined rule engine such as Drools to judge the alarm situation. The rules are based on expert experience and can be dynamically updated. The thresholds are adjusted based on historical data and are defined as condition-action pairs:

[0226] For example, the condition: water level rising speed v w ≥1cm / 10min, and smoke concentration c s ≥0.05 mg / m 3 .

[0227] Action: Determined as a medium-level emergency, it is recommended to "activate the drainage plan."

[0228] Alert perception based on rule engines has low computational complexity and is suitable for resource-constrained environments. The rules are transparent and easy to explain. Compared with machine learning models, alert perception does not require model training and relies on expert-defined logical rules rather than data-driven feature learning. However, it is not as adaptable to complex scenarios as machine learning and may require frequent rule updates.

[0229] Another preferred embodiment of this technology involves distributed perception based on edge computing, offloading alert awareness tasks to edge devices such as sensor nodes and cameras. Local inference is performed using lightweight models such as TinyML, with only preliminary results uploaded to the cloud. A simplified classification model is first run on the edge device to determine the alert level. The edge results are then aggregated in the cloud and combined with the knowledge base to generate recommendations.

[0230] Distributed perception based on edge computing can reduce communication delays, improve real-time performance, and reduce cloud computing pressure. Compared with the alert perception of machine learning models, distributed perception distributes perception tasks between the edge and the cloud rather than centralized processing. However, edge devices need to have certain computing capabilities, and the model accuracy may be limited.

[0231] Example 3: Reference Figure 4

[0232] Compared with the above embodiment, the difference is that the command center personnel can input their requirements into the command and dispatch intelligent agent. The intelligent agent uses the large model capability to understand and analyze the exception handling tasks input by the command personnel, and combines the user preferences and historical data in the memory module to accurately understand the task objectives. For example, the command center personnel require the intelligent agent to immediately notify the direct system and the nearest maintenance personnel to handle the problem immediately, and notify the systems and alarm receivers affected by the problematic equipment, and then find similar equipment with potential hidden dangers. Therefore, the input exception handling task in step S04 specifically includes:

[0233] S41. Exception handling task using the pre-trained language model BEST to parse input:

[0234] T = NLP_parse(input, θ NLP )

[0235] Among them, T is the task goal, θ NLP are model parameters;

[0236] S42. Combined with the memory module, calculate the similarity between the input and the historical task:

[0237] Sim(T,T hist )=cos(Emb(T),Emb(T hist ))

[0238] Sim(T,T hist ) represents the text T and T hist The similarity score between them, Emb(T) is the embedding vector of text T, Emb(T hist ) is the historical text T hist Embedding vector of , cos() is cosine similarity;

[0239] S43. Extract mission-critical targets using slot filling techniques:

[0240] S = {action, object, priority}

[0241] S represents the slot filling result;

[0242] S44. Output a structured task target list.

[0243] Example of the command and dispatch agent understanding and analyzing the exception handling tasks and structured task lists input by the command personnel:

[0244] 1. Commander input: "The water level in Area A has reached 15 centimeters and is still rising. Have Lao Wang lead a team to investigate immediately and close valve V101 upstream. Also, notify Section Chief Liu of the Emergency Office to analyze the equipment surrounding the leak to determine if there are any other potential hazards."

[0245] 2. Slot filling results (Slots) (simplified representation):

[0246] 1. Event description:

[0247] 1. Location: Area A

[0248] 2. Phenomenon: "Water level 15 cm", "still rising" (continues to rise)

[0249] 2. Instruction 1 (On-site verification and disposal):

[0250] 1. Action: "Notify and send someone" (implied: let someone go and take a look)

[0251] 2. Executor: "Lao Wang and his team" (person in charge surnamed Wang and his team)

[0252] 3. Task: "On-site inspection"

[0253] 4. Location: Area A

[0254] 5. Urgency: "Right now"

[0255] 3. Instruction 2 (remote operation):

[0256] 1. Action: "Turn off"

[0257] 2. Equipment: "Upstream valve V101"

[0258] 4. Directive 3 (Information Notification):

[0259] 1. Action: "Notify"

[0260] 2. Recipient: "Section Chief Liu from the Emergency Office"

[0261] 5. Instruction 4 (Analysis and Judgment):

[0262] 1. Action: Analysis

[0263] 2. Target: "Equipment surrounding the leak"

[0264] 3. Purpose: "To see if there are any other hidden dangers."

[0265] Output a structured task target list (based on the above slot filling results):

[0266]

[0267]

[0268] Example 4: Reference Figure 5

[0269] Compared with the above embodiment, the difference is that the command and dispatch agent has the ability to decompose complex tasks and can convert abstract task descriptions into specific action steps to ensure efficient execution of tasks. For example, the agent understands the above requirements of the commander and converts them into the following tasks:

[0270] 1. Notify the nearest equipment maintenance and inspection personnel based on their skills and tool availability;

[0271] 2. Based on the location and impact of the emergency, combined with real-time traffic conditions and the corridor's structural layout, the optimal maintenance entrance and route are planned, and the relevant information is sent to the maintenance inspector's mobile phone;

[0272] 3. Conduct a comprehensive analysis of the corresponding systems of the problem equipment to identify equipment and locations currently experiencing similar risks;

[0273] 4. Analyze potentially affected equipment based on equipment type, location, and other information, notify the relevant alarm receiving and handling system, and inform the relevant person in charge;

[0274] 5. Dynamically track the status of the device and the police situation in the area, and adjust the strategy in a timely manner if it is predicted that the police situation may escalate; 6........

[0276] In step S05, the exception handling task is broken down into subtasks, specifically including:

[0277] S51. Use the hierarchical task network HTN planning to decompose the exception handling task Y into subtasks:

[0278] Y={Y1,Y2,…,Y n}

[0279] Wherein, Y represents the exception handling task, Y i ={action, condition, resource}, i=1,2,…,n, Y i Represents a subtask;

[0280] S52. Assign subtasks to professional agents based on subtask type and resource constraints:

[0281] Agent i =argmin j C(Y i ,A j )

[0282] Among them, Agent i Represented as task Y i The selected agent, C(Y i ,A j ) is the cost function, which measures the task Y i By Agent Aj The cost of execution, A j represents the jth agent, j represents the agent index;

[0283] S53. Use dynamic programming to optimize the execution order of subtasks:

[0284]

[0285] Among them, t i For subtasks, c i is the resource cost;

[0286] S54. Output the subtask list and execution plan.

[0287] The following are examples of subtasks:

[0288] Subtask 1: Notify the nearest maintenance personnel.

[0289] Input: Personnel skills S p , tool equipped with E p , distance D p .

[0290] Optimization goal:

[0291] P * =argmin p (w d ·D p -w s ·S p -w e ·E p )

[0292] Subtask 2: Plan the maintenance route.

[0293] Use A * algorithm, combined with real-time traffic T r And the pipe gallery layout G:

[0294] Path=AStar(G,T r , starting point, end point)

[0295] Subtask 3: Analyze similar risk devices.

[0296] Analyze device association using graph neural network GNN:

[0297] R=GNN(G 设备 ,F 特征 )

[0298] Output: subtask list and execution plan.

[0299] Example 5: Reference Figure 6

[0300] After breaking down the exception handling task into several subtasks, the agent autonomously plans and executes each subtask. In step S06, the subtasks are executed, and the output of the subtask execution status specifically includes:

[0301] S61. Use the distributed multi-agent framework MADDPG to enable each agent to focus on a subtask:

[0302] π m =Policy(s m ,θ m )

[0303] Among them, π m is the agent's strategy in state s m The output under, Policy is the policy function, s m is the local state, θ m is the parameter of the policy function, m is the moment parameter;

[0304] S62, real-time perception of environmental changes and update of status:

[0305] s m =f(s m-1 ,z m )

[0306] Among them, z m is the new observation data;

[0307] S63. Use reinforcement learning to adjust the execution plan:

[0308] Q(s,a)←Q(s,a)+α[r+γmaxQ(s′,a′)-Q(a,a)]

[0309] Where Q(s,a) represents the action value function of taking action a in state s, ← is the assignment operation, α represents the learning rate, which is used to control the step size of each update, r represents the reward value of the environment feedback, γ represents the discount factor, and s′, a′ represent the next state;

[0310] S64. Use the timeline manager to track subtask status:

[0311] Progress={(Y i ,state,t i )}

[0312] Progress is a collection, Y i represents the i-th subtask, t i Indicates Y i timestamp;

[0313] S65: Output the real-time subtask execution status.

[0314] The command and dispatch intelligent agent adopts a multi-agent system, in which multiple professional intelligent agents work together to coordinate task allocation and progress management. Each intelligent agent focuses on a specific task. During the execution of subtasks, it displays the execution status of each subtask in real time. Finally, after all subtasks are completed, it outputs the execution status log and report of each subtask, and continuously perceives changes in the external environment, such as the impact of changes in traffic conditions on the arrival time of maintenance personnel, and adjusts the task execution plan in a timely manner.

[0315] In another preferred embodiment, this technology can also be used for task decomposition based on a workflow engine, using workflow management tools such as Apache Airflow and BPMN engine instead of HTN planning to decompose tasks into predefined workflow templates:

[0316] ① The task objectives are mapped to workflow nodes, with each node corresponding to a subtask such as "notify maintenance personnel";

[0317] ② The workflow engine automatically schedules subtasks based on dependencies and resource constraints;

[0318] ③Dynamic adjustment is achieved by re-arranging the workflow.

[0319] The development cycle of using workflow management tools is short and easy to integrate with existing enterprise systems, but they have weak adaptability to complex dynamic scenarios and require predefined templates. In addition, the workflow is based on fixed templates rather than dynamic planning, and has low intelligence.

[0320] In another preferred embodiment, this technology can also be based on single-agent centralized scheduling, instead of multi-agent collaboration, using a single central agent to be responsible for task decomposition, allocation and execution monitoring.

[0321] ① The central agent analyzes the task objectives and generates a list of subtasks;

[0322] ②Subtasks are assigned to execution units such as maintenance personnel and equipment through priority queues;

[0323] ③ Dynamic optimization is achieved by recalculating priorities through the central agent;

[0324] The centralized scheduling system based on a single agent has a simple architecture and is suitable for small-scale scenarios, but its scalability is relatively poor. The central agent may become a bottleneck. Centralized control replaces distributed collaboration, which reduces system complexity but also sacrifices parallel efficiency.

[0325] The two alternative solutions, workflow management tools and single-agent-based centralized scheduling, can achieve task decomposition and execution, which are suitable for applications with limited resources or simple scenarios, but may not fully match the efficient collaboration of the original solution in complex scenarios.

[0326] After the agent completes each subtask, it needs to test the execution quality. The quality test includes the following sub-agents: logic verification agent, fact-checking agent, and compliance review agent. The logic verification agent will test the causal rationality of the task chain. After the subtask is executed, the fact-checking agent cross-compares multiple sources to verify the authenticity of the data. Finally, the compliance review agent ensures that the output content complies with laws and regulations. In addition to verifying the authenticity of the data and the causal rationality of the task chain, it can also increase the evaluation of the task execution effect based on the quality inspection standards and indicator system of the expert knowledge base, such as the operating stability of the equipment after maintenance, the actual effectiveness of the alarm handling, etc.

[0327] Verify the logic, authenticity, and compliance of the agent's subtask execution, and evaluate the results:

[0328] 1. Logical Verification: Use formal verification to check task causality:

[0329] Valid=Check(Y i →Y i+1 ,rule)

[0330] 2. Fact checking: cross-validating data from multiple sources:

[0331]

[0332] Among them, d j is the data source, and Conf is the confidence level.

[0333] 3. Compliance review: Use the rule engine to match laws and regulations:

[0334] Compliance=RuleMatch(O,R 法规 )

[0335] Among them, O is the output content, R 法规 For the regulatory database.

[0336] 4. Effect evaluation: Define indicators based on expert knowledge base, such as equipment stability S e , alarm relief rate R n :

[0337] Score=w s ·S e +w r ·R a

[0338] Output quality inspection report, which includes the task's causal logic, authenticity, legal and regulatory compliance, and subtask execution results.

[0339] In another preferred embodiment, quality inspection can also be done through decentralized verification based on blockchain, using the blockchain network to verify the authenticity and compliance of task execution:

[0340] ①Task execution data such as sensor readings and execution logs are uploaded to the blockchain;

[0341] ② Smart contracts verify logical causality and regulatory compliance;

[0342] ③The effect evaluation is confirmed through a multi-party consensus mechanism.

[0343] The data of the blockchain-based decentralized verification solution cannot be tampered with, the verification is transparent and reliable, and it emphasizes data credibility rather than algorithm complexity. However, its deployment cost is high and the verification speed may be slow.

[0344] In another preferred embodiment of this technology, quality detection can also be quality control based on statistical analysis, using the statistical process control (SPC) method to evaluate the quality of task execution:

[0345] ① Define key performance indicators such as task completion time and alarm mitigation rate:

[0346] ② Use control charts such as X-bar charts to detect anomalies:

[0347] UCL=μ+3σ

[0348] LCL=μ-3σ

[0349] Where UCL / LCL are the upper and lower control limits, μ is the mean, and σ is the standard deviation;

[0350] ③Compliance is checked through keyword matching.

[0351] This solution is simple and efficient, suitable for standardized scenarios, and reduces technical complexity, but it is difficult to handle complex causal relationships or non-quantitative indicators.

[0352] Integrating the subtask execution results specifically includes:

[0353] Merge the subtasks and output the merged result R:

[0354]

[0355] Among them, R i is the subtask result;

[0356] By timestamp t i Record the execution process and generate a log Log: Log = (t i ,Yi i ,status,output);

[0357] Use dashboards or charts to display results, and users can provide feedback on the task breakdown, execution, and quality.

[0358] Based on user feedback, a feedback classification and priority management mechanism is established to respond to and handle urgent and important feedback in a timely manner. Reasonable intelligent agent execution plans are included in the historical case knowledge base of underground integrated pipeline corridors to achieve knowledge accumulation and inheritance, and continuously optimize the execution strategy and effect of the intelligent agent.

[0359] Optimize agent behavior and update knowledge base based on user feedback:

[0360] 1. Feedback classification: Use text classification models, such as SVM, to classify feedback into urgent, general, etc.

[0361] C=Classifier(F,θ C )

[0362] 2. Priority management: sorting based on feedback urgency and importance:

[0363] P f =w u ·U f +w i I f

[0364] Among them, U f For urgency, I f For importance.

[0365] 3. Knowledge base update: save the optimized solution into the case library:

[0366] KB←KB∪{scheme, effect, scenario}

[0367] 4. Model fine-tuning: Fine-tune the model using feedback data:

[0368]

[0369] Among them, L is the loss function and η is the learning rate.

[0370] Finally, the optimized agent behavior and updated knowledge base are output.

[0371] Example 6: Reference Figure 7

[0372] Compared with the above embodiment, the present invention also provides an IoT perception command and dispatch intelligent system based on a multimodal large model, including:

[0373] Monitoring module: used for multi-terminal on-site information monitoring of multi-scenario equipment;

[0374] Processing module: performs data preprocessing and multi-source data fusion on the multi-terminal on-site information;

[0375] Abnormal judgment module: When abnormalities are detected in multiple segments of on-site information, the command and dispatch intelligent agent determines the current alarm level and outputs the abnormal situation and preliminary handling suggestions;

[0376] Input module: The user inputs the exception handling task based on the exception situation and preliminary handling suggestions;

[0377] Disassembly module: disassembles the exception handling task into subtasks through the command and dispatch agent, and forms a subtask list;

[0378] Execution module: directs the dispatching agent to execute the subtasks according to the subtask list and outputs the subtask execution status;

[0379] Quality inspection module: performs quality inspection on the execution effect of subtasks and generates quality inspection reports;

[0380] Integration module: integrates the subtask execution results and generates execution logs and comprehensive reports;

[0381] Optimization module: optimizes the command and dispatch agent based on user feedback and updates the knowledge base.

[0382] The present invention provides an IoT perception command and dispatch intelligent system based on a multimodal large model, which provides capabilities such as equipment access and management, data fusion sharing and openness, business topic construction, and IoT perception intelligent models. It effectively overcomes the problems of difficult data integration, difficult cross-departmental communication, and low collaborative efficiency in urban comprehensive management scenarios. Through the IoT perception platform, it breaks the situation where each unit system is independent of each other, deeply integrates multiple data in comprehensive scenarios, and uses the IoT perception basic model capabilities to integrate urban knowledge networks to provide comprehensive and accurate knowledge support for urban management. At the same time, it can also give full play to the advantages of intelligent bodies, reshape work processes, achieve efficient cross-departmental collaboration, improve comprehensive decision-making efficiency, promote urban management towards intelligence and efficiency, and lead cities to achieve smart upgrades and sustainable development in the digital age.

[0383] This system has been determined to be applied to the Guangdong Unicom IoT Perception Platform, and will build basic model application and scenario model application capabilities on the Guangdong Unicom IoT Perception Platform, such as Figure 6 As shown in the figure, the architecture and module design of the IoT perception platform specifically include:

[0384] 1. Device access layer: supports direct connection of IoT devices, access of gateways and their sub-devices, and access of IoT application platforms, enabling access to all devices.

[0385] 2. Device management layer: supports standard object model management, object model mapping function, device life cycle management, device shadow, OTA upgrade and other device management capabilities.

[0386] 3. Data storage layer: Persists the business data of devices and devices, supports the storage and processing of various types of data such as structured data, semi-structured data, and unstructured data, and has massive data storage, as well as good scalability, high performance, and high availability.

[0387] 4. Data service layer: provides a series of capabilities such as IoT perception resource directory, data pre-quality inspection, data cataloging and connection, and data synchronization on the data sharing platform.

[0388] 5. Business enablement layer: The business enablement layer provides general cockpit capabilities, streaming capabilities, rule engine, alarm center, and other capabilities, based on which a series of industry SaaS applications can be built.

[0389] 6. Intelligent service layer: Intelligent services support access to large models and have the ability to perceive large models of the Internet of Things, including basic models of Internet of Things perception and scenario models of key industries.

[0390] 1) Large model access: Build model access and model management functions, supporting operations such as adding, switching, and configuring models;

[0391] ① General model access: supports access to general models such as Yuanjing, DeepSeek, and Qwen Tongyi;

[0392] ② Multimodal model access: such as Yuanjing multimodal large model, Qwen-VL, Volcano Ark, etc.;

[0393] ③Statistical analysis: conduct session statistics, token number statistics, session interaction number statistics, API call number statistics, cost consumption statistics, etc.

[0394] 2) IoT Perception Basic Model: Build IoT perception basic models and application capabilities to implement IoT perception-based AI applications;

[0395] ① IoT perception knowledge question and answer: IoT perception intelligent question and answer integrates IoT perception-related knowledge, covering the IoT perception sharing and exchange platform manual, IoT perception data resource directory specification, one-network sharing platform operation manual and other contents, providing users with timely and accurate knowledge feedback, convenient backtracking of historical questions and answers, and clear presentation of reply sources. Through the question and answer assistant, it effectively solves the difficulties faced by front-line personnel in industry units in using the IoT perception platform and sharing data, improves work efficiency, and releases data value. Secondly, the intelligent question and answer assistant can be used to improve the IoT perception knowledge base and promote the integration of the IoT perception knowledge network;

[0396] Among them, IoT perception intelligent data analysis provides basic statistical calculations of IoT perception data, anomaly monitoring analysis, and data analysis of prediction suggestions. Based on the data processed and analyzed by the platform, it uses data analysis and text generation technology to automatically generate business performance reports. The reports cover key indicator analysis, task execution status, benefit evaluation, etc., enabling related work such as data supervision and business assessment, and providing intuitive data support and reference basis for management decision-making;

[0397] ② Intelligent device access: Users can automatically map IoT device data into platform object models through interactive Q&A, simulate device reporting data, test device direct access debugging results, and improve data aggregation speed.

[0398] ③ Intelligent generation of object models: Based on the knowledge base standard object model template, upload the relevant files of the attributes, events, and services of the required object model, and the system will automatically generate the json file of the standard object model through intelligent calculation.

[0399] ④ Intelligent Data Quality Inspection: Utilizing the cataloging specifications and standards stored in the knowledge base, the big model can quickly understand the various rules and requirements in the cataloging specifications, such as data format, data type, and data association. When various industry units upload data cataloging resources, the big model automatically compares the resource content with the standards in the knowledge base, accurately identifying parts that do not meet the rules. For data that does not meet the standards, the big model combines the standard data templates and common problem solutions in the knowledge base to provide industry units with specific replacement and supplementary suggestions. These suggestions not only include how to correct erroneous data, but also how to supplement missing data and adjust data formats, helping industry units to quickly and accurately complete data pre-quality inspection.

[0400] ⑤ Intelligent equipment operation and maintenance: Based on the large model and multi-dimensional time series data analysis, it generates equipment maintenance suggestions by predicting equipment operation data trends, intelligently mining abnormal reporting values, and combining with the knowledge base.

[0401] 3) Key Industry Scenario Models: Targeting key industry scenarios, we build scenario models that can be customized and fine-tuned for specific scenarios. We also provide key industry intelligent agent services. The intelligent agents that empower vertical industries are autonomous systems driven by large models. By integrating knowledge bases, real-time data interfaces, and multimodal sensor perception of the environment, we leverage workflow orchestration to achieve task decomposition, intelligent analysis, and dynamic decision-making.

[0402] ① Situation Analysis Agent: The Situation Analysis Agent deeply analyzes the real-time and historical data of IoT sensing devices to assist in the prediction and early warning of IoT sensing event trends, such as potential hidden dangers and risk deduction. For example, in smart firefighting scenarios, based on the attributes and events reported by sensor devices, threshold management is used to comprehensively assess firefighting risks and issue early warning reminders.

[0403] ② Command and dispatch agent: The command and dispatch agent adopts a multi-agent architecture, and has the capabilities of perception (understanding of police situations), planning (task decomposition), execution (tool calling) and verification (result verification). It has model scheduling + tool chain integration + cross-end interaction capabilities. When the agent perceives and monitors an emergency, it automatically plans and executes command and dispatch strategies based on multimodal data and model analysis results. The agent performs the planning work and assists with event command suggestions through the historical expert knowledge base. The agent executes the call to connect to the police reception and handling systems of various units to achieve efficient emergency command and dispatch, such as commanding rescue forces and deploying materials in urban integrated pipeline corridor accidents.

[0404] 7. Platform presentation layer: provides IoT perception portal and IoT perception platform console for users.

[0405] 8. Capability exposure layer: supports capability exposure interfaces such as products, object models, devices, applications, and rule engines, and supports third-party applications to call the capabilities of the IoT perception platform.

[0406] 9. Lightweight scenario applications: Build lightweight scenario application capabilities to achieve rapid delivery and vertically connect the entire industry chain. Lightweight scenario applications include smart meter reading, smart buildings, smart campuses, and smart fire protection.

[0407] 10. Other Empowerments:

[0408] 1) Intelligent Applications for Urban Governance: The IoT sensing platform's device access capabilities, data service capabilities, IoT sensing foundational models, and key industry scenario models can all empower smart city integrated governance scenarios, such as urban integrated pipeline corridors, integrated water management, and urban fire management.

[0409] 2) One-network management of intelligent applications: The IoT perception platform's device access capabilities, data service capabilities, IoT perception basic models, and key industry scenario models can all enable digital government one-network management scenarios, such as comprehensive collaborative command and dispatch, multi-topic dynamic data acquisition, and multi-department collaborative event handling.

[0410] In another preferred embodiment, this technology can also be a centralized command and dispatch system based on cloud computing, replacing the multi-agent distributed system, using a centralized cloud computing platform such as AWS and Azure to handle all perception, scheduling, and optimization tasks:

[0411] ① Alarm situation awareness: Big data analysis engines such as Spark run in the cloud to process sensor and video data and combine pre-trained models to judge alarm situations;

[0412] ②Task decomposition and execution: Cloud-based task schedulers such as Kubernetes decompose tasks and assign them to execution units. Dynamic optimization is achieved through cloud-based algorithms such as genetic algorithms.

[0413] ③ Quality testing: A unified quality analysis module runs in the cloud, integrating rule checking and statistical evaluation;

[0414] ④ Feedback optimization: The cloud database stores historical data and updates the model regularly.

[0415] The centralized command and dispatch system based on cloud computing has concentrated computing resources, is easy to maintain and expand, and supports large-scale data processing. It emphasizes computing power rather than collaborative intelligence. Therefore, it has a high dependence on the network, may increase latency, and has a slow response in edge scenarios. It is suitable for urban governance applications with sufficient data center resources and concentrated scenarios, but may not be suitable for edge or high-real-time scenarios.

[0416] In another preferred embodiment, this technology can also be a semi-automatic scheduling system based on human assistance, using a semi-automatic framework of human-machine collaboration to replace the fully automated intelligent agent system, with intelligent modules assisting human decision-making:

[0417] ① Alarm perception: The intelligent module initially analyzes sensor data and generates an alarm report, which is manually reviewed to confirm the level and recommendations;

[0418] ②Task decomposition and execution: The intelligent module recommends task decomposition plans, and the commander manually adjusts and assigns tasks. The execution process is monitored by the system;

[0419] ③Quality testing: The intelligent module provides data verification and compliance prompts, and manual final verification is completed;

[0420] ④ Feedback optimization: Manually input feedback, and the system records and updates the rule base.

[0421] The semi-automated scheduling system based on human assistance makes full use of human experience, reduces the risk of system errors, and has low development costs. However, its efficiency is limited by human intervention and cannot be fully automated. It is more suitable for scenarios with weak technical infrastructure or those requiring strict human supervision, but its efficiency is lower than the original solution.

Claims

1. An intelligent method for IoT perception, command and dispatch based on a multimodal large model, characterized by: include: S01, command and dispatch intelligent agents to monitor multi-terminal on-site information of multi-scenario equipment; S02. Performing data preprocessing and multi-source data fusion on the multi-terminal on-site information; S03. When the multi-terminal on-site information is detected to be abnormal, the command and dispatch agent determines the current alarm level and outputs the abnormal situation and preliminary handling suggestions; S04. The user inputs an exception handling task based on the exception situation and preliminary handling suggestions; S05. Decompose the exception handling task into subtasks through the command and dispatch agent, and form a subtask list; S06, commanding and dispatching the intelligent agent to execute the subtasks according to the subtask list and outputting the subtask execution status; S07. Perform quality inspection on the execution effect of the subtask and generate a quality inspection report; S08. Integrate the subtask execution results to generate an execution log and a comprehensive report; S09. Optimize the command and dispatch agent based on user feedback and update the knowledge base.

2. The method of intelligent IoT perception, command and dispatch based on a multimodal large model according to claim 1 is characterized in that: The multi-terminal field data information in step S02 is real-time collection of two or more sensor data and video stream data from water immersion sensors, smoke sensors, gas sensors, displacement sensors, high-definition cameras and smart valves.

3. The data preprocessing in step S02 is to perform noise removal and standardization on the multi-terminal field information: in, x is the original sensor value, μ is the mean, σ is the standard deviation, x norm It is the standard value after data preprocessing; The multi-source data fusion fuses the sensor data and the video stream data through Kalman filtering: in, is the state estimation, K t is the Kalman gain, z t is the observation value, H is the observation matrix, and t represents the current time step.

4. The method of intelligent IoT perception, command and dispatch based on a multimodal large model according to claim 1 is characterized in that: The step S03 of determining the current emergency level through the command and dispatch agent specifically includes: S31. Use a neural network model combined with an expert knowledge base to classify police incidents; S32, let the alarm level be based on the feature vector F = [f1, f2, ..., f n ,], the current alarm level is determined by the maximum a posteriori probability MAP decision rule: L=argmax k P(L k |F,θ) Among them, L is the prediction result of the alarm level, argmax k It is used to find the input that makes the function reach its maximum value, θ is the model parameter, P(L k |F,θ) is the conditional probability; S33. Generate the preliminary processing suggestion based on knowledge graph query and reasoning: suggestion = KG_query(x1,x2,scene) Among them, KG_query() is the knowledge graph query function, and x1 and x2 are query parameters.

5. The method for IoT perception, command and dispatch based on a multimodal large model according to claim 3 is characterized in that: The abnormal situation output in step S03 includes the alarm level, alarm type, occurrence location, real-time status and impact range.

6. The method of intelligent IoT perception, command and dispatch based on a multimodal large model according to claim 1 is characterized in that: The input exception handling task in step S04 specifically includes: S41. Exception handling task using the pre-trained language model BEST to parse input: T = NLP_parse(input, θ NLP ) Among them, T is the task goal, θ NLP are model parameters; S42. Combined with the memory module, calculate the similarity between the input and the historical task: Sim(T,T hist )=cos(Emb(T),Emb(T hist )) Sim(T,T hist ) represents the text T and T hust The similarity score between them, Emb(T) is the embedding vector of text T, Emb(T hist ) is the historical text T hist Embedding vector of , cos() is cosine similarity; S43. Extract mission-critical targets using slot filling techniques: S = {action, object, priority} S represents the slot filling result; S44. Output a structured task target list.

7. The method of intelligent IoT perception, command and dispatch based on a multimodal large model according to claim 5 is characterized in that: In step S05, the exception handling task is broken down into subtasks, specifically including: S51. Use the hierarchical task network HTN planning to decompose the exception handling task Y into subtasks: Y={Y1,Y2,…,Y n } Wherein, Y represents the exception handling task, Y i ={action, condition, resource}, i=1,2,…,n, Y i Represents a subtask; S52. Assign subtasks to professional agents based on subtask type and resource constraints: Agent i =argmin j C(Y i ,A j ) Among them, Agent i Represented as task Y i The selected agent, C(Y i ,A j ) is the cost function, which measures the task Y i By Agent A j The cost of execution, A j represents the jth agent, j represents the agent index; S53. Use dynamic programming to optimize the execution order of subtasks: Among them, t i For subtasks, c i is the resource cost; S54. Output the subtask list and execution plan.

8. The method of intelligent IoT perception, command and dispatch based on a multimodal large model according to claim 6 is characterized in that: In step S06, the subtask is executed, and outputting the subtask execution status specifically includes: S61. Use the distributed multi-agent framework MADDPG to enable each agent to focus on a subtask: p m =Policy(s m ,i m ) Among them, π m is the agent's strategy in state s m The output under, Policy is the policy function, s m is the local state, θ m is the parameter of the policy function, m is the moment parameter; S62, real-time perception of environmental changes and update of status: s m =f(s m-1 ,z m ) Among them, z m is the new observation data; S63. Use reinforcement learning to adjust the execution plan: Q(s,a)←Q(s,a)+α[r+γmaxQ(s ′ ,a′)-Q(s,a)] Among them, Q(s,a) represents the action value function of taking action a in state s, ← is the assignment operation, α represents the learning rate, which is used to control the step size of each update, r represents the reward value of environmental feedback, γ represents the discount factor, and s ′ ,a′ represents the next state; S64. Use the timeline manager to track subtask status: Progress={(Y i ,state,t i )} Progress is a collection, Y i represents the i-th subtask, t i Indicates Y i timestamp; S65: Output the real-time subtask execution status.

9. The method of intelligent IoT perception, command and dispatch based on a multimodal large model according to claim 1 is characterized in that: Step S08 specifically includes: Merge the subtasks and output the merged result R: Among them, R i is the subtask result; By timestamp t i Record the execution process and generate a log Log: Log = (t i ,Y i ,status,output); Present the results using dashboards or charts.

10. An IoT perception command and dispatch intelligent system based on a multimodal large model, characterized by: include: Monitoring module: used for multi-terminal on-site information monitoring of multi-scenario equipment; Processing module: performs data preprocessing and multi-source data fusion on the multi-terminal on-site information; Abnormal judgment module: When abnormalities are detected in multiple segments of on-site information, the command and dispatch intelligent agent determines the current alarm level and outputs the abnormal situation and preliminary handling suggestions; Input module: The user inputs the exception handling task based on the exception situation and preliminary handling suggestions; Disassembly module: disassembles the exception handling task into subtasks through the command and dispatch agent, and forms a subtask list; Execution module: directs the dispatching agent to execute the subtasks according to the subtask list and outputs the subtask execution status; Quality inspection module: performs quality inspection on the execution effect of subtasks and generates quality inspection reports; Integration module: integrates the subtask execution results and generates execution logs and comprehensive reports; Optimization module: optimizes the command and dispatch agent based on user feedback and updates the knowledge base.

Images