A method and system for holographic situation awareness and collaborative command and dispatch of a smart park based on digital twinning and multi-modal AI inference

Abstract

The application discloses a kind of based on digital twin and multi-modal AI inference wisdom park holographic situation awareness and collaborative command scheduling method and system, belong to artificial intelligence technical field.The method constructs park digital twin model, collects and manages multi-modal data, carries out space-time correlation, fusion analysis and three-dimensional rendering to generate holographic situation;Based on holographic situation, identify abnormal events, determine warning level and carry out influence deduction;Combined with warning level, deduction result and resource state, generate collaborative disposal scheme, dispatch task and carry out evaluation and strategy update.The application realizes park running state real-time perception, abnormal event intelligent early warning and resource optimization allocation, improves event disposal efficiency and emergency collaborative ability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for holographic situational awareness and collaborative command and dispatch in smart parks based on digital twins and multimodal AI reasoning. Background Technology

[0002] As campuses, industrial parks, and commercial buildings continue to expand, the number of systems within these parks, including video surveillance, access control, fire protection, environmental monitoring, energy metering, vehicle management, and equipment maintenance, is constantly increasing. Currently, park management is typically handled by multiple independent business platforms. Each platform collects video images, equipment status, personnel locations, vehicle trajectories, energy consumption data, and work order data with different data formats, time bases, spatial coordinates, and object codes. Management personnel must switch between multiple system interfaces to view and manually compare data.

[0003] In existing technologies, some parks use two-dimensional maps, three-dimensional models, or large-screen visualizations to display equipment locations and alarm information. However, these display methods typically only aggregate data from different systems on a single page, lacking a unified spatiotemporal association corresponding to the park's building structure, floor space, equipment objects, and road topology. When abnormal events such as personnel intrusion, fires, equipment malfunctions, or traffic congestion occur, video alarms, sensor alarms, and changes in equipment status are difficult to accurately correlate with the same spatial location and associated objects. Management personnel find it difficult to quickly determine the actual impact range of the event, the distribution of surrounding personnel, accessible routes, and available resources.

[0004] Therefore, the technical problem with existing smart park management technologies is that heterogeneous sensing data and park spatial models lack a unified spatiotemporal correlation and fusion analysis mechanism, which makes it impossible to form a real-time holographic situation corresponding to the actual park space and resource status of abnormal events, thus making it difficult to achieve accurate judgment, impact prediction and collaborative handling of abnormal events. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the difficulty in unifying the spatiotemporal correlation between heterogeneous sensing data and park spatial models in existing smart parks, and to solve the problems of inaccurate situational assessment of abnormal events, difficulty in predicting impacts, and low efficiency of collaborative handling.

[0006] To address the aforementioned technical problems, the first aspect of this invention discloses a method for holographic situational awareness and collaborative command and dispatch in smart parks based on digital twins and multimodal AI reasoning, comprising: Acquire spatial data and equipment data of the park, process the spatial data and equipment data of the park, and construct a digital twin model of the park; Connect to sensing devices and business systems within the park to collect multimodal data from the park; Data governance is performed on the multimodal data of the park, and the governed multimodal data of the park is spatiotemporally correlated with the digital twin model of the park; The spatiotemporally correlated multimodal data of the park are fused, analyzed, and rendered in three dimensions to generate a holographic view of the park. Based on the holographic situational awareness of the park, the warning level of the abnormal event is determined, and the impact of the abnormal event is extrapolated. Based on the warning level of the abnormal event, the impact projection results, and the status of schedulable resources, a collaborative response plan is generated, response tasks are assigned, and evaluation and strategy updates are performed based on the response process data and response result data.

[0007] Furthermore, the park spatial data includes oblique photogrammetry data, building information model data, city information model data, and laser point cloud data; the equipment data includes equipment codes, equipment structural data, equipment installation location data, and equipment operating parameters. The construction of the digital twin model of the park includes: The oblique photography data is used to perform real-scene 3D reconstruction, and the laser point cloud data is denoised, registered, segmented and surface reconstructed. The reconstruction results are fused with the building information model data and city information model data according to a unified coordinate reference. Based on spatial topology, building component identification, floor identification and equipment coding, the semantic association relationship of objects is established to obtain the digital twin model of the park.

[0008] Furthermore, the multimodal data of the park includes at least two of the following: video image data, audio data, environmental monitoring data, access control data, personnel positioning data, vehicle positioning data, equipment status data, energy consumption data, fire protection data, asset data, inspection data, work order data, and contingency plan data. The sensing devices include at least one of video surveillance equipment, audio pickup equipment, access control equipment, environmental sensors, fire sensors, positioning terminals, energy metering equipment, and equipment control terminals.

[0009] Furthermore, the data governance includes data cleaning, data standardization, data quality verification, data storage, data service encapsulation, and permission identification configuration; The spatiotemporal association includes: configuring timestamps, spatial location identifiers, data source identifiers, and object identifiers for the multimodal data of the park; and mapping the multimodal data of the park to spatial objects, equipment objects, or business objects in the digital twin model of the park based on the timestamps, spatial location identifiers, and object identifiers.

[0010] Furthermore, the generation of the holographic situation of the park includes: The digital twin model of the park overlays personnel trajectories, vehicle trajectories, equipment operating status, energy consumption status, alarm information, and event information. Based on the display terminal parameters and viewing angle, perform light field holographic rendering or real-time 3D rendering, and output a holographic status interface of the park that includes spatial location, object attributes, real-time status and alarm indicators; The digital twin model of the park supports the subdivision of building structures, the unfolding of floors, and the display of the internal structure of equipment.

[0011] Furthermore, the identification of anomalous events includes: Visual recognition models are used to identify abnormal human behavior, abnormal vehicle behavior, smoke, intrusion, congestion, or abnormal equipment appearance. Speech recognition models are used to identify help requests, abnormal sounds, or alarm voices. Time series prediction models are used to predict abnormal trends in equipment operation data, energy consumption data, or environmental monitoring data. The abnormal events are determined based on the confidence level, temporal correlation, and spatial correlation of the output results of each model. Determining the warning level of the abnormal event includes calculating an event risk score R based on the event severity, the number of affected objects, the spatial range of the impact, the event duration, the event development trend, the importance of associated equipment, and the confidence level of the model inference. The event risk score R satisfies the following: R=w1S+w2N+w3A+w4T+w5D+w6I+w7C; Where S is the normalized value of event severity, N is the normalized value of the number of affected objects, A is the normalized value of the affected spatial range, T is the normalized value of event duration, D is the normalized value of event development trend, I is the normalized value of the importance of associated equipment, C is the model inference confidence, and w1 to w7 are the weight coefficients corresponding to S, N, A, T, D, I and C, respectively, and the sum of w1 to w7 is 1. The event risk score R is compared with a preset grading threshold to obtain the corresponding warning level.

[0012] Furthermore, the impact deduction of the abnormal event includes: The location and associated objects of the abnormal event are determined. Based on the spatial topology, personnel distribution, equipment status, road traffic status and resource location in the digital twin model of the park, the impact range, risk changes and resource access of the abnormal event within a preset time window are simulated, and at least one of the following projection results is output: evacuation route, isolation area, rescue route, list of linked equipment and disposal priority.

[0013] Furthermore, the generation of the collaborative handling plan includes: selecting candidate plans from the plan library according to the event type and warning level of the abnormal event; forming a resource candidate set according to the distance between the schedulable resources and the location where the abnormal event occurs, the estimated arrival time, the current task load, the availability status, the professional capabilities and the permission level; and using a reinforcement learning scheduling model to determine the resource allocation results, the task execution order, the handling path and the control instructions for the linkage equipment. The reinforcement learning scheduling model determines scheduling actions based on the current park situation, the attributes of the abnormal event, the status of candidate resources, the spatial location of resources, contingency rules, and historical handling results, and updates the scheduling strategy according to the reward value Q, wherein the reward value Q satisfies: Q=a1E+a2F+a3U-a4L-a5P; Where E is the reduction value of the event risk score before and after handling, F is the task completion degree, U is the compliance degree of the plan execution, L is the normalized value of the event response time, P is the normalized value of the resource allocation cost, and a1 to a5 are the weight coefficients corresponding to E, F, U, L and P respectively, and the sum of a1 to a5 is 1. The dispatching of disposal tasks includes generating an electronic work order containing a task identifier, event identifier, disposal location, disposal steps, time limit requirements, responsible party, and linkage instructions, and sending the electronic work order to the corresponding disposal terminal; The evaluation includes generating evaluation results based on event confirmation time, early warning generation time, task assignment time, resource arrival time, handling completion time, contingency plan execution compliance, task completion rate, and manual evaluation data, and updating contingency plan configuration, scheduling strategy, or model training data based on the evaluation results.

[0014] Secondly, the present invention also provides a smart park holographic situational awareness and collaborative command and dispatch system based on digital twins and multimodal AI reasoning, applied to any of the aforementioned smart park holographic situational awareness and collaborative command and dispatch methods based on digital twins and multimodal AI reasoning, comprising: The modeling module is used to acquire park spatial data and equipment data, process the park spatial data and equipment data, and construct a digital twin model of the park. The data acquisition module is used to connect to the sensing devices and business systems within the park and collect multimodal data from the park. The governance module is used to perform data governance on the multimodal data of the park and to spatiotemporally correlate the governed multimodal data of the park with the digital twin model of the park. The situation module is used to perform fusion analysis and 3D visualization rendering of spatiotemporally correlated multimodal data of the park to generate a holographic situation of the park. The analysis module is used to identify abnormal events based on the holographic situation of the park, determine the warning level of the abnormal events, and perform impact deduction on the abnormal events; The scheduling module is used to generate a collaborative handling plan, dispatch handling tasks, and evaluate and update strategies based on the warning level, impact projection results, and schedulable resource status of the abnormal event.

[0015] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor performs any of the above-described methods for holographic situational awareness and collaborative command and dispatch of smart parks based on digital twins and multimodal AI reasoning.

[0016] The beneficial technical effects of the present invention are at least as follows: Compared with existing technologies, this invention constructs a digital twin base for the park by integrating oblique photography models, BIM / CIM models, and laser point cloud data. It maps buildings, floors, roads, equipment, and spatial areas to a unified three-dimensional spatiotemporal coordinate system. It also accesses, manages, aligns in time, spatially locates, and associates heterogeneous sensing data such as video images, IoT sensors, access control, vehicles, energy consumption, and assets. This reduces data fragmentation between different business systems and enables multi-source alarm information to correspond to specific spatial locations, equipment objects, and associated areas within the park, thereby forming a holographic operational status that matches the physical park status.

[0017] Furthermore, this invention integrates and analyzes video recognition results, sensor detection results, equipment operating status, and historical handling data through multimodal artificial intelligence reasoning. This enables automatic identification, cross-validation, and tiered early warning of abnormal events, reducing the risk of missed or false alarms caused by relying on a single data source. Based on the spatial topology, floor structure, road traffic status, personnel distribution, and equipment status of the digital twin model, the impact range, associated risks, and handling paths of abnormal events can be visualized and simulated, providing managers with intuitive and accurate information for event assessment.

[0018] Furthermore, this invention utilizes a reinforcement learning-based collaborative command and dispatch mechanism. It comprehensively considers early warning levels, event types, impact ranges, resource locations, resource availability, and historical response results to generate task assignment and resource allocation schemes. This achieves closed-loop management of early warning, perception, assignment, response, evaluation, and improvement, enhancing the rationality of emergency resource allocation and the efficiency of cross-departmental collaborative response. Employing a microservices, containerization, and container orchestration architecture, each functional module can be deployed independently, scaled elastically, and updated on a rolling basis. This allows it to adapt to the concurrent access of tens of thousands of devices and real-time data updates, improving the system's scalability, stability, and ease of maintenance. Attached Figure Description

[0019] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a method for holographic situational awareness and collaborative command and dispatch in a smart park based on digital twin and multimodal AI reasoning, as disclosed in one embodiment of the present invention. Figure 2 This is a flowchart of a method for constructing a digital twin model of a park, as disclosed in one embodiment of the present invention; Figure 3 This is a flowchart of a method for multimodal data governance, spatiotemporal correlation, and holographic situation generation disclosed in one embodiment of the present invention; Figure 4 The flowchart below shows a method for abnormal event identification, early warning classification, and impact deduction according to one embodiment of the present invention. Figure 5 This is a flowchart of a collaborative processing scheme generation, task assignment, evaluation and strategy update method disclosed in one embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a smart park holographic situational awareness and collaborative command and dispatch system based on digital twin and multimodal AI reasoning, as disclosed in one embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] Figure 1 This is a flowchart illustrating a smart park holographic situational awareness and collaborative command and dispatch method based on digital twin and multimodal AI reasoning, provided as an embodiment of this application. Figure 1 As shown, this method can be executed by a smart park collaborative command and dispatch system, and specifically includes the following steps: S110. Acquire spatial and equipment data of the park and construct a digital twin model of the park.

[0025] The spatial data of the park may include one or more of the following: oblique photogrammetry data, Building Information Modeling (BIM) data, City Information Modeling (CIM) data, laser point cloud data, geographic information data, building floor plans, floor plans, and road topology data. Equipment data may include equipment codes, equipment types, equipment installation locations, equipment structural data, equipment operating parameters, equipment maintenance records, and equipment communication addresses.

[0026] In this embodiment, the digital twin model of the park is a three-dimensional mapping model of the physical park in digital space. This model is used not only to display the building exteriors, but also to represent the spatial and semantic relationships between buildings, floors, rooms, roads, pipelines, fire protection facilities, access control equipment, camera equipment, energy equipment, and other business objects within the park.

[0027] S120 connects to sensing devices and business systems within the park to collect multimodal data from the park.

[0028] The sensing devices may include at least one of the following: video surveillance equipment, audio pickup equipment, access control equipment, environmental sensors, fire sensors, smoke detectors, temperature and humidity sensors, positioning terminals, energy metering equipment, vehicle positioning equipment, parking management equipment, and equipment control terminals. The business systems may include at least one of the following: security management system, access control management system, fire management system, energy consumption management system, asset management system, equipment operation and maintenance system, work order management system, inspection management system, vehicle management system, and emergency response plan management system.

[0029] The multimodal data of the park may include at least two of the following: video image data, audio data, environmental monitoring data, access control data, personnel positioning data, vehicle positioning data, equipment status data, energy consumption data, fire alarm data, asset data, inspection data, work order data, contingency plan data, and historical event data.

[0030] For example, when a suspected fire occurs in the park, the system can simultaneously acquire smoke detection data, temperature data, video image data, fire equipment status, access control status, personnel location data, and nearby road traffic data of the target area, so as to avoid making event judgments based on data from a single sensor.

[0031] S130. Perform data governance on the multimodal data of the park, and then perform spatiotemporal correlation between the governed multimodal data of the park and the digital twin model of the park.

[0032] Data governance can include data cleaning, outlier removal, missing value imputation, format conversion, data standardization, data quality verification, data storage, data service encapsulation, access permission configuration, and data source identification configuration.

[0033] For example, video streams from different manufacturers can be uniformly converted into a preset encoding format; environmental sensor data with different sampling frequencies can be resampled according to a preset time window; and device status data lacking coordinate information can be retrieved from the device register based on the device code to obtain the corresponding installation location.

[0034] Spatiotemporal association includes configuring timestamps, spatial location identifiers, data source identifiers, and object identifiers for multimodal data in the park, and mapping the multimodal data in the park to spatial objects, equipment objects, or business objects in the park's digital twin model based on the timestamps, spatial location identifiers, and object identifiers.

[0035] Spatial location identifiers can be latitude and longitude coordinates, three-dimensional coordinates, building numbers, floor numbers, room numbers, road numbers, grid numbers, or area numbers. Object identifiers can be unique equipment codes, personnel identifiers, vehicle identifiers, asset identifiers, work order identifiers, or event identifiers.

[0036] S140. Perform fusion analysis and 3D visualization rendering on the spatiotemporally correlated multimodal data of the park to generate a holographic view of the park.

[0037] In this embodiment, the system overlays personnel trajectories, vehicle trajectories, equipment operating status, energy consumption status, alarm information, event information, and work order status onto the digital twin model of the park, and performs light field holographic rendering or real-time 3D rendering according to the display terminal parameters and the user's viewing angle to generate a holographic situational interface of the park.

[0038] The park's holographic status interface can display the building's external structure, floor distribution, room layout, equipment location, real-time operating status, personnel density, vehicle flow, alarm levels, and the scope of event impact. Users can also perform operations such as rotating, zooming, dividing, expanding floors, filtering areas, and viewing equipment structures on the building using a mouse, touch screen, voice commands, or virtual reality terminal.

[0039] S150. Based on the holographic situational awareness of the park, identify abnormal events, determine the warning level of the abnormal events, and perform impact simulations on the abnormal events.

[0040] Abnormal events can include intrusion, gathering of people, falls, abnormal loitering, vehicles driving in the wrong direction, traffic congestion, smoke and fire, abnormal sounds, equipment malfunctions, exceeding environmental parameters, abnormal energy consumption, abnormal fire-fighting equipment, and unauthorized access.

[0041] Specifically, the system can use a visual recognition model to analyze video image data to identify abnormal human behavior, abnormal vehicle behavior, smoke, intrusion, congestion, or abnormal equipment appearance; use a speech recognition model to analyze audio data to identify calls for help, sounds of breaking glass, explosions, abnormal impacts, or alarms; and use a time-series prediction model to predict equipment operation data, energy consumption data, and environmental monitoring data to identify equipment failure trends, abnormal energy trends, or abnormal environmental trends.

[0042] S160. Based on the warning level of the abnormal event, the impact simulation results, and the status of schedulable resources, generate a collaborative response plan, assign response tasks, and evaluate and update strategies based on the response process data and response result data.

[0043] Scheduled resources may include security personnel, property management personnel, equipment maintenance personnel, firefighters, medical rescue personnel, patrol vehicles, fire-fighting equipment, emergency supplies, broadcasting equipment, access control equipment, lighting equipment, smoke extraction equipment, and drones, etc. Resource status may include resource location, current task load, estimated arrival time, availability, professional capabilities, permission level, and communication status.

[0044] The system can select corresponding candidate plans from the plan library based on the event type and warning level, and then determine the resource allocation results, task execution sequence, disposal path and linkage equipment control instructions by combining the resource status and impact simulation results, and generate electronic work orders to send to the corresponding disposal terminals.

[0045] Figure 2 A flowchart illustrating a method for constructing a digital twin model of a park, as provided in an embodiment of this application. Figure 2 As shown, step S110 may include the following steps: S111. Acquire oblique photogrammetry data, BIM data, CIM data, laser point cloud data, and equipment data.

[0046] Among them, oblique photogrammetry data is used to reflect the real appearance of the park's buildings, roads, greenery and surrounding environment; BIM data is used to provide structured information on building components, floors, rooms, pipelines and equipment; CIM data is used to provide information on the relationship between the park and urban roads, public facilities and surrounding geographic space; and laser point cloud data is used to reflect the high-precision spatial outline of buildings, interior spaces and equipment objects.

[0047] S112. Perform real-scene 3D reconstruction on the oblique photogrammetry data, and perform denoising, registration, segmentation and surface reconstruction on the laser point cloud data.

[0048] In one implementation, multi-view image matching, dense point cloud generation, and texture mapping can be used to perform real-scene 3D reconstruction of oblique photogrammetry data to obtain real-scene 3D models of park buildings and sites.

[0049] For laser point cloud data, discrete noise can be removed first through statistical filtering, radius filtering, or voxel downsampling. Then, coordinate registration can be completed using iterative nearest-point algorithm, feature point matching algorithm, or marker point registration method. Afterward, point cloud segmentation is performed based on the normal vector, curvature, density, and spatial clustering relationship of the point cloud, and meshing and surface reconstruction are carried out to obtain a three-dimensional model of the building's interior space, equipment shell, or mechanical structure.

[0050] S113. Integrate real-world 3D models, point cloud reconstruction models, BIM data, and CIM data according to a unified coordinate benchmark.

[0051] In this embodiment, the park's geographic coordinate system can be used as a unified coordinate reference. For component coordinates in the BIM model, they can be converted to the park's geographic coordinate system using a coordinate transformation matrix; for real-world 3D models and point cloud models, spatial registration can be performed based on control points, building corner points, or road reference points.

[0052] S114. Establish spatial topological relationships and object semantic associations to obtain a digital twin model of the park.

[0053] Spatial topological relationships can include the inclusion relationship between buildings and floors, the inclusion relationship between floors and rooms, the connectivity relationship between roads and parking areas, the adjacency relationship between rooms and evacuation routes, and the positional relationship between equipment and installation areas.

[0054] The semantic association between objects can include the correspondence between building component identifiers, floor identifiers, room identifiers, equipment codes, equipment types, the system to which the equipment belongs, the communication address of the equipment, and the operating status of the equipment.

[0055] For example, if a camera device has the device code "CAM-01" and its installation location is "east side of the corridor on the third floor of Building A", the system can associate this device code with Building A, the third floor, the corridor area, and the corresponding three-dimensional coordinates. When the camera device detects an abnormal gathering of people, the system can directly locate the corresponding location in the park's digital twin model and query the surrounding access control, fire protection, lighting, and broadcasting equipment.

[0056] Figure 3 This is a flowchart illustrating a method for multimodal data governance, spatiotemporal correlation, and holographic situation generation provided in an embodiment of this application. Figure 3 As shown, steps S130 and S140 may include the following steps: S131. Perform unified access and protocol parsing for multimodal data in the park.

[0057] Different sensing devices and business systems can access the system through message queues, interface gateways, data buses, or edge computing nodes. Device communication protocols may include HTTP, MQTT, WebSocket, RTSP, ONVIF, Modbus, OPC UA, RESTful interfaces, or gRPC interfaces, etc.

[0058] S132. Clean, standardize, and verify the quality of multimodal data in the park.

[0059] For example, for environmental monitoring data, it can detect whether there are abnormal values ​​that exceed the sensor's range; for access control data, it can verify whether personnel identification, access control device identification, and passage time are complete; for device status data, it can verify whether the status code conforms to the preset device status dictionary; for video analysis results, it can verify whether the camera device code, detection time, and target coordinates are valid.

[0060] S133. Generate a unified event data structure for the governed data.

[0061] A unified event data structure can include at least the following: event identifier, data source identifier, object identifier, timestamp, spatial location identifier, data type, status value, confidence level, alarm level, and associated data index. The data source identifier indicates the device or business system from which the data originates; the confidence level indicates the reliability of the model's identification results or data judgment results.

[0062] S134. Based on time association, spatial association and object association, map multimodal data to the digital twin model of the park.

[0063] Time correlation can match data from different data sources according to a preset time window. For example, video recognition results, access control records, temperature change data, and equipment status change data within 30 seconds before and after the event can be used as correlated data for the same candidate event.

[0064] Spatial association can determine whether multiple data points correspond to the same spatial area based on 3D coordinates, building number, floor number, room number, or area number. Object association can determine the business relationship between data based on equipment code, personnel identifier, vehicle identifier, or asset identifier.

[0065] S135. Perform multimodal fusion analysis and 3D visualization rendering.

[0066] In one implementation, the system can configure fusion weights for different modalities of data based on data type and data quality. For the same anomalous event, if the video recognition result, smoke detection data, and temperature anomaly data all point to the same area, the overall confidence level of the anomalous event is increased; if only a single video recognition result exists and there is no other supporting data, the overall confidence level is decreased or the event is marked as requiring manual review.

[0067] In 3D visualization rendering, different colors, icons, dynamic effects, or transparency can be used to represent different object states. For example, normal equipment is marked in green, warning equipment in yellow, and faulty equipment in red; personnel trajectories are represented by dynamic lines; vehicle trajectories are represented by arrows; and the scope of an event's impact is represented by semi-transparent areas or heat maps.

[0068] Figure 4 A flowchart illustrating an abnormal event identification, early warning classification, and impact deduction method provided in this application embodiment. Figure 4 As shown, step S150 may include the following steps: S151. Identify candidate abnormal events through a multimodal artificial intelligence model.

[0069] Visual recognition models can employ object detection models, behavior recognition models, semantic segmentation models, or multi-object tracking models to identify people, vehicles, smoke, equipment appearance, and abnormal behavior. Speech recognition models can perform keyword recognition, voiceprint analysis, abnormal sound classification, or semantic understanding on audio data. Temporal prediction models can utilize recurrent neural networks, long short-term memory networks, Transformer models, or statistical prediction models to predict trends in equipment parameters, energy consumption data, temperature and humidity data, and fire safety parameters.

[0070] S152. Identify anomalous events based on the confidence level, temporal correlation, and spatial correlation of the model output results.

[0071] For example, within the same spatial area, if the video model detects smoke, the smoke sensor triggers an alarm, and the temperature data continues to rise, this information can be merged into a candidate fire event. If the video model only detects suspected smoke, but the smoke sensor and temperature data are not abnormal, the event can be marked as a low-confidence warning and pushed to the manual review terminal.

[0072] S153. Calculate the risk score R for abnormal events.

[0073] In this embodiment, the risk score R can be calculated according to the following formula: R=w1S+w2N+w3A+w4T+w5D+w6I+w7C; Where S is the normalized value of the event severity; N is the normalized value of the number of affected objects; A is the normalized value of the spatial range of the impact; T is the normalized value of the event duration; D is the normalized value of the event development trend; I is the normalized value of the importance of the associated equipment; C is the confidence level of the model inference; w1 to w7 are the weight coefficients corresponding to the above parameters, and the sum of w1 to w7 is 1.

[0074] Event severity can be determined based on the event type. For example, higher severity can be configured for fires, personal injuries, and fire equipment failures; lower severity can be configured for general equipment alarms. The importance of associated equipment can be determined based on the importance of the equipment to the security or operation of the park. For example, fire alarm control panels, power distribution equipment, core network equipment, or important entrance and exit access control equipment can be assigned higher importance.

[0075] S154. Compare the risk score R with the preset grading threshold to obtain the warning level.

[0076] For example, when R is less than the first threshold, it is determined as a warning level; when R is greater than or equal to the first threshold and less than the second threshold, it is determined as a general warning; when R is greater than or equal to the second threshold and less than the third threshold, it is determined as a higher warning; and when R is greater than or equal to the third threshold, it is determined as an emergency warning. Each threshold can be configured according to the park type, historical event records, and management requirements.

[0077] S155. Impact projection based on the digital twin model of the park.

[0078] Specifically, the system determines the location and associated objects of abnormal events, and simulates the impact range, risk changes and resource access status of abnormal events within a preset time window based on the building structure, spatial topology, personnel distribution, equipment status, road traffic status and resource location in the digital twin model of the park.

[0079] For example, in the event of a fire, the system can combine the location of the fire source, wind direction, temperature, building floor structure, evacuation route location, and personnel distribution to output potentially affected areas, recommended evacuation routes, areas to be isolated, fire rescue routes, and a list of smoke extraction equipment, broadcasting equipment, and access control equipment that should be activated. In the event of traffic congestion, the system can combine road topology, traffic density, parking space availability, and vehicle location to output alternative routes and traffic management suggestions.

[0080] Figure 5 This is a flowchart illustrating a method for generating a collaborative processing plan, assigning tasks, evaluating performance, and updating strategies, as provided in an embodiment of this application. Figure 5 As shown, step S160 may include the following steps: S161. Select candidate contingency plans from the contingency plan database based on the type of abnormal event and the warning level.

[0081] The contingency plan database can store contingency plans for fire, personnel intrusion, equipment failure, traffic congestion, extreme weather, abnormal energy consumption, and public safety incidents. Each plan may include the applicable event type, applicable warning level, response steps, responsible department, resource requirements, instructions for coordinated equipment, time limits, and escalation conditions.

[0082] S162. Form a resource candidate set based on the resource status.

[0083] The system can calculate the distance, estimated arrival time, current task load, availability, professional capabilities, and permission levels between schedulable resources and the location of abnormal events, and form a resource candidate set. The estimated arrival time can be calculated based on the resource's current location, road traffic status, elevator status, floor structure, and historical movement speed.

[0084] S163. Use reinforcement learning scheduling models to determine resource allocation results, task execution order, disposal path, and control instructions for linked equipment.

[0085] The state information of the reinforcement learning scheduling model can include the current park situation, abnormal event attributes, candidate resource status, resource spatial location, contingency plan rules, road traffic status, equipment status, and historical handling results. Scheduling actions can include selecting resources for handling, assigning tasks, adjusting task priorities, planning handling paths, controlling linked equipment, and triggering contingency plan upgrades.

[0086] In this embodiment, the reward value Q can be calculated according to the following formula: Q=a1E+a2F+a3U-a4L-a5P; Where E is the reduction value of the event risk score before and after the handling; F is the task completion degree; U is the compliance degree of the plan execution; L is the normalized value of the event response time; P is the normalized value of the resource allocation cost; a1 to a5 are the weight coefficients corresponding to the above parameters, and the sum of a1 to a5 is 1.

[0087] The greater the reduction in event risk score before and after handling, the higher the task completion rate, and the higher the compliance rate with the contingency plan, the greater the reward value; conversely, the longer the event response time or the higher the resource allocation cost, the smaller the reward value. The system can iteratively update the scheduling strategy based on the reward value, enabling subsequent similar events to obtain better resource allocation solutions.

[0088] S164. Generate an electronic work order and dispatch it to the corresponding processing terminal.

[0089] Electronic work orders may include task identifier, event identifier, handling location, handling steps, time limit, responsible party, task priority, recommended route, on-site photos or videos, associated equipment information, and linkage control commands. The handling terminal can be a mobile terminal, intercom terminal, vehicle terminal, duty workstation, or smart wearable device.

[0090] For example, when the system detects a high-level fire on a certain floor, it can assign corresponding tasks to security personnel, firefighters, and property management personnel respectively; at the same time, it can issue evacuation broadcast instructions to the broadcasting equipment, issue emergency opening instructions to the access control equipment, issue start instructions to the smoke exhaust equipment, and display rescue routes, evacuation routes, and resource locations in the park's holographic situation interface.

[0091] S165. Conduct evaluations and update strategies based on data from the handling process and the handling results.

[0092] The data for the handling process can include event confirmation time, early warning generation time, task assignment time, resource response time, resource arrival time, handling completion time, equipment linkage execution status, and work order feedback information. The data for the handling result can include whether the event has been eliminated, the degree of risk score reduction, task completion rate, compliance with the contingency plan, and human evaluation data.

[0093] The system can generate evaluation results based on the above data, and update contingency plan configurations, resource capability tags, scheduling strategies, risk score weights, and model training data according to the evaluation results. For example, if a certain type of event frequently experiences delayed warnings, the weight of the corresponding event development trend parameter can be increased; if a resource fails to arrive within the scheduled time multiple times, the estimated arrival time parameter of that resource can be corrected or its scheduling priority can be reduced.

[0094] refer to Figure 6This application also provides a smart park holographic situational awareness and collaborative command and dispatch system based on digital twins and multimodal AI inference. This system can be used to execute the methods of any of the above embodiments, including a modeling module, a data acquisition module, a governance module, a situational awareness module, an analysis module, and a dispatch module.

[0095] The modeling module acquires spatial and equipment data for the park and constructs a digital twin model of the park; the acquisition module connects to the park's sensing devices and business systems to collect multimodal data; the governance module cleans, standardizes, verifies the quality of, encapsulates, and spatiotemporally correlates the multimodal data; the situational awareness module performs multimodal fusion analysis and 3D visualization rendering to generate a holographic situational awareness of the park; the analysis module identifies abnormal events, calculates risk scores, determines warning levels, and performs impact simulations; and the scheduling module generates collaborative response plans, issues electronic work orders, controls linked equipment, evaluates responses, and updates scheduling strategies.

[0096] In one implementation, the above modules can be deployed using a microservice architecture. The modeling service, data access service, data governance service, AI inference service, 3D rendering service, contingency plan service, scheduling service, and work order service can each run in an independent container and communicate via REST interfaces, gRPC interfaces, or message queues. The containers can be orchestrated using Kubernetes to enable automatic service scaling, health checks, fault migration, and rolling updates.

[0097] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it causes the processor to perform the smart park holographic situational awareness and collaborative command and dispatch method based on digital twin and multimodal AI reasoning as described in any of the above embodiments.

[0098] Computer-readable storage media may include, but are not limited to, read-only memory, random access memory, magnetic disk, optical disk, flash memory, solid-state drive, mobile storage device, or other media capable of storing program instructions. The processor may be a general-purpose processor, central processing unit, graphics processing unit, artificial intelligence chip, field-programmable gate array, or application-specific integrated circuit.

[0099] Finally, it should be noted that the holographic situational awareness and collaborative command and dispatch method and system for smart parks based on digital twins and multimodal AI reasoning disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for holographic situational awareness and collaborative command and dispatch in smart parks based on digital twins and multimodal AI reasoning, characterized in that, include: Acquire spatial data and equipment data of the park, process the spatial data and equipment data of the park, and construct a digital twin model of the park; Connect to sensing devices and business systems within the park to collect multimodal data from the park; Data governance is performed on the multimodal data of the park, and the governed multimodal data of the park is spatiotemporally correlated with the digital twin model of the park; The spatiotemporally correlated multimodal data of the park are fused, analyzed, and rendered in three dimensions to generate a holographic view of the park. Based on the holographic situational awareness of the park, the warning level of the abnormal event is determined, and the impact of the abnormal event is extrapolated. Based on the warning level of the abnormal event, the impact projection results, and the status of schedulable resources, a collaborative response plan is generated, response tasks are assigned, and evaluation and strategy updates are performed based on the response process data and response result data.

2. The method for holographic situational awareness and collaborative command and dispatch of smart parks based on digital twins and multimodal AI reasoning as described in claim 1, characterized in that, The park spatial data includes oblique photogrammetry data, building information model data, city information model data, and laser point cloud data; the equipment data includes equipment codes, equipment structural data, equipment installation location data, and equipment operating parameters. The construction of the digital twin model of the park includes: The oblique photography data is used to perform real-scene 3D reconstruction, and the laser point cloud data is denoised, registered, segmented and surface reconstructed. The reconstruction results are fused with the building information model data and city information model data according to a unified coordinate reference. Based on spatial topology, building component identification, floor identification and equipment coding, the semantic association relationship of objects is established to obtain the digital twin model of the park.

3. The method for holographic situational awareness and collaborative command and dispatch of smart parks based on digital twins and multimodal AI reasoning as described in claim 1, characterized in that, The multimodal data of the park includes at least two of the following: video image data, audio data, environmental monitoring data, access control data, personnel positioning data, vehicle positioning data, equipment status data, energy consumption data, fire protection data, asset data, inspection data, work order data, and contingency plan data. The sensing devices include at least one of video surveillance equipment, audio pickup equipment, access control equipment, environmental sensors, fire sensors, positioning terminals, energy metering equipment, and equipment control terminals.

4. The method for holographic situational awareness and collaborative command and dispatch of smart parks based on digital twins and multimodal AI reasoning as described in claim 1, characterized in that, The data governance includes data cleaning, data standardization, data quality verification, data storage, data service encapsulation, and permission identification configuration; The spatiotemporal association includes: configuring timestamps, spatial location identifiers, data source identifiers, and object identifiers for the multimodal data of the park; and mapping the multimodal data of the park to spatial objects, equipment objects, or business objects in the digital twin model of the park based on the timestamps, spatial location identifiers, and object identifiers.

5. The method for holographic situational awareness and collaborative command and dispatch of smart parks based on digital twins and multimodal AI reasoning as described in claim 1, characterized in that, The generation of the park's holographic situation includes: The digital twin model of the park overlays personnel trajectories, vehicle trajectories, equipment operating status, energy consumption status, alarm information, and event information. Based on the display terminal parameters and viewing angle, perform light field holographic rendering or real-time 3D rendering, and output a holographic status interface of the park that includes spatial location, object attributes, real-time status and alarm indicators; The digital twin model of the park supports the subdivision of building structures, the unfolding of floors, and the display of the internal structure of equipment.

6. The method for holographic situational awareness and collaborative command and dispatch of smart parks based on digital twins and multimodal AI reasoning as described in claim 1, characterized in that, The identification of abnormal events includes: Visual recognition models are used to identify abnormal human behavior, abnormal vehicle behavior, smoke, intrusion, congestion, or abnormal equipment appearance. Speech recognition models are used to identify help requests, abnormal sounds, or alarm voices. Time series prediction models are used to predict abnormal trends in equipment operation data, energy consumption data, or environmental monitoring data. The abnormal events are determined based on the confidence level, temporal correlation, and spatial correlation of the output results of each model. Determining the warning level of the abnormal event includes calculating an event risk score R based on the event severity, the number of affected objects, the spatial range of the impact, the event duration, the event development trend, the importance of associated equipment, and the confidence level of the model inference. The event risk score R satisfies the following: R=w1S+w2N+w3A+w4T+w5D+w6I+w7C; Where S is the normalized value of event severity, N is the normalized value of the number of affected objects, A is the normalized value of the affected spatial range, T is the normalized value of event duration, D is the normalized value of event development trend, I is the normalized value of the importance of associated equipment, C is the model inference confidence, and w1 to w7 are the weight coefficients corresponding to S, N, A, T, D, I and C, respectively, and the sum of w1 to w7 is 1. The event risk score R is compared with a preset grading threshold to obtain the corresponding warning level.

7. The method for holographic situational awareness and collaborative command and dispatch of smart parks based on digital twins and multimodal AI reasoning as described in claim 1, characterized in that, The impact deduction of the abnormal event includes: The location and associated objects of the abnormal event are determined. Based on the spatial topology, personnel distribution, equipment status, road traffic status and resource location in the digital twin model of the park, the impact range, risk changes and resource access of the abnormal event within a preset time window are simulated, and at least one of the following projection results is output: evacuation route, isolation area, rescue route, list of linked equipment and disposal priority.

8. The method for holographic situational awareness and collaborative command and dispatch of smart parks based on digital twins and multimodal AI reasoning as described in claim 1, characterized in that, The generation of the collaborative response plan includes: selecting candidate plans from the plan library according to the event type and warning level of the abnormal event; forming a resource candidate set according to the distance between the schedulable resources and the location of the abnormal event, the estimated arrival time, the current task load, the availability status, the professional capabilities and the permission level; and using a reinforcement learning scheduling model to determine the resource allocation results, the task execution order, the response path and the control instructions for the linked equipment. The reinforcement learning scheduling model determines scheduling actions based on the current park situation, the attributes of the abnormal event, the status of candidate resources, the spatial location of resources, contingency rules, and historical handling results, and updates the scheduling strategy according to the reward value Q, wherein the reward value Q satisfies: Q=a1E+a2F+a3U-a4L-a5P; Where E is the reduction value of the event risk score before and after handling, F is the task completion degree, U is the compliance degree of the plan execution, L is the normalized value of the event response time, P is the normalized value of the resource allocation cost, and a1 to a5 are the weight coefficients corresponding to E, F, U, L and P respectively, and the sum of a1 to a5 is 1. The dispatching of disposal tasks includes generating an electronic work order containing a task identifier, event identifier, disposal location, disposal steps, time limit requirements, responsible party, and linkage instructions, and sending the electronic work order to the corresponding disposal terminal; The evaluation includes generating evaluation results based on event confirmation time, early warning generation time, task assignment time, resource arrival time, handling completion time, contingency plan execution compliance, task completion rate, and manual evaluation data, and updating contingency plan configuration, scheduling strategy, or model training data based on the evaluation results.

9. A smart park holographic situational awareness and collaborative command and dispatch system based on digital twins and multimodal AI reasoning, applied to the smart park holographic situational awareness and collaborative command and dispatch method based on digital twins and multimodal AI reasoning as described in any one of claims 1-8, characterized in that, include: The modeling module is used to acquire park spatial data and equipment data, process the park spatial data and equipment data, and construct a digital twin model of the park. The data acquisition module is used to connect to the sensing devices and business systems within the park and collect multimodal data from the park. The governance module is used to perform data governance on the multimodal data of the park and to spatiotemporally correlate the governed multimodal data of the park with the digital twin model of the park. The situation module is used to perform fusion analysis and 3D visualization rendering of spatiotemporally correlated multimodal data of the park to generate a holographic situation of the park. The analysis module is used to identify abnormal events based on the holographic situation of the park, determine the warning level of the abnormal events, and perform impact deduction on the abnormal events; The scheduling module is used to generate a collaborative handling plan, dispatch handling tasks, and evaluate and update strategies based on the warning level, impact projection results, and schedulable resource status of the abnormal event.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the smart park holographic situational awareness and collaborative command and dispatch method based on digital twin and multimodal AI reasoning as described in any one of claims 1 to 8.

Images