A multimodal property event recognition and global dispatch AI collaborative system

Abstract

The application discloses a kind of multimodal property event identification and global scheduling AI collaborative system, it is related to artificial intelligence and wisdom property management technical field, the system includes: perception understanding module fusion multimodal data generates structured description and cognitive index, instruction generation module retrieves historical cases and generates natural language scheduling instruction and mask, dispatches recovery module and recycles image binding mask, visual verification module compares benchmark image and calculates recovery deviation to determine state, decision archiving module is closed or secondary dispatch according to state and generates report;The application is unified by fusion multimodal data representation and quantifies risk, combined with semantic mask focusing verification area, improves identification and scheduling efficiency;By benchmark image comparison dynamic determination disposal state, according to event type difference fault tolerance, generate secondary instruction and archiving report, realize closed-loop control, reduce artificial verification cost, accumulate case and continuously optimize scheduling response capability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and smart property management technology, specifically to a multimodal property event recognition and full-domain scheduling AI collaborative system. Background Technology

[0002] As residential communities and commercial parks continue to expand, their managed areas are increasingly equipped with video surveillance and environmental sensors. Meanwhile, residents are increasingly using various channels such as WeChat private chats, WeChat group chats, 400 numbers, and landlines to report incidents online via voice and text. This generates a continuous stream of event data in three distinct formats: visual, sensor-based, and text-based. Emergencies such as fires, flooding, facility malfunctions, and security anomalies within these communities are characterized by their suddenness and simultaneous triggering by multiple clues. Data from a single source is insufficient to fully reconstruct the entire event. Relying solely on camera footage is susceptible to obstruction and lighting interference; relying solely on sensor readings cannot confirm the actual scene; and relying solely on verbal accounts from residents is prone to vague descriptions and location discrepancies. The industry is gradually introducing digital work order systems to handle various incident reports. However, most platforms can only store and display single types of data in a simple way, lacking the ability to uniformly convert and integrate cross-modal data. Various monitoring information forms data silos, making it impossible to automatically integrate clues from multiple channels to form standardized incident information. This makes it difficult to support the dispatch process in quickly and accurately matching resources for handling. In particular, it is impossible to obtain the current location of property staff in real time, resulting in blind dispatching or reliance on manual telephone confirmation during dispatching. It is difficult to ensure that the nearest personnel arrive at the scene as soon as possible. The handling of incidents on-site still relies heavily on staff to manually summarize and judge the severity of incidents. The overall level of automation in the incident handling chain is low.

[0003] The existing property incident handling system suffers from multiple technical shortcomings. First, multi-source data cannot achieve unified semantic mapping. Visual, sensor, and text data use independent parsing logics, and the system cannot automatically align information from different channels for the same event. It must rely on manual verification of equipment alarms, monitoring footage, and resident feedback, which is time-consuming and prone to information omissions. Second, traditional dispatch instructions rely solely on manual writing, lacking targeted labeling of key on-site verification areas and failing to intelligently dispatch orders based on real-time personnel locations. This often results in employees from distant locations being sent to the scene, delaying response. Furthermore, there are no standardized shooting guidelines for on-site verification images, leading to disorganized and unfocused uploaded images. Subsequent verification personnel must review all images frame by frame, resulting in low verification efficiency. Simultaneously, the existing solution lacks a standardized scene comparison verification mechanism. Work order completion relies entirely on subjective human judgment, with inconsistent standards among different staff, making it impossible to accurately distinguish between fully compliant, partially unrectified, and remaining potential hazards. For unrectified work orders, only general notifications can be repeatedly issued, failing to pinpoint specific non-compliant locations and easily leading to multiple invalid dispatches. In addition, the data throughout the entire process is stored in different functional modules. After the incident is handled, various materials need to be sorted out manually. It is impossible to automatically generate a complete time-series handling file, which is not conducive to reviewing and optimizing similar incidents. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a multimodal property event recognition and full-domain dispatch AI collaborative system. This system collects multimodal data, maps it to a unified semantic space to generate structured descriptions and event recognition indices, retrieves historical cases to generate natural language dispatch instructions and semantic attention mask matrices, and combines this with real-time employee location information to achieve precise, nearby task assignment. This data is then pushed to the executor's terminal and multi-angle image data is collected. A visual verification module retrieves pre-event baseline images, uses a visual judgment algorithm for semantic comparison, calculates scene recovery deviation values, and outputs the task completion status. A decision archiving module closes the work order or triggers a secondary dispatch based on the completion status, integrating all process data to generate a handling report.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal property event recognition and full-domain scheduling AI collaborative system, comprising: Perception and Understanding Module: Collects video stream data, IoT sensor alarm data, and voice notification text data, and maps them to a unified semantic space to generate a structured semantic description of the event scene and outputs an event cognition index; Instruction generation module: Receives structured semantic description and event cognition index, retrieves historical handling cases and operating procedures, generates natural language dispatch instructions containing handling actions, tool types, target locations and safety prompts, and constructs a semantic attention mask matrix; The dispatch and recycling module receives natural language dispatch instructions and sends them to the executor's terminal device. It also receives multi-angle image data of the on-site handling uploaded by the executor through the terminal device, binds the semantic attention mask matrix with the multi-angle image data, and then transmits it. Visual verification module: Receives semantic attention mask matrix and multi-angle image data, retrieves reference image data of the same scene before the event occurred, uses command-line visual judgment algorithm to perform semantic comparison between multi-angle image data and reference image data, calculates scene restoration deviation value and outputs task completion status; Decision archiving module: Receives task completion status. When the task completion status is "completed", the work order is closed. When the task completion status is "partially completed" or there are potential risks, a second task dispatch is triggered. At the same time, it integrates all data in the entire process to generate an event handling report.

[0006] Furthermore, in the perception and understanding module, the video stream data includes a continuous sequence of video frames collected by fixed cameras within the property area, along with the timestamp and spatial coordinates of each video frame. The video frame sequence contains visual features of flame targets, smoke targets, and personnel behavior trajectories. The IoT sensor alarm data includes smoke concentration values ​​reported by smoke detectors, real-time temperatures reported by temperature detectors and their positive deviation from the baseline temperature, real-time humidity reported by humidity detectors and their negative deviation from the baseline humidity, as well as the device number, installation location coordinates, and alarm timestamp of each detector. The voice reporting text data consists of audio uploaded by residents through WeChat one-on-one chats, WeChat group chats, 400 telephones, landlines, etc., which is processed by voice-to-text and includes a description of the location of the event, a description of the event phenomenon, and a reporting timestamp.

[0007] Furthermore, the unified semantic space is used to converge the original feature vectors of video stream data, IoT sensor alarm data, and voice reporting text data into the same high-dimensional representation space after semantic mapping, so that semantic units from different modalities that describe the same event phenomenon have similar vector distances; the structured semantic description is a structured data body containing event type labels, event location coordinates, event severity level, affected spatial range, and on-site environmental state description, and each field establishes an index mapping relationship with the coordinate point set in the unified semantic space.

[0008] Furthermore, in the perception and understanding module, the event cognition index is calculated using the formula... The calculation shows that E represents the event cognition index, which ranges from 0 to 1. A higher value indicates a higher severity of the event and a higher degree of confidence in the system's cognition. Represents the confidence weight of the visual modality. Represents the sensor modal confidence weights. The text modality confidence weights are represented by α, β, and γ, where the sum of these three values ​​equals 1. This represents the preset comprehensive weighting coefficient of visual hazard features, which includes the judgment criteria for flame outline features and smoke texture features. This represents the preset comprehensive weighting coefficient of sensor hazard characteristics, which includes the temperature change slope judgment benchmark and the smoke concentration peak judgment benchmark. This represents the preset comprehensive weighting coefficient of textual hazard features, which includes matching templates for charred and exploded keywords; This indicates that the i-th visual hazard feature value extracted from the current video frame is compared with... The normalized matching degree corresponding to the judgment benchmark. ,in This represents the i-th visual hazard feature vector extracted from the current video frame. express The preset baseline feature vector corresponding to the visual feature has a value range of 0 to 1; m is the total number of extracted visual hazard features; This indicates the magnitude by which the current reading of the j-th type of sensor deviates from its baseline value. The normalized ratio corresponding to the judgment criterion, , This represents the real-time reading of the j-th type of sensor, such as the current smoke concentration or current temperature. This represents the baseline value of the j-th type of sensor under normal operating conditions, such as the background smoke concentration value or the ambient temperature reference value under normal conditions. This represents the preset alarm threshold corresponding to the j-th type of sensor, i.e. The threshold corresponding to the judgment benchmark is used to trigger an alarm when the deviation reaches the threshold. This means taking the smaller value between the number inside the parentheses and 1, ensuring... Not exceeding 1; n is the total number of sensor types that trigger the deviation alarm; θ represents the semantic negative score of the voice report text data; τ represents the occurrence of [something] in the text data. The cumulative number of times the keyword is considered dangerous; (1+τ) represents logarithmic compression of τ, used to reduce the marginal contribution increment of high-frequency keywords to the event cognition index.

[0009] Furthermore, the instruction generation module uses the event type and event location in the structured semantic description as a joint retrieval index to retrieve standard operating procedures, essential tool lists, and safety precautions matching the event type from the historical maintenance knowledge base. It then concatenates the retrieval results with the structured semantic description to form an enhanced context, generating natural language dispatch instructions that conform to a predetermined sentence template. These instructions consist of a sequential combination of tool carrying requirements, target location indication, action sequence, and safety warning prompts. The semantic attention mask matrix extracts the target object nouns semantically indicated by verbs from the natural language dispatch instructions, marks the visual semantic dimensions corresponding to these target object nouns as attention dimensions, and marks the visual semantic dimensions corresponding to non-target object nouns as mask dimensions, thereby generating a binary semantic attention mask matrix.

[0010] Furthermore, when the dispatch and retrieval module pushes natural language dispatch instructions to the executor's terminal device, it first obtains the location information reported by all on-duty executor terminals. The location information includes GPS location data and WiFi location data. In outdoor environments, GPS location is prioritized, while WiFi location is used indoors or in areas where GPS signals are blocked. The module calculates the spatial straight-line distance or path distance between the coordinates corresponding to each location information and the location where the event occurred, selects the one with the smallest distance as the final dispatch target, and uses the terminal identifier of that target as the push target. At the same time, it pushes the semantic attention mask matrix to the local storage of the terminal device. After receiving the multi-angle image data uploaded by the executor after on-site handling, it performs clarity detection and illumination sufficiency detection on each image data. Image data with clarity lower than the resolution of 480×480 pixels and an average illumination value lower than the grayscale value of 30 are returned to the executor's terminal device and prompted to be reshot. Image data that meets all quality conditions are classified and stored according to the shooting angle. Then, according to the work order number, the classified and stored multi-angle image data is bound to the locally stored semantic attention mask matrix and transmitted to the visual verification module.

[0011] Furthermore, the multi-angle image data includes image files collected by the personnel following the shooting instructions displayed on the terminal device screen, from multiple preset perspectives around the disposal site. These preset perspectives include a frontal view of the disposal area, a left-side view of the disposal area, a right-side view of the disposal area, and a close-up view of key parts of the disposal area. The image files are in photo or short video format, and each image file includes a shooting angle label, a shooting timestamp, and the personnel identification. The semantic attention mask matrix is ​​bound to the multi-angle image data by using the work order number as the association key value, writing the binary data of the semantic attention mask matrix into the extended metadata field of each image file, so that each image file under the same work order carries the same semantic attention mask matrix and is then transmitted to the visual verification module together.

[0012] Furthermore, the visual verification module determines the monitoring device number based on the target location coordinates in the structured semantic description, and then traces back a preset time period based on the event timestamp. The preset time period is a time window of 1 to 5 minutes before the event timestamp. Within the time window, the image frame closest to the event time and without abnormal personnel gathering or equipment abnormality is preferentially captured as the reference image data. If the monitoring device has no recording records due to malfunction before the event, the cross-view images of adjacent monitoring devices on the same floor at the same time are used as the substitute reference image data, and the extracted reference scene semantic feature vector is normalized and calibrated based on the view difference of the substitute reference image data.

[0013] Furthermore, in the visual verification module, the mathematical expression for the instruction visual judgment algorithm is: In the formula, Δ represents the scene recovery deviation value; This represents the semantic feature vector of the baseline scene extracted from the baseline image data. It is a historical image of the same scene under normal conditions, taken by a fixed monitoring device before the event occurred. This represents the semantic feature vector of the processed scene extracted from the processed image data from the k-th angle, where K is the total number of multi-angle image data; ⊙ represents the selective attention operator, and σ represents the semantic attention mask matrix. The selective attention operation means setting the feature values ​​in the processed scene semantic feature vector that correspond to the masked dimensions marked in the semantic attention mask matrix to zero, and only retaining the feature values ​​marked as the attention dimensions to participate in subsequent calculations. λ represents the Euclidean distance norm, used to calculate the total semantic difference between the mean-processed post-processing scene semantic feature vector and the baseline scene semantic feature vector; λ represents the preset basic tolerance coefficient, which is a real number greater than 0; E is the event cognition index, and ln(1+E) represents the logarithmic operation with the natural constant e as the base, used to map the event cognition index to a dynamic tolerance adjustment factor. The product of the event cognition index and the basic tolerance coefficient constitutes the tolerable deviation dynamic threshold. The higher the event cognition index, the larger the tolerable deviation dynamic threshold.

[0014] Furthermore, after the visual verification module calculates the scene restoration deviation value Δ, it determines that the scene restoration is complete when Δ≤0, partially complete when Δ>0 and Δ≤1 preset threshold, and has remaining hidden dangers when Δ>1 preset threshold. The first preset threshold is a deviation tolerance upper limit value built into the decision archiving module, which is set according to the event type. The deviation tolerance upper limit value is 0.35 for fire events, 0.30 for water immersion events, 0.25 for security events, and 0.20 for cleaning and maintenance events. Based on the event type label in the structured semantic description, the deviation tolerance upper limit value corresponding to the event type label is retrieved as the first preset threshold. The deviation tolerance upper limit value is obtained by statistical analysis and calibration of the image data after the handling of each type of historical event. Specifically, the distribution of scene restoration deviation values ​​after the standard handling of each type of event is collected, and the upper limit of its 95% confidence interval is taken as the deviation tolerance upper limit value of the event type.

[0015] Furthermore, when the decision archiving module triggers a secondary dispatch, it obtains detailed feature difference data for each semantic dimension calculated during the current verification process from the visual verification module. This detailed feature difference data is a set of difference vectors calculated item by item along each dimension of the semantic feature space by the visual verification module when comparing the semantic feature vector of the scene after the treatment with the semantic feature vector of the baseline scene. Each element in this set of difference vectors corresponds to a difference value for a semantic dimension. The semantic dimensions include the flame presence dimension, smoke concentration dimension, power switch opening and closing status dimension, evacuation route unobstructed dimension, and water accumulation dimension. Based on the detailed feature difference data, the specific objects and spatial locations that do not meet the judgment criteria are located and filled into a predetermined text template of supplementary instructions or emergency instructions dynamically matched according to the event type and event cognition index in the structured semantic description, generating a differentiated secondary dispatch instruction. The differentiated secondary dispatch instruction includes a confirmation list of items completed in the previous treatment and operation instructions for items that need to be supplemented in this treatment. The executor's terminal device simultaneously displays a thumbnail of the previously uploaded image data and a schematic diagram of the items that need to be supplemented in this treatment.

[0016] Furthermore, in the decision archiving module, the data integrated after the work order is closed includes: the structured semantic description and event cognition index output by the perception and understanding module, the natural language scheduling instructions and semantic attention mask matrix output by the instruction generation module, all image data received by the dispatch and retrieval module and their quality inspection records, the scene restoration deviation value and task completion status output by the visual verification module, and all timestamps and personnel identification of the previous and second dispatches. The event handling report includes a text description and an image evidence attachment. The text description describes the event discovery process, the dispatch instruction issuance process, the on-site handling process, and the quality verification process in chronological order of the event occurrence. The image evidence attachment arranges a comparison group of the baseline image before handling and the image after handling in chronological order of shooting time.

[0017] Compared with existing technologies, this multimodal property event recognition and full-domain dispatch AI collaborative system has the following advantages: I. This invention constructs a unified semantic space by integrating three types of heterogeneous data: video, IoT sensing, and resident voice reports. This enables the unified representation of multimodal property event information, and quantifies the comprehensive risk level of events based on multimodal weighted calculation logic. It addresses the industry pain points of fragmented data formats and inconsistent risk assessment standards from different sources. It eliminates the need for manual sorting of various scattered reported information, and can autonomously complete standardized and structured output of event location, type, and scope of harm, supporting precise matching of handling solutions in the backend scheduling process. Simultaneously, it generates a semantic attention mask matrix to lock onto the core verification area on-site, focusing visual attention on the core problem points and avoiding interference from irrelevant scene information in the verification process. This reduces the manpower cost of manually screening images and verifying on-site issues, enabling fully automated flow of property emergencies from information collection to risk assessment, improving recognition response speed and information integration accuracy.

[0018] II. This invention establishes a visual verification logic based on benchmark image comparison, combined with a dynamic fault-tolerance adjustment mechanism to automatically determine the completion level of work orders, replacing the traditional manual verification mode of checking each on-site photo one by one. It matches differentiated deviation tolerance standards according to different event types, distinguishing between three work order statuses: fully resolved, partially resolved, and remaining hidden dangers. For work orders that do not meet the standards, it automatically extracts semantic dimension difference details, generates targeted secondary dispatch instructions, accurately marks unrectified points and supplementary operation requirements, and avoids duplicate and invalid dispatches. All data collected, dispatched, verified, and dispatched throughout the entire process are uniformly collected and archived, automatically generating a standardized handling report with complete time sequence and accompanying image comparison materials. This achieves closed-loop management of property work orders and accumulates standardized reference cases for the handling of similar events in the future, continuously optimizing the accuracy of overall dispatch matching and effectively reducing the cost of oversights and repetitive operations in property operation and maintenance.

[0019] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a framework diagram of a multimodal property event recognition and full-domain dispatch AI collaborative system; Figure 2 A flowchart of a multimodal property event recognition and full-domain dispatch AI collaborative system; Figure 3 This is a flowchart of the visual verification module of the present invention. Detailed Implementation

[0022] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0023] Example 1: This system enables fully automated handling of fire incidents involving open flames and smoke diffusion in the public corridors of high-rise residential buildings. Each floor of the building is equipped with high-definition panoramic surveillance cameras, combined smoke detection devices, and temperature sensors. The community's resident service mini-program includes a built-in voice reporting function. Residents can also report incidents via WeChat private chat, WeChat group chat, 400 telephone, and landline. The entire system relies on a perception and understanding module, an instruction generation module, a dispatch and retrieval module, a visual verification module, and a decision archiving module, sequentially connecting to complete a closed-loop work order system. All business processes utilize multimodal data fusion logic to achieve unattended automatic flow, fully covering the entire chain of business needs, including fire early warning, task dispatch, on-site rectification, effect verification, and data archiving. Figure 1 As shown.

[0024] The perception and understanding module simultaneously initiates a multi-source data acquisition operation across the entire domain, continuously receiving continuous video streams from fixed cameras in buildings and corridors, simultaneously retrieving standardized alarm data uploaded in real time by smoke and temperature sensors within the corridors, and receiving text reports from residents via a property management mini-program (recorded via voice-to-text), as well as voice or text reports from WeChat, 400 telephone numbers, and other channels. The module simultaneously performs feature extraction from these three types of heterogeneous data: extracting visual features corresponding to flame outlines and smoke textures from the video stream; extracting numerical features related to temperature deviation from the baseline and abnormal peak smoke concentration from the sensor data; and matching semantic features corresponding to hazard keywords such as "charred" and "fire" from resident reports. The module unifies the original features of the three different modalities—images, sensor values, and text—to a pre-defined unified semantic space, completing the convergence processing of semantic unit vector distances for similar events. This allows the three types of data features describing the same fire to form a close spatial relationship, eliminating the information fragmentation caused by different data formats. The module relies on the event cognition index calculation logic, fully loads the event cognition index calculation formula, and performs quantitative calculations. The formula expression is: The system pre-configures visual modal confidence weight α, sensor modal confidence weight β, and text modal confidence weight γ, and the sum of these three values ​​is always equal to 1; it then retrieves a preset visual hazard feature activation vector. Sensor hazard feature activation vector Textual hazard feature activation vector As the criterion for judgment; traverse the current video frame to extract m visual hazard features, and match each feature against the criterion to obtain m sets of matching degrees. After calculating the mean, and then... Multiply; statistically analyze the n types of sensors that trigger alarms, and record the ratio of the baseline deviation of each type of sensor to the corresponding reference as . After calculating the mean, and then... Multiplication; read the negative semantic score θ of the voice message text, count the cumulative occurrences τ of dangerous keywords in the text, and complete logarithmic compression. Later and Multiply by θ; multiply the weighted calculation results of the visual, sensor, and text modalities by the corresponding confidence weights α, β, and γ respectively, and then sum them to output the event cognition index E in the range of 0 to 1. Simultaneously output the corresponding structured semantic description of the fire event. The structured data fully marks the precise location coordinates of the fire, the hazard level, the affected corridor range, and the on-site environmental conditions such as ventilation and combustible material storage. All fields are indexed with the coordinate points of the unified semantic space. All subsequent downstream modules can rely on the index to quickly retrieve the original data of the corresponding modality without repeating the feature extraction operation, thus reducing the system computing power consumed by subsequent data retrieval.

[0025] The instruction generation module reads the structured semantic description output by the perception and understanding module. Using fire event type tags and precise spatial coordinates of the corridor as joint search indexes, it retrieves historical fire response cases and standardized operating procedures suitable for the building's fire scenario from the backend historical maintenance knowledge base. Simultaneously, it retrieves the corresponding fire extinguishing tool carrying list and on-site safety warnings. The module performs text concatenation processing between all the retrieved knowledge base content and the front-end output structured semantic description to generate enhanced context. The context is complete, including the fire location, on-site risk level, on-site environmental limitations, and key points of handling similar historical events, providing complete information support for dispatch instruction generation. The module outputs logically coherent and clearly hierarchical complete natural language dispatch instructions according to the system's preset sentence templates. The instruction content is strictly arranged in the order of tool carrying requirements, target location, sequence of handling actions, and safety warning prompts. Field personnel can carry out operations sequentially when reading the instructions, without missing any steps. The module synchronously extracts relevant target object terms related to fire extinguishing and hazard investigation from the command text content. It marks the visual semantic dimensions corresponding to flame locations, fire hydrants in corridors, combustibles in corridors, and evacuation routes as attention dimensions, and marks the visual semantic dimensions corresponding to non-target object terms such as corridor walls, unrelated appliances, and idle clutter as masking dimensions. Based on the two-dimensional division rules, a binary semantic attention mask matrix is ​​generated. This matrix will lock the visual areas that need to be compared in the subsequent image verification process and filter out verification interference caused by irrelevant image information.

[0026] The dispatch and recycling module receives natural language dispatch instructions and semantic attention mask matrices from the instruction generation module, and simultaneously obtains location information reported by the handheld terminals of all on-duty security personnel and maintenance personnel. In outdoor areas or areas with good GPS signals, the terminals obtain location coordinates via GPS; in areas with obstructed GPS signals, such as inside buildings or underground parking garages, the terminals automatically switch to WiFi positioning mode, scanning the MAC addresses and signal strength of surrounding wireless access points and matching them with the system's pre-stored WiFi fingerprint database to output indoor location coordinates. Based on the location information reported by each terminal (preferably using GPS data, and using WiFi positioning data when GPS signals are weak), combined with the coordinates of the building where the fire occurred, the module automatically calculates the straight-line distance between each employee and the fire location, and selects the three closest employees as priority dispatch targets. The complete dispatch instructions are pushed to the handheld terminals of on-duty security personnel and maintenance personnel, and the semantic attention mask matrix is ​​simultaneously stored in the local storage space of each receiving terminal. Field personnel do not need to maintain a continuous network connection during field operations; they can still retrieve the mask matrix to view key points captured on-site even offline. After arriving at the fire-stricken building and completing tasks such as extinguishing the open flames, dispersing the smoke, and clearing combustibles, field personnel collect multi-angle images of the building's front panoramic view, the left and right sides of the building, and close-up shots of the fire location, following the shooting guidelines built into their terminal devices. These images are then uploaded to the distribution and retrieval module. The module performs two quality checks on each uploaded image: clarity and lighting sufficiency. The image quality indicators are compared to the system's preset clarity and lighting thresholds to determine the quality. Image materials that do not meet the standards are returned to the corresponding field personnel's terminal devices, with a text prompt indicating the shooting defects and guiding the staff to reshoot compliant footage. This prevents low-quality images from entering the verification process and causing judgment errors. All image materials that meet the quality requirements are categorized and stored according to shooting angle tags. The system uses the work order number as a unique association key, writing the binary data of the semantic attention mask matrix into the extended metadata field of each image file. This binds the mask matrix to the multi-angle image data. After binding, the entire set of image data is transmitted to the visual verification module for subsequent verification. Three fire incidents occurred in the stairwells of this community at the same time. The above-mentioned data collection, quality inspection and binding process was carried out simultaneously. The image materials corresponding to the three work orders were all standardized, classified and bound, with no image loss or mask matching errors. The data link transmission was stable and uninterrupted.

[0027] The visual verification module receives the bound multi-angle image data and semantic attention mask matrix. First, it reads the fire target location coordinates stored in the structured semantic description. Through coordinate matching, it identifies the corresponding building's monitoring device number in the fire area. Following system rules, it retrieves recordings from the monitoring storage server within a preset time period, tracing back from the event timestamp. Within the captured time window, it selects clear image frames closest to the fire's occurrence, without abnormal personnel gathering or equipment tilting or obstructing the view, as baseline image data. These baseline images completely recreate a normal scene in the building before the fire, providing a standard reference for subsequent scene comparisons. If the corresponding building's monitoring device malfunctions during the fire's time and has no valid recordings, the module automatically retrieves cross-view images captured by adjacent monitoring devices on the same floor at the same time point as substitute baseline image data. Simultaneously, it performs viewpoint normalization calibration to eliminate feature comparison errors caused by different shooting angles. The module then calls the visual judgment algorithm and executes the corresponding calculation formula. During implementation, K processed multi-angle images are first read, and the processed scene semantic feature vector corresponding to each image is extracted frame by frame. The semantic attention mask matrix σ generated earlier is retrieved to perform selective attention operation ⊙. All feature values ​​in the masked dimension of each feature vector are set to zero, retaining only the features in the attention dimension for operation. The mean of all masked feature vectors is calculated, and the reference scene semantic feature vector is extracted from the reference image. Through the L2 norm The Euclidean distance between the mean handling feature vector and the baseline feature vector is calculated to obtain the total basic semantic difference. The system's preset basic tolerance coefficient λ is retrieved, and the event cognition index E output from the first step is read. A dynamic tolerance adjustment factor is generated using the natural logarithm operation ln(1+E). The two are multiplied to obtain the dynamic tolerable deviation threshold. The total basic semantic difference is subtracted from the dynamic tolerance threshold to finally output the scene recovery deviation value Δ. The module retrieves the deviation tolerance upper limit of 0.35 corresponding to fire-related events as the first preset threshold. The calculated scene recovery deviation value is compared with the threshold to accurately classify the task completion status into three categories: completed, partially completed, and with remaining hidden dangers. In this embodiment, the deviation value of the first fire work order is lower than the fire-related deviation tolerance upper limit, and the system directly determines that the task is completed. The deviation value of the second fire work order is in the middle range between zero and the threshold, and is determined to be partially completed. The deviation value of the third fire work order exceeds the threshold, and it is determined that there are remaining smoke hazards at the scene. The judgment results of the three types of work orders completely match the actual rectification situation at the scene, and no misjudgment or omission occurs.

[0028] The decision archiving module receives the task completion status judgment result output by the visual verification module and executes subsequent work order processing operations according to two types of logic. If the task completion status is judged as completed, it means that all hazards such as open flames, dense smoke, and combustibles in the corridor have been eliminated, and the module directly closes the current fire work order and terminates the current full-domain dispatch process. If the task completion status is judged as partially completed or there are remaining hazards, the module actively retrieves the detailed data of semantic dimension feature differences generated by the visual verification module during the comparison process. The detailed data includes the differences of all semantic dimensions such as flame presence dimension, smoke concentration dimension, and evacuation route unobstructed dimension. Based on the difference value, the module locates the specific spatial location and hazard type corresponding to the uneliminated hazard, fills the location information into the system's built-in supplementary instruction and emergency instruction pre-defined text template, and generates differentiated secondary dispatch instructions. The differentiated secondary dispatch instruction internally lists all confirmed items completed during the previous fieldwork, and also indicates the step-by-step operation guidance for the items that need to be supplemented in this operation. When field personnel receive the secondary dispatch instruction on their handheld terminals, the interface simultaneously displays thumbnails of the previously uploaded on-site images and schematic diagrams indicating unrectified potential hazards, helping staff quickly identify key areas for rectification. Regardless of whether the work order is completed in the first operation or the second dispatch completes the closed loop, the module will initiate a full-process data integration and collection process. The integrated data includes the structured semantic description and event cognition index output by the perception and understanding module, the natural language scheduling instructions and semantic attention mask matrix output by the instruction generation module, all on-site image data and image quality inspection records retained by the dispatch and collection module, the scene restoration deviation value and task completion status output by the visual verification module, as well as all operation timestamps and on-duty personnel identification for the initial and secondary dispatches. The module organizes the text description according to the complete timeline of the event. The text content records the entire process from fire discovery, multimodal data collection, dispatch instruction issuance, field on-site handling, and AI visual verification in sequence. It also organizes all on-site images and benchmark comparison images as image evidence attachments, and finally generates a complete fire event handling report for long-term archiving and storage. Community property management personnel can retrieve the report at any time for safety review and fire protection record registration in the park.

[0029] This embodiment comprehensively covers the entire chain of fire incidents in high-rise residential building corridors, from multi-source information collection, intelligent scheduling, on-site image transmission, automated verification to file archiving. It utilizes a unified semantic space to fuse three types of heterogeneous data, quantifies fire risk using event cognition index calculation logic, and employs a semantic attention mask matrix to pinpoint core areas for image verification. A visual judgment algorithm, combined with baseline images, objectively compares rectification effects, distinguishes work order status based on categorized deviation tolerance limits, and automatically generates precise secondary dispatch instructions for residual hazards. The entire process minimizes human intervention, uniformly collects all process data, and automatically generates standardized handling reports. This solves the problems of scattered information in traditional fire response, significant subjective bias in manual verification, ambiguous hazard rectification positioning, and difficulty in summarizing and retaining handling data, effectively improving the efficiency of fire response and the accuracy of rectification verification in the industrial park.

[0030] Example 2: The equipment room on the second basement level of the commercial complex is centrally equipped with water accumulation monitoring sensors, temperature and humidity sensors, and panoramic surveillance cameras. All merchants and offices within the complex can open a voice reporting channel through the property management's dedicated client, or report issues via WeChat one-on-one chat, WeChat group chat, 400 telephone, or landline. All front-end data collection devices are uniformly connected to this invention's system, and precise spatial coordinates are simultaneously recorded during the equipment installation phase to ensure that every piece of collected data can be accurately located to the corresponding equipment room area. The panoramic monitoring equipment in the equipment room continuously collects real-time video footage, capturing visual features such as pipe leaks, water accumulation on the ground, and equipment submerged in water, thus preserving complete visual information about water accumulation faults. Water and humidity sensors continuously collect environmental values, outputting in real-time the deviation of the current reading from the equipment baseline. Each set of sensor alarm data is bound to the equipment number, equipment room installation coordinates, and anomaly alarm timestamp, quantifying the environmental value changes caused by water accumulation faults. When merchants or enterprise staff discover water leakage or accumulation in the equipment room, they can record a voice description of the water accumulation area and the location of the leaking pipe through the client. The system automatically converts the voice to text, outputting a complete text report including the fault location, water accumulation phenomenon, and reporting timestamp, supplementing the fault details from the human perspective. Figure 2 As shown.

[0031] The perception and understanding module simultaneously receives three types of multimodal raw data: video streams, IoT sensor alarms, and voice report text. It initiates a unified semantic space mapping process, transforming the independent raw feature vectors of the three types of data into the same high-dimensional representation space. This allows different modal semantic units corresponding to the same water accumulation fault to form similar vector distances, breaking down information isolation between image, numerical, and text data. After mapping, a standardized structured semantic description is generated, fully recording the water immersion event type label, fault coordinates between devices, water accumulation severity level, water accumulation coverage area, and environmental status between devices. Each data field is indexed and mapped to a unified semantic space coordinate point set, facilitating subsequent modules to quickly retrieve fault-related information. Simultaneously, the event cognition index calculation logic is invoked to output the event cognition index corresponding to the water immersion event. This value synchronously reflects the severity of the water accumulation fault and the system's recognition confidence, providing a unified quantitative standard for subsequent hierarchical scheduling.

[0032] The instruction generation module uses the water immersion event types and equipment location coordinates stored in the structured semantic description as a joint retrieval index. It retrieves standardized handling procedures, on-site maintenance tool lists, and underground confined space operation safety specifications corresponding to pipe leaks and basement water accumulation from the system knowledge base. All the retrieved reference materials are spliced ​​with the structured semantic description to form an enhanced context. Based on the system's preset sentence templates, complete natural language dispatch instructions are generated. The instructions are arranged in the following order: tool carrying list, equipment fault location, water accumulation cleaning and pipe leak sealing actions, and underground space safety warnings. Engineering maintenance personnel can directly carry out standardized operations according to the instructions. The module extracts the target terms for handling such as seeping pipes, water-filled ground, and water-soaked equipment from the dispatch instructions. The visual semantic dimension matching the corresponding terms is marked as the attention dimension, and the dimensions of other irrelevant equipment and blank areas are marked as the masking dimension. Based on the two types of dimension markings, a binary semantic attention mask matrix is ​​generated to lock the water-related areas that need to be compared in subsequent image verification and filter out the image interference caused by irrelevant facilities between equipment.

[0033] The dispatch and recycling module pushes complete dispatch instructions to the mobile terminals of engineering and maintenance personnel. Before dispatching, it obtains the real-time location of all on-duty maintenance personnel, calculates the distance to the equipment, selects the two closest personnel for dispatch, and simultaneously sends the semantic attention mask matrix to the local storage of the terminal. The terminal page displays the system's preset multi-angle shooting specifications, requiring maintenance personnel to collect four types of image materials around the water accumulation fault area: front, sides, and close-up of the leaking pipes, standardizing the on-site shooting content. After the maintenance personnel arrive at the equipment room and complete all maintenance operations such as water drainage, sealing of damaged pipes, and drying of water-damaged equipment, they upload the photos through the terminal. The captured multi-angle image files are divided into two formats: photos and short videos. Each image is automatically bound with a shooting angle tag, shooting timestamp, and maintenance personnel identification. The module automatically performs image clarity and on-site lighting sufficiency detection. Images that do not meet the preset threshold are directly returned to the terminal and prompted to be reshot. Images with all image quality meeting the standards are stored according to shooting angle and associated with the work order number. The semantic attention mask matrix binary data is written into the extended metadata field of all image files to achieve one-to-one binding of images and mask matrices. After binding, they are uniformly transmitted to the visual verification module.

[0034] The visual verification module reads the target coordinates between devices within the structured semantic description, matches the corresponding monitoring device number, and backtracks a preset time window based on the fault reporting timestamp. It then selects image frames within the window that show no water accumulation or equipment malfunction as the baseline image, completely preserving the normal, dry scene between devices before the fault. If the corresponding monitoring device has no recording due to a fault, the system retrieves cross-view images from adjacent monitoring devices on the same floor during the same time period as a substitute baseline image. It performs viewpoint normalization calibration on the semantic feature vector of the baseline scene to address differences in image angles, eliminating comparison errors caused by viewpoint deviations. After calibration, it runs the command-line visual judgment algorithm, performing selective attention operations based on the semantic attention mask matrix. It sets all feature values ​​corresponding to the masked dimensions to zero, retaining only the features related to water accumulation and pipes for calculation. After averaging all multi-angle image feature vectors, it calculates the Euclidean distance norm with the baseline vector, combines it with the event cognition index to generate a dynamic tolerance adjustment factor, and superimposes a basic fault tolerance coefficient to obtain the scene recovery deviation value. It then retrieves the first preset threshold specific to water immersion and compares the deviation value with the threshold to classify the work order processing completion status. Figure 3 As shown.

[0035] The decision archiving module receives the work order judgment result. If it identifies that on-site water accumulation and pipeline damage have not been fully addressed, it retrieves detailed data on the differences in various semantic dimensions output by the visual verification module. Based on the differences in dimensions such as the presence of water accumulation and the on / off status of power switches, it locates the locations of uncleaned water accumulation points and unsealed damaged pipelines. The hazard information is then filled into the supplementary instruction template to generate a differentiated secondary dispatch instruction. The instruction includes a list of completed repairs from the previous round and operational guidelines for this supplementary handling. The repair personnel's terminal simultaneously displays historical image thumbnails and a schematic diagram of hazard point markings. If it determines that the on-site water immersion fault has been fully resolved... The system directly closes the current work order and collects all data from the entire process of perception, instruction, dispatch and recovery, and visual verification. This includes multimodal raw analysis data, scheduling instructions, mask matrix, image materials, quality inspection records, scene restoration deviation values, and two rounds of dispatch personnel and time records. All data is integrated to generate a standardized waterlogging handling archive. The archive consists of two parts: a text time-series main text and an image comparison attachment. It fully records the entire process of waterlogging fault from reporting to repair verification. The archive is encrypted and archived to the system database for long-term storage, facilitating property operation and maintenance management personnel to conduct equipment fault review and facility loss statistics.

[0036] In summary, the implementation plan for handling basement flooding faults fully utilizes a complete set of operational logic, including multimodal data collaborative analysis, AI automatic scheduling, standardized image acquisition, scene semantic comparison, differentiated secondary dispatch, and data archiving. It relies on a unified semantic space to break down information barriers between monitoring, sensing, and merchant voice reporting data. The plan quantifies the risk of flooding faults through event cognition index calculation logic, filters irrelevant visual interference using a semantic attention mask matrix, and automatically determines repair effectiveness using a visual judgment algorithm combined with flooding-specific thresholds. It accurately pushes supplementary repair instructions for residual water and damaged pipes, and automatically collects multi-dimensional business data throughout the entire process to generate standardized handling files. This reduces the workload of manual verification during the handling of leaks in underground equipment rooms and solves the industry problems of inaccurate identification of flooding hazards and inconsistent repair and acceptance standards in traditional models.

[0037] Example 3: The main pedestrian entrances and exits of the residential community and the perimeter wall passages are uniformly equipped with facial recognition panoramic monitoring and intrusion detection IoT sensors. All residents can enable the voice reporting function through the property management mini-program, or report issues through WeChat one-on-one chat, WeChat group chat, 400 telephone, landline, and other channels. All security data collection devices are uniformly connected to the system of this invention. During the equipment deployment phase, accurate spatial coordinates are entered to ensure that each piece of collected data is accurately located to the corresponding area of ​​the community entrance and exit. The entrance and exit monitoring equipment continuously records real-time video of pedestrian passages in the park, constantly extracting visual features corresponding to behaviors such as people loitering abnormally, climbing over walls, and damaging access control facilities, thus completely preserving visual information of security anomalies. After an intrusion sensor triggers an alarm, it simultaneously outputs the deviation of the current reading from the baseline. Each set of sensor alarm data is bound to the device number, entrance / exit installation coordinates, and alarm timestamp, quantifying the abnormal signal triggered by the security equipment. When residents discover unfamiliar personnel loitering at entrances / exits or damaged access control fences, they can record a voice report of the situation. The system automatically processes the voice into text, outputting a text report containing the location of the entrance / exit, the abnormal personnel's behavior, and the reporting timestamp, supplementing the details of security anomalies from the perspective of manual reporting.

[0038] The perception and understanding module receives three types of multimodal raw data: entrance / exit video streams, intrusion sensor alarm data, and resident voice reports. It performs a unified semantic space mapping operation to transform the raw feature vectors of the three types of data into the same high-dimensional representation space. This allows multimodal semantic units corresponding to the same security anomaly to form similar vector distances, eliminating the information fragmentation caused by differences in image, numerical, and text data formats. After mapping, it outputs a structured semantic description of the security event, fully recording the security event label, entrance / exit location coordinates, anomaly risk level, affected passage range, and entrance / exit environmental status. All fields are indexed and mapped to a unified semantic space coordinate point set, facilitating subsequent modules to quickly retrieve relevant security event information. Simultaneously, it calls the event cognition index calculation logic to output the event cognition index corresponding to the security event. This value reflects the level of security anomaly risk and the system's confidence level in identification, providing a unified quantitative basis for the graded dispatch of security personnel.

[0039] The instruction generation module uses the security event type and entrance / exit location coordinates within the structured semantic description as a joint retrieval index. It retrieves standard procedures for handling security incidents at park entrances / exits, on-site security alert regulations, and repair guidelines for damaged access control systems from the system's historical case database. All retrieved standard documents are combined with the structured semantic description to form an enhanced context. Based on the system's preset sentence templates, complete natural language security dispatch instructions are generated. The instructions are arranged sequentially according to the requirements for carrying security protective equipment, the target location at the entrance / exit, the sequence of actions for removing abnormal personnel and repairing facilities, and the on-site alert range. Park security personnel can directly complete standardized handling work according to the instructions. The module extracts security verification target terms such as pedestrian passages, fences, and access control equipment from the dispatch instructions. The visual semantic dimensions matching the corresponding terms are marked as attention dimensions, while irrelevant dimensions such as park greenery and unused roads are marked as shielding dimensions. Based on the two types of dimension markings, a semantic attention mask matrix is ​​generated to lock the core security areas that will be the focus of subsequent image verification and comparison, filtering out visual interference from irrelevant scenery at entrances / exits.

[0040] The dispatch and recycling module pushes security dispatch instructions to the handheld terminals of security personnel in the park. Based on the real-time location of all security personnel, it calculates the distance to the incident entrance / exit and automatically dispatches the order to the two nearest security personnel. Simultaneously, the semantic attention mask matrix is ​​sent to the local storage of the terminal. The terminal page displays the system's preset shooting guidelines, requiring security personnel to collect multi-angle images of the abnormal area around the entrance / exit, including the front, sides, and close-up of damaged facilities. After arriving at the entrance / exit and completing tasks such as removing strangers, repairing damaged fences and access control, and managing the passageway order, the security personnel upload multi-angle image files through the terminal. The image files support both photo and short video formats. Each image is automatically bound with a shooting angle tag, shooting timestamp, and security personnel identification. The module automatically verifies the image clarity and on-site lighting sufficiency. Images that do not meet the preset threshold are returned to the terminal and prompted to be reshot. Images with all qualified images are stored according to shooting angle, with the work order number as the associated key value. The binary data of the semantic attention mask matrix is ​​written into the extended metadata field of all image files. After the images and mask matrix are bound together, they are uniformly transmitted to the visual verification module.

[0041] The visual verification module reads the coordinates of the entrance / exit targets within the structured semantic description, matches the corresponding monitoring device number, and uses the security anomaly reporting timestamp as a baseline to backtrack a preset time window. It then filters image frames within the window that are free of unfamiliar personnel and have intact facilities as baseline images, completely preserving the normal scene before the security anomaly at the entrance / exit. If the corresponding monitoring device is faulty and has no recording, the system retrieves cross-view images from adjacent park roads during the same time period as alternative baseline images. It performs viewpoint normalization calibration on the semantic feature vector of the baseline scene to eliminate comparison errors caused by viewpoint deviations. After calibration, it executes the instruction visual judgment algorithm, which performs selective attention operations based on the semantic attention mask matrix, masking irrelevant dimension feature values ​​and retaining only security-related dimension features such as fences, access control, and evacuation routes for calculation. After averaging all multi-angle image feature vectors, it calculates the Euclidean distance norm with the baseline vector, constructs a dynamic tolerance adjustment factor based on the event cognition index, and superimposes the basic fault tolerance coefficient to obtain the scene recovery deviation value. It then retrieves the security-specific first preset threshold and compares the deviation value with the threshold to determine the completion status of the work order.

[0042] The decision archiving module receives the work order judgment result. If security risks still exist at the entrance / exit, it retrieves the detailed data on the differences in various semantic dimensions output by the visual verification module. Based on the differences in the dimensions of evacuation passage accessibility and facility integrity, it locates unrepaired fences and areas where stranded personnel have not been cleared. The risk information is then filled into the supplementary instruction template to generate a differentiated secondary dispatch instruction. The instruction includes a list of completed security measures from the previous round and guidance for this supplementary measure. The security terminal simultaneously displays historical image thumbnails and a diagram showing the marked risk points. If the security status of the entrance / exit is determined to have fully recovered, the system directly closes the gate / exit. Closing the current work order, all business data from the entire process of perception, instruction, dispatch and retrieval, and visual verification is collected, including multimodal parsing data, dispatch instructions, mask matrix, image materials, image quality inspection records, scene restoration deviation values, and personnel and time records for two rounds of dispatch. All data is integrated to generate a security incident handling report, which consists of two parts: a time-series text body and an image comparison attachment. The report fully records the entire process of security anomalies from reporting, dispatching, on-site handling to AI automatic verification, and is encrypted and archived to the system database for long-term storage, supporting the periodic statistics of security risks in the park and the optimization of security handling processes.

[0043] In summary, the implementation plan for handling security anomalies in public areas of the park fully implements the collaborative operation logic of all functional modules of this invention. It relies on a unified semantic space to integrate three types of security data: entrance and exit monitoring, intrusion sensing, and resident voice reporting. It quantifies the security risks at entrances and exits through the event cognition index calculation logic, uses a semantic attention mask matrix to lock core verification areas such as fences and access control, and uses a visual judgment algorithm to automatically complete the on-site rectification effect judgment based on security-specific thresholds. It generates targeted secondary dispatch instructions for security risks that have not been eliminated, and automatically retains a complete security incident handling report throughout the entire process. The entire system realizes autonomous identification, automatic scheduling, and unmanned acceptance of security anomalies at park entrances and exits, solving the practical pain points of traditional security management models, such as the inability to synchronize and integrate information from multiple channels, low efficiency of manual on-site verification, and inadequate follow-up on hazard rectification.

[0044] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multimodal property event recognition and full-domain scheduling AI collaborative system, characterized in that, The system includes: Perception and Understanding Module: Collects video stream data, IoT sensor alarm data, and voice notification text data, and maps them to a unified semantic space to generate a structured semantic description of the event scene and outputs an event cognition index; Instruction generation module: Receives structured semantic description and event cognition index, retrieves historical handling cases and operating procedures, generates natural language scheduling instructions and constructs a semantic attention mask matrix; The dispatch and recycling module receives natural language dispatch instructions and pushes them to the terminal devices of the personnel. It also receives multi-angle image data of the on-site handling uploaded by the personnel through the terminal devices, binds the semantic attention mask matrix with the multi-angle image data, and then transmits it. Visual verification module: Receives semantic attention mask matrix and multi-angle image data, retrieves reference image data of the same scene before the event occurred, uses command-line visual judgment algorithm to perform semantic comparison between multi-angle image data and reference image data, calculates scene restoration deviation value and outputs task completion status; Decision archiving module: Receives task completion status. When the task completion status is "completed", the work order is closed. When the task completion status is "partially completed" or there are potential risks, a second task dispatch is triggered. At the same time, it integrates all data in the entire process to generate an event handling report.

2. The multimodal property event recognition and full-domain scheduling AI collaborative system according to claim 1, characterized in that, In the perception and understanding module, the video stream data includes a continuous sequence of video frames collected by fixed cameras within the property area, along with the timestamp and spatial coordinates of each video frame. The video frame sequence contains visual features of flame targets, smoke targets, and personnel behavior trajectories. The IoT sensor alarm data includes smoke concentration values ​​reported by smoke detectors, real-time temperatures reported by temperature detectors and their positive deviation from the baseline temperature, real-time humidity reported by humidity detectors and their negative deviation from the baseline humidity, as well as the device number, installation location coordinates, and alarm timestamp of each detector. The voice reporting text data is the text content obtained by converting audio uploaded by the owner into text, which includes a description of the location of the event, a description of the event phenomenon, and a reporting timestamp.

3. The multimodal property event recognition and full-domain scheduling AI collaborative system according to claim 1, characterized in that, In the perception and understanding module, the event cognition index is expressed by the formula... The calculation yields E, where E represents the event perception index. Represents the confidence weight of the visual modality. Represents the sensor modal confidence weights. Indicates the text modality confidence weight; This represents the pre-defined comprehensive weighting coefficient for visual hazard features; This represents the preset comprehensive weighting coefficient for the sensor's hazard characteristics; This represents the pre-defined comprehensive weighting coefficient for textual hazard features; This indicates that the i-th visual hazard feature value extracted from the current video frame is compared with... The normalized matching degree of the corresponding judgment benchmark; m is the total number of extracted visual hazard features; This indicates the magnitude by which the current reading of the j-th type of sensor deviates from its baseline value. The normalized ratio corresponding to the judgment benchmark; n is the total number of sensor types that trigger the deviation alarm; θ represents the semantic negative score of the voice report text data; τ represents the occurrence of [missing information] in the text data. The cumulative number of times dangerous keywords were used.

4. The multimodal property event recognition and full-domain scheduling AI collaborative system according to claim 1, characterized in that, The instruction generation module uses the event type and event location in the structured semantic description as a joint retrieval index. It retrieves standard operating procedures, essential tool lists, and safety precautions matching the event type from the historical maintenance knowledge base. The retrieval results are then concatenated with the structured semantic description to form an enhanced context, generating natural language dispatch instructions that conform to a predetermined sentence template. These instructions consist of a sequential combination of tool carrying requirements, target location indication, action sequence, and safety warning prompts. The semantic attention mask matrix extracts the target object nouns that the verbs in the natural language dispatch instructions refer to. It marks the visual semantic dimensions corresponding to these target object nouns as attention dimensions and marks the visual semantic dimensions corresponding to non-target object nouns as mask dimensions, thereby generating a binary semantic attention mask matrix.

5. The multimodal property event recognition and full-domain scheduling AI collaborative system according to claim 1, characterized in that, When the dispatch and retrieval module pushes natural language dispatch instructions to the executor's terminal device, it first obtains the location information reported by all on-duty executor terminals. The location information includes GPS location data and WiFi location data. GPS location is prioritized in outdoor environments, while WiFi location is used indoors or in areas where GPS signals are blocked. The module calculates the spatial straight-line distance or path distance between the coordinates corresponding to each location information and the location where the event occurred, and selects the one with the smallest distance as the final dispatch target. At the same time, it pushes a semantic attention mask matrix to the local storage of the terminal device. After receiving the multi-angle image data of the on-site handling uploaded by the executor, it performs sharpness detection and lighting sufficiency detection on each image data. Image data that does not meet the preset sharpness threshold and preset lighting threshold is returned to the executor's terminal device and a prompt is made to retake the image. Image data that meets all quality conditions is classified and stored according to the shooting angle.

6. The multimodal property event recognition and full-domain scheduling AI collaborative system according to claim 1, characterized in that, The visual verification module determines the monitoring device number based on the target location coordinates in the structured semantic description, and then traces back a preset time period based on the event timestamp. It prioritizes capturing the image frame closest to the event time that has no abnormal gathering of people or abnormal equipment status in the image as the reference image data. When the monitoring device has no recording record due to malfunction before the event, the cross-view image of adjacent monitoring devices on the same floor at the same time is used as the substitute reference image data. The extracted reference scene semantic feature vector is then normalized and calibrated based on the viewpoint difference of the substitute reference image data.

7. The multimodal property event recognition and full-domain scheduling AI collaborative system according to claim 1, characterized in that, In the visual verification module, the mathematical expression for the instruction visual judgment algorithm is: In the formula, Δ represents the scene recovery deviation value; This represents the semantic feature vector of the baseline scene extracted from the baseline image data; σ represents the semantic feature vector of the scene after processing, extracted from the on-site processing image data from the k-th angle, where K is the total number of multi-angle image data; σ represents the semantic attention mask matrix; λ represents the preset basic fault tolerance coefficient, which is a real number greater than 0; and E is the event cognition index.

8. The multimodal property event recognition and full-domain scheduling AI collaborative system according to claim 1, characterized in that, After calculating the scene restoration deviation value Δ, the visual verification module determines that the scene restoration is complete when Δ≤0, partially complete when Δ>0 and Δ≤a first preset threshold, and has remaining hidden dangers when Δ>a first preset threshold. The first preset threshold is a deviation tolerance upper limit built into the decision archiving module, which is set according to the event type. The deviation tolerance upper limit for fire events is higher than that for water immersion events, which is higher than that for security events, which is higher than that for cleaning and maintenance events. Based on the event type label in the structured semantic description, the deviation tolerance upper limit corresponding to the event type label is retrieved as the first preset threshold.

9. The multimodal property event recognition and full-domain scheduling AI collaborative system according to claim 1, characterized in that, When the decision archiving module triggers a secondary dispatch, it obtains detailed feature difference data for each semantic dimension calculated during the current verification process from the visual verification module. Based on the detailed feature difference data, it locates the specific objects and spatial locations that do not meet the judgment criteria and fills them into a predefined text template of supplementary instructions dynamically matched according to the event type and event cognition index in the structured semantic description, generating a differentiated secondary dispatch instruction. The differentiated secondary dispatch instruction includes a confirmation list of previously completed items and operational guidelines for items that need to be supplemented this time. The executor's terminal device simultaneously displays a thumbnail of the previously uploaded image data and a schematic diagram of the items that need to be supplemented this time.

10. The multimodal property event recognition and full-domain scheduling AI collaborative system according to claim 1, characterized in that, In the decision archiving module, the incident handling report includes a text description and an image evidence attachment. The text description describes the incident discovery process, the dispatch instruction issuance process, the on-site handling process, and the quality verification process in chronological order of the incident occurrence. The image evidence attachment arranges a comparison group of the baseline image before handling and the image after handling in chronological order of shooting time.

Images