Cloud-edge collaborative multi-modal reasoning method and system for park governance

Abstract

The application discloses a cloud-edge collaborative multi-modal reasoning method and system for park management, relates to the field of multi-modal intelligent reasoning, and comprises the following steps: collecting and processing local multi-modal sensing data through multiple edge nodes, performing single-modal reasoning based on a lightweight AI model, and generating multiple edge event description information; based on a dynamic triggering mechanism, evaluating the multiple edge event description information to determine an event processing strategy and determining whether to trigger a collaborative reasoning request; for a marked edge node triggering a request, based on an attention mechanism, fusing and reasoning multi-modal feature information uploaded from the marked edge node to generate a joint reasoning result; the cloud end generates a management instruction according to the joint reasoning result and delivers the management instruction to the marked edge node to drive the execution of park management actions. The application solves the technical problems of insufficient real-time performance and reliability of the existing multi-modal reasoning for park management, and achieves the technical effect of improving the real-time performance and reliability of park management.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal intelligent reasoning, in particular to a cloud-edge collaborative multi-modal reasoning method and system for park governance. BACKGROUND

[0002] The efficiency of park governance is crucial for improving the overall operation efficiency of the park and ensuring safety and order. Currently, park governance mainly relies on centralized cloud computing architecture, which uploads multi-modal data collected by various sensors in the park to the cloud for processing and analysis, and then issues governance instructions from the cloud. However, this architecture faces problems such as high data transmission delay, high cloud computing pressure, and strong dependence on the network, which makes it difficult to meet the demand for governance response speed in real-time scenarios, and may cause system performance bottlenecks when data volume surges.

[0003] In the current related technology, the multi-modal reasoning for park governance has the technical problems of insufficient real-time performance and reliability. SUMMARY

[0004] The present application provides a cloud-edge collaborative multi-modal reasoning method and system for park governance, which collects and processes local multi-modal sensor data through multiple edge nodes in the park, generates edge event description information using lightweight AI model single-modal reasoning, and determines whether to trigger a collaborative reasoning request based on a dynamic trigger mechanism. The marked edge node that triggers the request uses attention mechanism to fuse the multi-modal feature information uploaded by it for reasoning to obtain a joint result. The cloud generates governance instructions based on this and issues them to the marked edge node to execute governance actions. This solves the technical problems of insufficient real-time performance and reliability in existing multi-modal reasoning for park governance, and achieves the technical effect of improving the real-time performance and reliability of park governance.

[0005] The present application provides a cloud-edge collaborative multi-modal reasoning method for park governance, which includes: collecting and processing local multi-modal sensor data through multiple edge nodes deployed in the park, and generating multiple edge event description information based on pre-set lightweight AI model single-modal reasoning; evaluating the multiple edge event description information based on a pre-set dynamic trigger mechanism to determine an event processing strategy and determine whether to trigger a collaborative reasoning request; for the marked edge node that triggers the collaborative reasoning request, fusing and reasoning the multi-modal feature information uploaded from the marked edge node based on an attention mechanism to generate a joint reasoning result, the multi-modal feature information being a feature vector extracted from the edge event description information; and the cloud generating governance instructions based on the joint reasoning result and issuing them to the marked edge node to drive execution of park governance actions.

[0006] In a possible implementation, the single-modal inference is performed based on the preset lightweight AI model, a plurality of edge event description information is generated, and the following processing is performed: the multi-modal sensor data at least includes video data, audio data, and Internet of Things sensor data; after the multi-modal sensor data is preprocessed, the multi-modal sensor data is input into the lightweight AI model embedded in the corresponding edge node for inference, and each edge model inference result is output; wherein the lightweight AI model corresponding to the video data is a target detection model, which outputs a target category and carries a confidence; the lightweight AI model corresponding to the audio data is an abnormal sound classification model, which outputs a sound event category and carries a confidence; the lightweight AI model corresponding to the Internet of Things sensor data is a rule engine, which outputs a state anomaly identifier and carries a confidence; a high-dimensional feature vector is extracted from the intermediate layer of the lightweight AI model, and is encapsulated with each edge model inference result, a geographical position of data collection, a timestamp, and a device identifier to generate the plurality of edge event description information.

[0007] In a possible implementation, the plurality of edge event description information is evaluated based on a preset dynamic triggering mechanism in the plurality of edge nodes to determine an event processing strategy and determine whether to trigger a collaborative inference request, and the following processing is performed: based on a predefined event location-event type-priority mapping table, an initial priority score is assigned to each event type in the plurality of edge event description information to generate a plurality of initial event scores; the confidence in the plurality of edge event description information and the plurality of initial event scores are respectively weighted to obtain a plurality of comprehensive evaluation scores; the plurality of comprehensive evaluation scores are compared with a preset high threshold and a preset low threshold to determine whether to trigger a collaborative inference request.

[0008] In a possible implementation, the plurality of comprehensive evaluation scores are compared with a preset high threshold and a preset low threshold to determine whether to trigger a collaborative inference request, and the following processing is performed: if any comprehensive evaluation score is lower than the preset low threshold, it is determined that the corresponding event processing strategy is processed locally by the edge node and discarded; if any comprehensive evaluation score is higher than the preset high threshold, it is determined that the corresponding event processing strategy is triggered by the edge node to trigger a local response and is reported to the cloud; if any comprehensive evaluation score is between the preset low threshold and the preset high threshold, it is determined that the event processing strategy is to trigger a collaborative inference request.

[0009] In a possible implementation, for a marked edge node triggering a collaborative inference request, multi-modal feature information uploaded from the marked edge node is fused and inferred based on an attention mechanism to generate a joint inference result, and the following processing is performed: the marked edge node retrieves event description information of other modalities generated in a same geographical area within a preset forward pushing time window in a local cache, and judges whether the event description information of other modalities is associated with current edge event description information in a time sequence and semantics; if there is an association, a feature vector of the current edge event and feature vectors of all associated events are assembled into a composite data packet; and the composite data packet is uploaded to the cloud as a payload of the collaborative inference request to perform fusion and inference based on the attention mechanism to generate the joint inference result.

[0010] In a possible implementation, the composite data packet is uploaded to the cloud as a payload of the collaborative inference request to perform fusion and inference based on the attention mechanism to generate the joint inference result, and the following processing is performed: the composite data packet is mapped to a unified feature space according to a modal classification to generate a feature sequence of each modality; the feature sequence of each modality is input into a fusion network based on the attention mechanism, and a correlation weight between different modal features is calculated through a cross-attention mechanism; the feature sequence of each modality is weighted and fused according to the correlation weight to obtain a fused context feature representation; and the joint inference result is output by a regressor based on the context feature representation.

[0011] In a possible implementation, after obtaining the fused context feature representation, the following processing is further performed: a pre-constructed park knowledge graph is queried to obtain context information related to the joint inference result, and the park knowledge graph at least includes park area function attributes, device layout information, personnel permission information, and historical event records; the joint inference result is logically associated with the context information, including association of risk levels, semantic connotations, and disposal priorities, and the joint inference result is optimized.

[0012] In a possible implementation, the following processing is performed: the step of processing and discarding locally by the edge node includes: without triggering any alarm or response action, recording the edge event and related data in a local log, and automatically covering or deleting after reaching a preset storage time limit or storage space threshold.

[0013] In a possible implementation, the following processing is performed: the step of triggering a local response by the edge node and reporting to the cloud includes: performing a predefined local response action according to the type and level of the edge event; extracting key information of the edge event to generate an event report, including at least event type, location, time, confidence, associated device ID, and snapshot or segment of field evidence; and uploading the event report to the cloud through an asynchronous communication mode.

[0014] The application also provides a cloud-edge collaborative multi-modal reasoning system for park governance, comprising: a single-modal reasoning module, configured to collect and process local multi-modal sensing data through a plurality of edge nodes deployed in the park, and perform single-modal reasoning based on a preset lightweight AI model to generate a plurality of edge event description information; a collaborative reasoning trigger determination module, configured to evaluate the plurality of edge event description information based on a preset dynamic trigger mechanism at the plurality of edge nodes to determine whether to trigger a collaborative reasoning request according to an event processing strategy; a joint reasoning module, configured to perform fusion reasoning on multi-modal feature information uploaded from a marked edge node based on an attention mechanism to generate a joint reasoning result, the multi-modal feature information being a feature vector extracted from the edge event description information; and a governance instruction generation module, configured to generate a governance instruction according to the joint reasoning result at the cloud end and deliver the governance instruction to the marked edge node to drive execution of a park governance action.

[0015] The cloud-edge collaborative multi-modal reasoning method and system for park governance provided by the application first collect and process local multi-modal sensing data through a plurality of edge nodes deployed in the park, and perform single-modal reasoning based on a preset lightweight AI model to generate a plurality of edge event description information, then evaluate the plurality of edge event description information based on a preset dynamic trigger mechanism at the plurality of edge nodes to determine whether to trigger a collaborative reasoning request according to an event processing strategy, then perform fusion reasoning on multi-modal feature information uploaded from a marked edge node based on an attention mechanism to generate a joint reasoning result, the multi-modal feature information being a feature vector extracted from the edge event description information, and finally generate a governance instruction according to the joint reasoning result at the cloud end and deliver the governance instruction to the marked edge node to drive execution of a park governance action. The real-time performance and reliability of park governance are improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings of the embodiments of the application will be briefly introduced below. In the present application, a flowchart is used to illustrate the operations performed by the system according to the embodiments of the application. It should be understood that the foregoing or the following operations are not necessarily performed in sequence. On the contrary, various steps can be processed in reverse order or simultaneously as needed. Meanwhile, other operations can be added to these processes, or a step or several steps can be removed from these processes.

[0017] Figure 1 The flowchart of the cloud-edge collaborative multi-modal reasoning method for park governance provided by the embodiments of the application.

[0018] Figure 2 A structural schematic diagram of a cloud-edge collaborative multi-modal reasoning system for park governance is provided in the embodiments of the present application.

[0019] Label explanation: single-modal reasoning module 10, collaborative reasoning trigger judgment module 20, joint reasoning module 30, governance instruction generation module 40. DETAILED DESCRIPTION

[0020] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined invention purposes, the specific embodiments, structures, features and effects according to the present application are described in detail below in combination with the drawings and preferred embodiments.

[0021] The embodiments of the present application provide a cloud-edge collaborative multi-modal reasoning method for park governance, as shown in the following Figure 1 The method comprises the following steps:

[0022] In step S100, a plurality of edge nodes deployed in the park respectively collect and process local multi-modal sensor data, and perform single-modal reasoning based on a pre-set lightweight AI model to generate a plurality of edge event description information.

[0023] Specifically, a special lightweight AI model for different modal data is deployed on each edge node. These models are optimized by model compression techniques such as pruning, quantization or knowledge distillation to adapt to the limited computing resources of edge devices. Among them, the edge node is an embedded device integrated with GPU or NPU acceleration computing capability. The edge node reads the raw data from the connected sensors such as camera, microphone array, temperature and humidity sensor in real time, performs format standardization, noise reduction, normalization and other preprocessing, and inputs the data into the corresponding AI model for reasoning. The reasoning result and its deep feature are extracted and encapsulated into structured event description information together with the metadata. Among them, the metadata includes the location obtained by the GPS / Beidou positioning module, the system clock timestamp, the device unique ID, etc.

[0024] The acquisition of the lightweight AI model includes an offline process of model selection, optimization and retraining. First, a benchmark model balanced in accuracy and speed is selected as a teacher model from a public model library or a basic large model. Then, the teacher model is fully trained using a large-scale dataset collected and labeled from the target park scene to obtain a high-precision benchmark model. Model compression techniques are then applied to optimize the benchmark model. Specifically, rules or importance-based algorithms are used to identify and remove unimportant connections or entire neurons in the network, generating a sparse model architecture. The model weights and activation values are converted from 32-bit floating-point numbers to low-precision data formats, such as 8-bit integers. These optimization operations are completed through a deep learning optimization framework. Finally, to compensate for the accuracy loss caused by compression, a smaller labeled dataset from the real scene of the park is used to perform a round of knowledge distillation or fine-tuning on the pruned and quantized lightweight model, making it suitable for the edge computing environment and specific park tasks.

[0025] In a possible implementation, based on the preset lightweight AI model, single-modal inference is performed to generate multiple edge event description information. Step S100 further includes step S110. The multi-modal sensor data includes at least video data, audio data, and Internet of Things sensor data. After preprocessing, the multi-modal sensor data is input into the lightweight AI model embedded in the corresponding edge node for inference, and the inference results of each edge model are output. The lightweight AI model corresponding to the video data is a target detection model, which outputs the target category and carries the confidence. The lightweight AI model corresponding to the audio data is an abnormal sound classification model, which outputs the sound event category and carries the confidence. The lightweight AI model corresponding to the Internet of Things sensor data is a rule engine, which outputs the state abnormality identifier and carries the confidence.

[0026] Specifically, the edge device runs a lightweight AI inference framework such as TensorFlow Lite, PyTorchMobile, or OpenVINO Toolkit. For video data, the pre-processing includes frame sampling, image scaling, and color space conversion. The pre-processed video frames are input to a lightweight object detection model such as a compressed YOLO-V5s or MobileNet-SSD model, which outputs bounding box coordinates, object classes, and confidence scores, where the object classes include people, vehicles, etc. For audio data, the pre-processing includes pre-emphasis, framing, windowing, and Fast Fourier Transform to extract Mel-frequency cepstral coefficient features. These features are input to a lightweight abnormal sound classification model such as an audio classification model based on the MobileNetV2 architecture, which outputs event classes and confidence scores, where the event classes include glass breaking sound, fire alarm sound, etc. For Internet of Things sensor data, such as values from smoke sensors and door magnets, the pre-processing includes data cleaning and sliding window average filtering, followed by judgment through an IF-THEN rule set, for example: if temperature > 60°C, output high temperature anomaly status identifier and confidence score.

[0027] Step S120, extracting high-dimensional feature vectors from the intermediate layers of the lightweight AI model, packaging with the inference results of each edge model, the geographical location of data collection, the timestamp, and the device identifier to generate the multiple edge event description information. Specifically, during model inference, high-dimensional feature vectors are extracted from the fully connected layer or after the last convolutional pooling layer of the model through the framework interface. For example, a 512-dimensional floating-point vector is extracted from the end of the backbone network of the YOLO model. These feature vectors capture the abstract semantic information of the data. The packaging program on the edge node serializes the high-dimensional feature vectors, model inference results, geographical location obtained from the GPS module, timestamp obtained from the system clock, and unique identifier burned in the device at the time of factory shipment into a standard format data structure such as JSON or Protocol Buffers format, forming a complete edge event description information.

[0028] Step S200, based on the preset dynamic trigger mechanism, the multiple edge nodes evaluate the multiple edge event description information to determine the event processing strategy and decide whether to trigger a collaborative inference request.

[0029] Specifically, a decision engine is running on each edge node, which loads the pre-defined strategy configuration. The engine receives the edge event description information and compares it with the built-in scoring rules and thresholds to automatically decide whether the event should be handled locally, reported to the cloud directly, or need to initiate cloud-edge collaborative inference.

[0030] In a possible implementation, the plurality of edge nodes evaluate the plurality of edge event description information based on a preset dynamic triggering mechanism to determine whether an event processing strategy triggers a collaborative reasoning request, and step S200 further includes step S210 of assigning an initial priority score to each event type in the plurality of edge event description information based on a predefined event location-event type-priority mapping table to generate a plurality of initial event scores. Specifically, the edge node locally stores or synchronizes from the cloud a priority mapping table, which is a data structure such as a hash table or a relational database table, the key of which is composed of a geographic location area code and an event type enumeration value, and the value is an integer or floating point priority score. When the edge event description information arrives, the decision engine parses its geographic location and event type fields as keys to query the table to obtain the corresponding initial priority score.

[0031] The event location-event type-priority mapping table is obtained based on quantitative analysis of park business rules and historical operation data. The park management party defines a list of core event types and geographic location partitions based on domain knowledge, and assigns an initial priority score to each location-event type combination by one or more of expert experience assignment, historical data analysis, and risk assessment model. For example, if the historical data analysis method is used, the park historical event record database is analyzed to count the frequency of occurrence of different types of events in a specific area, the average disposal time, and the actual loss caused, and the loss degree or urgency is mapped to the priority score. Finally, after integrating and normalizing these scores, a structured query table is formed and preloaded in the edge node or uniformly issued and updated by the cloud.

[0032] Step S220, the confidence of the plurality of edge event description information and the plurality of initial event scores are respectively weighted and calculated to obtain a plurality of comprehensive evaluation scores. Specifically, the decision engine performs weighted summation calculation, and the formula is: comprehensive evaluation score = a x initial priority score + b x model confidence, wherein a and b are predefined weight coefficients, and the sum of a and b is 1. The model confidence is the probability value directly output by the AI model, and is between 0 and 1. The calculation is performed on the CPU of the edge node, and is completed independently for each event to generate a comprehensive evaluation score.

[0033] Step S230, the plurality of comprehensive evaluation scores are compared with a preset high threshold and a preset low threshold to determine whether a collaborative reasoning request is triggered. Specifically, the decision engine compares each comprehensive evaluation score obtained by calculation with two statically configured threshold values, one of which is a preset high threshold and the other of which is a preset low threshold. According to the comparison result, a preset conditional branching logic is executed to assign a processing strategy to each event.

[0034] The determination of the preset high and low thresholds is set by statistical analysis on the comprehensive evaluation scores of historical event samples and combined with business decision requirements. A large number of comprehensive evaluation scores of historical events with final confirmation results are collected, a distribution diagram of the scores is drawn, and the thresholds are selected in combination with business targets such as expected false alarm rate and missed alarm rate. The preset high threshold is set at a score level capable of filtering out most false positives and low-risk true events, and events higher than the threshold are considered as high-confidence and high-priority real emergency events that need to be immediately responded and reported. The preset low threshold is set at a score level capable of identifying obviously worthless events, and events lower than the threshold are considered as unimportant or low-confidence noises that can be directly and safely ignored. The two thresholds jointly act to divide the event stream into three clear processing channels, realizing intelligent grading and filtering of edge-side events.

[0035] In a possible implementation, the plurality of comprehensive evaluation scores are compared with the preset high threshold and the preset low threshold to determine whether to trigger a collaborative reasoning request, and step S230 further includes step S231. If any comprehensive evaluation score is lower than the preset low threshold, it is determined that the corresponding event processing strategy is to be processed locally by the edge node and discarded, wherein the step of being processed locally by the edge node and discarded includes: not triggering any alarm or response action, recording the edge event and related data in a local log, and automatically covering or deleting after reaching a preset storage time limit or storage space threshold. Specifically, when the event score is less than the preset low threshold, low-value event mute processing is performed, that is, the decision engine of the edge node calls the log recording module to write the summary information of the event, including event ID, type, time, location, score, etc., into the log file of the local storage device. A separate daemon process or timing task is set by the system to periodically scan the log file, and according to the preset strategy, such as the log file size exceeding 10 MB or the log recording time being earlier than the current time by 30 days, the oldest log entries are automatically deleted or covered to avoid storage space exhaustion. In this process, no network upload or alarm action is triggered.

[0036] Step S232, if any comprehensive evaluation score is higher than the preset high threshold, it is determined that the corresponding event processing strategy is to trigger a local response by the edge node and report to the cloud, wherein the step of triggering a local response by the edge node and reporting to the cloud includes: performing a predefined local response action according to the type and level of the edge event; extracting key information of the edge event to generate an event report, including at least event type, location, time, confidence, associated device ID, and snapshot or fragment of field evidence; and uploading the event report to the cloud through an asynchronous communication mode.

[0037] Specifically, when the event score is greater than the preset high threshold, the rapid response and reporting processing of the high-confidence emergency event is performed, that is, the decision engine first queries the local response action mapping table according to the event type and level, and executes the corresponding action. For example, for a fire alarm event, a connected sound and light alarm can be triggered to ring through a GPIO interface. At the same time, the reporting module extracts key fields from the event description information, including event type, latitude and longitude, UTC timestamp, confidence, associated camera ID, etc., and attaches a piece of field evidence, such as a 5-second video H.264 encoding segment or a JPEG format snapshot of the scene before and after triggering the event, to assemble an event report. Using an asynchronous communication library, the report is uploaded to a message queue in the cloud through HTTP / HTTPS or MQTT protocol, and after uploading, the edge node does not wait for a response from the cloud and continues to process other tasks.

[0038] The local response action mapping table is obtained by manual arrangement and configuration based on the preset emergency plan and automatic disposal process of the park. The table is a structured configuration file defined by park operation and management personnel according to standard operating procedures, which contains the mapping relationship between event type, event level and response action instruction list, and specifies that when the edge node independently determines the occurrence of a high-confidence event, a series of graded, specific hardware control instructions or software logic should be automatically executed according to the severity of the event. The table exists in the form of a configuration file or a database table and is issued by the central management system to each edge node. The control program on the node parses and loads the mapping table to establish the binding relationship between event type, level and underlying drive interface. For example, for a fire and smoke identification event, when the level is high, the system will trigger multiple emergency responses including starting a sound and light alarm, broadcasting evacuation instructions to devices in the area, recording a serious level log, and synchronously reporting to the cloud. If the level is medium, the response is to play a warning voice, record a warning log, and report event information. For an illegal intrusion event of access control, if it is an emergency level, it will perform forced locking, upload field video evidence, and send real-time alarm; if it is a medium level, it will display warning information locally and report to the cloud to notify security personnel to check.

[0039] Step S233, if any of the comprehensive evaluation scores is between the preset low threshold and the preset high threshold, the event processing strategy is determined to trigger a cooperative reasoning request. Specifically, when the event score is greater than or equal to the preset low threshold and less than or equal to the preset high threshold, further analysis and processing of the uncertain event is performed, that is, the decision engine labels or sets a specific state flag for the event description information, indicating that the cooperative reasoning process needs to be started for the event.

[0040] Step S300, for the marked edge node triggering the cooperative inference request, based on the attention mechanism, the multi-modal feature information uploaded from the marked edge node is fused and inferred to generate a joint inference result, and the multi-modal feature information is a feature vector extracted from edge event description information.

[0041] Specifically, the marked edge node collects recent data from other modalities that may be related to the current event, and uploads all related features to the cloud. The cloud is deployed with a more powerful multi-modal fusion model based on attention mechanism, which is used to weigh the importance of different modal features, perform deep fusion, and obtain a more accurate joint judgment.

[0042] In one possible implementation, for the marked edge node triggering the cooperative inference request, based on the attention mechanism, the multi-modal feature information uploaded from the marked edge node is fused and inferred to generate a joint inference result, and step S300 further includes step S310, the marked edge node retrieves event description information of other modalities generated in the same geographical area within a preset forward time window in the local cache, and judges whether the event description information of other modalities and the current edge event description information exist in time sequence and semantics. Specifically, the edge node maintains a time-effective local cache database, when the event is marked as cooperative inference, the node program queries the cache with the geographical position and timestamp of the current event as index, and queries all event description information from other types of sensors generated in the same geographical position within a preset time window, wherein the preset time window can be 60 seconds before the current time. Based on the pre-defined spatio-temporal and semantic association rules, the edge node performs a multi-condition filtering query to retrieve all other modal events within the time window and geographical space range from the local cache. The event set retrieved is judged by applying a set of semantic association rules, which are defined in advance by logical expressions, to check whether the current event and the candidate event exist in a meaningful combination in type.

[0043] Step S320, if there is an association, the feature vector of the current edge event and the feature vector of all associated events are assembled into a composite data packet. Specifically, the data assembly program on the edge node splices or encapsulates the feature vector of the current event and the feature vector of all retrieved associated events into a new data structure according to a predetermined order. At the same time, the metadata of these events is also included.

[0044] Step S330, upload the composite data packet to the cloud for fusion inference based on attention mechanism as the payload of the collaborative inference request, and generate the joint inference result. Specifically, the cloud service receives the composite data packet and inputs it into a pre-trained multi-modal fusion network based on attention mechanism. The network dynamically allocates weights by calculating the correlation between different modal features, and finally outputs a comprehensive joint inference result.

[0045] In one possible implementation, the composite data packet is uploaded to the cloud for fusion inference based on attention mechanism as the payload of the collaborative inference request to generate the joint inference result, and step S330 further includes step S331 of mapping the composite data packet to a unified feature space according to modal classification to generate a feature sequence of each modal. Specifically, feature alignment and standardization are performed. The cloud model includes multiple modal-specific encoders. The feature vector of each modal is projected into a unified dimensional vector space through its corresponding encoder, and these projected vectors are arranged in the order of the source events to form a feature sequence of each modal.

[0046] Step S332, input the feature sequence of each modal into a fusion network based on attention mechanism to calculate the correlation weight between different modal features through cross-attention mechanism. Specifically, the fusion network adopts a Transformer encoder structure to concatenate the feature sequences of all modes and input them into a multi-head self-attention layer. In this layer, the attention weight matrix is solved by calculating the query, key, and value vectors obtained by linear transformation of the input features, which quantifies the correlation between any two features in the sequence. For example, the smoke feature in the video and the high temperature feature of the temperature sensor will obtain a higher attention weight.

[0047] Step S333, according to the correlation weight, weighted fusion is performed on the feature sequence of each modal to obtain a fused context feature representation. Specifically, the calculated attention weight matrix is multiplied by the original feature sequence after value vector transformation, that is, a weighted sum of all input features is performed, and the weight is the importance. A new feature representation that integrates all modal context information is output, and the feature representation has the same dimension as the input sequence.

[0048] Step S334, based on the context feature representation, output the joint inference result through a regressor. Specifically, the fused context feature representation is input into a regressor, which is composed of several fully connected layers, and the output layer outputs a score representing the global risk level of the event or a multi-class probability distribution, such as the probabilities of normal, low-risk, and high-risk, according to the task definition.

[0049] In a possible implementation, after obtaining the fused context feature representation, step S330 further includes step S335 of querying a pre-constructed park knowledge graph to obtain context information related to the joint inference result, the park knowledge graph at least including park area function attributes, device layout information, personnel permission information, and historical event records. Specifically, the cloud system maintains a park knowledge graph, which is stored using a graph database. The nodes in the park knowledge graph represent entities, such as Building A, a production workshop, General Manager Zhang San, camera C001, and the like, and the edges represent relationships, such as located in, belongs to, has permission, and the like. When the joint inference result is generated, the system queries the park knowledge graph through a graph query language to obtain related entities and relationships. For example, when the joint inference result identifies that there is an unauthorized person at the entrance of Zone B, the system queries the park knowledge graph to obtain information such as the responsible department associated with the entrance of Zone B, the list of personnel permissions allowed to pass during the time period, and the layout of nearby security devices.

[0050] Step S336 logically associates the joint inference result with the context information, including the association of risk levels, semantic connotations, and disposal priorities, to optimize the joint inference result. Specifically, the cloud system is provided with a rule engine written based on expert experience or business logic, which takes the joint inference result and the context information obtained from the park knowledge graph as input and executes the rules. For example, the rule is: IF event type is unauthorized person AND personnel identity is scheduled visitor AND current time is more than 30 minutes earlier than scheduled time THEN the optimization result is early arrival of the visitor, the risk level is adjusted to low, and the disposal priority is adjusted to medium. Through such logical association, the inference result is more consistent with the business scenario.

[0051] Step S400: The cloud generates a governance instruction according to the joint inference result and delivers it to the marking edge node to drive the execution of park governance actions.

[0052] Specifically, the instruction generation module of the cloud generates specific and executable instructions according to the joint inference result. These instructions are structured commands that are delivered to the marking edge node that initially reported the event through the cloud-edge communication channel. The instruction executor on the edge node parses and executes these instructions, thereby driving the connected physical devices to complete specific park governance actions.

[0053] The embodiments of the present application collect and process local multi-modal sensor data through multiple edge nodes in the park, generate edge event description information using a lightweight AI model single-modal inference, multiple edge nodes evaluate the information according to a dynamic triggering mechanism to determine whether to trigger a collaborative inference request, the marked edge node of the triggering request uses an attention mechanism to fuse the multi-modal feature information uploaded by it to perform inference to obtain a joint result, the cloud generates a governance instruction according to the joint result and issues it to the marked edge node to execute governance actions and other technical means, which solves the technical problems of insufficient real-time and reliability of the existing multi-modal inference for park governance, and achieves the technical effects of improving the real-time and reliability of park governance.

[0054] In the foregoing, the multi-modal inference method for park governance is described in detail according to the embodiments of the present application. Next, the multi-modal inference system for park governance according to the embodiments of the present application will be described with reference to the accompanying drawings. Figure 1 The multi-modal inference method for park governance according to the embodiments of the present application is described in detail. Next, the multi-modal inference system for park governance according to the embodiments of the present application will be described with reference to the accompanying drawings. Figure 2 The multi-modal inference system for park governance according to the embodiments of the present application is described in detail. Next, the multi-modal inference system for park governance according to the embodiments of the present application will be described with reference to the accompanying drawings.

[0055] The multi-modal inference system for park governance according to the embodiments of the present application is used to solve the technical problems of insufficient real-time and reliability of the existing multi-modal inference for park governance, and achieve the technical effects of improving the real-time and reliability of park governance. The multi-modal inference system for park governance includes a single-modal inference module 10, a collaborative inference triggering judgment module 20, a joint inference module 30, and a governance instruction generation module 40.

[0056] The single-modal inference module 10 is used to collect and process local multi-modal sensor data through multiple edge nodes deployed in the park, and perform single-modal inference based on a pre-set lightweight AI model to generate multiple edge event description information. The collaborative inference triggering judgment module 20 is used to evaluate the multiple edge event description information based on a pre-set dynamic triggering mechanism in the multiple edge nodes to determine whether to trigger a collaborative inference request according to an event processing strategy. The joint inference module 30 is used to perform fusion inference on multi-modal feature information uploaded from a marked edge node based on an attention mechanism to generate a joint inference result, and the multi-modal feature information is a feature vector extracted from the edge event description information. The governance instruction generation module 40 is used to generate a governance instruction according to the joint inference result in the cloud and issue it to the marked edge node to drive execution of park governance actions.

[0057] The detailed description of the specific configuration of the single-modal inference module 10 is explained as follows: as described above, based on the pre-set lightweight AI model, the single-modal inference is performed to generate a plurality of edge event description information, and the single-modal inference module 10 can further include: an inference unit for the multi-modal sensing data at least including video data, audio data and Internet of Things sensor data, after pre-processing the multi-modal sensing data, the pre-processed data is input into the corresponding lightweight AI model embedded in the edge node to perform inference, and the inference result of each edge model is output, wherein the lightweight AI model corresponding to the video data is a target detection model, which outputs the target category and carries the confidence; the lightweight AI model corresponding to the audio data is an abnormal sound classification model, which outputs the sound event category and carries the confidence; the lightweight AI model corresponding to the Internet of Things sensor data is a rule engine, which outputs the state abnormality identifier and carries the confidence; a data encapsulation unit is used to extract a high-dimensional feature vector from the intermediate layer of the lightweight AI model, and the high-dimensional feature vector is encapsulated with each edge model inference result, the geographical position, the time stamp and the device identifier of data collection to generate the plurality of edge event description information.

[0058] The detailed description of the specific configuration of the collaborative inference trigger judgment module 20 is explained as follows: as described above, based on the pre-set dynamic trigger mechanism, the plurality of edge nodes evaluate the plurality of edge event description information to determine whether to trigger a collaborative inference request, and the collaborative inference trigger judgment module 20 can further include: an initial priority score assignment unit for assigning an initial priority score to each event type in the plurality of edge event description information based on a pre-defined event location-event type-priority mapping table to generate a plurality of initial event scores; a weighted calculation unit for weighting the confidence in the plurality of edge event description information and the plurality of initial event scores to obtain a plurality of comprehensive evaluation scores; a judgment unit for comparing the plurality of comprehensive evaluation scores with a pre-set high threshold and a pre-set low threshold to determine whether to trigger a collaborative inference request.

[0059] Among them, the plurality of comprehensive evaluation scores are compared with the pre-set high threshold and the pre-set low threshold to determine whether to trigger a collaborative inference request, and the judgment unit can further include: a local processing sub-unit for determining that the corresponding event processing strategy is processed by the edge node locally and discarded if any comprehensive evaluation score is lower than the pre-set low threshold; a reporting sub-unit for determining that the corresponding event processing strategy is triggered by the edge node to trigger a local response and reported to the cloud if any comprehensive evaluation score is higher than the pre-set high threshold; a collaborative inference request triggering sub-unit for determining that the event processing strategy is to trigger a collaborative inference request if any comprehensive evaluation score is between the pre-set low threshold and the pre-set high threshold.

[0060] The detailed description of the specific configuration of the joint inference module 30 is explained as follows: as described above, for the marked edge node triggering the collaborative inference request, the multi-modal feature information uploaded from the marked edge node is fused and inferred based on the attention mechanism to generate a joint inference result. The joint inference module 30 can further include: an association judgment unit for the marked edge node to search for event description information of other modalities generated in the same geographical area within a preset forward pushing time window in the local cache, and judge whether the event description information of other modalities is associated with the current edge event description information in time sequence and semantics; a vector assembly unit for assembling the feature vector of the current edge event and the feature vector of all associated events into a composite data packet if there is an association; a fusion inference unit for uploading the composite data packet as the payload of the collaborative inference request to the cloud for fusion inference based on the attention mechanism to generate the joint inference result.

[0061] Among them, the composite data packet is uploaded as the payload of the collaborative inference request to the cloud for fusion inference based on the attention mechanism to generate the joint inference result. The fusion inference unit can further include: a feature sequence generation subunit for mapping the composite data packet to a unified feature space according to modal classification to generate a feature sequence of each modality; a correlation weight calculation subunit for inputting the feature sequence of each modality into a fusion network based on the attention mechanism, calculating the correlation weight between different modal features through cross-attention mechanism; a weighted fusion subunit for weighting and fusing the feature sequence of each modality according to the correlation weight to obtain a fused context feature representation; and a joint inference result output subunit for outputting the joint inference result through a regressor based on the context feature representation.

[0062] Among them, after obtaining the fused context feature representation, the fusion inference unit can further include: a park knowledge graph query subunit for querying a pre-constructed park knowledge graph to obtain context information related to the joint inference result, the park knowledge graph at least including park area function attributes, device layout information, personnel permission information and historical event records; and an inference result optimization subunit for logically associating the joint inference result with the context information, including the association of risk level, semantic connotation and disposal priority, and optimizing the joint inference result.

[0063] Among them, the local processing subunit can further include: the steps of local processing and discarding by the edge node include: without triggering any alarm or response action, recording the edge event and related data in the local log, and automatically covering or deleting after reaching the preset storage time limit or storage space threshold.

[0064] The reporting subunit can further include: the step of triggering a local response by the edge node and reporting to the cloud includes: according to the type and level of the edge event, performing a predefined local response action; extracting the key information of the edge event to generate an event report, including at least event type, location, time, confidence, associated device ID and snapshot or fragment of field evidence; uploading the event report to the cloud through an asynchronous communication mode.

[0065] The cloud-edge collaborative multi-modal reasoning system for park management provided by the embodiments of the present application can execute the cloud-edge collaborative multi-modal reasoning method for park management provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0066] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, however, any number of different modules can be used and run on the user terminal and / or server, and each unit and module included is only divided according to the functional logic, but is not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific name of each functional unit is only for easy mutual differentiation, and does not limit the protection scope of the present application.

[0067] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with the preferred embodiment, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content to obtain equivalent embodiments with slight changes or modifications, as long as they do not deviate from the technical solution of the present application. Any modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application are still within the scope of the technical solution of the present application.

Claims

1. A cloud-edge collaborative multimodal reasoning method for park governance, characterized in that, include: By deploying multiple edge nodes within the park, local multimodal sensing data is collected and processed, and single-modal inference is performed based on a pre-built lightweight AI model to generate multiple edge event description information. Based on a preset dynamic triggering mechanism, the description information of the multiple edge events is evaluated at multiple edge nodes to determine whether the event processing strategy should trigger a collaborative reasoning request. For the marked edge nodes that trigger collaborative reasoning requests, based on the attention mechanism, the multimodal feature information uploaded from the marked edge nodes is fused and reasoned to generate a joint reasoning result. The multimodal feature information is a feature vector extracted from the edge event description information. Based on the joint inference results, the cloud generates governance instructions and sends them to the marked edge nodes to drive the execution of park governance actions.

2. The cloud-edge collaborative multimodal reasoning method for park governance as described in claim 1, characterized in that, Based on a pre-built lightweight AI model, single-modal reasoning is performed to generate multiple edge event descriptions, including: The multimodal sensing data includes at least video data, audio data, and IoT sensor data; After preprocessing the multimodal sensing data, it is input into the lightweight AI model embedded in the corresponding edge node for inference, and the inference results of each edge model are output. Among them, the lightweight AI model corresponding to the video data is an object detection model, which outputs the object category and carries a confidence score; the lightweight AI model corresponding to the audio data is an abnormal sound classification model, which outputs the sound event category and carries a confidence score; and the lightweight AI model corresponding to the IoT sensor data is a rule engine, which outputs an abnormal status identifier and carries a confidence score. High-dimensional feature vectors are extracted from the intermediate layer of the lightweight AI model and encapsulated with the inference results of each edge model, the geographical location of data collection, timestamps, and device identifiers to generate the description information of the multiple edge events.

3. The cloud-edge collaborative multimodal reasoning method for park governance as described in claim 1, characterized in that, Based on a preset dynamic triggering mechanism, multiple edge node descriptions are evaluated to determine whether a collaborative reasoning request should be triggered according to the event handling strategy. This includes: Based on a predefined event location-event type-priority mapping table, an initial priority score is assigned to each event type in the description information of the multiple edge events, generating multiple initial event scores; The confidence levels in the multiple edge event descriptions are weighted and calculated with the multiple initial event scores to obtain multiple comprehensive evaluation scores; The multiple comprehensive evaluation scores are compared with preset high thresholds and preset low thresholds to determine whether a collaborative reasoning request is triggered.

4. The cloud-edge collaborative multimodal reasoning method for park governance as described in claim 3, characterized in that, The multiple comprehensive evaluation scores are compared with preset high thresholds and preset low thresholds to determine whether to trigger a collaborative reasoning request, including: If any comprehensive evaluation score is lower than the preset low threshold, the corresponding event handling strategy is determined to be local processing by the edge node and discarded. If any comprehensive evaluation score is higher than the preset high threshold, the corresponding event handling strategy is determined to be that the edge node triggers a local response and reports it to the cloud; If any comprehensive evaluation score falls between the preset low threshold and the preset high threshold, the event handling strategy is determined to be to trigger a collaborative reasoning request.

5. The cloud-edge collaborative multimodal reasoning method for park governance as described in claim 1, characterized in that, For the marked edge nodes that trigger collaborative reasoning requests, based on an attention mechanism, multimodal feature information uploaded from the marked edge nodes is fused and reasoned to generate joint reasoning results, including: The marked edge node is stored in the local cache. It retrieves event description information of other modalities generated in the same geographical area within a preset forward time window, and determines whether the event description information of other modalities is related to the current edge event description information in terms of time sequence and semantics. If there is a correlation, the feature vector of the current edge event and the feature vectors of all related events are combined into a composite data packet. The composite data packet is used as the payload of the collaborative reasoning request and uploaded to the cloud for fusion reasoning based on an attention mechanism to generate the joint reasoning result.

6. The cloud-edge collaborative multimodal reasoning method for park governance as described in claim 5, characterized in that, The composite data packet is used as the payload of the collaborative reasoning request and uploaded to the cloud for fusion reasoning based on an attention mechanism to generate the joint reasoning result, including: The composite data packets are classified and mapped to a unified feature space according to their modalities, generating feature sequences for each modality; The feature sequences of each modality are input into the attention-based fusion network, and the correlation weights between features of different modalities are calculated through cross-attention mechanism. Based on the relevance weights, the feature sequences of each modality are weighted and fused to obtain the fused context feature representation; Based on the contextual feature representation, the joint inference result is output through a regressor.

7. The cloud-edge collaborative multimodal reasoning method for park governance as described in claim 6, characterized in that, After obtaining the fused contextual feature representation, the following is also included: Query the pre-built park knowledge graph to obtain contextual information related to the joint reasoning result. The park knowledge graph includes at least the park area functional attributes, equipment layout information, personnel permission information, and historical event records. The joint reasoning results are logically correlated with the contextual information, including the correlation of risk level, semantic connotation, and handling priority, and the joint reasoning results are optimized.

8. The cloud-edge collaborative multimodal reasoning method for park governance as described in claim 4, characterized in that, The steps for local processing and discarding by edge nodes include: Without triggering any alarms or response actions, edge events and related data are recorded in the local log, and automatically overwritten or deleted after the preset storage time limit or storage space threshold is reached.

9. The cloud-edge collaborative multimodal reasoning method for park governance as described in claim 4, characterized in that, The steps for an edge node to trigger a local response and report it to the cloud include: Execute predefined local response actions based on the type and level of the edge event; Extract key information from edge events to generate event reports, including at least the event type, location, time, confidence level, associated device ID, and snapshots or fragments of on-site evidence; The event report is uploaded to the cloud via asynchronous communication.

10. A cloud-edge collaborative multimodal reasoning system for park governance, characterized in that: The system is used to implement the cloud-edge collaborative multimodal reasoning method for park governance as described in any one of claims 1-9, and the system includes: The single-modal inference module is used to collect and process local multimodal sensing data through multiple edge nodes deployed in the park, and perform single-modal inference based on a pre-built lightweight AI model to generate multiple edge event description information. The collaborative reasoning trigger determination module is used to evaluate the description information of multiple edge events at multiple edge nodes based on a preset dynamic triggering mechanism, so as to determine whether the event processing strategy should trigger a collaborative reasoning request. The joint reasoning module is used to perform fusion reasoning on the multimodal feature information uploaded from the marked edge nodes that trigger the collaborative reasoning request, based on an attention mechanism, to generate a joint reasoning result. The multimodal feature information is a feature vector extracted from the edge event description information. The governance instruction generation module is used to generate governance instructions in the cloud based on the joint inference results and send them to the marked edge nodes to drive the execution of park governance actions.

Images