An intelligent park multi-modal risk early warning method and device based on artificial intelligence

Abstract

The application discloses an intelligent park multi-modal risk early warning method and device based on artificial intelligence, belongs to the technical field of intelligent parks, and evaluates the modal confidence of multi-modal heterogeneous data, obtains multi-modal fusion features and multi-modal risk probabilities according to the multi-modal confidence obtained through evaluation, determines the decision inconsistency degree according to the multi-modal risk probabilities, and performs adaptive risk early warning path selection and risk identification according to the multi-modal risk probabilities and the decision inconsistency degree to obtain a target risk probability; the multi-modal fusion features are input into an artificial intelligence model to predict a risk spreading path, and a target risk spreading path is obtained; hierarchical early warning and linkage response are performed according to the target risk probability and the target risk spreading path, the problems of low timeliness of early warning, incapability of decision making when evidences conflict and lack of physical interpretability in the prior art are solved, and the accuracy and intelligent level of park safety early warning are significantly improved.

Description

Technical Field

[0001] This application belongs to the field of smart park technology, specifically relating to a method and device for multimodal risk early warning in smart parks based on artificial intelligence. Background Technology

[0002] With the development of IoT and AI technologies, smart park security management systems are transforming from traditional passive monitoring to proactive early warning. Parks are characterized by dense populations, complex equipment, and dynamically changing operations, resulting in numerous security risks and hazards such as fires, leaks, unauthorized personnel operations, and intrusions.

[0003] Existing park risk early warning systems typically analyze data using a single modality. Examples include image recognition systems based on video surveillance or threshold alarm systems based on IoT sensors. However, single-modality early warning methods have significant limitations: First, they lack robustness; video surveillance is susceptible to lighting and obstruction, and sensors are prone to false alarms due to environmental interference. Second, the information is incomplete; for example, a smoke sensor alarm alone cannot determine whether it's a real fire or dust interference, and seeing people gathering in video footage alone cannot determine if it's accompanied by abnormal sounds. Third, existing systems are mostly reactive, meaning they only trigger an alarm after the risk has occurred and reached a certain threshold, lacking the ability to anticipate the spatiotemporal spread of risks.

[0004] While existing technologies propose fusing multi-source heterogeneous data, the commonly used general feature-layer fusion or decision-layer fusion methods cannot dynamically capture the correlation importance of different modalities in complex environments, and lack effective mechanisms to dynamically reduce the weight of disturbed modalities. Furthermore, existing technologies employ LSTM (Long Short-Term Memory) networks and TransE graph embedding techniques for temporal situation prediction. However, the prediction performance drops sharply in data-sparse areas (such as corners of a park without historical fire records). In addition, these methods rely on predefined ontology relationships and static rules, making it difficult to accurately quantify and extrapolate the dynamic spread of risk in the physical space over time. Summary of the Invention

[0005] The purpose of this application is to provide a smart park multimodal risk early warning method and device based on artificial intelligence, so as to solve the problems of false alarms and missed alarms in single-modal early warning, lack of dynamic weight allocation mechanism in multimodal fusion and / or inability to make decisions when evidence conflicts.

[0006] The first aspect of this application provides a multimodal risk early warning method for smart parks based on artificial intelligence, including:

[0007] Multimodal heterogeneous data is collected by multi-source heterogeneous sensors deployed in the smart park, and the multimodal heterogeneous data is preprocessed to obtain multimodal data features.

[0008] Modality confidence is evaluated on the multimodal heterogeneous data to determine the multimodal confidence corresponding to the multimodal heterogeneous data, and multi-source heterogeneous feature fusion is performed based on the multimodal confidence and the multimodal data features to obtain multimodal fusion features;

[0009] The multimodal risk probability is determined by the multimodal confidence level, and the degree of decision inconsistency is determined based on the multimodal risk probability. Based on the multimodal risk probability and the degree of decision inconsistency, adaptive risk warning path selection and risk identification are performed to obtain the target risk probability.

[0010] The multimodal fusion features are input into a pre-set artificial intelligence model to predict the risk propagation path, thereby obtaining the target risk propagation path; the artificial intelligence model is set as a situation inference model based on a physical information spatiotemporal graph convolutional network.

[0011] Based on the target risk probability and the target risk spread path, a graded early warning and coordinated response are implemented to realize a smart park multimodal risk early warning method based on artificial intelligence.

[0012] Furthermore, multimodal heterogeneous data is collected through multi-source heterogeneous sensors deployed within the smart park, and the multimodal heterogeneous data is preprocessed to obtain multimodal data features, including:

[0013] Multimodal heterogeneous data is obtained by collecting visual monitoring video streams, audio streams, and sensing data from IoT sensors through multi-source heterogeneous sensors deployed in the smart park; the IoT sensors include temperature sensors, smoke concentration sensors, and combustible gas concentration sensors.

[0014] Using a unified clock provided by a Network Time Protocol (NTP) server, the image frames in the visual monitoring video stream at the current moment are determined to obtain visual modal data; based on the current moment, the audio stream is extracted in the form of a time window to obtain sound modal data; based on the current moment, sensor data is extracted in the form of a time window to obtain supplementary modal data.

[0015] The visual features are obtained by acquiring the features of the visual modality data through a visual feature extraction model.

[0016] The time-domain data and frequency-domain data of the sound modal data are obtained, and the time-domain data and frequency-domain data are concatenated to obtain concatenated data; the features of the concatenated data are obtained through a sound feature extraction model to obtain sound features;

[0017] The supplementary features are obtained by acquiring the features of the supplementary modality data through a supplementary feature extraction model.

[0018] The visual features, sound features, and supplementary features are collectively used as multimodal data features.

[0019] Further, modal confidence assessment is performed on the multimodal heterogeneous data to determine the multimodal confidence corresponding to the multimodal heterogeneous data, including:

[0020] The visual modality confidence level corresponding to the visual modality data is obtained as follows:

[0021]

[0022] in, Indicates the confidence level of the visual modality. Represents a sigmoid function, and x represents the variable of the sigmoid function, k represents the steepness coefficient of the sigmoid function, e represents the natural constant, and I represents the current illumination, which is collected by the light sensor deployed at the camera location. This indicates the preset minimum effective illuminance threshold. H represents the preset optimal illumination threshold, and H represents the current image information entropy. This indicates the preset minimum information entropy threshold. This indicates the preset maximum information entropy threshold;

[0023] The sound modal confidence level corresponding to the sound modal data is obtained as follows:

[0024]

[0025] in, SNR represents the signal-to-noise ratio, which indicates the confidence level of the sound modality. This indicates the preset minimum effective signal-to-noise ratio threshold. This indicates the preset optimal signal-to-noise ratio threshold. Represents the background noise energy spectrum. This represents the energy spectrum of the current audio signal;

[0026] The supplementary modal confidence level corresponding to the supplementary modal data is obtained as follows:

[0027]

[0028] in, Indicates the supplementary modal confidence level. This indicates the cumulative number of communication verification errors within a preset time window. This indicates the total number of communications within the preset time window. This represents a very small constant to prevent division by zero, and is set to 0.001; This represents the chain multiplication operator. Denotes the Gaussian decay function, and z represents the variable of the Gaussian decay function. This represents the current measured value of the i-th sensor parameter. This represents the center value of the normal range for the i-th sensor parameter. The normal range half-width of the i-th sensor parameter is represented by m, and m represents the total number of IoT sensors.

[0029] The visual modality confidence, the audio modality confidence, and the supplementary modality confidence are collectively used as the multimodal confidence corresponding to the multimodal heterogeneous data.

[0030] Further, based on the multimodal confidence level and the multimodal data features, multi-source heterogeneous feature fusion is performed to obtain multimodal fused features, including:

[0031] Using the visual features in the multimodal data features as the query and the sound features as the key and value, a cross-attention mechanism is used to enhance the visual features, resulting in enhanced visual features.

[0032] Using the supplementary features as queries and the enhanced visual features as keys and values, a cross-attention mechanism is employed to perform feature fusion, resulting in multimodal fusion features.

[0033] Furthermore, determining the multimodal risk probability through the multimodal confidence level includes:

[0034] The confidence level of the visual modality is multiplied into the visual features to obtain the visual feature vector with enhanced confidence.

[0035] The confidence level of the sound modality is multiplied into the sound features to obtain the sound feature vector with enhanced confidence.

[0036] The confidence scores of the supplementary modes are multiplied into the supplementary features to obtain the supplementary feature vector with enhanced confidence.

[0037] The multi-path fully connected layer processes the confidence-enhanced visual feature vector, the confidence-enhanced audio feature vector, and the confidence-enhanced supplementary feature vector to obtain the first risk probability, the second risk probability, and the third risk probability, thus obtaining the multimodal risk probability.

[0038] Furthermore, the degree of decision inconsistency is determined based on the multimodal risk probability as follows:

[0039]

[0040] in, Indicates the degree of inconsistency in decision-making. This represents the first risk probability obtained through the visual feature vector enhanced with confidence. This represents the second risk probability obtained through the confidence-enhanced sound feature vector. This represents the third risk probability obtained through the supplementary feature vector enhanced with confidence.

[0041] Furthermore, based on the multimodal risk probability and the degree of decision inconsistency, adaptive risk warning path selection and risk identification are performed to obtain the target risk probability, including:

[0042] If the degree of decision inconsistency is less than a preset inconsistency threshold, then the target risk probability is obtained based on the multimodal risk probability as follows:

[0043]

[0044] in, Indicates the probability of target risk. Indicates the confidence level of the visual modality. Indicates the confidence level of the sound modality. Indicates the supplementary modal confidence level. This represents the risk probability output by the dominant path with the highest confidence level. This represents the attenuation coefficient of the non-dominant path. Indicates the dominant path, which can be a visual path, an audio path, or a supplementary path; This represents the confidence level corresponding to the j-th path. This represents the confidence level corresponding to the j-th path in the multimodal risk probability;

[0045] If the degree of decision inconsistency is greater than or equal to a preset inconsistency threshold, then the DS evidence theory is used to fuse the multimodal risk probabilities to determine the target risk probability.

[0046] Furthermore, the multimodal fusion features are input into a pre-set artificial intelligence model to predict the risk propagation path, thereby obtaining the target risk propagation path, including:

[0047] The smart park is divided into N logical regions, each of which is considered a graph node. Assuming a connected path exists between any two graph nodes, an edge is constructed between them, with the edge weight set as follows:

[0048]

[0049] in, Represents graph nodes With graph nodes Edge weights between them Represents graph nodes With graph nodes Minimum cross-sectional area of ​​the passage between them Represents graph nodes With graph nodes The physical distance between them Represents graph nodes With graph nodes The preset ventilation coefficient between;

[0050] The graph nodes, edges, and edge weights are used to construct a target graph, and the features of the graph nodes are set as multiple consecutive historical multimodal fusion features and the current multimodal fusion features.

[0051] The target map is constructed as the input of the pre-set artificial intelligence model, and a control vector is inserted into the intermediate layer of the pre-set artificial intelligence model to output a risk heat map;

[0052] Based on the aforementioned risk heatmap, the target risk propagation path is analyzed.

[0053] Furthermore, based on the target risk probability and the target risk spread path, a tiered early warning and coordinated response are implemented, including:

[0054] If the probability of the target risk is less than the first preset risk threshold, it is determined to be low risk, and a pop-up window will appear on the system interface to remind security personnel to pay close attention.

[0055] If the target risk probability is greater than or equal to the first preset risk threshold and less than the second preset risk threshold, it is determined to be a medium risk, and an early warning is triggered. The audio and video footage of the medium risk incident location will automatically pop up, and a broadcast will be made to notify people near the medium risk incident location to pay attention; the first preset risk threshold is less than the second preset risk threshold.

[0056] If the probability of the target risk is greater than or equal to the second preset risk threshold, it is determined to be a high risk, triggering an emergency alarm, automatically activating the sprinkler system, opening the fire door, cutting off the non-explosion-proof power supply, and broadcasting the target risk spread path.

[0057] The second aspect of this application provides a smart park multimodal risk early warning device based on artificial intelligence, comprising:

[0058] The multimodal data processing module is used to collect multimodal heterogeneous data through multi-source heterogeneous sensors deployed in the smart park, and to preprocess the multimodal heterogeneous data to obtain multimodal data features.

[0059] The multimodal feature fusion module is used to evaluate the modal confidence of the multimodal heterogeneous data, determine the multimodal confidence corresponding to the multimodal heterogeneous data, and perform multi-source heterogeneous feature fusion based on the multimodal confidence and the features of the multimodal data to obtain multimodal fused features.

[0060] The risk probability analysis module is used to determine the multimodal risk probability through the multimodal confidence level, determine the degree of decision inconsistency based on the multimodal risk probability, and perform adaptive risk warning path selection and risk identification based on the multimodal risk probability and the degree of decision inconsistency to obtain the target risk probability.

[0061] The risk contagion analysis module is used to input the multimodal fusion features into a pre-set artificial intelligence model to predict the risk contagion path and obtain the target risk contagion path; the artificial intelligence model is set as a situation inference model based on a physical information spatiotemporal graph convolutional network.

[0062] The hierarchical linkage processing module is used to perform hierarchical early warning and linkage response based on the target risk probability and the target risk spread path, so as to realize a smart park multimodal risk early warning method based on artificial intelligence.

[0063] The beneficial effects of this application are as follows:

[0064] This application proposes an AI-based multimodal risk early warning method for smart parks. It assesses the modal confidence of multimodal heterogeneous data and obtains multimodal fusion features and multimodal risk probabilities based on the assessed multimodal confidence. Then, it determines the degree of decision inconsistency based on the multimodal risk probabilities and performs adaptive risk early warning path selection and risk identification based on the multimodal risk probabilities and the degree of decision inconsistency to obtain the target risk probability. The multimodal fusion features are input into an AI model to predict the risk propagation path, resulting in the target risk propagation path. Based on the target risk probability and the target risk propagation path, it performs hierarchical early warning and coordinated response, realizing an AI-based multimodal risk early warning method for smart parks. This method solves the problems of low early warning timeliness, inability to make decisions when evidence conflicts, and lack of physical interpretability in existing technologies, significantly improving the accuracy and intelligence level of park safety early warning. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0066] Figure 1 A flowchart illustrating a multimodal risk early warning method for smart parks based on artificial intelligence, provided for this application;

[0067] Figure 2 A schematic diagram of the structure of a smart park multimodal risk early warning device based on artificial intelligence provided in this application;

[0068] Among them, 201-Multimodal data processing module, 202-Multimodal feature fusion module, 203-Risk probability analysis module, 204-Risk probability analysis module, and 205-Hierarchical linkage processing module.

[0069] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0071] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0072] like Figure 1 As shown in the figure, this application provides a method for multimodal risk early warning in smart parks based on artificial intelligence, including:

[0073] S101. Collect multimodal heterogeneous data through multi-source heterogeneous sensors deployed in the smart park, and preprocess the multimodal heterogeneous data to obtain multimodal data features;

[0074] S102. The modal confidence of the multimodal heterogeneous data is evaluated to determine the multimodal confidence corresponding to the multimodal heterogeneous data, and multi-source heterogeneous feature fusion is performed based on the multimodal confidence and the features of the multimodal data to obtain multimodal fusion features.

[0075] S103. Determine the multimodal risk probability through the multimodal confidence level, determine the degree of decision inconsistency based on the multimodal risk probability, and perform adaptive risk warning path selection and risk identification based on the multimodal risk probability and the degree of decision inconsistency to obtain the target risk probability.

[0076] S104. Input the multimodal fusion features into a pre-set artificial intelligence model to predict the risk propagation path and obtain the target risk propagation path; the artificial intelligence model is set as a situation inference model based on a physical information spatiotemporal graph convolutional network.

[0077] S105. Based on the target risk probability and the target risk propagation path, perform graded early warning and linkage response to realize a smart park multimodal risk early warning method based on artificial intelligence.

[0078] In some possible embodiments, multimodal heterogeneous data is collected by multi-source heterogeneous sensors deployed in a smart park, and the multimodal heterogeneous data is preprocessed to obtain multimodal data features, including:

[0079] Multimodal heterogeneous data is obtained by collecting visual monitoring video streams, audio streams, and sensing data from IoT sensors through multi-source heterogeneous sensors deployed in the smart park; the IoT sensors include temperature sensors, smoke concentration sensors, and combustible gas concentration sensors.

[0080] For example, multi-source heterogeneous sensors deployed in smart parks can collect multimodal heterogeneous data in real time. This multimodal heterogeneous data can include: visual surveillance video streams collected by high-definition network cameras, auditory audio streams collected by microphone arrays, and sensor data collected by IoT sensors.

[0081] Using a unified clock provided by a Network Time Protocol (NTP) server, the image frames in the visual monitoring video stream at the current moment are determined to obtain visual modal data; based on the current moment, the audio stream is extracted in the form of a time window to obtain sound modal data; based on the current moment, sensor data is extracted in the form of a time window to obtain supplementary modal data.

[0082] For example, if the time window length is L and the current time is t, then the audio stream and sensor data within the range [tL, t] can be acquired. For the sensor data, it can be collected at a preset fixed data sampling frequency to form time-series data.

[0083] The visual features are obtained by acquiring the features of the visual modality data through a visual feature extraction model.

[0084] For example, the visual feature extraction model can be set as a convolutional neural network model. This model can be pre-trained with two labels: "fire present" and "fire not present," and the fixed-dimensional feature vectors output by the fully connected layers of the trained convolutional neural network model can be used as visual features. However, it's worth noting that the visual feature extraction model can also be set as other object recognition models, such as the YOLO model.

[0085] The time-domain data and frequency-domain data of the sound modal data are obtained, and the time-domain data and frequency-domain data are concatenated to obtain concatenated data; the features of the concatenated data are obtained through a sound feature extraction model to obtain sound features;

[0086] For example, the sound feature extraction model can be set as a long short-term memory network model or a convolutional neural network model, and the training process of the sound feature extraction model is the same as that of the visual feature extraction model, which will not be described in detail here.

[0087] The supplementary features are obtained by acquiring the features of the supplementary modality data through a supplementary feature extraction model.

[0088] For example, the supplementary feature extraction model can be set as a long short-term memory network model or a convolutional neural network model, and the training process of the supplementary feature extraction model is the same as that of the visual feature extraction model, which will not be described in detail here.

[0089] The visual features, sound features, and supplementary features are collectively used as multimodal data features.

[0090] In some possible embodiments, modal confidence assessment is performed on the multimodal heterogeneous data to determine the multimodal confidence corresponding to the multimodal heterogeneous data, including:

[0091] The visual modality confidence level corresponding to the visual modality data is obtained as follows:

[0092]

[0093] in, represents the confidence level of the visual modality, is dimensionless, and takes values ​​in the range [0,1]. Represents a sigmoid function, and , used to map the normalized physical quantity to the interval [0,1]; x represents the variable of the sigmoid function; k represents the steepness coefficient of the sigmoid function, and is set to 10 to control the steepness of the mapping curve; e represents the natural constant; I represents the current illumination, collected by the light sensor deployed at the camera location, in lux. This represents the preset minimum effective illuminance threshold. Below this value, visual data is essentially invalid. =50 lux; This represents the preset optimal illuminance threshold. When this value is reached or exceeded, the reliability of visual data tends to saturate. =500 lux; H represents the current image information entropy, and ; This represents the probability distribution of grayscale value g in an image, expressed in bits. This indicates the preset minimum information entropy threshold, which is set to 3 bits. Below this value, the image content is too monotonous or too dark. This indicates the preset maximum information entropy threshold, which is set to 7 bits. When this value is reached or exceeded, the image information richness meets the requirements for reliable recognition.

[0094] The sound modal confidence level corresponding to the sound modal data is obtained as follows:

[0095]

[0096] in, represents the confidence level of the sound modality, is dimensionless, and ranges from [0,1]; SNR (Sound Noise Ratio) is in decibels. This indicates the preset minimum effective signal-to-noise ratio threshold, which is set to 10dB. Below this value, the audio signal will be submerged in noise. This indicates the preset optimal signal-to-noise ratio threshold, set to a value between 30-60dB. When this value is reached or exceeded, the audio signal quality meets the requirements for reliable recognition. The background noise energy spectrum is calculated by performing a fast Fourier transform on the audio during a quiet period without any events, and the sum of squares of the energy in each frequency band is given in joules. This represents the energy spectrum of the current audio signal, which is obtained by calculating the sum of squares of the energy in each frequency band after performing a fast Fourier transform on the real-time acquired audio, and the unit is joules;

[0097] The supplementary modal confidence level corresponding to the supplementary modal data is obtained as follows:

[0098]

[0099] in, This represents the supplementary modal confidence level, which is dimensionless and ranges from [0,1]. This indicates the cumulative number of communication verification errors within a preset time window, which is 60 seconds. This indicates the total number of communications within the preset time window. This represents a very small constant to prevent division by zero, and is set to 0.001; This represents the chain multiplication operator. Denotes the Gaussian decay function, and z represents the variable of the Gaussian decay function. This represents the current measured value of the i-th sensor parameter, in degrees Celsius (°C), parts per million (ppm), or percentage of the lower explosive limit (%LEL), depending on the parameter type. This represents the center value of the normal range for the i-th sensor parameter, where temperature is taken as... =25 Smoke concentration is taken =0ppm, combustible gas concentration taken =0%LEL; This represents the half-width of the normal range for the i-th sensor parameter, where temperature is taken as... =5°C, smoke concentration taken =50ppm, combustible gas concentration taken as =5%LEL; m represents the total number of IoT sensors;

[0100] The visual modality confidence, the audio modality confidence, and the supplementary modality confidence are collectively used as the multimodal confidence corresponding to the multimodal heterogeneous data.

[0101] In some possible embodiments, multi-source heterogeneous feature fusion is performed based on the multimodal confidence and the multimodal data features to obtain multimodal fused features, including:

[0102] Using the visual features in the multimodal data features as the query and the sound features as the key and value, a cross-attention mechanism is used to enhance the visual features, resulting in enhanced visual features.

[0103] Using the supplementary features as queries and the enhanced visual features as keys and values, a cross-attention mechanism is employed to perform feature fusion, resulting in multimodal fusion features.

[0104] For example, it can be based on visual features For querying, based on audio features Given a key and a value, the enhanced visual features are calculated as follows:

[0105]

[0106] in, Indicates enhanced visual features, This represents the cross-attention mechanism, where Q indicates the use of visual features. The constructed query, K, indicates the use of audio features. The constructed key, V, indicates the use of audio features. The constructed value.

[0107] It can be used to supplement features Query, with enhanced visual features Given keys and values, the multimodal fusion features are obtained as follows:

[0108]

[0109] in, Indicates multimodal fusion features, Indicates the use of supplementary features The constructed query Indicates the use of enhanced visual features The key to be constructed Indicates the use of enhanced visual features The constructed value.

[0110] In some possible embodiments, determining the multimodal risk probability through the multimodal confidence level includes:

[0111] The visual modality confidence score is multiplied into the visual features to obtain the confidence-enhanced visual feature vector; for example, the confidence-enhanced visual feature vector can be obtained as follows: ;

[0112] The confidence level of the sound modality is multiplied into the sound features to obtain the confidence-enhanced sound feature vector; for example, the confidence-enhanced sound feature vector can be obtained as follows: ;

[0113] The supplementary modality confidence is multiplied into the supplementary features to obtain the confidence-enhanced supplementary feature vector; for example, the confidence-enhanced supplementary feature vector can be obtained as follows: ;

[0114] The multi-path fully connected layer processes the confidence-enhanced visual feature vector, the confidence-enhanced audio feature vector, and the confidence-enhanced supplementary feature vector to obtain the first risk probability, the second risk probability, and the third risk probability, thus obtaining the multimodal risk probability.

[0115] For example, the first risk probability, the second risk probability, and the third risk probability can be respectively:

[0116]

[0117] in, Indicates the probability of the first risk. Indicates the probability of the second risk. This represents the probability of the third risk. Softmax is a normalized exponential function, and the output is a two-dimensional probability distribution. The weight matrix of the fully connected layer representing the visual path. The weight matrix of the fully connected layer representing the audio path. The weight matrix of the fully connected layer represents the sensing path; The bias vector representing the visual path. The bias vector representing the audio path. The bias vector represents the sensing path.

[0118] However, it's worth noting that a backpropagation (BP) neural network can also be used to train the risk probability identification model. This model can be trained by using the enhanced visual feature vector, enhanced sound feature vector, or enhanced supplementary feature vector as input, and by employing pre-defined labels indicating either a fire presence or absence. In subsequent identification processes, the trained risk probability identification model can be directly used.

[0119] In some possible embodiments, the degree of decision inconsistency is determined based on the multimodal risk probability as follows:

[0120]

[0121] in, It represents the degree of inconsistency in decision-making, is dimensionless, and takes values ​​in the range [0,1]. This represents the first risk probability obtained through the visual feature vector enhanced with confidence. This represents the second risk probability obtained through the confidence-enhanced sound feature vector. This represents the third risk probability obtained through the supplementary feature vector enhanced with confidence.

[0122] In some possible embodiments, based on the multimodal risk probability and the degree of decision inconsistency, adaptive risk warning path selection and risk identification are performed to obtain the target risk probability, including:

[0123] If the degree of decision inconsistency is less than a preset inconsistency threshold, then the target risk probability is obtained based on the multimodal risk probability as follows:

[0124]

[0125] in, Indicates the probability of target risk. Indicates the confidence level of the visual modality. Indicates the confidence level of the sound modality. Indicates the supplementary modal confidence level. This represents the risk probability output by the dominant path with the highest confidence level. This represents the attenuation coefficient of the non-dominant path. Indicates the dominant path, which can be a visual path, an audio path, or a supplementary path; This represents the confidence level corresponding to the j-th path. This represents the confidence level corresponding to the j-th path in the multimodal risk probability;

[0126] Optionally, the inconsistency threshold can be set to 0.4. When the degree of decision inconsistency is greater than or equal to 0.4, it can be considered that there is a significant contradiction between the evidence from different modalities. At this time, decision-level fusion based on Dempster-Shafer evidence theory is initiated.

[0127] If the degree of decision inconsistency is greater than or equal to a preset inconsistency threshold, then the DS evidence theory is used to fuse the multimodal risk probabilities to determine the target risk probability.

[0128] In the execution of the DS evidence theory, the evidence uncertainty can be set to 0.1, which represents the basic confidence level of the model in its output. The remaining 10% confidence is allocated to the uncertainty, and the rest can be fused according to existing technology.

[0129] In some possible embodiments, the multimodal fusion features are input into a pre-set artificial intelligence model to predict the risk propagation path, thereby obtaining the target risk propagation path, including:

[0130] The smart park is divided into N logical regions, each of which is considered a graph node. Assuming a connected path exists between any two graph nodes, an edge is constructed between them, with the edge weight set as follows:

[0131]

[0132] in, Represents graph nodes With graph nodes The edge weights between the two regions are used to quantify the efficiency of heat or mass exchange between the two regions. Represents graph nodes With graph nodes Minimum cross-sectional area of ​​the passage between them Represents graph nodes With graph nodes The physical distance between them Represents graph nodes With graph nodes The preset ventilation coefficient is dimensionless and ranges from [0,1]. It is determined by the duct design drawings or the measured wind speed. The higher the ventilation efficiency, the larger this value.

[0133] For example, a smart park can be divided into N logical regions, which are represented as graph nodes v∈V in the graph G=(V,E), where V represents the set of graph nodes and N represents the total number of graph nodes. If two regions are physically connected by a path, including corridors, doors, pipes, and vents, then an edge e∈E is established between the corresponding nodes, where E represents the set of edges. Here, a logical region refers to a basic monitoring unit obtained by discretizing the physical space of the smart park according to function, structure, or risk monitoring requirements. Each logical region, as a node in the graph structure, represents the smallest spatial granularity for risk situation deduction and propagation direction analysis.

[0134] The graph nodes, edges, and edge weights are used to construct a target graph, and the features of the graph nodes are set as multiple consecutive historical multimodal fusion features and the current multimodal fusion features.

[0135] For example, the features of a graph node can be set as follows:

[0136]

[0137] in, Representing the characteristics of graph nodes, Indicates the first Multimodal fusion features of graph nodes at time t Indicates the first Each graph node at time... Multimodal fusion features, Indicates the first Each graph node at time... The multimodal fusion features are defined by T, which represents the length of the historical time window. T = 10 time steps are taken, with each time step spaced 1 second apart.

[0138] The target map is constructed as the input of the pre-set artificial intelligence model, and a control vector is inserted into the intermediate layer of the pre-set artificial intelligence model to output a risk heat map;

[0139] For example, suppose the artificial intelligence model is set as a situation inference model based on a spatiotemporal graph convolutional network of physical information, and the situation inference model is implemented by alternating stacking of spatial graph convolution and temporal convolution. Then the features output by the spatial graph convolution can be fused with the control vector and used as the input of the temporal convolution.

[0140] The control vector can be defined as:

[0141]

[0142] in, Represents the control vector. Indicates the spray status, with a value of 0 (not started) or 1 (started); This represents the smoke exhaust velocity, measured in meters per second (m / s), with a value range of [0,3]. The ventilation status is 0 (off) or 1 (on). It's worth noting that the smoke exhaust velocity can also be normalized. The parameters in the control vector can be directly read from the fire control system.

[0143] The control vector is concatenated with the features output by the spatial graph convolution, and then mapped back to the original dimension through a fully connected layer. This data is used as input to the temporal convolution, enabling the AI ​​model to output a risk heatmap, which provides the fire risk for each graph node.

[0144] Based on the aforementioned risk heatmap, the target risk propagation path is analyzed.

[0145] For example, we can first calculate the risk probability gradient between adjacent nodes as follows: , Indicates the first The graph node to the first The risk probability gradient of the nth graph node, where a negative value indicates that the risk starts from the nth node. The graph node to the first Propagation of graph nodes; Indicates the first Risk probability of each graph node Indicates the first Risk probability of each graph node Indicates the first The graph node and the first The physical distance between nodes in the graph can be extracted from the building information model of the park.

[0146] For each graph node, the rate of risk change can be obtained as follows:

[0147]

[0148] in, Indicates the first The rate of risk change for each node in the graph is positive if it indicates an increase in risk. Indicates the first time at time t Risk probability of each graph node Indicates time Upper Risk probability of each graph node This represents the time step, and its value is 1 second.

[0149] When the The graph node to the first The risk probability gradient of the nth graph node is negative, and the nth node is negative. If the rate of risk change of a node in a graph is positive, then the node with the highest risk is determined. The graph node points to the first The process involves spreading through all graph nodes to determine the path of the target risk's spread.

[0150] Alternatively, the loss function can be constructed as follows:

[0151]

[0152]

[0153] in, Represents the loss function. This represents the root mean square error. This represents the loss weighting coefficient. This represents the physical information loss function, where N represents the total number of graph nodes. Indicates the first The predicted temperature value of each graph node at time t. Indicates the first The predicted temperature value of each graph node at time t+1. Indicates the first The predicted temperature value of each graph node at time t. Indicates the first The set of neighboring nodes of a graph node This represents the thermal diffusivity, and is set to 0.01. It is worth noting that, in order to enable the various formulas to perform calculations effectively, the various parameters can also be dimensionless.

[0154] This loss function can be used to train an artificial intelligence model, thereby obtaining a pre-set artificial intelligence model.

[0155] In some possible embodiments, graded early warning and coordinated response are performed based on the target risk probability and the target risk spread path, including:

[0156] If the probability of the target risk is less than the first preset risk threshold, it is determined to be low risk, and a pop-up window will appear on the system interface to remind security personnel to pay close attention.

[0157] If the target risk probability is greater than or equal to the first preset risk threshold and less than the second preset risk threshold, it is determined to be a medium risk, and an early warning is triggered. The audio and video footage of the medium risk incident location will automatically pop up, and a broadcast will be made to notify people near the medium risk incident location to pay attention; the first preset risk threshold is less than the second preset risk threshold.

[0158] If the probability of the target risk is greater than or equal to the second preset risk threshold, it is determined to be a high risk, triggering an emergency alarm, automatically activating the sprinkler system, opening the fire door, cutting off the non-explosion-proof power supply, and broadcasting the target risk spread path.

[0159] Optionally, personnel can be guided to evacuate along a predicted safe route via broadcast. This safe route is planned backwards from the predicted risk spread path. Alternatively, a risk threshold can be pre-set, and all graph nodes with a target risk probability greater than the threshold can be designated as obstacle nodes. The safe exit can be designated as the destination node, and the node where the personnel are located can be designated as the starting node. An ant colony algorithm can be used to search for a safe route with the goal of minimizing the total target risk probability of the safe route.

[0160] Based on the above technical solution, this application has the following technical effects:

[0161] 1. Physical quantity-driven modal confidence assessment significantly improves robustness.

[0162] This application does not rely on black-box attention learning, but instead directly calculates the reliability confidence of each modality's data through physical quantities such as illumination, signal-to-noise ratio, and sensor self-diagnostic status. Compared to existing technologies that use static feature layer stitching or data-driven weighted fusion, this application can automatically reduce the weight of the interfered modalities under extreme interference conditions such as camera obstruction or sudden drops in microphone signal-to-noise ratio. According to embodiments, when the video modality completely fails, the system can maintain the overall risk identification accuracy through the audio and sensor-dominated paths.

[0163] 2. Quantification of conflicting evidence and resolution of Dempster-Shafer evidence theory to solve the dilemma of contradictory decision-making.

[0164] This application defines an evidence conflict coefficient for the first time in multimodal park early warning and designs a decision-level fusion path based on Dempster-Shafer evidence theory specifically for high-conflict scenarios. Existing technologies cannot make effective decisions when multimodal evidence is contradictory (such as sensor alarms but no video anomalies), while this application can explicitly express uncertain states and automatically switch fusion strategies through the conflict coefficient, significantly improving decision reliability.

[0165] 3. Physically information-driven spatiotemporal graph convolutional situational deduction enables physically interpretable predictions.

[0166] This application embeds the heat conduction equation as prior physical knowledge into the loss function of a spatiotemporal graph convolutional network, ensuring that the model output strictly follows physical laws. Compared to the purely data-driven method using LSTM and graph embedding in existing technologies, this application reduces prediction errors in data-sparse regions (corners of parks without historical fire data).

[0167] 4. The conditional encoder enables the simulation of intervention effects and supports proactive decision-making.

[0168] This application, through the injection of control vectors for sprinklers and smoke extraction using conditional encoders, achieves for the first time a positive simulation of the extent to which a fire will be suppressed if the fire suppression system is activated. Existing early warning systems only output risk identification results and lack the ability to simulate intervention effects. This application, through proactive decision-making, allows park managers to rehearse the control effects of different strategies before emergency response.

[0169] like Figure 2 As shown in the figure, this application provides a smart park multimodal risk early warning device based on artificial intelligence, including:

[0170] The multimodal data processing module 201 is used to collect multimodal heterogeneous data through multi-source heterogeneous sensors deployed in the smart park, and to preprocess the multimodal heterogeneous data to obtain multimodal data features.

[0171] The multimodal feature fusion module 202 is used to evaluate the modal confidence of the multimodal heterogeneous data, determine the multimodal confidence corresponding to the multimodal heterogeneous data, and perform multi-source heterogeneous feature fusion based on the multimodal confidence and the features of the multimodal data to obtain multimodal fused features.

[0172] The risk probability analysis module 203 is used to determine the multimodal risk probability through the multimodal confidence level, determine the degree of decision inconsistency based on the multimodal risk probability, and perform adaptive risk warning path selection and risk identification based on the multimodal risk probability and the degree of decision inconsistency to obtain the target risk probability.

[0173] The risk contagion analysis module 204 is used to input the multimodal fusion features into a pre-set artificial intelligence model to predict the risk contagion path and obtain the target risk contagion path; the artificial intelligence model is set as a situation inference model based on a physical information spatiotemporal graph convolutional network.

[0174] The hierarchical linkage processing module 205 is used to perform hierarchical early warning and linkage response based on the target risk probability and the target risk spread path, so as to realize a smart park multimodal risk early warning method based on artificial intelligence.

[0175] The multimodal risk early warning device for smart parks based on artificial intelligence provided in this application embodiment can execute the method described in any of the above embodiments. Its principle and beneficial effects are similar, and will not be repeated here.

[0176] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0177] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0179] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0180] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A multimodal risk early warning method for smart parks based on artificial intelligence, characterized in that, include: Multimodal heterogeneous data is collected by multi-source heterogeneous sensors deployed in the smart park, and the multimodal heterogeneous data is preprocessed to obtain multimodal data features. Modality confidence is evaluated on the multimodal heterogeneous data to determine the multimodal confidence corresponding to the multimodal heterogeneous data, and multi-source heterogeneous feature fusion is performed based on the multimodal confidence and the multimodal data features to obtain multimodal fusion features; The multimodal risk probability is determined by the multimodal confidence level, and the degree of decision inconsistency is determined based on the multimodal risk probability. Based on the multimodal risk probability and the degree of decision inconsistency, adaptive risk warning path selection and risk identification are performed to obtain the target risk probability. The multimodal fusion features are input into a pre-set artificial intelligence model to predict the risk propagation path, thereby obtaining the target risk propagation path; The artificial intelligence model is set as a situation inference model based on a physical information spatiotemporal graph convolutional network; Based on the target risk probability and the target risk spread path, a graded early warning and coordinated response are implemented to realize a smart park multimodal risk early warning method based on artificial intelligence.

2. The AI-based smart park multimodal risk early warning method according to claim 1, characterized in that, Multimodal heterogeneous data is collected by multi-source heterogeneous sensors deployed in the smart park, and the multimodal heterogeneous data is preprocessed to obtain multimodal data features, including: Multimodal heterogeneous data is obtained by collecting visual monitoring video streams, audio streams, and sensing data from IoT sensors through multi-source heterogeneous sensors deployed in the smart park; the IoT sensors include temperature sensors, smoke concentration sensors, and combustible gas concentration sensors. Using a unified clock provided by a Network Time Protocol (NTP) server, the image frames in the visual monitoring video stream at the current moment are determined to obtain visual modal data; based on the current moment, the audio stream is extracted in the form of a time window to obtain sound modal data; based on the current moment, sensor data is extracted in the form of a time window to obtain supplementary modal data. The visual features are obtained by acquiring the features of the visual modality data through a visual feature extraction model. The time-domain data and frequency-domain data of the sound modal data are obtained, and the time-domain data and frequency-domain data are concatenated to obtain concatenated data; the features of the concatenated data are obtained through a sound feature extraction model to obtain sound features; The supplementary features are obtained by acquiring the features of the supplementary modality data through a supplementary feature extraction model. The visual features, sound features, and supplementary features are collectively used as multimodal data features.

3. The AI-based smart park multimodal risk early warning method according to claim 2, characterized in that, Modality confidence assessment is performed on the multimodal heterogeneous data to determine the multimodal confidence level corresponding to the multimodal heterogeneous data, including: The visual modality confidence level corresponding to the visual modality data is obtained as follows: in, Indicates the confidence level of the visual modality. Represents a sigmoid function, and x represents the variable of the sigmoid function, k represents the steepness coefficient of the sigmoid function, e represents the natural constant, and I represents the current illumination, which is collected by the light sensor deployed at the camera location. This indicates the preset minimum effective illuminance threshold. H represents the preset optimal illumination threshold, and H represents the current image information entropy. This indicates the preset minimum information entropy threshold. This indicates the preset maximum information entropy threshold; The sound modal confidence level corresponding to the sound modal data is obtained as follows: in, SNR represents the signal-to-noise ratio, which indicates the confidence level of the sound modality. This indicates the preset minimum effective signal-to-noise ratio threshold. This indicates the preset optimal signal-to-noise ratio threshold. Represents the background noise energy spectrum. This represents the energy spectrum of the current audio signal; The supplementary modal confidence level corresponding to the supplementary modal data is obtained as follows: in, Indicates the supplementary modal confidence level. This indicates the cumulative number of communication verification errors within a preset time window. This indicates the total number of communications within the preset time window. This represents a very small constant to prevent division by zero, and is set to 0.001; This represents the chain multiplication operator. Denotes the Gaussian decay function, and z represents the variable of the Gaussian decay function. This represents the current measured value of the i-th sensor parameter. This represents the center value of the normal range for the i-th sensor parameter. The normal range half-width of the i-th sensor parameter is represented by m, and m represents the total number of IoT sensors. The visual modality confidence, the audio modality confidence, and the supplementary modality confidence are collectively used as the multimodal confidence corresponding to the multimodal heterogeneous data.

4. The AI-based smart park multimodal risk early warning method according to claim 2, characterized in that, Multi-source heterogeneous feature fusion is performed based on the multimodal confidence level and the multimodal data features to obtain multimodal fused features, including: Using the visual features in the multimodal data features as the query and the sound features as the key and value, a cross-attention mechanism is used to enhance the visual features, resulting in enhanced visual features. Using the supplementary features as queries and the enhanced visual features as keys and values, a cross-attention mechanism is employed to perform feature fusion, resulting in multimodal fusion features.

5. The AI-based smart park multimodal risk early warning method according to claim 4, characterized in that, Determining the multimodal risk probability using the multimodal confidence level includes: The confidence level of the visual modality is multiplied into the visual features to obtain the visual feature vector with enhanced confidence. The confidence level of the sound modality is multiplied into the sound features to obtain the sound feature vector with enhanced confidence. The confidence scores of the supplementary modes are multiplied into the supplementary features to obtain the supplementary feature vector with enhanced confidence. The multi-path fully connected layer processes the confidence-enhanced visual feature vector, the confidence-enhanced audio feature vector, and the confidence-enhanced supplementary feature vector to obtain the first risk probability, the second risk probability, and the third risk probability, thus obtaining the multimodal risk probability.

6. The method for multimodal risk early warning of smart parks based on artificial intelligence according to claim 1, characterized in that, The degree of decision inconsistency is determined based on the multimodal risk probability as follows: in, Indicates the degree of inconsistency in decision-making. This represents the first risk probability obtained through the visual feature vector enhanced with confidence. This represents the second risk probability obtained through the confidence-enhanced sound feature vector. This represents the third risk probability obtained through the supplementary feature vector enhanced with confidence.

7. The method for multimodal risk early warning of smart parks based on artificial intelligence according to claim 1, characterized in that, Based on the multimodal risk probability and the degree of decision inconsistency, adaptive risk warning path selection and risk identification are performed to obtain the target risk probability, including: If the degree of decision inconsistency is less than a preset inconsistency threshold, then the target risk probability is obtained based on the multimodal risk probability as follows: in, Indicates the probability of target risk. Indicates the confidence level of the visual modality. Indicates the confidence level of the sound modality. Indicates the supplementary modal confidence level. This represents the risk probability output by the dominant path with the highest confidence level. This represents the attenuation coefficient of the non-dominant path. Indicates the dominant path, which can be a visual path, an audio path, or a supplementary path; This represents the confidence level corresponding to the j-th path. This represents the confidence level corresponding to the j-th path in the multimodal risk probability; If the degree of decision inconsistency is greater than or equal to a preset inconsistency threshold, then the DS evidence theory is used to fuse the multimodal risk probabilities to determine the target risk probability.

8. The method for multimodal risk early warning of smart parks based on artificial intelligence according to claim 7, characterized in that, The multimodal fusion features are input into a pre-set artificial intelligence model to predict the risk propagation path, resulting in the target risk propagation path, including: The smart park is divided into N logical regions, each of which is considered a graph node. Assuming a connected path exists between any two graph nodes, an edge is constructed between them, with the edge weight set as follows: in, Represents graph nodes With graph nodes Edge weights between them Represents graph nodes With graph nodes Minimum cross-sectional area of ​​the passage between them Represents graph nodes With graph nodes The physical distance between them Represents graph nodes With graph nodes The preset ventilation coefficient between; The graph nodes, edges, and edge weights are used to construct a target graph, and the features of the graph nodes are set as multiple consecutive historical multimodal fusion features and the current multimodal fusion features. The target map is constructed as the input of the pre-set artificial intelligence model, and a control vector is inserted into the intermediate layer of the pre-set artificial intelligence model to output a risk heat map; Based on the aforementioned risk heatmap, the target risk propagation path is analyzed.

9. The method for multimodal risk early warning of smart parks based on artificial intelligence according to claim 8, characterized in that, Based on the target risk probability and the target risk spread path, a tiered early warning and coordinated response are implemented, including: If the probability of the target risk is less than the first preset risk threshold, it is determined to be low risk, and a pop-up window will appear on the system interface to remind security personnel to pay close attention. If the target risk probability is greater than or equal to the first preset risk threshold and less than the second preset risk threshold, it is determined to be a medium risk, and an early warning is triggered. The audio and video footage of the medium risk incident location will automatically pop up, and a broadcast will be made to notify people near the medium risk incident location to pay attention; the first preset risk threshold is less than the second preset risk threshold. If the probability of the target risk is greater than or equal to the second preset risk threshold, it is determined to be a high risk, triggering an emergency alarm, automatically activating the sprinkler system, opening the fire door, cutting off the non-explosion-proof power supply, and broadcasting the target risk spread path.

10. A smart park multimodal risk early warning device based on artificial intelligence, characterized in that, include: The multimodal data processing module is used to collect multimodal heterogeneous data through multi-source heterogeneous sensors deployed in the smart park, and to preprocess the multimodal heterogeneous data to obtain multimodal data features. The multimodal feature fusion module is used to evaluate the modal confidence of the multimodal heterogeneous data, determine the multimodal confidence corresponding to the multimodal heterogeneous data, and perform multi-source heterogeneous feature fusion based on the multimodal confidence and the features of the multimodal data to obtain multimodal fused features. The risk probability analysis module is used to determine the multimodal risk probability through the multimodal confidence level, determine the degree of decision inconsistency based on the multimodal risk probability, and perform adaptive risk warning path selection and risk identification based on the multimodal risk probability and the degree of decision inconsistency to obtain the target risk probability. The risk contagion analysis module is used to input the multimodal fusion features into a pre-set artificial intelligence model to predict the risk contagion path and obtain the target risk contagion path; the artificial intelligence model is set as a situation inference model based on a physical information spatiotemporal graph convolutional network. The hierarchical linkage processing module is used to perform hierarchical early warning and linkage response based on the target risk probability and the target risk spread path, so as to realize a smart park multimodal risk early warning method based on artificial intelligence.

Images