Monitoring and early warning method and device based on multi-modal large model, equipment and medium

Abstract

The invention discloses a monitoring and early warning method, device and equipment based on a multi-modal large model and a medium, and relates to the technical field of monitoring and early warning, and the early warning method comprises the following specific steps: S100, multi-modal data collection and fusion: through sensors and cameras deployed in shopping malls, elevators and surrounding road areas, carrying out multi-modal data collection and fusion; the method comprises the following steps: comprehensively collecting multi-modal data of personnel flow, elevator operation and traffic conditions, preprocessing original data, and integrating different modal data into a unified feature vector by using a modal entropy weight fusion function; through a multi-modal data acquisition and fusion technology, the system can comprehensively capture complex environment information of shopping malls, elevators and surrounding road areas, data of different modals are efficiently integrated into a unified feature vector by using a modal entropy weight fusion function, the problem of multi-source heterogeneous data processing is solved, and the system can be widely applied to the field of multi-source heterogeneous data processing. And the accuracy and efficiency of data fusion are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of monitoring and early warning technology, and specifically to a monitoring and early warning method, device, equipment and medium based on a multimodal large model. Background Art

[0002] With the acceleration of urbanization and the continuous increase in the flow of people in public places, security monitoring and early warning in shopping malls, elevators and surrounding road areas have become important links in ensuring public safety. Traditional monitoring systems mostly rely on single-modal data collection, such as video monitoring through cameras alone or monitoring the operation status of elevators through sensors alone. There may be limitations in fully capturing multi-dimensional information in complex environments. With the rapid development of big data, artificial intelligence and Internet of Things technologies, multimodal data fusion and intelligent analysis technologies have provided new ideas for solving this problem. Multimodal data can contain richer scene information, such as personnel flow, equipment operation status, and traffic conditions. In-depth mining and analysis of these data through intelligent algorithms will help to achieve more accurate risk warning and disposal, and improve the intelligence level of public safety monitoring.

[0003] First, traditional systems usually adopt a single-modal data collection method, and the data source is relatively single, which may not be able to fully reflect the risk factors in a complex environment, resulting in the need to improve the accuracy of early warnings and a high false alarm and missed alarm rate. Secondly, the intelligence level of traditional technologies in data processing and analysis needs to be improved, and they rely more on manual experience or preset rules to make risk judgments. Their adaptability may be insufficient when facing complex and changing scenario requirements. Furthermore, the threshold setting of traditional early warning systems usually adopts a static method, which is difficult to dynamically adjust according to real-time data, which may lead to insufficient warning sensitivity and challenges in timely intervention in the early stages of risks. These factors affect the application effect of traditional monitoring and early warning technologies in the field of public safety.

[0004] In summary, there is a practical need to develop a new technology that can efficiently integrate multimodal data, intelligently analyze risks, and dynamically adjust warning thresholds. Monitoring and early warning methods, devices, equipment, and media based on multimodal large models aim to improve the intelligence level of monitoring and early warning systems, reduce the possibility of accidents, and provide more effective solutions for the public safety field. Summary of the Invention

[0005] The purpose of the present invention is to make up for the shortcomings of the existing technology and provide a monitoring and early warning method, device, equipment and medium based on a multimodal large model. By comprehensively collecting and integrating multimodal data, including information on personnel flow, elevator operation and traffic conditions, and using algorithm models to perform cross-scenario causal correlation analysis, the present invention can accurately identify potential risk trends. Furthermore, by constructing an interpretable causal graph, the present invention realizes risk tracing and provides managers with an intuitive risk analysis tool. At the same time, combined with temporal data fusion prediction technology, the present invention can dynamically adjust the early warning threshold to ensure the timeliness and accuracy of the early warning.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: First, a monitoring and early warning method based on a multimodal large model, the specific steps of the early warning method are:

[0007] S100, multimodal data collection and fusion: Sensors and cameras deployed in shopping malls, elevators, and surrounding road areas comprehensively collect multimodal data on personnel flow, elevator operation, and traffic conditions. After preprocessing the collected data, the modal entropy weight fusion function is used to integrate the different modal data into a unified feature vector.

[0008] S200, cross-scenario causal association analysis: The fused data is input into a cross-scenario association model built based on spatiotemporal causal logic to explore potential causal relationships between scenario data. Risk assessment is performed on the association results based on association rules to identify potential risk trends. Preliminary analysis results are output, including risk scenarios, association characteristics, and risk levels.

[0009] S300, Causal Graph Construction and Risk Tracing: Based on a multimodal large model, an interpretable causal graph is constructed with scenario events as nodes and causal strength as edges. Through counterfactual simulation, the change in risk probability after removing specific factors is deduced to locate the root cause of the risk.

[0010] S400, Temporal Data Fusion Prediction: This integrates the current frame data of the surveillance video, historical time period data, and preliminary prediction data. It uses a time series analysis model to mine the temporal dimension features in the data, combining historical patterns with current status to predict the changing trends of data in various scenarios within a short period of time.

[0011] S500, dynamic threshold warning and disposal: Integrates cross-scenario correlation analysis, causal reasoning conclusions, and temporal prediction results to dynamically adjust the warning thresholds for various risks. When real-time monitoring data reaches or exceeds the dynamic threshold, an alert is immediately triggered, and a report containing risk causal attribution is generated, providing managers with precise risk disposal guidance.

[0012] Furthermore, in the multimodal data acquisition and fusion step S100, the modal entropy weight fusion function is used to integrate the different modal data into a unified feature vector, and the calculation formula is: Among them F fusion is the fused multimodal feature vector, F m is the original eigenvector of the mth mode, m=1,2,…,M, M is the total number of modes, m is the mode index, indicating different types of data modes, ω m is the fusion weight of the mth modality, and the formula is: H(F m ) is the information entropy of the mth modal eigenvector, which is used to measure the uncertainty of the mode, and k is the index used to traverse all modes.

[0013] Furthermore, in the cross-scenario causal correlation analysis in S200, a cross-scenario correlation model is constructed by combining the spatiotemporal causal strength function, and the formula is: Among them C i→j is the causal strength of scenario event i on event j, is the loss function, D is the dataset containing all scene data, X i Represents the feature data corresponding to scene event i, Y j represents the feature data corresponding to scene event j, ||S i -S j || is the spatial distance between events i and j, S i is the spatial location of event i, S j is the spatial position of event j, i and j are index identifiers used to distinguish events in different scenes, and σ is the spatial influence radius.

[0014] Furthermore, the association rules in the cross-scenario causal association analysis in S200 are:

[0015] Personnel mobility risk rules:

[0016] When the population density in a certain area of ​​a shopping mall exceeds 150% of the historical average density of the area during the same period within 10 minutes, it is marked as a local crowd-dense risk;

[0017] When the overall flow of people in a mall increases by more than 30% of the expected flow of people for the day within 30 minutes, and the average length of stay of people entering the mall increases by more than 20 minutes, it is determined to be a mall congestion risk;

[0018] Elevator operation risk rules:

[0019] When the elevator fault alarm reaches 3 times or more within 1 hour, or there are more than two abnormal stops in a row, it is identified as an elevator fault risk;

[0020] When the number of people in the elevator car reaches 90% of the rated load and the elevator continues to run for more than 15 minutes, and the average waiting time of people on the floor where the elevator is located increases by more than 5 minutes compared to normal days, it is marked as a risk of insufficient elevator capacity;

[0021] Traffic situation risk rules:

[0022] When the congestion index of surrounding roads exceeds 0.7 for 20 consecutive minutes (the congestion index ranges from 0 to 1, with higher values ​​indicating more severe congestion), and the congested road section is within 1 km of the main entrance and exit of the shopping mall, it is determined to be a traffic congestion-related risk;

[0023] When the remaining parking space rate in the parking lot around the mall is less than 20% and the length of the vehicle queue at the parking lot entrance exceeds 50 meters, it is marked as a parking resource shortage risk;

[0024] Cross-scenario comprehensive risk rules:

[0025] When surrounding roads are severely congested, the flow of people at the mall entrance increases by 40% within 15 minutes, and the elevator failure rate increases by 30% compared to normal days, it is identified as a comprehensive traffic-personnel-equipment risk;

[0026] When people gather in a certain area of ​​a shopping mall and elevators near the area frequently malfunction, and the traffic speed of the passage leading to the outside of the mall on the corresponding floor decreases by more than 50%, it is marked as a comprehensive safety risk in the local area.

[0027] Furthermore, the specific steps of constructing an interpretable causal graph with scenario events as nodes and causal strengths as edges based on a multimodal large model in S300, causal graph construction and risk tracing, are as follows:

[0028] (1) Disassemble the preliminary risk analysis results and abstract the scenario information and implicit events into causal graph nodes;

[0029] (2) Combine domain knowledge and preset rules to screen potential causal relationships between nodes;

[0030] (3) Quantitatively evaluate the causal strength between nodes from multiple dimensions, including time sequence, scope of influence, and intervention response;

[0031] (4) Construct a directed weighted hierarchical causal graph with scenario events as nodes and causal strength as edge weights;

[0032] (5) Use real-time data to verify the causal graph and update the graph structure and edge weights based on the new data.

[0033] Furthermore, the calculation formula for constructing the time series analysis model in the temporal data fusion prediction in S400 is: in is the predicted feature at time τ in the future, X tis the eigenvector at time t, X t-h is the historical feature vector at time th, h is the historical time step, α t is the attention weight at historical moment t, and is calculated as: r is the sum index, X current is the eigenvector at the current moment, The attention weight at the next τ time, is the feature vector at the future τ-1 moment, Transformer is the Transformer model structure, MLP is the multi-layer perceptron, ⊙ is element-by-element multiplication, t represents the time index of the current moment, and τ represents the time span of the future prediction.

[0034] Furthermore, in the step S500, the warning thresholds of various risks are dynamically adjusted through the risk entropy flow threshold model in the dynamic threshold warning and disposal. The calculation formula is: Where T dynamic is the dynamic warning threshold, T0 is the basic warning threshold, λ is the entropy flow adjustment coefficient, H risk (t) is the risk entropy at time t, C i→risk is the causal strength of scenario event i on the occurrence of risk, N is the total number of events that affect the risk, t represents the time index of the current moment, and i is the index of the scenario event.

[0035] In the second aspect, the device includes: a multimodal data acquisition and fusion module, a cross-scenario causal association analysis module, a causal graph construction and risk tracing module, a temporal data fusion prediction module, and a dynamic threshold warning and disposal module;

[0036] The multimodal data acquisition and fusion module collects multimodal data on personnel flow, elevator operation, and traffic conditions through sensors and cameras deployed in shopping malls, elevators, and surrounding road areas. After preprocessing the raw data, the module uses a modal entropy weight fusion function to integrate the different modal data into a unified feature vector.

[0037] The cross-scenario causal association analysis module inputs the unified feature vector output by the fusion module into a cross-scenario association model built based on spatiotemporal causal logic, explores potential causal relationships between scenario data, and conducts risk assessment on the association results based on the association rules of personnel flow, elevator operation, traffic conditions, and cross-scenario comprehensive risks, identifies potential risk trends, and outputs preliminary analysis results including risk scenarios, association characteristics, and risk levels.

[0038] The causal graph construction and risk tracing module: Based on a multimodal large model, key scenario information and implicit events in the preliminary analysis results are abstracted into causal graph nodes, and a directed weighted hierarchical causal graph is constructed with scenario events as nodes and causal strength as edge weights. Through counterfactual simulation, the risk probability changes after removing specific factors are deduced to locate the root cause of the risk;

[0039] The temporal data fusion prediction module: integrates the current frame data of the surveillance video, the historical time period data and the preliminary prediction data, uses the time series analysis model to mine the time dimension features, combines historical rules and current status, and predicts the data change trend of each scene in the short term in the future;

[0040] The dynamic threshold warning and disposal module: integrates cross-scenario correlation analysis results, causal reasoning conclusions and temporal prediction results, dynamically adjusts various risk warning thresholds through the risk entropy flow threshold model, triggers a warning when real-time monitoring data reaches or exceeds the dynamic threshold, generates a report containing risk causal attribution, and provides managers with accurate risk disposal guidance.

[0041] In the third aspect, the early warning device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the monitoring and early warning method based on the multimodal large model described in the first aspect is implemented.

[0042] In a fourth aspect, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the monitoring and early warning method based on a multimodal large model described in the first aspect.

[0043] Compared with the existing technology, the monitoring and early warning method, device, equipment and medium based on the multimodal large model have the following beneficial effects:

[0044] 1. Through multimodal data acquisition and fusion technology, the present invention's system can comprehensively capture the complex environmental information of shopping malls, elevators, and surrounding road areas, and use the modal entropy weight fusion function to efficiently integrate data of different modalities into a unified feature vector, which not only solves the problem of multi-source heterogeneous data processing, but also greatly improves the accuracy and efficiency of data fusion. The cross-scene causal correlation analysis module further explores the potential causal relationship between scene data. The correlation model constructed based on spatiotemporal causal logic can accurately identify risk trends, provide early warnings for managers, and effectively reduce the possibility of accidents. This method is highly integrated and intelligent, and can dynamically adjust the warning threshold according to real-time data, realize the intelligence and personalization of risk warnings, and provide strong guarantees for public safety and operational efficiency.

[0045] 2. The present invention uses a directed weighted hierarchical causal graph with scenario events as nodes and causal strength as edges to intuitively display the internal logic and propagation path of risk occurrence, which is convenient for managers to quickly locate the root cause of risk and take effective measures. The application of counterfactual simulation technology further enhances the accuracy of risk tracing and provides a scientific basis for risk disposal. At the same time, the dynamic threshold warning mechanism dynamically adjusts the warning threshold through the risk entropy flow threshold model, ensuring the timeliness and accuracy of the warning and avoiding the limitations of the traditional fixed threshold warning mechanism. The implementation of this method not only improves the response speed and disposal efficiency of the monitoring and warning system, but also greatly reduces the false alarm and missed alarm rates, providing solid technical support for public safety and operational stability.

[0046] Other advantages, objects and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be learned from the practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0048] Figure 1 This is a framework diagram of the monitoring and early warning method based on a multimodal large model;

[0049] Figure 2 This is a flow chart of a monitoring and early warning device based on a multimodal large model. DETAILED DESCRIPTION

[0050] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0051] Example 1:

[0052] Specific application of shopping mall monitoring and early warning method based on multimodal large model.

[0053] Multimodal data collection and fusion: Deploy cameras and sensors at the mall entrance, corridors on each floor, inside elevators, and on surrounding roads to continuously collect data on people's walking trajectories, elevator operating status, and vehicle flow on surrounding roads. After denoising and unifying the format of the collected video images and sensor values, use the modal entropy weight fusion function to integrate different types of data into a feature vector that can reflect the overall status of the mall, such as Figure 1 As shown in Figure 2, the calculation formula of the modal entropy weight fusion function is: Among them F fusion is the fused multimodal feature vector, F m is the original eigenvector of the mth mode, m=1,2,…,M, M is the total number of modes, m is the mode index, indicating different types of data modes, ω m is the fusion weight of the mth modality, and the formula is: H(F m ) is the information entropy of the mth mode eigenvector, which is used to measure the uncertainty of the mode, and k is the index used to traverse all modes for subsequent analysis.

[0054] Cross-scenario causal correlation analysis: The integrated feature vector is input into the cross-scenario correlation model built based on spatiotemporal causal logic. The calculation formula is: Among them C i→j is the causal strength of scenario event i on event j, is the loss function, D is the dataset containing all scene data, X i Represents the feature data corresponding to scene event i, Y j represents the feature data corresponding to scene event j, ||S i -S j || is the spatial distance between events i and j, S i is the spatial location of event i, S j is the spatial location of event j, i and j are index identifiers used to distinguish events in different scenarios, and σ is the spatial impact radius. The potential causal relationship between the flow of people in the mall, elevator operation and surrounding traffic conditions is analyzed. According to the association rules of people flow risk rules and elevator operation risk rules, the analysis results are evaluated for risk. For example, when it is found that the density of people in a certain area of ​​the mall increases significantly in a short period of time and the congestion index of the surrounding roads increases at the same time, the possible risk trend of people congestion is identified, and the preliminary analysis results including risk scenarios, association characteristics and risk levels are output.

[0055] Causal graph construction and risk tracing: Based on a multimodal large model, the densely populated areas, abnormal elevator operation scenario information and possible hidden events (such as a brand promotion attracting a large number of customers) in the preliminary analysis results are abstracted into causal graph nodes. Combined with the domain knowledge and expert experience of shopping mall operations, the potential causal relationship between nodes is screened, and the causal strength between nodes is quantitatively evaluated from multiple dimensions such as time sequence and impact range. A directed weighted hierarchical causal graph is constructed with scenario events as nodes and causal strength as edge weights. Through counterfactual simulation, if it is assumed that the promotion activity does not exist, the change in mall population density after removing this factor is deduced, and the root cause of the risk of dense crowds is located.

[0056] Temporal data fusion prediction: Fusion of current surveillance video data, historical data from the past period, and previous preliminary prediction data. Use the time series analysis model to mine the temporal dimension features in the data and analyze the temporal changes in personnel flow and elevator usage data. The calculation formula is: in is the predicted feature at time τ in the future, X t is the eigenvector at time t, X t-h is the historical feature vector at time th, h is the historical time step, α t is the attention weight at historical moment t, and is calculated as: r is the sum index, X current is the eigenvector at the current moment, The attention weight at the next τ time, is the feature vector at the next τ-1 moment, Transformer is the Transformer model structure, MLP is the multi-layer perceptron, ⊙ is element-by-element multiplication, t represents the time index of the current moment, and τ represents the time span of the future prediction. Combining historical patterns and the current actual status of the mall, it predicts the changing trend of data in various scenarios in the short term in the future, such as the change in the number of people in each area of ​​the mall and the operating load of the elevators in the next 30 minutes.

[0057] Dynamic threshold warning and disposal: Based on the risk trends obtained from cross-scenario correlation analysis, the root causes obtained from causal diagram construction and risk tracing, and the future change trends obtained from temporal data fusion prediction, the risk entropy flow threshold model is used to dynamically adjust the warning thresholds for various risks. The calculation formula is: Where T dynamic is the dynamic warning threshold, T0 is the basic warning threshold, λ is the entropy flow adjustment coefficient, H risk (t) is the risk entropy at time t, C i→riskis the causal strength of scenario event i on the occurrence of risk, N is the total number of events that affect the risk, t represents the time index of the current moment, and i is the index of the scenario event. When the real-time monitored population density and elevator failure frequency data reach or exceed the dynamically adjusted threshold, the early warning system is immediately triggered, and a detailed report containing the causal attribution of the risk is generated. For example, if the dense population is caused by a promotion activity in a certain area and the surrounding traffic is congested, it provides managers with accurate risk management guidance, such as recommending increasing security personnel in the area and guiding customers to disperse their flow.

[0058] In summary, the shopping mall monitoring and early warning method based on a multimodal large model collects multimodal data through sensors and cameras during peak holiday traffic scenes. After integration through the modal entropy weight fusion function, it inputs the cross-scenario association model to analyze the causal relationship, combines the association rules to assess the risk, and then constructs a causal graph and traces the source through counterfactual simulation. The time series analysis model is used to predict the trend, and finally the risk entropy flow threshold model dynamically adjusts the threshold to achieve accurate early warning. This method improves the timeliness of shopping mall risk identification and the targeted disposal through multi-model collaboration and dynamic strategies, providing an intelligent solution for safe operations.

[0059] Example 2:

[0060] Application of monitoring and early warning devices based on multimodal large models in transportation hubs.

[0061] Security monitoring and early warning of the railway station and surrounding areas.

[0062] Multimodal data collection and fusion module: Deploy cameras, pedestrian flow sensors, and vehicle detectors in the waiting hall, entry and exit passages, elevators, escalators, and surrounding roads and parking lots of the railway station to collect multimodal data on the distribution of waiting passengers, elevator operation status, vehicle traffic on surrounding roads, and parking space usage in real time. After cleaning and standardizing the collected raw data, the modal entropy weight fusion function is used. The calculation formula of the modal entropy weight fusion function is as follows: The data of different modes are integrated into a unified feature vector so that the data can more comprehensively reflect the actual conditions of the railway station and surrounding areas, such as Figure 2 shown.

[0063] Cross-scenario causal association analysis module: The unified feature vector output by the multimodal data acquisition and fusion module is input into the cross-scenario association model built based on spatiotemporal causal logic. The calculation formula of the cross-scenario association model is: Deeply explore the potential causal relationships between different scenario data such as personnel flow, elevator operation, and surrounding traffic conditions in the railway station, and conduct risk assessment on the associated results according to the personnel flow risk rules, elevator operation risk rules, traffic condition risk rules and cross-scenario comprehensive risk rules. For example, when it is detected that the population density in a certain area of ​​the railway station waiting hall suddenly increases, and at the same time, elevators near the area frequently malfunction and the speed of the passage leading to the outside of the station on the corresponding floor decreases, the possible local area comprehensive safety risk trend is identified, and preliminary analysis results including risk scenarios, correlation characteristics and risk levels are output.

[0064] Causal graph construction and risk tracing module: Based on a multimodal large model, the module abstracts the densely populated areas in the waiting hall, frequent elevator failures, and possible hidden events (such as a train delay causing a large number of passengers to be stranded) in the preliminary analysis results into causal graph nodes. Combined with the domain knowledge of railway station operations and the rules formulated by experts, the module screens the potential causal relationships between nodes, quantitatively evaluates the causal strength between nodes from multiple dimensions such as time sequence, scope of influence, and intervention response, and constructs a directed, weighted hierarchical causal graph with scenario events as nodes and causal strength as edge weights. Through counterfactual simulation, for example, assuming that the train arrives on time, the module deduces the changes in the waiting hall population density and elevator operation status after removing this factor, and locates the root cause of the risk.

[0065] Temporal data fusion prediction module: This module integrates current surveillance video data, personnel flow data within a historical time period, elevator operation data, and previous preliminary prediction data. It uses a time series analysis model to mine the temporal dimension features in the data, analyze the temporal variation patterns of personnel in and out of the station, and the frequency of elevator use. Combining historical patterns with the actual operating status of the railway station, it predicts the changing trends of data in various scenarios in the short term. The calculation formula for the time series analysis model is: Such as the change in the number of people in the waiting hall in the next hour and the probability of elevator failure.

[0066] Dynamic threshold warning and disposal module: This module integrates the risk trends obtained by the cross-scenario causal association analysis module, the root causes obtained by the causal graph construction and risk tracing module, and the future change trends obtained by the temporal data fusion prediction module. It dynamically adjusts the warning thresholds of various risks through the risk entropy flow threshold model. The calculation formula of the risk entropy flow threshold model is: When the real-time monitored data on population density, number of elevator failures, and surrounding road congestion index reach or exceed the dynamically adjusted threshold, the early warning mechanism is quickly triggered, and a report containing risk causal attribution is generated. For example, if the crowds are densely populated due to train delays and elevator failures causing passengers to be stranded, precise risk management guidance can be provided to managers, such as adding temporary waiting areas, speeding up elevator maintenance, and coordinating with transportation departments to clear surrounding roads.

[0067] In summary, when the monitoring and early warning device based on the multimodal large model is applied in transportation hubs, it collects and integrates multimodal data through multi-module collaboration, explores cross-scenario causal relationships, constructs causal graphs, and forms a closed loop from data collection to disposal guidance, effectively improving passenger diversion efficiency and providing a quantifiable and explainable intelligent monitoring and early warning solution for the safe and efficient operation of large transportation hubs.

[0068] Example 3:

[0069] Application of monitoring and early warning equipment based on multimodal large models in industrial parks.

[0070] Hardware deployment: Various sensors (such as temperature sensors, smoke sensors, and pressure sensors) and high-definition cameras are deployed in the production workshops, storage areas, logistics channels, and surrounding roads of the industrial park. Edge computing devices and central servers are also configured. Sensors are used to collect equipment operating parameters and environmental indicator data. Cameras are used to monitor personnel operating behavior and real-time regional images. Edge computing devices perform preliminary preprocessing on the collected data, and central servers are used to store and further analyze the data.

[0071] Multimodal data acquisition and fusion: Edge computing devices perform denoising and format conversion pre-processing on the equipment operation data (such as temperature, pressure, and speed) collected by sensors and the video data captured by cameras, and then transmit them to the central server through the network. The central server uses the modal entropy weight fusion function to integrate data from different modalities into a unified feature vector to comprehensively reflect the safe production status of the industrial park.

[0072] Cross-scenario causal correlation analysis: The integrated feature vector is input into a cross-scenario correlation model built based on spatiotemporal causal logic to analyze the potential causal relationship between the operating status of production equipment, personnel operating behavior, environmental indicators and logistics transportation conditions. Based on preset risk rules, such as abnormal temperature rise of equipment and improper operation of nearby personnel, and congestion in logistics channels resulting in the inability of emergency rescue equipment to arrive in time, possible safety risk trends are identified and preliminary analysis results are output.

[0073] Causal graph construction and risk tracing: Based on a multimodal large model, the key scenario information of abnormal equipment operation, personnel illegal operation and possible hidden events (such as untimely equipment maintenance and inadequate personnel training) in the preliminary analysis results are abstracted into causal graph nodes. Combined with the domain knowledge of industrial production and expert experience, the potential causal relationship between nodes is screened, and the causal strength is quantitatively evaluated from the dimensions of time sequence and impact degree. A causal graph is constructed. Through counterfactual simulation, such as assuming regular equipment maintenance, the change in the probability of equipment failure after removing this factor is deduced to locate the root cause of the risk.

[0074] Temporal data fusion prediction: This method integrates real-time data at the current moment, production data from historical time periods, and previous forecast data. It uses a time series analysis model to mine time dimension features, analyze the changing patterns of equipment operating parameters and personnel operation frequency data, and predict the changing trends of data in various scenarios in the short term. For example, it predicts the temperature change of a certain device in the next two hours and the probability of congestion in the logistics channel.

[0075] Dynamic threshold warning and disposal: Integrating cross-scenario correlation analysis, causal reasoning, and temporal prediction results, the warning threshold is dynamically adjusted through the risk entropy flow threshold model. When real-time monitoring data reaches or exceeds the dynamic threshold, the warning system is triggered, and a report containing risk causal attribution is generated. For example, if abnormal equipment temperature is caused by untimely maintenance and improper operation by personnel, precise disposal guidance is provided to management personnel, such as immediately arranging equipment maintenance, retraining relevant personnel, and adjusting logistics channel access plans.

[0076] To summarize, when monitoring and early warning equipment based on multimodal large models is applied in industrial parks, it realizes the collection and fusion of multimodal data through the combination of sensors, cameras, edge computing devices, and central server hardware. It uses cross-scenario correlation models and causal graphs to explore and trace risk causal relationships, combines time series analysis to predict trends, and dynamically adjusts early warning thresholds to achieve accurate early warnings. The equipment forms a complete closed loop from data collection, analysis to disposal, providing an intelligent and automated monitoring and early warning solution for safe production in industrial parks, effectively improving the park's risk prevention and control capabilities and production management efficiency.

[0077] Example 4:

[0078] Application of monitoring and early warning media based on multimodal large models in smart campuses.

[0079] Media storage and deployment: Use high-performance computer-readable storage media (such as solid-state drives and disk arrays) to store monitoring and early warning system programs, historical data, and model parameters based on multimodal large models. Deploy the storage media on the campus data center server and connect it to the campus sensor network (such as pedestrian flow sensors, access control sensors), cameras, and broadcast system equipment to ensure real-time data transmission and interaction.

[0080] Multimodal data collection and fusion: Sensors collect real-time personnel flow and access control entry and exit records in various areas of the campus, and cameras collect video images. After these data are transmitted to the server, the system program uses the modal entropy weight fusion function to preprocess and fuse them to form a unified feature vector to reflect the personnel flow and security status information on campus.

[0081] Cross-scenario causal correlation analysis: The fused feature vector is input into the cross-scenario correlation model to analyze the potential causal relationship between personnel flow, access control status, and abnormal behavior (such as climbing over walls and staying in remote areas for a long time). Based on the set risk rules, if the population density in a certain area suddenly increases and abnormal behavior occurs, and the nearby access control records show that outsiders have broken in, potential security risk trends are identified and preliminary analysis results are output.

[0082] Causal graph construction and risk tracing: Based on a multimodal large model, the areas of personnel gathering, scene information of abnormal behavior events, and possible hidden events (such as campus activities attracting a large number of people and unauthorized personnel entering) in the preliminary analysis results are abstracted into causal graph nodes. Combined with the domain knowledge and expert experience of campus safety management, the causal relationship between nodes is screened, the causal strength is quantitatively evaluated, and a causal graph is constructed. Through counterfactual simulation, such as assuming there is no campus activity, the changes in the personnel gathering situation are deduced to locate the root cause of the risk.

[0083] Temporal data fusion prediction: Integrate current real-time data, personnel flow data from historical time periods, and access control records, use time series analysis models to mine time dimension characteristics, analyze the changing trends of personnel entry and exit patterns and the frequency of abnormal behavior data, and predict changes in data in various scenarios in the short term in the future, such as the personnel flow at the school gate in the next 1 hour and the probability of abnormal behavior in a remote area.

[0084] Dynamic threshold warning and disposal: Based on the results of cross-scenario correlation analysis, causal reasoning and temporal prediction, the warning threshold is dynamically adjusted through the risk entropy flow threshold model. When the real-time monitoring data reaches or exceeds the dynamic threshold, the warning is triggered, and the system program automatically generates a report containing risk causal attribution, such as whether the gathering of people is caused by campus activities and the intrusion of outsiders. At the same time, the broadcasting system is linked to broadcast warnings and provide disposal guidance to management personnel, such as dispatching additional security personnel to relevant areas, strengthening access control management, and investigating abnormal personnel.

[0085] In summary, in the application of smart campuses, the monitoring and early warning medium based on the multimodal large model stores system programs and data through computer-readable storage media, combines with sensors and cameras on campus to realize the collection and fusion of multimodal data, and uses cross-scenario correlation analysis and causal graph construction to accurately identify risks and trace their sources. It predicts trends through time series analysis and dynamically adjusts thresholds to achieve early warning. This medium is combined with the campus security system to form an intelligent and automated monitoring and early warning system, which provides strong technical support for the security management of smart campuses and effectively improves the campus security prevention level and management efficiency.

[0086] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as above in terms of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, make some changes or modifications to equivalent embodiments using the technical contents disclosed above. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A monitoring and early warning method based on a multimodal large model, characterized in that: The specific steps of this early warning method are: S100, multimodal data collection and fusion: Sensors and cameras deployed in shopping malls, elevators, and surrounding road areas comprehensively collect multimodal data on personnel flow, elevator operation, and traffic conditions. After preprocessing the collected data, the modal entropy weight fusion function is used to integrate the different modal data into a unified feature vector. S200, cross-scenario causal association analysis: The fused data is input into a cross-scenario association model built based on spatiotemporal causal logic to explore potential causal relationships between scenario data. Risk assessment is performed on the association results based on association rules to identify potential risk trends. Preliminary analysis results are output, including risk scenarios, association characteristics, and risk levels. S300, Causal Graph Construction and Risk Tracing: Based on a multimodal large model, an interpretable causal graph is constructed with scenario events as nodes and causal strength as edges. Through counterfactual simulation, the change in risk probability after removing specific factors is deduced to locate the root cause of the risk. S400, Temporal Data Fusion Prediction: This integrates the current frame data of the surveillance video, historical time period data, and preliminary prediction data. It uses a time series analysis model to mine the temporal dimension features in the data, combining historical patterns with current status to predict the changing trends of data in various scenarios within a short period of time. S500, dynamic threshold warning and disposal: Integrates cross-scenario correlation analysis, causal reasoning conclusions, and temporal prediction results to dynamically adjust the warning thresholds for various risks. When real-time monitoring data reaches or exceeds the dynamic threshold, an alert is immediately triggered, and a report containing risk causal attribution is generated, providing managers with precise risk disposal guidance.

2. The monitoring and early warning method based on a multimodal large model according to claim 1 is characterized in that: In the above S100, the modal entropy weight fusion function is used in the multimodal data acquisition and fusion to integrate the different modal data into a unified feature vector. The calculation formula is: Among them F fusion is the fused multimodal feature vector, F m is the original eigenvector of the mth mode, m=1,2,…,M, M is the total number of modes, m is the mode index, indicating different types of data modes, ω m is the fusion weight of the mth modality, and the formula is: H(F m ) is the information entropy of the mth modal eigenvector, which is used to measure the uncertainty of the mode, and k is the index used to traverse all modes.

3. The monitoring and early warning method based on a multimodal large model according to claim 1 is characterized in that: In the above S200, a cross-scenario correlation model is constructed by combining the spatiotemporal causal strength function in the cross-scenario causal correlation analysis, and the formula is: Among them C i→j is the causal strength of scenario event i on event j, is the loss function, D is the dataset containing all scene data, X i Represents the feature data corresponding to scene event i, Y j represents the feature data corresponding to scene event j, ||S i -S j || is the spatial distance between events i and j, S i is the spatial location of event i, S j is the spatial position of event j, i and j are index identifiers used to distinguish events in different scenes, and σ is the spatial influence radius.

4. The monitoring and early warning method based on a multimodal large model according to claim 1 is characterized in that: In S200, the association rules in the cross-scenario causal association analysis are: Personnel mobility risk rules: When the population density in a certain area of ​​a shopping mall exceeds 150% of the historical average density of the area during the same period within 10 minutes, it is marked as a local crowd-dense risk; When the overall flow of people in a mall increases by more than 30% of the expected flow of people for the day within 30 minutes, and the average length of stay of people entering the mall increases by more than 20 minutes, it is determined to be a mall congestion risk; Elevator operation risk rules: When the elevator fault alarm reaches 3 times or more within 1 hour, or there are more than two abnormal stops in a row, it is identified as an elevator fault risk; When the number of people in the elevator car reaches 90% of the rated load and the elevator continues to run for more than 15 minutes, and the average waiting time of people on the floor where the elevator is located increases by more than 5 minutes compared to normal days, it is marked as a risk of insufficient elevator capacity; Traffic situation risk rules: When the congestion index of surrounding roads exceeds 0.7 for 20 consecutive minutes (the congestion index ranges from 0 to 1, with higher values ​​indicating more severe congestion), and the congested road section is within 1 km of the main entrance and exit of the shopping mall, it is determined to be a traffic congestion-related risk; When the remaining parking space rate in the parking lot around the mall is less than 20% and the length of the vehicle queue at the parking lot entrance exceeds 50 meters, it is marked as a parking resource shortage risk; Cross-scenario comprehensive risk rules: When surrounding roads are severely congested, the flow of people at the mall entrance increases by 40% within 15 minutes, and the elevator failure rate increases by 30% compared to normal days, it is identified as a comprehensive traffic-personnel-equipment risk; When people gather in a certain area of ​​a shopping mall and elevators near the area frequently malfunction, and the traffic speed of the passage leading to the outside of the mall on the corresponding floor decreases by more than 50%, it is marked as a comprehensive safety risk in the local area.

5. The monitoring and early warning method based on a multimodal large model according to claim 1 is characterized in that: The specific steps of constructing an interpretable causal graph with scenario events as nodes and causal strengths as edges based on a multimodal large model in S300, causal graph construction and risk tracing, are as follows: (1) Disassemble the preliminary risk analysis results and abstract the scenario information and implicit events into causal graph nodes; (2) Combine domain knowledge and preset rules to screen potential causal relationships between nodes; (3) Quantitatively evaluate the causal strength between nodes from multiple dimensions, including time sequence, scope of influence, and intervention response; (4) Construct a directed weighted hierarchical causal graph with scenario events as nodes and causal strength as edge weights; (5) Use real-time data to verify the causal graph and update the graph structure and edge weights based on the new data.

6. The monitoring and early warning method based on a multimodal large model according to claim 1 is characterized in that: The calculation formula for the construction of the time series analysis model in the temporal data fusion prediction in S400 is: in is the predicted feature at time τ in the future, X t is the eigenvector at time t, X t-h is the historical feature vector at time th, h is the historical time step, α t is the attention weight at historical moment t, and is calculated as: r is the sum index, X current is the eigenvector at the current moment, The attention weight at the next τ time, is the feature vector at the future τ-1 moment, Transformer is the Transformer model structure, MLP is the multi-layer perceptron, ⊙ is element-by-element multiplication, t represents the time index of the current moment, and τ represents the time span of the future prediction.

7. The monitoring and early warning method based on a multimodal large model according to claim 1 is characterized in that: In the above S500, the early warning thresholds of various risks are dynamically adjusted through the risk entropy flow threshold model in the dynamic threshold early warning and disposal. The calculation formula is: Where T dynamic is the dynamic warning threshold, T0 is the basic warning threshold, λ is the entropy flow adjustment coefficient, H risk (t) is the risk entropy at time t, C i→risk is the causal strength of scenario event i on the occurrence of risk, N is the total number of events that affect the risk, t represents the time index of the current moment, and i is the index of the scenario event.

8. A monitoring and early warning device based on a multimodal large model, the system being applicable to the monitoring and early warning method based on a multimodal large model as described in any one of claims 1 to 7, characterized in that: The device includes: a multimodal data acquisition and fusion module, a cross-scenario causal association analysis module, a causal graph construction and risk tracing module, a temporal data fusion prediction module, and a dynamic threshold warning and disposal module; The multimodal data acquisition and fusion module collects multimodal data on personnel flow, elevator operation, and traffic conditions through sensors and cameras deployed in shopping malls, elevators, and surrounding road areas. After preprocessing the raw data, the module uses a modal entropy weight fusion function to integrate the different modal data into a unified feature vector. The cross-scenario causal association analysis module inputs the unified feature vector output by the fusion module into a cross-scenario association model built based on spatiotemporal causal logic, explores potential causal relationships between scenario data, and conducts risk assessment on the association results based on the association rules of personnel flow, elevator operation, traffic conditions, and cross-scenario comprehensive risks, identifies potential risk trends, and outputs preliminary analysis results including risk scenarios, association characteristics, and risk levels. The causal graph construction and risk tracing module: Based on a multimodal large model, key scenario information and implicit events in the preliminary analysis results are abstracted into causal graph nodes, and a directed weighted hierarchical causal graph is constructed with scenario events as nodes and causal strength as edge weights. Through counterfactual simulation, the risk probability changes after removing specific factors are deduced to locate the root cause of the risk; The temporal data fusion prediction module: integrates the current frame data of the surveillance video, the historical time period data and the preliminary prediction data, uses the time series analysis model to mine the time dimension features, combines historical rules and current status, and predicts the data change trend of each scene in the short term in the future; The dynamic threshold warning and disposal module: integrates cross-scenario correlation analysis results, causal reasoning conclusions and temporal prediction results, dynamically adjusts various risk warning thresholds through the risk entropy flow threshold model, triggers a warning when real-time monitoring data reaches or exceeds the dynamic threshold, generates a report containing risk causal attribution, and provides managers with accurate risk disposal guidance.

9. A monitoring and early warning device based on a multimodal large model, characterized in that: The early warning device includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the monitoring and early warning method based on the multimodal large model described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the monitoring and early warning method based on a multimodal large model as described in any one of claims 1 to 7.

Images