Safety monitoring method and system based on multi-modal large model and terminal equipment

Abstract

The invention relates to the technical field of safety monitoring and prediction, in particular to a safety monitoring method and system based on a multi-modal large model and terminal equipment, and the method comprises the steps: collecting multi-source heterogeneous data through a data collection interface, and cleaning the multi-source heterogeneous data to obtain multi-modal associated data; inputting the multi-modal associated data into a multi-modal large model for processing, and outputting a safety prediction result of the current coal conveying and unloading scene; and transmitting the safety prediction result to a management terminal, so that the management terminal performs corresponding early warning control operation according to the safety prediction result. By fusing the multi-source heterogeneous data, deep fusion and semantic unification of the multi-source heterogeneous data are realized, and the accuracy and interpretability of risk identification are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of security monitoring and prediction technology, and in particular to a security monitoring method, system and terminal equipment based on a multimodal large model. Background Technology

[0002] Safety hazard management in power production and construction remains a core challenge in the power industry's safety production system. Coal unloading scenarios, due to their dense equipment, complex operational processes, and interwoven multi-source data, are of paramount importance for safety management and harbor significant potential safety risks. However, existing safety monitoring and prediction methods still have significant technical limitations, with weak single-modal analysis capabilities. Text analysis is susceptible to fluctuations in the quality of unstructured text data, leading to insufficient accuracy in risk discovery. Visual inspection can only achieve coarse-grained localization of abnormal scenes and cannot integrate multi-source heterogeneous data such as logs and sensors for comprehensive risk assessment. Existing interactive question-and-answer systems mostly rely on template matching or small-scale model fine-tuning, making it difficult to accommodate multimodal input requirements such as images, videos, voice, and text, and unable to achieve collaborative interaction of cross-modal information. Existing methods mostly employ periodic offline analysis modes, making it difficult to respond in real-time to the dynamic changes in coal unloading operations, and unable to accurately predict the probability and scope of potential faults, let alone provide decision support for operation scheduling optimization. Summary of the Invention

[0003] In view of this, embodiments of this application provide a security monitoring method, system, and terminal device based on a multimodal large model, which can effectively solve the problem of weak single-modal analysis capabilities.

[0004] In a first aspect, embodiments of this application provide a security monitoring method based on a multimodal large model, including: Multi-source heterogeneous data is collected through a data acquisition interface, and the multi-source heterogeneous data is cleaned to obtain multimodal correlation data. The multimodal correlation data is input into a large multimodal model for processing, and the safety prediction results for the current coal conveying and unloading scenario are output. The security prediction results are transmitted to the management terminal so that the management terminal can perform corresponding early warning control operations based on the security prediction results.

[0005] In some embodiments, the multi-source heterogeneous data includes: textual knowledge data, visual data, and sensor data; The process of collecting multi-source heterogeneous data and cleaning the multi-source heterogeneous data to obtain multimodal correlation data includes: The text knowledge data is subjected to regular expression matching to remove irrelevant fields and generate text feature data in a uniform format. The visual data is cropped in key regions and video frame features are extracted. The extracted video frame features are then averaged and pooled along the time dimension to unify the visual feature dimensions and obtain visual feature data. The sensor data is segmented through a sliding window, then normalized to eliminate dimensional differences, and the trend information of numerical features is extracted through a temporal convolutional network. After temporal alignment, temporal feature data is obtained. The text feature data, the visual feature data, and the temporal feature data are aligned to obtain multimodal association data.

[0006] In some embodiments, the method further includes: The text knowledge data is encoded using a large language model as a text encoder to generate high-dimensional text vector type text feature data. The sensor data is encoded using a fully connected neural network as a numerical encoder to generate time-series feature data of the type of numerical embedding vector; The visual feature data is encoded by a visual encoder of a pre-trained model that compares text-image pairs to generate visual feature data of the visual embedding vector type.

[0007] In some embodiments, the method further includes: When a question is received from a user, the multimodal big data model analyzes the question through text recognition and provides corresponding search results or answers based on the question.

[0008] In some embodiments, aligning the text feature data, the visual feature data, and the temporal feature data to obtain multimodal association data includes: Calculate the cosine similarity between the text feature data and the visual feature data, and use the text feature data whose cosine similarity is greater than a preset value as the semantic label of the corresponding visual feature data; By using cross-modal contrastive learning, the text feature data, the visual feature data, and the temporal feature data are mapped to a semantic space, resulting in multimodal associated data in which the text feature data, the visual feature data, and the temporal feature data are interconnected based on the semantic space.

[0009] In some embodiments, the method further includes: Entity extraction is performed on the text feature data to obtain a keyword set; The keyword set is semantically similar to the nodes in the current knowledge graph, and keywords with similarity lower than a preset value are filtered out to obtain the core keyword set. The entity association paths of the input keyword set are constructed by the path connection method, and the association attributes of the keyword set are supplemented by the neighbor expansion method to update the knowledge graph.

[0010] Secondly, this application also provides a security monitoring system based on a multimodal large model, comprising: The data acquisition module is used to acquire multi-source heterogeneous data through the data acquisition interface, and to clean the multi-source heterogeneous data to obtain multimodal correlation data. The model recognition module is used to input the multimodal correlation data into the multimodal large model for processing and output the safety prediction result of the current coal conveying and unloading scenario; The alarm control module is used to transmit the security prediction results to the management terminal, so that the management terminal can perform corresponding early warning control operations based on the security prediction results.

[0011] In some embodiments, the system further includes an edge deployment module; The edge deployment module includes an edge computing device, which is located within the coal conveying and unloading scenario. The edge computing device is equipped with the distillation model of the multimodal large model.

[0012] Thirdly, this application also provides a terminal device, which includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the security monitoring method based on a multimodal large model.

[0013] Fourthly, this application also provides a readable storage medium storing a computer program that, when executed on a processor, implements the aforementioned security monitoring method based on a multimodal large model.

[0014] The embodiments of this application have the following beneficial effects: This application breaks down information silos by deeply integrating and semantically unifying multi-source heterogeneous data, effectively solving the problem of the separation between visual information and textual rules in traditional methods. It enables the system to automatically associate the corresponding hazard type, handling standards and historical cases from a single defect image, significantly improving the accuracy and interpretability of risk identification. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This paper illustrates a flowchart of a security monitoring method based on a multimodal large model according to an embodiment of this application. Figure 2 This illustration shows a data processing flow diagram of a security monitoring method based on a multimodal large model according to an embodiment of this application. Figure 3 A schematic diagram of the multimodal data processing flow of a safety monitoring system based on a multimodal large model, according to an embodiment of this application, is presented. Figure 4 The diagram shows a schematic of a safety monitoring system based on a multimodal large model according to an embodiment of this application. Detailed Implementation

[0017] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0018] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0020] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0021] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0022] To address the shortcomings of existing technologies, this application provides a safety monitoring method based on a multimodal large model. The method involves collecting multi-source heterogeneous data through a data acquisition interface, cleaning the data to obtain multimodal correlated data, inputting this data into a multimodal large model for processing, and outputting a safety prediction result for the current coal conveying and unloading scenario. This safety prediction result is then transmitted to a management terminal, enabling the terminal to perform corresponding early warning and control operations based on the prediction. This method achieves better safety monitoring by fusing and processing multi-source heterogeneous data through a multimodal large model.

[0023] The following examples illustrate the safety monitoring method based on a multimodal large model.

[0024] Figure 1 A flowchart of a security monitoring method based on a multimodal large model according to an embodiment of this application is shown. Exemplarily, the security monitoring method based on a multimodal large model includes the following steps: Step S100: Collect multi-source heterogeneous data through the data acquisition interface, and clean the multi-source heterogeneous data to obtain multimodal correlation data.

[0025] The method in this embodiment is applied to safety monitoring in coal conveying and unloading scenarios, and data collection is performed for this purpose. The data collected in this embodiment is multi-source heterogeneous data, that is, data is collected from multiple different data sources to form a dataset.

[0026] For example, by accessing the corresponding knowledge database through a search engine, textual knowledge data can be collected. This knowledge data includes static knowledge texts such as laws and regulations, standards and norms, accident cases, operation manuals, and project implementation paths, as well as dynamic operational texts such as coal conveying and unloading site operation logs (belt speed, coal flow) and maintenance reports (equipment wear and fault repair records).

[0027] Visual data can also be collected through cameras set up in coal conveying and unloading scenarios, such as fixed monitoring videos of core areas like coal conveyor belt corridors and unloading ditches, aerial images of the stockpile taken by drones, and equipment defect images uploaded by inspection personnel via their mobile devices.

[0028] For example, sensor data such as IoT sensor data (time-series data such as temperature, vibration, flow, etc.), equipment parameters (model, rated power), and hazard logs (hazard type, rectification records), etc.

[0029] The three types of data mentioned above have different sources and corresponding data structures, and there is no obvious correlation between them. However, text knowledge data mainly contains textual knowledge related to the coal conveying and unloading scenario, which can provide corresponding guidance for safety monitoring. The difference between visual data and sensor data is that visual data mainly consists of image and video data, while sensor data is non-image structured information data. Both types of data can reflect the current state of the coal conveying and unloading scenario.

[0030] The acquired data needs to be further preprocessed and then integrated to form multimodal association data that can be recognized by the model.

[0031] First, regular expression matching is performed on the text knowledge data to remove irrelevant fields and generate text feature data in a unified format.

[0032] We collected full text data from coal conveying and unloading scenarios, including knowledge-based texts such as laws and regulations, standards and specifications, accident cases, operation manuals, and project implementation paths, as well as operational texts such as on-site operation logs (e.g., belt speed records, coal flow data) and maintenance reports (e.g., equipment wear levels, historical fault repair records). Using NLP tools (e.g., jieba word segmentation, BERT entity recognition model), we performed word segmentation and key entity extraction on the text corpus (e.g., "equipment number," "fault type," "numerical threshold," "violation operation," "handling plan," etc.). We then used regular expression matching to remove irrelevant administrative records, duplicate descriptions, and other redundant fields, ultimately generating a unified format text feature sequence, which was stored in the text database.

[0033] The visual data is cropped in key areas, and video frame features are extracted. The extracted video frame features are then averaged and pooled along the time dimension to unify the visual feature dimensions and obtain visual feature data.

[0034] Visual data includes fixed monitoring videos of areas such as coal conveyor belt corridors, unloading trenches, and crushers; panoramic images of the stockpile taken by drones; and images of equipment defects uploaded by inspection personnel via mobile devices. The images / videos are preprocessed first: key areas are cropped (e.g., belt joints, motor bearings), the resolution is adjusted to 1024×1024, and noise is eliminated using Gaussian filtering. A lightweight visual Transformer model (e.g., MobileViT) is used to extract local features from the images (e.g., conveyor belt cracks, coal spillage outlines, bolt loosening marks); dynamic features (e.g., personnel movement trajectories when crossing safety lines, abnormal equipment rotation trajectories) are extracted from the video stream using optical flow methods (e.g., Farneback algorithm), and the video frame features are averaged and pooled along the time dimension to unify the visual feature dimensions. The preprocessed visual data is temporarily stored in a visual database for subsequent cross-modal alignment.

[0035] The sensor data is segmented using a sliding window, then normalized to eliminate dimensional differences, and the trend information of numerical features is extracted through a temporal convolutional network. After temporal alignment, temporal feature data is obtained.

[0036] Data from IoT sensors on coal conveying and unloading equipment is collected, including real-time time-series data such as coal flow rate from belt weighing sensors, equipment vibration frequency from vibration sensors, and bearing temperature from temperature sensors. Outliers are removed, data is segmented using a sliding window, and then normalized to eliminate dimensional differences. Finally, a temporal convolutional network (TCN) is used to extract trend information of numerical features (such as sudden temperature rises, abnormal fluctuations in vibration frequency, and sudden drops in coal flow rates), and after time-series alignment, the data is stored in a numerical database.

[0037] The text feature data, visual feature data, and temporal feature data are then aligned to obtain multimodal association data. This multimodal association data can be used for security monitoring and analysis, as well as for updating security knowledge graphs.

[0038] The alignment here is cross-modal alignment, which can be achieved by introducing the CLIP model to align features extracted from visual data with semantics in text data. Furthermore, the cosine similarity between visual features and text entities is calculated, and highly correlated text entities are selected as semantic labels for visual features. This transforms visual information into semantic representations that can be used for decision-making, thus addressing the problem of the disconnect between visual and textual knowledge.

[0039] In this way, cross-modal comparative learning can be used to map the three types of embedded vectors—text, numerical, and visual—to the semantic space, solving the problem of the separation between visual information and textual knowledge, and providing support for subsequent linked reasoning.

[0040] In addition, entity extraction will be performed on the text feature data to obtain a keyword set. These keywords may include entity types such as "device name," "fault type," "violation of regulations," "handling measures," and "numerical threshold."

[0041] The keyword set is semantically similar to the nodes in the current knowledge graph. Keywords with similarity scores below a preset value are filtered out to obtain the core keyword set. This involves a filtering operation on the keyword set to remove keywords with low relevance.

[0042] The entity association path of the delivered keyword set is constructed by the path connection method, such as "conveyor belt crack → equipment overload → shutdown for maintenance → belt replacement". The association attributes of the keyword set are supplemented by the neighbor expansion method. For example, the neighbor entities of "abnormal vibration frequency" include "bearing wear" and "insufficient lubricating oil" to update the knowledge graph.

[0043] Step S200: Input the multimodal correlation data into the multimodal large model for processing, and output the safety prediction result of the current coal conveying and unloading scenario.

[0044] The multimodal large model is a pre-trained model used in this embodiment to predict safety conditions in coal conveying and unloading scenarios. This model includes a basic capability layer and a core model layer; the basic capability layer includes an image recognition model, a text understanding model, and a speech-to-text model. By inheriting these basic single-modal basic models, underlying capability support can be provided.

[0045] The core model layer includes a multimodal vision-language large model, a hybrid expert architecture, and a self-attention mechanism framework.

[0046] By using a multimodal vision-language large model as the core, equipped with a hybrid expert architecture, and employing a self-attention mechanism framework for unified modeling, the model's adaptability to multiple scenarios and tasks in coal transportation and unloading is improved.

[0047] During the fine-tuning phase, the model's encoder-decoder layers and backbone network are pruned (e.g., the number of encoder layers is reduced), and the attention mechanism and feature fusion strategy are optimized. LoRA technology can also be used for lightweight fine-tuning of the model, while full-parameter fine-tuning of the core decision layer is performed, reducing deployment costs while maintaining accuracy. Furthermore, iterative reinforcement learning can be employed, such as introducing a safety reward model and PPO reinforcement learning algorithm, using historical hazard handling results (e.g., identification accuracy, solution matching degree, rectification completion rate) as feedback signals to continuously optimize the model's risk perception and decision-making capabilities, achieving autonomous reasoning and iterative learning.

[0048] Multimodal correlation data reflects the current state of the coal conveying and unloading scenario. This data serves as input and is analyzed by the aforementioned multimodal large model. For multimodal correlation data, this embodiment of the model will provide three types of safety prediction results, including: Hazard assessment: Determine whether there are any hazards in the current scenario, such as "the coal conveyor belt is running off-center, which is a level one hazard".

[0049] Fault location: Accurately locate the location and cause of the hidden danger, such as "the cause of belt misalignment is insufficient belt tension, located in the middle section of belt No. 3".

[0050] Solution: Provide a handling procedure that complies with the company's safety regulations (e.g., "Stop the machine immediately, adjust the tension wheel pressure to 0.3MPa, and restart after re-inspection").

[0051] Step S300: The security prediction result is transmitted to the management terminal so that the management terminal can perform corresponding early warning control operations based on the security prediction result.

[0052] The aforementioned safety prediction results are transmitted to a high-level management terminal. This management terminal is a device that manages the entire coal conveying and unloading scenario; it can be a central computer or a cloud-based server, and it has the authority to adjust the coal conveying and unloading scenario. Based on the aforementioned safety prediction results, this embodiment can generate the existing problems in the current coal conveying and unloading scenario, their locations, and suggested solutions. For the management terminal, operators can evaluate these safety prediction results and then decide whether to implement the solution or perform manual adjustments.

[0053] No adjustments will be made when the safety prediction result indicates normalcy. Adjustments will only be made when the safety prediction result indicates a problem. These adjustments can be performed manually or by a program. During adjustments, the progress can be tracked in real time, thus achieving a closed-loop management system from automatic identification to manual review and rectification tracking.

[0054] In addition, a visual interface can display three main components: first, real-time analysis results (hazard type, location, and level); second, risk trend charts (such as equipment temperature change curves over the past 7 days and hazard incidence statistics); and third, suggested operating procedures (displayed in a combination of text and graphics, and supported for download and printing). Simultaneously, safety trend analysis reports are generated regularly to support management decision-making.

[0055] like Figure 2 The diagram illustrates the multimodal large language model-based safety monitoring and prediction process for coal conveying and unloading scenarios. It showcases the data flow and model processing during safety monitoring using a multimodal large language model in these scenarios. Taking image data from the coal conveying and unloading scenario as an example, after alignment and preprocessing, it is input into the model for further analysis. After obtaining safety prediction results, the model can undergo iterative reinforcement learning based on a safety reward model to update itself in real time, ensuring it doesn't fall behind.

[0056] Figure 3 This primarily demonstrates the multimodal data processing flow in a safety monitoring system based on a multimodal large model. After collection and processing, the data enters the model for further processing. Depending on whether a retrieval is required, the data will either enter a retrieval or generation process. For example, after acquiring various image information of the current coal unloading scenario, the system needs to generate safety monitoring results based on this information, so it will enter the generation process, ultimately generating the corresponding safety prediction result. Another example is when employees use the system to retrieve safety regulations; they might input a question into the system, in which case the data will enter the retrieval process for searching. Figure 3 The index building and document parsing shown are used to obtain search results and reply them to employees.

[0057] Furthermore, the model in this embodiment also has retrieval and question-answering functions. When it receives a question input by a user, the multimodal big data model analyzes the question through text recognition and provides corresponding retrieval results or answers based on the question.

[0058] This embodiment's method breaks down information silos by deeply fusing and semantically unifying multi-source heterogeneous data. It systematically integrates textual knowledge such as laws and regulations and accident cases, visual data such as surveillance videos and drone images, and structured data such as sensor time-series signals and equipment parameters to form a unified multimodal relational database. Furthermore, this embodiment's method introduces the CLIP model to achieve cross-modal semantic alignment between image features and entities in the safety knowledge graph, effectively solving the problem of the separation between visual information and textual rules in traditional methods. This allows the system to automatically associate a defective image with corresponding hazard types, handling standards, and historical cases, significantly improving the accuracy and interpretability of risk identification. The model adopts a technical route centered on a multimodal large model and supported by a hybrid expert architecture. Combined with lightweight fine-tuning and reinforcement learning mechanisms, the model maintains efficient deployment capabilities while possessing the ability to internalize knowledge and optimize strategies for specific coal transportation and unloading scenarios. By introducing a safety reward function based on historical handling effects, the model can continuously iterate and optimize its risk judgment logic and handling recommendation strategies, achieving a shift from passive matching to proactive evolution, significantly reducing the frequency of manual intervention and model maintenance costs.

[0059] Figure 4 A schematic diagram of a security monitoring system based on a multimodal large model according to an embodiment of this application is shown. Exemplarily, the security monitoring system based on a multimodal large model includes: The data acquisition module 10 is used to acquire multi-source heterogeneous data through the data acquisition interface, and clean the multi-source heterogeneous data to obtain multimodal correlation data. Model recognition module 20 is used to input the multimodal correlation data into a multimodal large model for processing and output the safety prediction result of the current coal conveying and unloading scenario; The alarm control module 30 is used to transmit the security prediction result to the management terminal so that the management terminal can perform corresponding early warning control operations based on the security prediction result.

[0060] As an example, the system also includes an edge deployment module 40. The edge deployment module 40 includes an edge computing device located within the coal conveying and unloading scenario, and the edge computing device is equipped with the distillation model of the multimodal large model.

[0061] It is understood that the device in this embodiment corresponds to the security monitoring method based on a multimodal large model in the above embodiments. The options in the above embodiments are also applicable to this embodiment, so they will not be described again here.

[0062] This application also provides a terminal device, exemplary of which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the functions of the various modules in the above-described security monitoring method based on a multimodal large model or the above-described security monitoring system based on a multimodal large model.

[0063] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0064] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0065] This application also provides a readable storage medium for storing the computer program used in the aforementioned terminal device.

[0066] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0067] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0068] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0069] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A safety monitoring method based on a multimodal large model, characterized in that, include: Multi-source heterogeneous data is collected through a data acquisition interface, and the multi-source heterogeneous data is cleaned to obtain multimodal correlation data. The multimodal correlation data is input into a large multimodal model for processing, and the safety prediction results for the current coal conveying and unloading scenario are output. The security prediction results are transmitted to the management terminal so that the management terminal can perform corresponding early warning control operations based on the security prediction results.

2. The safety monitoring method based on a multimodal large model according to claim 1, characterized in that, The multi-source heterogeneous data includes: text knowledge data, visual data, and sensor data; The process of collecting multi-source heterogeneous data and cleaning the multi-source heterogeneous data to obtain multimodal correlation data includes: The text knowledge data is subjected to regular expression matching to remove irrelevant fields and generate text feature data in a uniform format. The visual data is cropped in key regions and video frame features are extracted. The extracted video frame features are then averaged and pooled along the time dimension to unify the visual feature dimensions and obtain visual feature data. The sensor data is segmented and normalized to obtain dimensionless data. The trend information of the dimensionless data is then extracted and time-aligned using a temporal convolutional network to obtain temporal feature data. The text feature data, the visual feature data, and the temporal feature data are aligned to obtain multimodal association data.

3. The safety monitoring method based on a multimodal large model according to claim 2, characterized in that, Also includes: The text knowledge data is encoded using a large language model as a text encoder to generate high-dimensional text vector type text feature data. The sensor data is encoded using a fully connected neural network as a numerical encoder to generate time-series feature data of the type of numerical embedding vector; The visual feature data is encoded by a visual encoder of a pre-trained model that compares text-image pairs to generate visual feature data of the visual embedding vector type.

4. The safety monitoring method based on a multimodal large model according to claim 1, characterized in that, Also includes: When a question is received from a user, the multimodal large model is used to perform text recognition, analyze the question, and provide corresponding search results or answers based on the question.

5. The safety monitoring method based on a multimodal large model according to claim 2, characterized in that, The step of aligning the text feature data, the visual feature data, and the temporal feature data to obtain multimodal association data includes: Calculate the cosine similarity between the text feature data and the visual feature data, and use the text feature data whose cosine similarity is greater than a preset value as the semantic label of the corresponding visual feature data; By using cross-modal contrastive learning, the text feature data, the visual feature data, and the temporal feature data are mapped to a semantic space, resulting in multimodal associated data in which the text feature data, the visual feature data, and the temporal feature data are interconnected based on the semantic space.

6. The safety monitoring method based on a multimodal large model according to claim 2, characterized in that, Also includes: Entity extraction is performed on the text feature data to obtain a keyword set; The keyword set is semantically similar to the nodes in the current knowledge graph, and keywords with similarity lower than a preset value are filtered out to obtain the core keyword set. The entity association paths of the input keyword set are constructed by the path connection method, and the association attributes of the keyword set are supplemented by the neighbor expansion method to update the knowledge graph.

7. A safety monitoring system based on a multimodal large model, characterized in that, include: The data acquisition module is used to acquire multi-source heterogeneous data through the data acquisition interface, and to clean the multi-source heterogeneous data to obtain multimodal correlation data. The model recognition module is used to input the multimodal correlation data into the multimodal large model for processing and output the safety prediction result of the current coal conveying and unloading scenario; The alarm control module is used to transmit the security prediction results to the management terminal, so that the management terminal can perform corresponding early warning control operations based on the security prediction results.

8. The safety monitoring system based on a multimodal large model according to claim 7, characterized in that, It also includes an edge deployment module; The edge deployment module includes an edge computing device, which is located within the coal conveying and unloading scenario. The edge computing device is equipped with the distillation model of the multimodal large model.

9. A terminal device, characterized in that, The terminal device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the security monitoring method based on a multimodal large model as described in any one of claims 1-6.

10. A readable storage medium, characterized in that, It stores a computer program, which, when executed on a processor, implements the security monitoring method based on a multimodal large model according to any one of claims 1-6.

Images