Remote alarm risk research and judgment method and device based on coal mine underground multi-modal data

Abstract

The present application provides a kind of remote alarm risk research and judgment method and device based on coal mine underground multi-modal data, it is related to coal mine safety technical field, method includes: in the case where detecting coal mine underground sensor data exception, according to the type of abnormal sensor, the time series data of abnormal sensor in coal mine underground area is acquired, the associated time series data of multiple associated sensors influencing abnormal sensor data, the state and behavior of feature object influencing coal mine underground safety in image video, personnel's work trajectory;Determine the alarm reason of sensor data exception, and carry out multi-angle alarm risk research and judgment by searching enhanced generation large model, obtain the risk level of alarm reason and corresponding alarm reason coping strategy.Therefore, through multi-modal data fusion analysis of sensor data, image video and work trajectory in coal mine underground area, the intelligent risk research and judgment of coal mine safety alarm is realized, alarm risk research and judgment time is shortened, and alarm risk research and judgment precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of coal mine safety technology, and in particular to a remote alarm risk analysis method and device based on multimodal data in underground coal mines. Background Art

[0002] Currently, the work on coal mine safety alarms mainly relies on coal mine personnel combining their experience and the alarm information uploaded by the coal mine to manually screen and query relevant monitoring videos and measurement point data to conduct alarm risk assessment. However, due to the large number of mines and large differences in production conditions, a large number of alarms require manual assessment every day, and the alarm data and causes are complex. The timeliness and accuracy of the assessment are insufficient, and it cannot systematically solve the problems of alarm authenticity verification and comprehensive risk assessment. Summary of the Invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of the present invention is to propose a remote alarm risk assessment method based on multimodal data in coal mines. By performing multimodal data fusion analysis on sensor data, image videos and work trajectories in the underground area of ​​the coal mine, intelligent risk assessment of coal mine safety alarms can be realized, the alarm risk assessment time can be shortened, and the accuracy of alarm risk assessment can be improved.

[0005] The second object of the present invention is to propose a remote alarm risk analysis device based on multimodal data in coal mines.

[0006] A third object of the present invention is to provide an electronic device.

[0007] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions.

[0008] To achieve the above objectives, a first embodiment of the present invention proposes a remote alarm risk assessment method based on multimodal data in coal mines, the method comprising:

[0009] When abnormal sensor data is detected in the coal mine, time series data of the abnormal sensor in the coal mine area corresponding to the sensor data and / or associated time series data of multiple associated sensors in the coal mine area that affect the abnormal sensor data are obtained according to the type of the abnormal sensor;

[0010] Acquiring images and videos within the underground coal mine area, and performing cross-modal recognition on the images and videos to identify states and behaviors of characteristic objects in the images and videos that affect safety in the underground coal mine;

[0011] Obtaining a work trajectory composed of job type information and positioning data of personnel in the underground area of ​​the coal mine;

[0012] Based on the time series data, associated time series data, the status and behavior of characteristic objects, and the working trajectory, the alarm cause when the sensor data is abnormal is determined, and a large model is generated through retrieval enhancement to conduct multi-angle alarm risk analysis on the time series data, associated time series data, the status and behavior of characteristic objects, and the working trajectory, determine the risk level of the alarm cause, and generate response strategies for alarm causes of different risk levels.

[0013] To achieve the above-mentioned purpose, a second embodiment of the present invention proposes a remote alarm risk analysis device based on multimodal data of underground coal mines, the device comprising:

[0014] A first acquisition module is configured to, when abnormal sensor data is detected in an underground coal mine, acquire, based on the type of the abnormal sensor, time series data of the abnormal sensor in the underground coal mine corresponding to the sensor data, and / or associated time series data of multiple associated sensors in the underground coal mine that affect the abnormal sensor data;

[0015] A second acquisition module is configured to acquire images and videos within the underground coal mine area, and perform cross-modal recognition on the images and videos to identify states and behaviors of characteristic objects in the images and videos that affect safety in the underground coal mine;

[0016] The third acquisition module is used to obtain the work trajectory of the personnel in the underground area of ​​the coal mine, which is composed of the work type information and positioning data;

[0017] The alarm and risk assessment module is used to determine the cause of the alarm when the sensor data is abnormal based on the time series data, associated time series data, the state and behavior of the characteristic objects, and the working trajectory, and to perform multi-angle alarm risk assessment on the time series data, associated time series data, the state and behavior of the characteristic objects, and the working trajectory through retrieval enhancement to determine the risk level of the alarm cause and generate response strategies for alarm causes of different risk levels.

[0018] The embodiments of the present invention provide a remote alarm risk assessment method, device, electronic device, and storage medium based on multimodal data from underground coal mines. When abnormal sensor data is detected in underground coal mines, the method, device, electronic device, and storage medium are used to obtain, based on the type of abnormal sensor, the time series data of the abnormal sensor in the underground coal mine area, the associated time series data of multiple associated sensors that affect the abnormal sensor data, the state and behavior of characteristic objects in the image and video that affect underground coal mine safety, and the work trajectory of personnel; determine the cause of the alarm, and perform multi-angle alarm risk assessment by searching and enhancing the generation of a large model to obtain the risk level of the alarm cause and the corresponding alarm cause response strategy. Thus, by performing multimodal data fusion analysis on sensor data, image and video, and work trajectory in the underground coal mine area, intelligent risk assessment of coal mine safety alarms is achieved, the alarm risk assessment time is shortened, and the accuracy of alarm risk assessment is improved.

[0019] To achieve the above-mentioned purpose, the third aspect embodiment of the present invention proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0020] In order to achieve the above-mentioned objectives, an embodiment of the fourth aspect of the present invention proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0021] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flow chart of a remote alarm risk assessment method based on multimodal data in underground coal mines provided by an embodiment of the present invention;

[0024] Figure 2 This is an application flow chart of a remote alarm risk assessment method based on multimodal data in underground coal mines provided by an embodiment of the present invention;

[0025] Figure 3 A schematic structural diagram of a remote alarm risk analysis device based on multimodal data in underground coal mines provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0027] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of the present invention comply with the relevant provisions of relevant laws and regulations.

[0028] The following describes a remote alarm risk assessment method and device based on multimodal data in underground coal mines according to an embodiment of the present invention with reference to the accompanying drawings.

[0029] Figure 1 A flow chart of a remote alarm risk assessment method based on multimodal data in underground coal mines provided by an embodiment of the present invention.

[0030] like Figure 1 As shown, the method includes the following steps:

[0031] Step 101, when abnormal sensor data is detected in the coal mine underground, obtain the time series data of the abnormal sensor in the coal mine underground area corresponding to the sensor data, and / or the associated time series data of multiple associated sensors in the coal mine underground area that affect the abnormal sensor data, according to the type of the abnormal sensor.

[0032] In some possible implementations, coal mine underground sensor data may include, but is not limited to, carbon monoxide and methane.

[0033] Specifically, taking the case of a carbon monoxide sensor in a coal mine as an example, based on practical experience and business logic in coal mine safety production, carbon monoxide alarms are often accompanied by rising temperatures and increased smoke concentrations. Therefore, once a carbon monoxide sensor issues an abnormal signal, temperature and smoke time series data from temperature sensors and smoke sensors in the same area as the carbon monoxide sensor are immediately collected to serve as correlated time series data for multiple associated sensors in the coal mine that affect carbon monoxide.

[0034] Taking methane as an example, the transmission and distribution of methane underground in coal mines is closely related to ventilation systems, mining operations, and other factors. When a methane sensor at a specific location sounds an alarm, the methane concentration data from sensors upstream and downstream of the sensor is collected as time series data, and a time series curve is constructed over time. If multiple correlated sensors affecting the methane sensor data are wind direction and air volume sensors in the ventilation system, the correlated time series data is the wind direction and air volume time series data from the wind direction and air volume sensors in the same area as the methane sensor. This allows for joint analysis of multiple sensors at different locations to accurately pinpoint the source of the problem.

[0035] Thus, by sensor data time series analysis, multi-type and multi-site sensor data fusion analysis and business mechanism model are packaged into lightweight algorithm modules, and the algorithm modules have high scalability and flexibility.

[0036] In step 102, image videos in the coal mine underground area are acquired, and cross-modal recognition is performed on the image videos to identify the state and behavior of the feature objects in the image videos that affect the safety of the coal mine underground.

[0037] In some possible implementation manners, acquiring image videos in the coal mine underground area and performing cross-modal recognition on the image videos to identify the state and behavior of the feature objects in the image videos that affect the safety of the coal mine underground include: acquiring image videos in the coal mine underground area, and setting scene prompt words under different operating conditions of the coal mine underground; performing frame-by-frame cross-modal recognition on the image videos based on the scene prompt words to identify the features and motion trajectories of the feature objects in the image videos, and judging the state and behavior of the feature objects according to the features and motion trajectories.

[0038] Specifically, in order to make the analysis process of the large model more logical and interpretable, the thinking chain technology is introduced to decompose complex problems into a series of logical steps to build a multi-modal large model. In image video analysis, the multi-modal large model first scans the image video frames based on the prompt words to identify the feature objects in the image videos, such as equipment, personnel, smoke, etc. Then, according to the features and motion trajectories of the feature objects, their state and behavior are judged. For example, whether the equipment is in a normal operating state, whether the personnel have irregular operations, etc. With the help of the multi-modal large model technology, through the optimized prompt words and thinking chain technology, the image videos in the coal mine underground area are analyzed in depth to realize accurate identification and judgment of multiple scenes, and to provide key support for alarm risk research and judgment.

[0039] In step 103, the work trajectory of the personnel in the coal mine underground area is acquired according to the work type information and positioning data of the personnel.

[0040] In some possible implementation manners, acquiring the work trajectory of the personnel in the coal mine underground area according to the work type information and positioning data of the personnel includes: acquiring the work type information and positioning data of the personnel in the coal mine underground area through the coal mine underground monitoring system; and depicting the work trajectory of the personnel in the coal mine underground area according to the work type information and positioning data of the personnel, wherein the work trajectory includes multiple coal mine underground areas passed through and entry and exit times of each coal mine underground area.

[0041] Optionally, for alarms caused by underground work, such as the risk of alarms of excessive methane concentrations due to forklift operations, or excessive alarms due to misoperation of the methane calibration, it is necessary to verify the authenticity of the cause of the alarm based on the upstream and downstream methane concentration, wind direction, wind volume time series data, methane calibration operations, and methane concentration data, and then combined with the methane receiving action trajectory (alarm time and location).

[0042] Step 104, based on the time series data, associated time series data, the state and behavior of the characteristic objects, and the working trajectory, determine the alarm cause when the sensor data is abnormal, and generate a large model through retrieval enhancement to conduct multi-angle alarm risk analysis on the time series data, associated time series data, the state and behavior of the characteristic objects, and the working trajectory, determine the risk level of the alarm cause, and generate alarm cause response strategies for different risk levels.

[0043] In some possible implementations, the method further includes: obtaining time series data within a time threshold before and after the abnormal sensor alarm, and obtaining associated time series data of multiple associated sensors within the same time threshold based on the type of the abnormal sensor; performing data fusion analysis on the time series data within the time threshold and the associated time series data based on the business mechanism model corresponding to the abnormal sensor to deduce the alarm cause of the abnormal sensor.

[0044] The business mechanism model corresponding to the abnormal sensor refers to how the abnormal sensor works in the business system and its relationship with the business logic.

[0045] Optionally, in the case of abnormal carbon monoxide sensor data, the alarm causes include fire hazards underground, abnormal carbon monoxide sensor, and exhaust emissions from vehicles in the same area as the carbon monoxide sensor; in the case of abnormal methane sensor data, the alarm causes include the presence of a gas leakage source, abnormal coal mine ventilation system, and incorrect calibration of the methane sensor.

[0046] Optionally, in the case where the time series data is carbon monoxide, the associated time series data is the temperature and smoke concentration time series data of the temperature sensor and the smoke sensor in the same area of ​​the carbon monoxide sensor, the state and behavior of the characteristic object are the vehicle path and the exhaust gas emitted by the vehicle in the same area of ​​the carbon monoxide sensor, and the work trajectory is the movement trajectory of the vehicle workers in the same area of ​​the carbon monoxide sensor; the correlation analysis algorithm is used to calculate the correlation coefficient between the carbon monoxide concentration and the temperature and smoke concentration time series data. When the correlation coefficient exceeds the set threshold, it is judged that the cause of the alarm is caused by the fire hazard generated underground; when the correlation coefficient is less than the set threshold, there is no abnormal fluctuation in the temperature and smoke concentration time series data of the smoke sensor, then the cause of the alarm may be that the carbon monoxide sensor is abnormal or there is exhaust gas emitted by a vehicle in the same area of ​​the carbon monoxide sensor. For example, there is a vehicle path in the same area of ​​the carbon monoxide sensor, and it coincides with the corresponding work trajectory of the vehicle workers, then the cause of the alarm is that the exhaust gas emitted by the vehicle causes the carbon monoxide sensor data to be abnormal.

[0047] Furthermore, in the case where the time series data is methane, the associated time series data is the wind direction and wind volume time series data of the wind direction and wind volume sensors of the coal mine ventilation system in the same area as the methane sensor. The state and behavior of the characteristic object are the methane calibration operation in the same area of ​​the methane sensor and the methane concentration data recorded under the methane calibration operation. The methane calibration operation corresponds to the methane receiving action trajectory of the workers in the same area of ​​the methane sensor. Based on the methane concentration, wind direction, wind volume time series data, methane calibration operation, methane concentration data and methane receiving action trajectory corresponding to the upstream and downstream of the methane sensor, the alarm cause when the methane sensor data is abnormal is determined.

[0048] Specifically, if the upstream methane concentration is normal but the downstream methane concentration suddenly increases, and the ventilation direction and air volume time series data show no significant changes, the alarm is initially suspected to be caused by a gas leak near the abnormal location (methane sensor). If both upstream and downstream methane concentrations increase abnormally, and the ventilation volume decreases in the ventilation direction and air volume time series data, the alarm may be caused by a malfunction in the coal mine ventilation system. Different response measures are taken depending on the severity of the alarm.

[0049] A pattern matching model is constructed based on the standard process and logic of methane calibration operations. During the methane calibration process, the methane sensor receives standard gas according to a specific time sequence and operating procedures, and records the corresponding changes in methane concentration data. When a methane alarm occurs, the sensor data before and after the alarm is compared with the standard data pattern of the methane calibration operation. If the data change pattern is highly similar to the standard data pattern, it can be preliminarily determined that the methane alarm may be caused by an incorrect calibration operation.

[0050] In some possible implementation manners, the risk of the alarm cause is judged and the alarm cause coping strategy of different risk levels is generated by searching the enhanced large model to perform multi-angle alarm risk judgment on the time series data, the associated time series data, the state and behavior of the feature object, and the work trajectory, judging the risk level of the alarm cause, and generating the alarm cause coping strategy of different risk levels, including: searching the enhanced large model to perform context analysis on the time series data, the associated time series data, the state and behavior of the feature object, and the work trajectory based on a preset coal mine risk hidden vector knowledge base to obtain an initial search result; performing multi-angle alarm risk judgment on the adjusted prompt in the initial search result by the set multi-angle coal industry risk judgment large model, judging the risk level of the alarm cause, and generating the alarm cause coping strategy of different risk levels; wherein the coal mine risk hidden vector knowledge base is constructed based on coal mine hidden accident cases and coal mine related legal regulation data, coal mine industry standard data, and coal mine safety regulation data.

[0051] The context analysis result search and the mixed search can improve the accuracy and recall rate of the search result.

[0052] In other possible implementation manners, the multi-modal large model for cross-modal recognition of image videos, the search enhanced large model, and the multi-angle coal industry risk judgment large model are parameter optimized according to the alarm cause coping strategy and the data of remote alarm. Specifically, the alarm cause coping strategy and the data of remote alarm in each link of the judgment process are collected by setting a link tracking module, which are displayed to professional staff through a visualization platform, and the multi-modal large model for cross-modal recognition of image videos, the search enhanced large model, and the multi-angle coal industry risk judgment large model are parameter optimized according to the feedback of the staff, to realize the virtuous cycle of continuous feedback and iteration of the judgment capability.

[0053] The remote alarm risk judgment method based on the multi-modal data in the coal mine underground area of the embodiment of the application, in the case that the sensor data in the coal mine underground area is detected to be abnormal, acquires the time series data of the abnormal sensor in the coal mine underground area, the associated time series data of the multiple associated sensors affecting the abnormal sensor data, the state and behavior of the feature object affecting the safety of the coal mine underground area in the image video, and the work trajectory of the personnel according to the type of the abnormal sensor; judges the alarm cause, and performs multi-angle alarm risk judgment by the search enhanced large model, judges the risk level of the alarm cause, and generates the alarm cause coping strategy of different risk levels. Therefore, the multi-modal data fusion analysis is performed on the sensor data, the image video, and the work trajectory in the coal mine underground area, the intelligent risk judgment of the coal mine safety alarm is realized, the alarm risk judgment time is shortened, and the alarm risk judgment accuracy is improved.

[0054] The present invention also proposes an application flow chart of a remote alarm risk assessment method based on multimodal data in coal mines, such as Figure 2 As shown in the figure, the application process includes field data access, alarm information preprocessing, intelligent analysis and judgment, and user feedback management. Specifically:

[0055] On-site data access: Connect to the coal mine database (safety monitoring system, personnel positioning system, industrial video monitoring system), collect coal mine alarm data in real time (time series data of sensor data anomalies and related time series data), personnel's job information and positioning data in the coal mine underground area, and images and videos in the coal mine underground area, and open data permissions to provide a data basis for alarm risk analysis.

[0056] Alarm information preprocessing: After the user enters the alarm information, the analysis scenario to which the alarm information (time series data and associated time series data when sensor data is abnormal) belongs is identified based on the coal industry macro model (based on the introduction of a thinking chain based on the coal mine alarm report to generate a specific analysis task process). At the same time, key data (such as the work trajectory of underground personnel of the mine and the status and behavior of characteristic objects in the underground coal mine area) is automatically identified from the alarm information reported by the mine, providing an important basis for subsequent alarm risk analysis. For analysis scenarios that require information other than the alarm, the natural language query (text2sql) function is used to dynamically query and complete relevant data. The analysis scenario is identified by organizing the analysis scenarios of different types of alarms and matching the corresponding scenarios by analyzing the alarm information. The key data is identified by extracting the work trajectory of underground personnel of the mine and the status and behavior of characteristic objects in the underground coal mine area from the alarm information, namely key data such as time, location, and personnel. The query and completion of relevant data is based on the automatic query of the coal mine database based on text2sql technology to supplement the additional necessary information for the analysis.

[0057] Intelligent Assessment: Based on the necessary assessment information (time series data and associated time series data, personnel job information and location data, and images and videos) identified through alarm information preprocessing, assessment tasks (such as image and video recognition, identification and analysis of time series data and associated time series data, and personnel trajectory verification) are assigned. Combined with the coal mine risk vector knowledge base, comprehensive risk assessment is performed to obtain a comprehensive assessment result (determining the cause of the alarm when sensor data is abnormal). Intermediate assessment results are retained and uploaded to the visualization platform for user review and confirmation.

[0058] User feedback management: By providing a visual management platform, users can conduct real-time review (user review behavior) of the processing status (rejected, unconfirmed / unprocessed, confirmed / unprocessed, unconfirmed / processed, confirmed / processed) of the comprehensive risk assessment process (user review confirmation, user review rejection, generation of the risk level of the alarm cause and the corresponding alarm cause response strategy and feedback). After the risk level of the alarm cause and the corresponding alarm cause response strategy and feedback are generated, the risk level of the alarm cause and the corresponding alarm cause response strategy and feedback are manually archived / archived by date (supports viewing / exporting archived data operations) and recorded in the assessment record library.

[0059] In order to implement the above embodiment, the present invention also proposes a remote alarm risk analysis device based on multimodal data of coal mine underground.

[0060] Figure 3 A schematic structural diagram of a remote alarm risk analysis device based on multimodal data in underground coal mines provided by an embodiment of the present invention.

[0061] like Figure 3 As shown, the remote alarm risk analysis device 30 based on multimodal data of underground coal mines includes: a first acquisition module 31, a second acquisition module 32, a third acquisition module 33, and an alarm and risk analysis module 34.

[0062] The first acquisition module 31 is configured to, when abnormal sensor data is detected in the coal mine, acquire, based on the type of the abnormal sensor, time series data of the abnormal sensor in the coal mine area corresponding to the sensor data, and / or associated time series data of multiple associated sensors in the coal mine area that affect the abnormal sensor data;

[0063] A second acquisition module 32 is configured to acquire images and videos in the underground coal mine area and perform cross-modal recognition on the images and videos to identify states and behaviors of characteristic objects in the images and videos that affect safety in the underground coal mine;

[0064] The third acquisition module 33 is used to obtain the work trajectory of the personnel in the underground area of ​​the coal mine, which is composed of the work type information and positioning data;

[0065] The alarm and risk assessment module 34 is used to determine the cause of the alarm when the sensor data is abnormal based on the time series data, associated time series data, the state and behavior of the characteristic object, and the working trajectory, and to perform multi-angle alarm risk assessment on the time series data, associated time series data, the state and behavior of the characteristic object, and the working trajectory by retrieval enhancement to generate a large model, determine the risk level of the alarm cause, and generate response strategies for alarm causes of different risk levels.

[0066] Furthermore, in a possible implementation of the embodiment of the present invention, the second acquisition module 32 is specifically configured to:

[0067] Acquire images and videos of the underground coal mine area, and set scene prompt words under different operating conditions of the underground coal mine;

[0068] Based on the scene prompt words, the image video is subjected to frame-by-frame cross-modal recognition to identify the features and motion trajectories of characteristic objects in the image video that affect the safety of coal mines underground, and the state and behavior of the characteristic objects are determined based on the features and motion trajectories.

[0069] Furthermore, in a possible implementation of the embodiment of the present invention, the third acquisition module 33 is specifically configured to:

[0070] Obtaining job type information and location data of personnel in the underground area of ​​the coal mine through the coal mine underground monitoring system;

[0071] The work trajectory of the personnel in the underground coal mine area is depicted through the personnel's job information and positioning data, wherein the work trajectory includes multiple underground coal mine areas passed through and the entry and exit time of each underground coal mine area.

[0072] Furthermore, in a possible implementation of the embodiment of the present invention, the apparatus further includes:

[0073] The fourth acquisition module is used to obtain time series data within a time threshold before and after the abnormal sensor alarm, and obtain the associated time series data of multiple associated sensors within the same time threshold according to the type of the abnormal sensor;

[0074] The fusion analysis module is used to perform data fusion analysis on the time series data and the associated time series data within the time threshold based on the business mechanism model corresponding to the abnormal sensor, and deduce the alarm cause of the abnormal sensor.

[0075] Furthermore, in a possible implementation of the embodiment of the present invention, the alarm and risk assessment module 34 is further specifically configured to:

[0076] The retrieval enhancement generation large model performs context analysis result retrieval and hybrid retrieval on the time series data, associated time series data, the state and behavior of feature objects, and work trajectory based on a preset coal mine risk hidden danger vector knowledge base to obtain initial retrieval results;

[0077] Through the multi-angle coal industry risk assessment model, the adjusted prompt word Prompt in the initial search results is used to conduct multi-angle alarm risk assessment, determine the risk level of the alarm cause, and generate response strategies for alarm causes of different risk levels;

[0078] The coal mine risk hidden danger vector knowledge base is constructed based on coal mine hidden danger accident cases and coal mine related laws and regulations data, coal mine industry standard data, and coal mine safety regulations data.

[0079] Furthermore, in a possible implementation of the embodiment of the present invention, the apparatus further includes:

[0080] The feedback optimization module is used to optimize the parameters of the multimodal large model for cross-modal recognition of the image and video, the retrieval enhancement generation large model, and the multi-angle coal industry risk assessment large model based on the alarm cause response strategy and remote alarm data.

[0081] The remote alarm risk assessment device based on multimodal data in underground coal mines according to the embodiment of the present invention, upon detecting abnormal sensor data in underground coal mines, obtains the time series data of the abnormal sensor in the underground coal mine area, the associated time series data of multiple associated sensors that affect the abnormal sensor data, the state and behavior of characteristic objects that affect underground coal mine safety in the image and video, and the work trajectory of personnel, based on the type of abnormal sensor; determines the cause of the alarm, and performs multi-angle alarm risk assessment by searching and enhancing the generation of a large model, determines the risk level of the alarm cause, and generates response strategies for alarm causes of different risk levels. Thus, by performing multimodal data fusion analysis on sensor data, image and video, and work trajectory in the underground coal mine area, intelligent risk assessment of coal mine safety alarms is achieved, the alarm risk assessment time is shortened, and the accuracy of alarm risk assessment is improved.

[0082] In order to implement the above embodiment, the present invention further provides an electronic device, including:

[0083] at least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0084] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the aforementioned method.

[0085] In order to implement the above embodiment, the present invention further proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the above method.

[0086] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0087] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0088] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0089] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" is any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (not exhaustive) of computer-readable media include: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0090] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any of the following technologies known in the art, or a combination thereof, may be used: a discrete logic circuit having logic gates for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gates, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0091] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0092] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0093] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A remote alarm risk assessment method based on multimodal data in coal mines, characterized in that: The method comprises: When abnormal sensor data is detected in the coal mine, time series data of the abnormal sensor in the coal mine area corresponding to the sensor data and / or associated time series data of multiple associated sensors in the coal mine area that affect the abnormal sensor data are obtained according to the type of the abnormal sensor; Acquiring images and videos within the underground coal mine area, and performing cross-modal recognition on the images and videos to identify states and behaviors of characteristic objects in the images and videos that affect safety in the underground coal mine; Obtaining a work trajectory composed of job type information and positioning data of personnel in the underground area of ​​the coal mine; According to the time series data, associated time series data, the state and behavior of the characteristic object, and the work trajectory, the alarm cause when the sensor data is abnormal is judged, and the time series data, associated time series data, the state and behavior of the characteristic object, and the work trajectory are subjected to multi-angle alarm risk analysis and judgment by the retrieval enhancement generation large model to judge the risk level of the alarm cause, and generate alarm cause response strategies of different risk levels, including: the retrieval enhancement generation large model performs context analysis result retrieval and hybrid retrieval on the time series data, associated time series data, the state and behavior of the characteristic object, and the work trajectory based on a preset coal mine risk hidden danger vector knowledge base to obtain an initial retrieval result, and the prompt word Prompt adjusted in the initial retrieval result is subjected to multi-angle alarm risk analysis and judgment by the set multi-angle coal industry risk analysis large model to judge the risk level of the alarm cause, and generate alarm cause response strategies of different risk levels, wherein the coal mine risk hidden danger vector knowledge base is constructed based on coal mine hidden danger accident cases and coal mine-related laws and regulations data, coal mine industry standard data, and coal mine safety regulations data; It also includes: obtaining time series data within a time threshold before and after the abnormal sensor alarm, and obtaining associated time series data of multiple associated sensors within the same time threshold according to the type of the abnormal sensor, performing data fusion analysis on the time series data within the time threshold and the associated time series data based on the business mechanism model corresponding to the abnormal sensor, and deducing the alarm cause of the abnormal sensor, wherein the business mechanism model corresponding to the abnormal sensor refers to how the abnormal sensor works in the business system and its association with the business logic.

2. The method according to claim 1, characterized in that The acquiring of images and videos in the underground coal mine area and performing cross-modal recognition on the images and videos to identify states and behaviors of characteristic objects in the images and videos that affect safety in the underground coal mine include: Acquire images and videos of the underground coal mine area, and set scene prompt words under different operating conditions of the underground coal mine; Based on the scene prompt words, the image video is subjected to frame-by-frame cross-modal recognition to identify the features and motion trajectories of characteristic objects in the image video that affect the safety of coal mines underground, and the state and behavior of the characteristic objects are determined based on the features and motion trajectories.

3. The method according to claim 1, characterized in that The obtaining of a work trajectory formed by the job type information and positioning data of personnel in the underground area of ​​the coal mine includes: Obtaining job type information and location data of personnel in the underground area of ​​the coal mine through the underground coal mine monitoring system; The work trajectory of the personnel in the underground coal mine area is depicted through the personnel's job information and positioning data, wherein the work trajectory includes multiple underground coal mine areas passed through and the entry and exit time of each underground coal mine area.

4. The method according to claim 1, wherein The method further comprises: According to the alarm cause response strategy and remote alarm data, the parameters of the multimodal large model for cross-modal recognition of the image and video, the retrieval enhancement generation large model, and the multi-angle coal industry risk assessment large model are optimized.

5. A remote alarm risk analysis device based on multimodal data in coal mines, characterized in that: The device comprises: A first acquisition module is configured to, when abnormal sensor data is detected in an underground coal mine, acquire, based on the type of the abnormal sensor, time series data of the abnormal sensor in the underground coal mine corresponding to the sensor data, and / or associated time series data of multiple associated sensors in the underground coal mine that affect the abnormal sensor data; A second acquisition module is configured to acquire images and videos within the underground coal mine area, and perform cross-modal recognition on the images and videos to identify states and behaviors of characteristic objects in the images and videos that affect safety in the underground coal mine; The third acquisition module is used to obtain the work trajectory of the personnel in the underground area of ​​the coal mine, which is composed of the work type information and positioning data; The alarm and risk assessment module is used to determine the alarm cause when the sensor data is abnormal based on the time series data, associated time series data, the state and behavior of the characteristic object, and the work trajectory, and to perform multi-angle alarm risk assessment on the time series data, associated time series data, the state and behavior of the characteristic object, and the work trajectory through retrieval enhancement and generation of a large model, determine the risk level of the alarm cause, and generate alarm cause response strategies of different risk levels. It is also specifically used to: retrieve and enhance the generation of a large model based on a preset coal mine risk hidden danger vector knowledge base to perform context analysis result retrieval and hybrid retrieval on the time series data, associated time series data, the state and behavior of the characteristic object, and the work trajectory to obtain an initial retrieval result, and perform multi-angle alarm risk assessment on the prompt word Prompt adjusted in the initial retrieval result through the set multi-angle coal industry risk assessment large model, determine the risk level of the alarm cause, and generate alarm cause response strategies of different risk levels, wherein the coal mine risk hidden danger vector knowledge base is constructed based on coal mine hidden danger accident cases and coal mine-related laws and regulations data, coal mine industry standard data, and coal mine safety regulations data; Also includes: The fourth acquisition module is used to obtain time series data within a time threshold before and after the abnormal sensor alarm, and obtain the associated time series data of multiple associated sensors within the same time threshold according to the type of the abnormal sensor; The fusion analysis module is used to perform data fusion analysis on the time series data and the associated time series data within the time threshold based on the business mechanism model corresponding to the abnormal sensor, and deduce the alarm cause of the abnormal sensor, wherein the business mechanism model corresponding to the abnormal sensor refers to how the abnormal sensor works in the business system and its association with the business logic.

6. The device according to claim 5, characterized in that The second acquisition module is specifically configured to: Acquire images and videos of the underground coal mine area, and set scene prompt words under different operating conditions of the underground coal mine; Based on the scene prompt words, the image video is subjected to frame-by-frame cross-modal recognition to identify the features and motion trajectories of characteristic objects in the image video that affect the safety of coal mines underground, and the state and behavior of the characteristic objects are determined based on the features and motion trajectories.

7. The device according to claim 5, characterized in that The third acquisition module is specifically configured to: Obtaining job type information and location data of personnel in the underground area of ​​the coal mine through the underground coal mine monitoring system; The work trajectory of the personnel in the underground coal mine area is depicted through the personnel's job information and positioning data, wherein the work trajectory includes multiple underground coal mine areas passed through and the entry and exit time of each underground coal mine area.

8. The device according to claim 5, characterized in that The device further comprises: The feedback optimization module is used to optimize the parameters of the multimodal large model for cross-modal recognition of the image and video, the retrieval enhancement generation large model, and the multi-angle coal industry risk assessment large model based on the alarm cause response strategy and remote alarm data.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-4.

Images