Remote alarm risk studying and judging method and device based on underground coal mine multi-modal data

Abstract

The invention provides a remote alarm risk studying and judging method and device based on underground coal mine multi-mode data, and relates to the technical field of coal mine safety. Obtaining time sequence data of abnormal sensors in an underground coal mine area, associated time sequence data of a plurality of associated sensors influencing the data of the abnormal sensors, states and behaviors of feature objects influencing underground coal mine safety in image videos, and working tracks of personnel; and judging the alarm reason of the sensor data abnormity, and performing multi-angle alarm risk research and judgment through a retrieval enhancement generation large model to obtain the risk level of the alarm reason and the corresponding alarm reason coping strategy. Therefore, multi-modal data fusion analysis is performed through the sensor data, the image video and the working track in the underground coal mine area, intelligent risk research and judgment of coal mine safety alarm are realized, the alarm risk research and judgment time is shortened, and the alarm risk research and judgment precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of coal mine safety, and in particular, to a remote alarm risk judgment method and device based on multi-modal data in coal mines. Background Art

[0002] Currently, the work of coal mine safety alarm mainly relies on coal mine personnel to manually screen and query relevant data such as monitoring videos and measuring points by combining experience and the alarm information uploaded by the coal mine side for alarm risk judgment. However, due to the large number of mines and significant differences in production conditions, a large number of alarms need to be manually judged every day, and the alarm data and reasons are complex. The timeliness and accuracy of the judgment are insufficient, and the problems of alarm authenticity verification and comprehensive judgment of risk hazards cannot be systematically solved. Summary of the Invention

[0003] The present invention aims to at least solve one of the technical problems in the related art to some extent.

[0004] To this end, the first object of the present invention is to propose a remote alarm risk judgment method based on multi-modal data in coal mines. Through the fusion analysis of multi-modal data such as sensor data, image videos, and work trajectories in the coal mine underground area, intelligent risk judgment of coal mine safety alarms is realized, the alarm risk judgment time is shortened, and the alarm risk judgment accuracy is improved.

[0005] The second object of the present invention is to propose a remote alarm risk judgment device based on multi-modal data in coal mines.

[0006] The third object of the present invention is to propose an electronic device.

[0007] The fourth object of the present invention is to propose a non-transitory computer-readable storage medium storing computer instructions.

[0008] To achieve the above object, the first aspect embodiment of the present invention proposes a remote alarm risk judgment method based on multi-modal data in coal mines, and the method includes: When detecting abnormal sensor data in the coal mine underground, according to the type of the abnormal sensor, obtain the time series data of the abnormal sensor corresponding to the coal mine underground area where the sensor data is located, and / or the associated time series data of multiple associated sensors affecting the abnormal sensor data in the coal mine underground area; Obtain the image videos in the coal mine underground area, and perform cross-modal recognition on the image videos to identify the states and behaviors of the characteristic objects affecting coal mine underground safety in the image videos; Obtain the work trajectory formed by the job type information and positioning data of personnel in the coal mine underground area; Based on the timing data, associated timing data, the states and behaviors of characteristic objects, and the working trajectories, determine the alarm reasons when the sensor data is abnormal, and conduct multi-angle alarm risk research and judgment on the timing data, associated timing data, the states and behaviors of characteristic objects, and the working trajectories through a retrieval-augmented generation large model, determine the risk level of the alarm reasons, and generate response strategies for alarm reasons with different risk levels.

[0009] To achieve the above object, a second aspect embodiment of the present invention proposes a remote alarm risk research and judgment device based on multi-modal data in a coal mine. The device includes: A first acquisition module, configured to, when detecting abnormal sensor data in a coal mine, according to the type of the abnormal sensor, acquire the timing data of the abnormal sensor in the area of the coal mine corresponding to the sensor data, and / or the associated timing data of multiple associated sensors affecting the abnormal sensor data in the area of the coal mine; A second acquisition module, configured to acquire the image and video in the area of the coal mine, and perform cross-modal recognition on the image and video to identify the states and behaviors of characteristic objects affecting the safety of the coal mine in the image and video; A third acquisition module, configured to acquire the working trajectory formed by the work type information and positioning data of personnel in the area of the coal mine; An alarm and risk research and judgment module, configured to, based on the timing data, associated timing data, the states and behaviors of characteristic objects, and the working trajectories, determine the alarm reasons when the sensor data is abnormal, and conduct multi-angle alarm risk research and judgment on the timing data, associated timing data, the states and behaviors of characteristic objects, and the working trajectories through a retrieval-augmented generation large model, determine the risk level of the alarm reasons, and generate response strategies for alarm reasons with different risk levels.

[0010] The remote alarm risk research and judgment method, device, electronic device, and storage medium according to the embodiments of the present invention, when detecting abnormal sensor data in a coal mine, according to the type of the abnormal sensor, acquire the timing data of the abnormal sensor in the area of the coal mine, the associated timing data of multiple associated sensors affecting the abnormal sensor data, the states and behaviors of characteristic objects affecting the safety of the coal mine in the image and video, and the working trajectory of personnel; determine the alarm reasons, and conduct multi-angle alarm risk research and judgment through a retrieval-augmented generation large model to obtain the risk level of the alarm reasons and the corresponding response strategies for the alarm reasons. Thus, through multi-modal data fusion analysis of sensor data, image and video, and working trajectories in the area of the coal mine, intelligent risk research and judgment of coal mine safety alarms is realized, the alarm risk research and judgment time is shortened, and the alarm risk research and judgment accuracy is improved.

[0011] To achieve the above object, an embodiment of the third aspect of the present invention provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method described in the first aspect.

[0012] To achieve the above object, an embodiment of the fourth aspect of the present invention provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method described in the first aspect.

[0013] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the following description of the embodiments in conjunction with the accompanying drawings, where: Figure 1 FIG. is a schematic flowchart of a method for remotely alarming risk judgment based on multi-modal data in a coal mine underground provided by an embodiment of the present invention; Figure 2 FIG. is an application flowchart of a method for remotely alarming risk judgment based on multi-modal data in a coal mine underground provided by an embodiment of the present invention; Figure 3 FIG. is a schematic structural diagram of a device for remotely alarming risk judgment based on multi-modal data in a coal mine underground provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.

[0016] It should be noted that, in the technical solution of the present invention, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of relevant laws and regulations.

[0017] The method and device for remotely alarming risk judgment based on multi-modal data in a coal mine underground according to embodiments of the present invention will be described below with reference to the accompanying drawings.

[0018] Figure 1 FIG. is a schematic flowchart of a method for remotely alarming risk judgment based on multi-modal data in a coal mine underground provided by an embodiment of the present invention.

[0019] As shown Figure 1 below, the method includes the following steps: Step 101, when detecting abnormal sensor data in the coal mine underground, according to the type of the abnormal sensor, obtain the time-series data of the abnormal sensor in the area of the coal mine underground corresponding to the sensor data, and / or the associated time-series data of multiple associated sensors affecting the abnormal sensor data in the area of the coal mine underground.

[0020] In some possible implementation manners, the sensor data in the coal mine underground may include but is not limited to carbon monoxide and methane.

[0021] Specifically, taking the carbon monoxide sensor in the coal mine underground as an example, according to the actual experience and business logic of coal mine safety production, carbon monoxide alarm is often accompanied by the phenomena of rising temperature and increasing smoke concentration. Therefore, once the carbon monoxide sensor emits an abnormal signal, immediately collect the temperature and smoke time-series data of the temperature sensor and the smoke sensor in the same area as the carbon monoxide sensor, as the associated time-series data of multiple associated sensors affecting carbon monoxide in the area of the coal mine underground.

[0022] Taking the methane sensor in the coal mine underground as an example, the transmission and distribution of methane underground are closely related to the ventilation system, mining operations, etc. When the methane sensor at a certain position alarms, obtain the methane concentration data of the methane sensors at the upstream and downstream positions of the methane sensor as time-series data, and construct a time-series change curve; when multiple associated sensors affecting the methane sensor data in the area of the coal mine underground are the wind direction and air volume sensors of the ventilation system, the associated time-series data is the wind direction and air volume time-series data of the wind direction and air volume sensors in the same area as the methane sensor, realizing the joint analysis of multiple sensors at different locations and accurately positioning the source of the problem.

[0023] Thus, through the time-series analysis of sensor data, the fusion analysis of multi-type and multi-location sensor data and the business mechanism model are encapsulated into a lightweight algorithm module, and the algorithm module has high scalability and flexibility.

[0024] Step 102, obtain the image and video in the area of the coal mine underground, and perform cross-modal recognition on the image and video to identify the states and behaviors of the characteristic objects affecting the safety of the coal mine underground.

[0025] In some possible implementation manners, image and video in the underground coal mine area are acquired, and cross-modal recognition is performed on the image and video to identify the states and behaviors of characteristic objects affecting the safety in the underground coal mine in the image and video, including: acquiring the image and video in the underground coal mine area, and setting scene prompt words under different operating conditions in the underground coal mine; performing frame-by-frame cross-modal recognition on the image and video based on the scene prompt words to identify the characteristics and movement trajectories of the characteristic objects affecting the safety in the underground coal mine in the image and video, and judging the states and behaviors of the characteristic objects according to the characteristics and movement trajectories.

[0026] Specifically, in order to make the analysis process of the large model more logical and interpretable, the technique of chain of thought is introduced to decompose complex problems into a series of logical steps to construct a multi-modal large model. In the analysis of image and video by the multi-modal large model, first, frame-by-frame scanning of the image and video frames is performed based on the prompt words to identify the characteristic objects in the image and video, such as equipment, personnel, smoke, etc. Then, according to the characteristics and movement trajectories of the characteristic objects, their states and behaviors are judged. For example, it is judged whether the equipment is in a normal operating state and whether the personnel have illegal operations, etc. With the multi-modal large model technology, through the optimized prompt words and the chain of thought technology, in-depth analysis of the image and video in the underground coal mine area is carried out to achieve accurate recognition and judgment of various scenarios, providing key support for alarm risk judgment.

[0027] Step 103, acquire the work trajectories formed by the job types and positioning data of personnel in the underground coal mine area.

[0028] In some possible implementation manners, acquiring the work trajectories formed by the job types and positioning data of personnel in the underground coal mine area includes: acquiring the job types and positioning data of personnel in the underground coal mine area through the underground coal mine monitoring system; depicting the work trajectories of personnel in the underground coal mine area based on the job types and positioning data of the personnel, where the work trajectories include multiple underground coal mine areas passed through and the access times of each underground coal mine area.

[0029] Optionally, for alarms caused by underground work, such as the alarm risk of excessive methane concentration caused by forklift operation and the over-limit alarm caused by misoperation of methane calibration, it is necessary to verify the authenticity of the alarm cause based on the upstream and downstream methane concentration, wind direction, air volume time series data, methane calibration operation, and methane concentration data, and then combine the methane reception action trajectory (alarm time and location).

[0030] Step 104: Based on the timing data, associated timing data, the states and behaviors of characteristic objects, and the working trajectories, determine the alarm reasons when sensor data is abnormal, and conduct multi-angle alarm risk research and judgment on the timing data, associated timing data, the states and behaviors of characteristic objects, and the working trajectories through a retrieval-augmented generation large model, determine the risk levels of the alarm reasons, and generate response strategies for alarm reasons with different risk levels.

[0031] In some possible implementation manners, it further includes: obtaining the timing data within a period threshold before and after the abnormal sensor alarm, and obtaining the associated timing data of multiple associated sensors within the same time threshold according to the type of the abnormal sensor; the timing data and the associated timing data within the time threshold are subjected to data fusion analysis based on the service mechanism model corresponding to the abnormal sensor to deduce the alarm reasons of the abnormal sensor.

[0032] Among them, the service mechanism model corresponding to the abnormal sensor refers to how the abnormal sensor works in the service system and its association with the service logic.

[0033] Optionally, in the case of abnormal carbon monoxide sensor data, the alarm reasons include potential fire hazards in the mine, abnormal carbon monoxide sensors, and vehicle exhaust emissions in the same area as the carbon monoxide sensor; in the case of abnormal methane sensor data, the alarm reasons include the existence of a gas leakage source, abnormal coal mine ventilation system, and incorrect calibration operation of the methane sensor.

[0034] Optionally, when the timing data is carbon monoxide, the associated timing data is the temperature and smoke concentration timing data of the temperature sensor and the smoke sensor in the same area as the carbon monoxide sensor, the states and behaviors of the characteristic objects are the vehicle paths and vehicle exhaust emissions in the same area as the carbon monoxide sensor, and the working trajectory is the action trajectory of the workers of the vehicle type in the same area as the carbon monoxide sensor; using the correlation analysis algorithm, calculate the correlation coefficient between the concentration of carbon monoxide and the temperature and smoke concentration timing data, and when the correlation coefficient exceeds the set threshold, determine that the alarm reason is caused by potential fire hazards in the mine; when the correlation coefficient is less than the set threshold and there is no abnormal fluctuation in the temperature and smoke concentration timing data of the smoke sensor, the alarm reason may be an abnormal carbon monoxide sensor or vehicle exhaust emissions in the same area as the carbon monoxide sensor. For example, if there is a vehicle path in the same area as the carbon monoxide sensor and it coincides with the corresponding working trajectory of the workers of the vehicle type, the alarm reason is that the vehicle exhaust emissions cause the abnormal carbon monoxide sensor data.

[0035] Further, when the time-series data is methane, the associated time-series data is the wind direction and air volume time-series data of the wind direction and air volume sensors in the coal mine ventilation system in the same area as the methane sensor. The states and behaviors of the characteristic objects are the methane calibration operations in the same area as the methane sensor and the methane concentration data recorded under the methane calibration operations. The methane calibration operations correspond to the methane receiving action trajectories of the workers in the same area as the methane sensor; based on the methane concentration, wind direction, air volume time-series data, methane calibration operations, methane concentration data, and methane receiving action trajectories corresponding to the upstream and downstream of the methane sensor, determine the alarm cause when the methane sensor data is abnormal.

[0036] Specifically, if the methane concentration in the upstream is normal, the methane concentration in the downstream suddenly increases, and there are no obvious changes in the wind direction and air volume time-series data, the preliminary judgment of the alarm cause is that there is a gas leakage source near the abnormal location (methane sensor); if the methane concentrations in both the upstream and downstream increase abnormally, and the air volume in the wind direction and air volume time-series data decreases, the alarm cause may be a malfunction in the coal mine ventilation system. Different countermeasures are taken according to the severity of the alarm cause.

[0037] Among them, based on the standard process and logic of the methane calibration operation, a pattern matching model is constructed. During the methane calibration operation, the methane sensor receives the standard gas in a specific time sequence and operation steps, and records the corresponding changes in methane concentration data. When a methane alarm occurs, compare the sensor data before and after the current alarm time with the standard data pattern of the methane calibration operation. If the data change pattern is highly similar to the standard data pattern, it can be preliminarily judged that the cause of the methane alarm may be due to incorrect calibration operations.

[0038] In some possible implementation manners, a retrieval-enhanced generation large model is used to conduct multi-angle alarm risk research and judgment on the time-series data, associated time-series data, states and behaviors of characteristic objects, and work trajectories, judge the risk level of the alarm cause, and generate alarm cause countermeasures for different risk levels, including: the retrieval-enhanced generation large model conducts context analysis result retrieval and hybrid retrieval on the time-series data, associated time-series data, states and behaviors of characteristic objects, and work trajectories based on a preset coal mine risk hidden danger vector knowledge base to obtain an initial retrieval result; a multi-angle coal industry risk research and judgment large model conducts multi-angle alarm risk research and judgment on the adjusted prompt words Prompt in the initial retrieval result, judges the risk level of the alarm cause, and generates alarm cause countermeasures for different risk levels; among them, the coal mine risk hidden danger vector knowledge base is constructed based on coal mine hidden danger accident cases, coal mine-related laws and regulations data, coal mine industry standard data, and coal mine safety regulations data.

[0039] Among them, context analysis result retrieval and hybrid retrieval can improve the accuracy and recall rate of the retrieval results.

[0040] In some other possible embodiments, according to the alarm cause response strategy and the data of remote alarms, the parameters of the multi-modal large model for cross-modal recognition of image and video, the retrieval enhanced generation large model, and the multi-angle coal industry risk judgment large model are optimized. Specifically, by setting up a link tracking module, the alarm cause response strategy and the data of remote alarms in each link during the judgment process are collected, and they are displayed to professional staff through a visualization platform. Then, according to the feedback from the staff, the parameters of the multi-modal large model for cross-modal recognition of image and video, the retrieval enhanced generation large model, and the multi-angle coal industry risk judgment large model are optimized, realizing a virtuous cycle of continuous feedback and iteration of the judgment ability.

[0041] In the remote alarm risk judgment method based on multi-modal data in the embodiment of the present invention, when abnormal sensor data in the coal mine underground is detected, according to the type of the abnormal sensor, the time-series data of the abnormal sensor in the coal mine underground area, the associated time-series data of multiple associated sensors affecting the abnormal sensor data, the states and behaviors of the characteristic objects affecting the safety in the coal mine underground in the image and video, and the work trajectories of the personnel are obtained; the alarm cause is judged, and multi-angle alarm risk judgment is carried out through the retrieval enhanced generation large model, the risk level of the alarm cause is judged, and alarm cause response strategies with different risk levels are generated. Thus, through multi-modal data fusion analysis of the sensor data, image and video, and work trajectories in the coal mine underground area, intelligent risk judgment of coal mine safety alarms is realized, the alarm risk judgment time is shortened, and the alarm risk judgment accuracy is improved.

[0042] The present invention also proposes an application flowchart of a remote alarm risk judgment method based on multi-modal data in the coal mine underground, as Figure 2 shown. This application process includes on-site data access, alarm information preprocessing, intelligent judgment, and user feedback management. Specifically: On-site data access: Connect to the coal mine database (safety monitoring system, personnel positioning system, industrial video monitoring system), collect real-time coal mine alarm data (time-series data and associated time-series data of abnormal sensor data), the type information and positioning data of the personnel in the coal mine underground area, and the image and video in the coal mine underground area, and open data permissions to provide a data basis for alarm risk judgment.

[0043] Alarm Information Preprocessing: After the user inputs the alarm information, based on the large model of the coal industry (introducing the chain of thought based on the coal mine alarm report to generate the specific judgment task process), identify the judgment scenario to which the alarm information (the time series data and associated time series data when the sensor data is abnormal) belongs. At the same time, automatically identify the key data from the alarm information reported by the mine (such as the work trajectories of underground personnel in the mine and the states and behaviors of characteristic objects in the underground area of the coal mine), providing an important basis for subsequent alarm risk judgment. For judgment scenarios that require other information in addition to the alarm, relevant data is dynamically queried and complemented through the natural language query (text2sql) function. Among them, identifying the judgment scenario is achieved by sorting out the judgment scenarios of different types of alarms and analyzing the alarm information to match the corresponding scenario; identifying the key data is to extract the work trajectories of underground personnel in the mine and the states and behaviors of characteristic objects in the underground area of the coal mine from the alarm information, that is, key data such as time, location, and personnel; querying and complementing the relevant data is to automatically query and supplement the additional necessary information required for judgment from the coal mine database based on the text2sql technology.

[0044] Intelligent Judgment: Distribute judgment tasks (image and video recognition, recognition and analysis of time series data and associated time series data, personnel trajectory verification, etc.) according to the necessary judgment information (time series data and associated time series data, work type information and positioning data of personnel, images and videos) identified by the alarm information preprocessing, and conduct comprehensive risk judgment in combination with the coal mine risk and hidden danger vector knowledge base to obtain the comprehensive judgment result (judging the alarm reason when the sensor data is abnormal), and at the same time retain the intermediate comprehensive judgment result. Then upload the comprehensive judgment result to the visualization platform for users to review and confirm.

[0045] User Feedback Management: Enable users to review (user review behavior) and intervene in the processing status (rejected, unconfirmed / unhandled, confirmed / unhandled, unconfirmed / handled, confirmed / handled) of the comprehensive risk judgment process (user review and confirmation, user review and rejection, generating the risk level of the alarm reason and the corresponding countermeasures and feedback opinions for the alarm reason) through the provided visualization management platform. Among them, after generating the risk level of the alarm reason and the corresponding countermeasures and feedback opinions for the alarm reason, manually file / archive by date (supporting operations such as viewing / exporting the archived data) the risk level of the alarm reason and the corresponding countermeasures and feedback opinions for the alarm reason, and record and save them in the judgment record library.

[0046] To implement the above embodiments, the present invention also proposes a remote alarm risk judgment device based on multi-modal data in a coal mine underground.

[0047] Figure 3 Schematic diagram of the structure of a remote alarm risk judgment device based on multi-modal data in a coal mine underground provided by the embodiment of the present invention.

[0048] As Figure 3 shown, the remote alarm risk judgment device 30 based on multi-modal data in coal mines includes: a first acquisition module 31, a second acquisition module 32, a third acquisition module 33, and an alarm and risk judgment module 34.

[0049] The first acquisition module 31 is configured to, when detecting abnormal sensor data in a coal mine, obtain the time-series data of abnormal sensors in the area of the coal mine corresponding to the sensor data according to the type of the abnormal sensor, and / or the associated time-series data of multiple associated sensors affecting the abnormal sensor data in the area of the coal mine; The second acquisition module 32 is configured to acquire the image and video in the area of the coal mine, perform cross-modal recognition on the image and video to identify the states and behaviors of the characteristic objects affecting the safety of the coal mine in the image and video; The third acquisition module 33 is configured to acquire the work trajectories formed by the work type information and positioning data of personnel in the area of the coal mine; The alarm and risk judgment module 34 is configured to, according to the time-series data, the associated time-series data, the states and behaviors of the characteristic objects, and the work trajectories, determine the alarm reason when the sensor data is abnormal, and perform multi-angle alarm risk judgment on the time-series data, the associated time-series data, the states and behaviors of the characteristic objects, and the work trajectories through a retrieval-enhanced generation large model, determine the risk level of the alarm reason, and generate response strategies for alarm reasons with different risk levels.

[0050] Further, in a possible implementation manner of the embodiment of the present invention, the second acquisition module 32 is specifically configured to: Acquire the image and video in the area of the coal mine and set scene prompt words under different operating conditions in the coal mine; Perform frame-by-frame cross-modal recognition on the image and video based on the scene prompt words to identify the characteristics and movement trajectories of the characteristic objects affecting the safety of the coal mine in the image and video, and determine the states and behaviors of the characteristic objects according to the characteristics and movement trajectories.

[0051] Further, in a possible implementation manner of the embodiment of the present invention, the third acquisition module 33 is specifically configured to: Acquire the work type information and positioning data of personnel in the area of the coal mine through the coal mine monitoring system; Depict the work trajectories of personnel in the area of the coal mine through the work type information and positioning data of the personnel, where the work trajectories include multiple areas of the coal mine passed through and the entry and exit times of each area of the coal mine.

[0052] Further, in a possible implementation manner of the embodiment of the present invention, the device further includes: A fourth acquisition module, configured to acquire the time-series data within a period threshold before and after the abnormal sensor alarm, and acquire the associated time-series data of multiple associated sensors within the same time threshold according to the type of the abnormal sensor; A fusion analysis module, configured to perform data fusion analysis on the time-series data and the associated time-series data within the time threshold based on the service mechanism model corresponding to the abnormal sensor, and deduce the alarm reason of the abnormal sensor.

[0053] Further, in a possible implementation manner of the embodiment of the present invention, the alarm and risk judgment module 34 is further specifically configured to: The retrieval enhanced generation large model performs context analysis result retrieval and hybrid retrieval on the time-series data, the associated time-series data, the states and behaviors of the feature objects, and the work trajectories based on a preset coal mine risk and hidden danger vector knowledge base to obtain an initial retrieval result; Perform multi-angle alarm risk judgment on the adjusted prompt words Prompt in the initial retrieval result through a set multi-angle coal industry risk judgment large model, judge the risk level of the alarm reason, and generate response strategies for alarm reasons with different risk levels; Wherein, the coal mine risk and hidden danger vector knowledge base is constructed based on coal mine hidden danger accident cases, coal mine-related laws and regulations data, coal mine industry standard data, and coal mine safety regulation data.

[0054] Further, in a possible implementation manner of the embodiment of the present invention, the device further includes: A feedback optimization module, configured to optimize the parameters of the multi-modal large model for cross-modal recognition of the image and video, the retrieval enhanced generation large model, and the multi-angle coal industry risk judgment large model according to the alarm reason response strategy and the remote alarm data.

[0055] The remote alarm risk judgment device based on multi-modal data in the coal mine underground in the embodiment of the present invention, when detecting abnormal sensor data in the coal mine underground, according to the type of the abnormal sensor, acquires the time-series data of the abnormal sensor in the coal mine underground area, the associated time-series data of multiple associated sensors affecting the abnormal sensor data, the states and behaviors of the feature objects affecting the coal mine underground safety in the image and video, and the work trajectories of the personnel; judges the alarm reason, and performs multi-angle alarm risk judgment through the retrieval enhanced generation large model, judges the risk level of the alarm reason, and generates response strategies for alarm reasons with different risk levels. Thus, through multi-modal data fusion analysis of the sensor data, image and video, and work trajectories in the coal mine underground area, intelligent risk judgment of coal mine safety alarms is realized, the alarm risk judgment time is shortened, and the alarm risk judgment accuracy is improved.

[0056] To implement the above embodiments, the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the foregoing method.

[0057] To implement the above embodiments, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the foregoing method.

[0058] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without conflict, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0059] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0060] Any process or method description in a flowchart or described in other ways herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of the present invention includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in an order opposite to that shown or discussed, according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0061] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definable list of executable instructions for implementing logical functions, and can be embodied specifically in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in conjunction with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then storing it in a computer memory.

[0062] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0063] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the methods of the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0064] In addition, each functional unit in various embodiments of the present invention may be integrated into one processing module, may exist separately as individual physical units, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0065] The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disc, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A remote alarm risk judgment method based on multi-modal data in coal mines, characterized in that The method comprises: When abnormal sensor data is detected in the coal mine, according to the type of abnormal sensor, the time series data of the abnormal sensor in the coal mine area corresponding to the sensor data and / or the associated time series data of multiple associated sensors in the coal mine area that affect the abnormal sensor data are obtained; Acquire an image video in the underground area of ​​the coal mine, and perform cross-modal recognition on the image video to identify the state and behavior of characteristic objects in the image video that affect the safety of the underground coal mine; Obtaining a work trajectory consisting of job type information and positioning data of personnel in the underground area of ​​the coal mine; Based on the time series data, related time series data, the state and behavior of characteristic objects, and the working trajectory, the alarm cause when the sensor data is abnormal is determined, and a large model is generated through retrieval enhancement to conduct multi-angle alarm risk analysis on the time series data, related time series data, the state and behavior of characteristic objects, and the working trajectory, determine the risk level of the alarm cause, and generate alarm cause response strategies of different risk levels.

2. The method according to claim 1, wherein The step of acquiring the image video in the underground area of ​​the coal mine and performing cross-modal recognition on the image video to identify the state and behavior of characteristic objects in the image video that affect the safety of the underground coal mine includes: Acquire image videos in the underground area of ​​the coal mine, and set scene prompt words under different operating conditions in the underground coal mine; Based on the scene prompt words, the image video is recognized frame by frame across modalities to identify the features and motion trajectories of feature objects in the image video that affect the safety of underground coal mines, and the state and behavior of the feature objects are determined based on the features and motion trajectories.

3. The method according to claim 1, wherein The obtaining of the work trajectory formed by the work type information and positioning data of the personnel in the underground area of ​​the coal mine includes: Obtaining the type of work information and location data of personnel in the underground area of ​​the coal mine through the underground coal mine monitoring system; The work trajectory of the personnel in the underground area of ​​the coal mine is depicted through the type of work information and positioning data of the personnel, wherein the work trajectory includes multiple underground areas of the coal mine passed through and the entry and exit time of each underground area of ​​the coal mine.

4. The method according to claim 1, wherein The method further comprises: Obtain the time series data within a time threshold before and after the abnormal sensor alarm, and obtain the associated time series data of multiple associated sensors within the same time threshold according to the type of abnormal sensor; The time series data within the time threshold and the associated time series data are subjected to data fusion analysis based on the business mechanism model corresponding to the abnormal sensor to deduce the alarm cause of the abnormal sensor.

5. The method according to claim 1, wherein The large model generated by retrieval enhancement performs multi-angle alarm risk analysis on the time series data, associated time series data, the state and behavior of characteristic objects, and the work trajectory, determines the risk level of the alarm cause, and generates response strategies for alarm causes of different risk levels, including: The retrieval enhancement generation large model performs context analysis result retrieval and hybrid retrieval on the time series data, associated time series data, the state and behavior of feature objects, and work trajectory based on the preset coal mine risk hidden danger vector knowledge base to obtain initial retrieval results; Using the set multi - angle risk judgment large model for the coal industry to conduct multi - angle alarm risk judgment on the adjusted prompt words in the initial retrieval results, determine the risk level of the alarm reason, and generate coping strategies for alarm reasons with different risk levels; Among them, the coal mine risk and hidden danger vector knowledge base is constructed based on coal mine hidden danger accident cases, coal mine - related laws and regulations data, coal mine industry standard data, and coal mine safety regulation data.

6. The method according to claim 5, wherein The method further includes: According to the coping strategies for alarm reasons and the data of remote alarms, optimize the parameters of the multi - modal large model for cross - modal recognition of the image and video, the retrieval - enhanced generation large model, and the multi - angle risk judgment large model for the coal industry.

7. A remote alarm risk judgment device based on multi-modal data in underground coal mines, characterized in that, The device includes: The first acquisition module is used to, when detecting abnormal data of coal mine underground sensors, obtain the time - series data of the abnormal sensors in the coal mine underground area corresponding to the sensor data according to the type of the abnormal sensors, and / or the associated time - series data of multiple associated sensors affecting the abnormal sensor data in the coal mine underground area; The second acquisition module is used to acquire the image and video in the coal mine underground area, conduct cross - modal recognition on the image and video to identify the states and behaviors of the characteristic objects affecting the safety of the coal mine underground in the image and video; The third acquisition module is used to acquire the work trajectories formed by the job types and positioning data of personnel in the coal mine underground area; The alarm and risk judgment module is used to, according to the time - series data, associated time - series data, states and behaviors of characteristic objects, and work trajectories, determine the alarm reason when the sensor data is abnormal, and conduct multi - angle alarm risk judgment on the time - series data, associated time - series data, states and behaviors of characteristic objects, and work trajectories through the retrieval - enhanced generation large model, determine the risk level of the alarm reason, and generate coping strategies for alarm reasons with different risk levels.

8. The device according to claim 7, characterized in that, The second acquisition module is specifically used for: Acquire the image and video in the coal mine underground area and set scene prompt words under different operating conditions in the coal mine underground; Based on the scene prompt words, conduct frame - by - frame cross - modal recognition on the image and video to identify the characteristics and movement trajectories of the characteristic objects affecting the safety of the coal mine underground in the image and video, and determine the states and behaviors of the characteristic objects according to the characteristics and movement trajectories.

9. The device according to claim 7, characterized in that, The third acquisition module is specifically used for: Obtain the job types and positioning data of personnel in the coal mine underground area through the coal mine underground monitoring system; Depict the work trajectories of personnel in the coal mine underground area through the job types and positioning data of the personnel, where the work trajectories include multiple coal mine underground areas passed through and the entry and exit times of each coal mine underground area.

10. The device according to claim 7, characterized in that, The device further includes: The fourth acquisition module is used to acquire the time - series data within a certain time threshold before and after the abnormal sensor alarm, and according to the type of the abnormal sensor, acquire the associated time - series data of multiple associated sensors within the same time threshold; The fusion analysis module is used to conduct data fusion analysis on the time - series data and associated time - series data within the time threshold based on the business mechanism model corresponding to the abnormal sensor, and deduce the alarm reason of the abnormal sensor.

11. The device according to claim 7, characterized in that, The alarm and risk judgment module is further specifically configured to: The retrieval-augmented generation large model performs context analysis result retrieval and hybrid retrieval on the time-series data, associated time-series data, states and behaviors of feature objects, and work trajectories based on a preset coal mine risk and hidden danger vector knowledge base to obtain an initial retrieval result; Through a set multi-angle coal industry risk judgment large model, it conducts multi-angle alarm risk judgment on the adjusted prompt words (Prompts) in the initial retrieval result, determines the risk level of the alarm cause, and generates response strategies for alarm causes with different risk levels; Among them, the coal mine risk and hidden danger vector knowledge base is constructed based on coal mine hidden danger accident cases, coal mine-related laws and regulations data, coal mine industry standard data, and coal mine safety regulation data.

12. The device according to claim 11, wherein The device further includes: A feedback optimization module, configured to optimize the parameters of the multi-modal large model for cross-modal recognition of the image and video, the retrieval-augmented generation large model, and the multi-angle coal industry risk judgment large model according to the alarm cause response strategy and the data of remote alarms.

13. An electronic device, characterized in that, Including: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

Images