Underground equipment state detection method based on multi-modal data fusion

Abstract

The invention provides an underground equipment state detection method based on multi-modal data fusion, and relates to the technical field of coal mine underground equipment detection. The method comprises the following steps: acquiring start-stop state text information and monitoring video information of underground equipment; inputting the start-stop state text information into a pre-trained text and graph generation model to obtain a first image; acquiring video frame information in the monitoring video information; performing synthesis processing on the first image and the video frame information to generate an image set; and detecting the image set by adopting a visual detection algorithm to obtain a first start-stop state detection result of the underground equipment. The starting and stopping states of the underground equipment are comprehensively judged through various kinds of information, data sources are enriched, and the accuracy of equipment state detection is improved. And when the video information goes wrong, the start-stop state text information is used for data supplementation, so that the problem that the start-stop state of the equipment cannot be detected or cannot be detected accurately due to loss of the video information can be avoided to a certain extent, and the construction safety is ensured.

Description

Technical Field

[0001] The present disclosure relates to the technical field of underground equipment detection in coal mines, and in particular to an underground equipment status detection method based on multi-modal data fusion. Background Art

[0002] Most of the existing coal mines are monitored manually by dispatchers 24 hours a day through the main transport subsystem, comprehensive mining subsystem or management and control platform. The equipment status, such as start and stop, failure, etc., is judged by the eyes, and the real-time data monitoring of the mine equipment status is combined. This method causes great labor costs and it is difficult to ensure the accuracy of monitoring. Some coal mines have put video monitoring systems online, but due to poor algorithms, long system algorithm calculation cycles, and single data sources, the judgment accuracy is low, the misjudgment rate is high, and the monitoring results are delayed. In addition, due to the single data source, when there is a problem with the network, or the sensor for uploading production data fails, the camera freezes, etc., the system may make wrong judgments, affecting production, and the operators on the well may misoperate the equipment according to the wrong state, such as misstart and stop. More seriously, if there is maintenance by personnel, the sudden start of the equipment will cause casualties. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first embodiment of the present disclosure proposes a method for detecting downhole equipment status based on multimodal data fusion, comprising the following steps:

[0005] Obtain the start / stop status text information and monitoring video information of the underground equipment at the same time;

[0006] Inputting the start / stop status text information into a pre-trained text-image generation model to obtain a plurality of first images containing the downhole equipment; the text-image generation model has learned the ability to generate corresponding first images according to the start / stop status text information of the downhole equipment;

[0007] Extracting image frames from the monitoring video information to obtain multiple video frame information;

[0008] Performing synthesis processing on the multiple first images and the multiple video frame information to generate an image set;

[0009] A visual detection algorithm is used to perform status detection on the downhole equipment in the image set to obtain a first start / stop status detection result of the downhole equipment.

[0010] In some embodiments of the present disclosure, the start-stop status text information includes at least one of the following: the start-stop status text information of the downhole equipment detected during the inspection process, the start-stop status text information reported by the downhole equipment, and the start-stop status text information entered by the operator.

[0011] In some embodiments of the present disclosure, the text-image generation model is a CLIPBert model.

[0012] In some embodiments of the present disclosure, the method further includes: acquiring sensor data of the downhole equipment; determining a second start-stop state detection result of the downhole equipment according to the sensor data;

[0013] The start / stop state of the downhole equipment is determined by combining the first start / stop state detection result and the second start / stop state detection result.

[0014] The second embodiment of the present disclosure proposes a downhole equipment status detection device based on multimodal data fusion, comprising:

[0015] An acquisition module is used to obtain the start / stop status text information and monitoring video information of the underground equipment at the same time;

[0016] A generation module, used for inputting the start / stop status text information into a pre-trained text-image generation model to obtain a plurality of first images containing the downhole equipment; the text-image generation model has learned the ability to generate corresponding first images according to the start / stop status text information of the downhole equipment;

[0017] An extraction module is used to extract image frames from the monitoring video information to obtain multiple video frame information;

[0018] A synthesis module, used for synthesizing the plurality of first images and the plurality of video frame information to generate an image set;

[0019] The first detection module is used to perform status detection on the downhole equipment in the image set by using a visual detection algorithm to obtain a first start-stop status detection result of the downhole equipment.

[0020] In some embodiments of the present disclosure, the start-stop status text information includes at least one of the following: the start-stop status text information of the downhole equipment detected during the inspection process, the start-stop status text information reported by the downhole equipment, and the start-stop status text information entered by the operator.

[0021] In some embodiments of the present disclosure, the text-image generation model is a CLIPBert model.

[0022] In some embodiments of the present disclosure, the device also includes a second detection module; wherein the second detection module is used to: obtain sensor data of the downhole equipment; determine a second start-stop state detection result of the downhole equipment based on the sensor data; and determine the start-stop state of the downhole equipment by combining the first start-stop state detection result and the second start-stop state detection result.

[0023] A third aspect of the present disclosure provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0024] The memory stores computer-executable instructions;

[0025] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0026] The fourth aspect of the present disclosure provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0027] The underground equipment status detection method based on multimodal data fusion provided by the present disclosure comprehensively judges the start and stop status of the underground equipment based on the start and stop status text information of the underground equipment and the monitoring video information, enriches the data source, and improves the accuracy of equipment status detection. In addition, when there is a problem with the video information, the start and stop status text information is used to supplement the data, which can avoid the problem of the inability to detect or inaccurate detection of the start and stop status of the equipment due to the loss of video information to a certain extent. The present disclosure helps to reduce the loss of personnel and property at the coal mine site, reduce the possibility of production accidents, and ensure the safety of workers underground in the coal mine.

[0028] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A schematic diagram of a flow chart of a method for detecting downhole equipment status based on multimodal data fusion provided by an embodiment of the present disclosure;

[0031] Figure 2 A flow chart of another method for detecting downhole equipment status based on multimodal data fusion provided by an embodiment of the present disclosure;

[0032] Figure 3A schematic diagram of a downhole equipment status detection device based on multimodal data fusion provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0034] Specifically, the downhole equipment status detection method based on multimodal data fusion according to an embodiment of the present disclosure is described below with reference to the accompanying drawings. Figure 1 A flow chart of a method for detecting downhole equipment status based on multimodal data fusion provided in an embodiment of the present disclosure.

[0035] Step 101, obtaining the start / stop status text information and monitoring video information of the downhole equipment at the same time.

[0036] In some embodiments of the present disclosure, the start-stop status text information may be at least one of the following: the start-stop status text information of the downhole equipment detected during the inspection process, the start-stop status text information reported by the downhole equipment, and the start-stop status text information entered by the operator.

[0037] Among them, the text information on the start and stop status of the underground equipment detected during the inspection process refers to the start and stop status information determined when the inspection robot inspects along the specified route. The inspection detection result can be the start and stop text information directly generated or recorded, or it can also be a voice broadcast or a fixed prompt tone of the start and stop status. If the inspection detection result is a voice broadcast of the start and stop status issued by the inspection robot, the start and stop status information contained in the voice broadcast can be identified by a large language model to generate the start and stop status text information. If the inspection detection result is a fixed prompt tone, the start and stop status of the underground equipment can be determined according to the pre-established mapping relationship between the prompt tone and the start and stop status, and then the start and stop status text information is generated. The start and stop status text information reported by the underground equipment is a text description of the start and stop status generated by the underground equipment according to its own parameters, and is reported to the system or platform related to the corresponding equipment of the coal mine.

[0038] If the start-stop status text information includes multiple types of the start-stop status text information of the downhole equipment detected during the inspection, the start-stop status text information reported by the downhole equipment, and the start-stop status text information entered by the operator, the multiple types of text information are compared, and if the information is consistent, any type of text information can be used as the start-stop status text information. If the information is inconsistent, the start-stop status text information of the downhole equipment detected during the inspection is used as the start-stop status text information.

[0039] The monitoring video information is the downhole equipment monitoring video collected by a video acquisition device (such as a camera), and the video includes all downhole equipment or part of the downhole equipment.

[0040] Step 102: Input the start / stop status text information into a pre-trained text-image generation model to obtain a plurality of first images containing downhole equipment. The text-image generation model has learned the ability to generate corresponding first images according to the start / stop status text information of the downhole equipment.

[0041] It should be noted that the text-image generation model is pre-trained using historical images and historical start-stop status text information of downhole equipment, and can generate multiple first images of corresponding downhole equipment based on the start-stop status text information. Among them, the position of a certain marker in the multiple first images can reflect the start-stop status of the downhole equipment. As an example, assuming that the downhole equipment is a belt, and the start-stop status text information is the stop state, the displacement of a certain marker on the belt remains basically unchanged between the multiple first images generated based on the text information. Optionally, the text-image generation model can adopt the CLIPBERT model based on sparse sampling video and language learning.

[0042] Step 103: extract image frames from the monitoring video information to obtain multiple video frame information.

[0043] Step 104: synthesize the plurality of first images and the plurality of video frame information to generate an image set.

[0044] As an example, N video frame information is extracted from the monitoring video information, and the text-image generation model generates X consecutive first images based on the start-stop status text information. The N video frame information and the X consecutive first images are synthesized to generate an image set containing N+X images. Since the start-stop status text information and the monitoring video information are collected at the same time, there is no requirement for the order of the first image and video frame information in the image set. The first image and video frame information are synthesized to increase the data source for start-stop status recognition. The start-stop status of the downhole equipment can be jointly judged through multi-modal data, which can improve the detection accuracy.

[0045] In actual coal mine scenarios, there are situations such as poor camera signals, obstacles, unstable data, and failures. If the camera fails to collect monitoring, the monitoring video information is empty, N = 0, and the image set only includes X consecutive first images generated by the text-image generation model based on the start-stop status text information. Therefore, the monitoring video information is supplemented with the start-stop status text information to avoid the problem of the inability to detect the start-stop status of the equipment due to the loss of video information.

[0046] Step 105 , using a visual detection algorithm to perform status detection on the downhole equipment in the image set, and obtaining a first start / stop status detection result of the downhole equipment.

[0047] In some embodiments of the present disclosure, a visual detection algorithm may be used to detect the downhole equipment in the image set to determine the start and stop status of the downhole equipment. As a possible implementation method, the first start and stop status detection result of the downhole equipment may be obtained by using the LK pyramid optical flow algorithm combined with the wide flow time series feature analysis detection method. Alternatively, other visual detection algorithms may be used, which are not limited in the present disclosure.

[0048] In order to further improve the accuracy of downhole equipment status detection, in some embodiments of the present disclosure, in addition to the start / stop status text information and monitoring video information, sensor data sources may be added to jointly determine the start / stop status of the downhole equipment. Figure 2 FIG. 1 is a flow chart of another method for detecting downhole equipment status based on multimodal data fusion provided by an embodiment of the present disclosure. Figure 2 As shown, the start-stop status text information, monitoring video information and sensor data of the downhole equipment at the same time are obtained. Sensors such as laser radars, equipment real-time data sensors, etc. collect the real-time operating speed, start-stop status, etc. of the downhole equipment. The second start-stop status detection result of the downhole equipment is determined according to the sensor data. If the status data is 1, it means start, 0 means stop, the speed is greater than 0, it means start, and the speed is less than or equal to 0, it means stop. Combined with the first start-stop status detection result and the second start-stop status detection result, the start-stop status of the downhole equipment is determined. For example, the first start-stop status detection result and the second start-stop status detection result can be weighted and summed to determine the start-stop status of the downhole equipment. Alternatively, the first start-stop status detection result and the second start-stop status detection result can be compared. If the results are consistent, the detection result is output. If they are inconsistent, an error message is generated and fed back to the terminal to remind the staff to ensure the safety of the personnel in construction.

[0049] By implementing the embodiments of the present disclosure, the start / stop status of the underground equipment is comprehensively judged based on the start / stop status text information and monitoring video information of the underground equipment, enriching the data source and improving the accuracy of equipment status detection. In addition, when there is a problem with the video information, the start / stop status text information is used to supplement the data, which can avoid the problem of the equipment start / stop status being unable to be detected or inaccurately detected due to the loss of video information to a certain extent. The present disclosure helps to reduce the loss of personnel and property at the coal mine site, reduce the possibility of production accidents, and ensure the safety of workers underground in the coal mine.

[0050] Figure 3 Schematic diagram of a downhole equipment status detection device based on multimodal data fusion provided by an embodiment of the present disclosure. Figure 3 As shown, the downhole equipment status detection device based on multimodal data fusion includes: an acquisition module 301, a generation module 302, an extraction module 303, a synthesis module 304 and a first detection module 305.

[0051] The acquisition module 301 is used to acquire the start / stop status text information and monitoring video information of the downhole equipment at the same time.

[0052] In some embodiments of the present disclosure, the start-stop status text information includes at least one of the following: the start-stop status text information of the downhole equipment detected during the inspection process, the start-stop status text information reported by the downhole equipment, and the start-stop status text information entered by the operator.

[0053] The generation module 302 is used to input the start / stop status text information into a pre-trained text-image generation model to obtain a plurality of first images containing downhole equipment. The text-image generation model has learned the ability to generate corresponding first images according to the start / stop status text information of the downhole equipment.

[0054] In some embodiments of the present disclosure, the text-image generation model is a CLIPBert model.

[0055] The extraction module 303 is used to extract image frames from the monitoring video information to obtain multiple video frame information.

[0056] The synthesis module 304 is used to perform synthesis processing on the multiple first images and the multiple video frame information to generate an image set.

[0057] The first detection module 305 is used to perform status detection on the downhole equipment in the image set by using a visual detection algorithm to obtain a first start-stop status detection result of the downhole equipment.

[0058] In some embodiments of the present disclosure, Figure 3 On the basis of the above, the downhole equipment state detection device based on multimodal data fusion also includes a second detection module. The second detection module is used to: obtain sensor data of the downhole equipment; determine the second start-stop state detection result of the downhole equipment according to the sensor data; and determine the start-stop state of the downhole equipment by combining the first start-stop state detection result and the second start-stop state detection result.

[0059] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0060] In order to implement the above embodiments, the present disclosure also proposes an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0061] In order to implement the above embodiments, the present disclosure further proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0062] In order to implement the above embodiments, the present disclosure also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0063] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0064] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0065] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0066] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0067] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0068] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0069] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0070] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A method for detecting downhole equipment status based on multimodal data fusion, characterized in that: The following steps are involved: Obtain the start and stop status text information and monitoring video information of underground equipment at the same time; Inputting the start / stop status text information into a pre-trained text-image generation model to obtain a plurality of first images containing the downhole equipment; the text-image generation model has learned to generate corresponding first images based on the start / stop status text information of the downhole equipment; Extracting image frames from the surveillance video information to obtain multiple video frame information; Performing synthesis processing on the plurality of first images and the plurality of video frame information to generate an image set; A visual detection algorithm is used to perform status detection on the downhole equipment in the image set to obtain a first start-stop status detection result of the downhole equipment.

2. The method according to claim 1, characterized in that The start / stop status text information includes at least one of the following: the start / stop status text information of the downhole equipment detected during the inspection process, the start / stop status text information reported by the downhole equipment, and the start / stop status text information entered by the operator.

3. The method according to claim 1, characterized in that The text-image generation model is the CLIPBert model.

4. The method according to claim 1, wherein Also includes: Acquiring sensor data of the downhole equipment; determining a second start / stop state detection result of the downhole equipment according to the sensor data; The start / stop state of the downhole equipment is determined by combining the first start / stop state detection result and the second start / stop state detection result.

5. A downhole equipment status detection device based on multimodal data fusion, characterized in that: include: An acquisition module is used to obtain the start / stop status text information and monitoring video information of the downhole equipment at the same time; a generation module, configured to input the start / stop status text information into a pre-trained text-image generation model to obtain a plurality of first images containing the downhole equipment; the text-image generation model having learned the ability to generate corresponding first images based on the start / stop status text information of the downhole equipment; An extraction module is used to extract image frames from the monitoring video information to obtain multiple video frame information; a synthesis module, configured to perform synthesis processing on the plurality of first images and the plurality of video frame information to generate an image set; The first detection module is used to perform status detection on the downhole equipment in the image set using a visual detection algorithm to obtain a first start-stop status detection result of the downhole equipment.

6. The device according to claim 5, characterized in that The start / stop status text information includes at least one of the following: the start / stop status text information of the downhole equipment detected during the inspection process, the start / stop status text information reported by the downhole equipment, and the start / stop status text information entered by the operator.

7. The device according to claim 5, characterized in that The text-image generation model is the CLIPBert model.

8. The device according to claim 5, characterized in that Also includes a second detection module; wherein the second detection module is configured to: Acquiring sensor data of the downhole equipment; determining a second start / stop state detection result of the downhole equipment according to the sensor data; The start / stop state of the downhole equipment is determined by combining the first start / stop state detection result and the second start / stop state detection result.

9. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 4 when executed by a processor.

Images