Intelligent underground operation state monitoring system and method

Abstract

The embodiment of the invention discloses an intelligent downhole operation state monitoring system and method, and relates to the technical field of monitoring data processing. The method comprises the following steps: acquiring multi-modal monitoring data for reflecting an underground operation state; performing fluctuation analysis on each piece of time sequence data in the multi-modal monitoring data to obtain a corresponding fluctuation analysis result; the correlation degree between the fluctuation analysis results corresponding to the different time series data is calculated based on a grey correlation analysis method, and fluctuation correlation information is obtained; performing information fusion on the fluctuation analysis result and corresponding time sequence data through a time window alignment mode to obtain target monitoring data; taking the fluctuation associated information as a weight parameter of an attention mechanism, performing multi-modal feature coding on the target monitoring data through a coding network, and outputting a corresponding operation state representation vector; and inputting the operation state representation vector into a decoding network to carry out downhole operation state anomaly analysis to obtain intelligent downhole operation state monitoring data.

Description

Technical Field

[0001] The present application relates to the technical field of monitoring data processing, and in particular to an intelligent downhole operation status monitoring system and method. Background Art

[0002] With the increasing complexity of mining and underground engineering operations, real-time monitoring of underground operation status has become a core link to ensure safe production. Traditional monitoring systems mainly rely on single-modal data (such as video monitoring, environmental sensor data, etc.) to judge local operation status. However, the underground operation environment has the characteristics of strong dynamics, multiple interference sources, and limited space (limited space may cause more blind spots in video monitoring). A single data source is susceptible to noise interference, and it is difficult to fully reflect the overall picture of the operation status, and it is impossible to capture the coordinated abnormal state in complex situations.

[0003] For example, when the movement trajectory of underground workers is completely synchronized with the movement trajectory of underground coal mine transportation equipment, it means that the workers may have taken the coal mine transportation equipment (which is not allowed according to regulations), and if this behavior is monitored based on single-mode data (such as surveillance video), it is difficult to detect abnormalities. For another example, during coal mining, coal seam collapse will lead to a large amount of gas release. Therefore, excessive gas concentration underground is usually accompanied by various phenomena such as flying coal dust and abnormal noise. These phenomena may appear as isolated events in single-mode data, and it is difficult to accurately judge the real state reflected based on single-mode data.

[0004] Based on this, it is necessary to study an intelligent downhole operation status monitoring method based on correlation analysis of multimodal monitoring data to capture more subtle abnormal operation conditions and improve the monitoring accuracy and safety management level of downhole operation conditions. Summary of the invention

[0005] To solve the above problems, one aspect of an embodiment of this specification provides an intelligent downhole operation status monitoring method, the method comprising:

[0006] Acquire multimodal monitoring data for reflecting the downhole operation status, wherein the multimodal monitoring data includes time series data respectively obtained based on video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data;

[0007] Performing time-frequency domain fluctuation analysis on each time series data in the multimodal monitoring data to obtain fluctuation analysis results corresponding to each time series data;

[0008] Based on the grey correlation analysis method, the correlation between the fluctuation analysis results corresponding to different time series data is calculated to obtain the fluctuation correlation information including the correlation weight matrix;

[0009] The fluctuation analysis result is fused with the corresponding time-series data in the multi-modal monitoring data through time-window alignment to obtain target monitoring data;

[0010] Taking the fluctuation correlation information as the weight parameter of the attention mechanism, the target monitoring data is encoded with multi-modal features through the encoding network of a pre-trained machine learning model, and a job status representation vector corresponding to the multi-modal monitoring data is output, where the job status representation vector at least includes index data corresponding to the equipment operation status, environmental safety status, and personnel status;

[0011] The job status representation vector is input into the decoding network of the pre-trained machine learning model for abnormal analysis of the underground operation status, and intelligent underground operation status monitoring data is obtained.

[0012] In some embodiments, the obtaining of the multi-modal monitoring data for reflecting the underground operation status includes:

[0013] The video monitoring data is collected through a camera array deployed in the underground roadway, and a multi-object tracking algorithm based on deep learning is used to extract the attention information from the video monitoring data to obtain the first time-series data corresponding to the video monitoring data, where the attention information at least includes personnel operation specification information, equipment movement trajectory information, and material stacking status information;

[0014] The sound monitoring data is collected through a distributed acoustic sensor network, and Mel-frequency cepstral coefficients are combined with a convolutional recurrent neural network for voiceprint feature extraction to obtain the second time-series data corresponding to the sound monitoring data.

[0015] In some embodiments, the convolutional recurrent neural network includes a time-frequency map convolutional layer and a bidirectional LSTM layer. The using of Mel-frequency cepstral coefficients combined with a convolutional recurrent neural network for voiceprint feature extraction to obtain the second time-series data corresponding to the sound monitoring data includes:

[0016] Constructing an acoustic feature template that at least includes an abnormal sound library, a personnel voice library, and an environmental noise library;

[0017] Extracting the short-time spectrum features from the sound monitoring data through the time-frequency map convolutional layer;

[0018] Using the bidirectional LSTM layer to obtain the long-time sequence dependence features in the sound monitoring data;

[0019] Based on the acoustic feature template, voiceprint matching is performed on the extracted short-time spectrum features and long-time sequence dependence features to identify the event information contained in the sound monitoring data;

[0020] Obtain the second time-series data corresponding to the sound monitoring data according to the event type, occurrence time, and continuous state corresponding to the event information.

[0021] In some embodiments, the time-frequency domain fluctuation analysis of each time-series data in the multimodal monitoring data includes:

[0022] Use the sliding time window algorithm to determine the fluctuation feature vector corresponding to each time-series data in the multimodal monitoring data within a time window of a preset time step, where the fluctuation features at least include the fluctuation amplitude, fluctuation frequency, and phase offset corresponding to each time-series data.

[0023] In some embodiments, the calculation of the correlation degree between the fluctuation analysis results corresponding to different time-series data based on the grey relational analysis method to obtain the fluctuation correlation information including the correlation weight matrix includes:

[0024] Perform normalization processing on the fluctuation analysis results corresponding to each time-series data to obtain the normalized fluctuation analysis sequence corresponding to each time-series data;

[0025] Determine the target sequence and the comparison sequence from the normalized fluctuation analysis sequence;

[0026] Calculate the grey correlation coefficients between each comparison sequence and the target sequence;

[0027] Perform weighted averaging on the grey correlation coefficients corresponding to each time point to obtain the correlation degree matrix, and determine the weight coefficients corresponding to each fluctuation feature by the entropy weight method to obtain the weight coefficient matrix;

[0028] Perform the Hadamard product operation on the correlation degree matrix and the weight coefficient matrix to generate a correlation weight matrix reflecting the dynamic coupling relationship between the multimodal monitoring data.

[0029] In some embodiments, the information fusion of the fluctuation analysis results and the corresponding time-series data in the multimodal monitoring data by the way of time window alignment to obtain the target monitoring data includes:

[0030] Stitch the fluctuation feature vector within each time window in the fluctuation analysis results with the original data of the corresponding time-series data in the multimodal monitoring data within the same time window to form the target monitoring data with dimension expansion and feature enhancement.

[0031] In some embodiments, the encoding network includes a multimodal parallel encoding module, a cross-modal attention fusion module, a feature extraction module, and an output module; where

[0032] The multi-modal parallel encoding module includes branch networks respectively used for encoding target monitoring data corresponding to the video monitoring data, sound monitoring data, environmental monitoring data, device operation parameters, device positioning data, and personnel positioning data;

[0033] The cross-modal attention fusion module is used to use the fluctuation correlation information as an attention weight to dynamically weight and fuse the multi-modal features corresponding to the target monitoring data;

[0034] The feature extraction module is used to perform deep feature extraction on the fused multi-modal features to obtain multi-scale deep features corresponding to the target monitoring data;

[0035] The output module is used to map the multi-scale deep features to a job status representation vector corresponding to the multi-modal monitoring data, where the job status representation vector at least includes index data corresponding to the device operation status, environmental safety status, and personnel status.

[0036] In some embodiments, the decoding network includes a deep learning model, and the deep learning model is used to decode the job status representation vector to output a corresponding abnormal analysis result of the underground operation status based on the job status representation vector.

[0037] In some embodiments, the machine learning model is trained based on the following method:

[0038] Obtain multiple groups of sample multi-modal monitoring data in underground operation scenarios, and simultaneously determine the actual job status corresponding to each group of sample multi-modal monitoring data;

[0039] Determine corresponding sample target monitoring data based on the sample multi-modal monitoring data, and use the actual job status corresponding to the sample multi-modal monitoring data as the label of the sample target monitoring data to obtain multiple training samples;

[0040] Input the training samples into the encoding network and decoding network of the machine learning model for processing to obtain an abnormal analysis result of the underground operation status;

[0041] Optimize the network parameters of the encoding network and the decoding network based on the difference between the abnormal analysis result of the underground operation status and the label until the preset training conditions are met, and obtain the trained machine learning model.

[0042] Another aspect of the embodiments of this specification also provides an intelligent underground operation status monitoring system, and this system includes:

[0043] The multi-modal monitoring data acquisition module is used to acquire multi-modal monitoring data for reflecting the underground operation status. Among them, the multi-modal monitoring data includes time-series data respectively obtained based on video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data;

[0044] The fluctuation analysis module is used to perform time-frequency domain fluctuation analysis on each time-series data in the multi-modal monitoring data to obtain the fluctuation analysis results corresponding to each time-series data;

[0045] The fluctuation correlation information determination module is used to calculate the correlation degree between the fluctuation analysis results corresponding to different time-series data based on the grey correlation analysis method to obtain the fluctuation correlation information including the correlation weight matrix;

[0046] The target monitoring data generation module is used to perform information fusion on the fluctuation analysis results and the corresponding time-series data in the multi-modal monitoring data by means of time window alignment to obtain the target monitoring data;

[0047] The operation status characterization vector determination module is used to use the fluctuation correlation information as the weight parameter of the attention mechanism, perform multi-modal feature extraction on the target monitoring data through the encoding network, and output the operation status characterization vector corresponding to the multi-modal monitoring data. Among them, the operation status characterization vector at least includes index data corresponding to the equipment operation status, environmental safety status, and personnel status;

[0048] The operation status monitoring data determination module is used to input the operation status characterization vector into a pre-trained machine learning model for underground operation status anomaly analysis to obtain intelligent underground operation status monitoring data.

[0049] The beneficial effects that the intelligent underground operation status monitoring system and method provided by the embodiments of this specification may bring at least include:

[0050] (1) By performing fluctuation analysis on each time-series data in the multi-modal monitoring data to obtain the corresponding fluctuation analysis results respectively, and then performing information fusion on the fluctuation analysis results and the corresponding time-series data in the multi-modal monitoring data to obtain the target monitoring data, it can make the target monitoring data contain more features than the original data, thus facilitating the identification of more subtle underground operation abnormal situations;

[0051] (2) By calculating the correlation degree between the fluctuation analysis results corresponding to different time-series data based on the grey correlation analysis method, the fluctuation correlation information including the correlation weight matrix is obtained. Then, taking the fluctuation correlation information as the weight parameter of the attention mechanism, the multi-modal feature encoding of the target monitoring data is performed through the encoding network of the pre-trained machine learning model, and the job status representation vector corresponding to the multi-modal monitoring data is output. Finally, inputting the job status representation vector into the decoding network of the pre-trained machine learning model for underground job status anomaly analysis, the intelligent underground job status monitoring data can be obtained, and the dynamic coupling relationship between different modalities of monitoring data can be combined to realize the identification of abnormal states, so as to more sensitively capture the changes in the underground job status and more accurately realize the monitoring of the underground job status.

[0052] Additional features will be partly described in the following description. For those skilled in the art, it will become obvious by referring to the following content and the drawings, or can be understood by generating or operating examples. The features of this specification can be realized and obtained by practicing or using various aspects of the methods, tools and combinations described in the following detailed examples. Brief Description of the Drawings

[0053] This specification will be further described in the form of exemplary embodiments, and these exemplary embodiments will be described in detail through the drawings. These embodiments are not restrictive. In these embodiments, the same numbers represent the same structures, where:

[0054] Figure 1 is an exemplary flowchart of the intelligent underground job status monitoring method shown in some embodiments of this specification;

[0055] Figure 2 is an exemplary sub-step flowchart of the intelligent underground job status monitoring method shown in some embodiments of this specification;

[0056] Figure 3 is an exemplary training step flowchart of the machine learning model shown in some embodiments of this specification;

[0057] Figure 4 is an exemplary module diagram of the intelligent underground job status monitoring system shown in some embodiments of this specification. Detailed Description of the Embodiments

[0058] To more clearly illustrate the technical solutions of the embodiments of this specification, the following will briefly introduce the attached drawings required for the description of the embodiments. Obviously, the attached drawings in the following description are only some examples or embodiments of this specification. For those of ordinary skill in the art, without creative efforts, this specification can also be applied to other similar scenarios based on these drawings. Unless obvious from the language context or otherwise stated, the same reference numerals in the drawings represent the same structure or operation.

[0059] It should be understood that the "system", "device", "unit" and / or "module" used in this specification are a way to distinguish different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the said words can be replaced by other expressions.

[0060] As shown in this specification and the claims, unless the context clearly indicates an exception, words such as "a", "an", "one" and / or "the" are not specifically singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0061] Flowcharts are used in this specification to illustrate the operations performed by the systems according to the embodiments of this specification. It should be understood that the previous or subsequent operations do not necessarily need to be executed precisely in sequence. On the contrary, they can be executed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or several steps can be removed from these processes.

[0062] The following will provide a detailed description of the intelligent underground operation status monitoring system and method provided by the embodiments of this specification in conjunction with the attached drawings.

[0063] Figure 1 is an exemplary flowchart of an intelligent underground operation status monitoring method shown according to some embodiments of this specification. In some embodiments, the intelligent underground operation status monitoring method can be executed by a processing logic, which can include hardware (such as circuits, dedicated logic, programmable logic, microcode, etc.), software (instructions running on a processing device to perform hardware simulation), etc. or any combination thereof. In some embodiments, Figure 1 One or more operations in the flowchart of the intelligent underground operation status monitoring method shown can be implemented by a processing device and / or a terminal device. For example, the intelligent underground operation status monitoring method can be stored in a storage device in the form of a computer program and / or instructions, and be called and / or executed by a processing device and / or a terminal device.

[0064] Refer toFigure 1 In the embodiments of the present application, the intelligent underground operation status monitoring method provided may include the following steps S110 to S160:

[0065] Step S110: Obtain multimodal monitoring data for reflecting the underground operation status, where the multimodal monitoring data includes time-series data respectively obtained based on video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data. In some embodiments, step S110 may be executed by the multimodal monitoring data acquisition module 210 mentioned later.

[0066] In some embodiments of the present application, multimodal monitoring data may be obtained by deploying various sensors and monitoring devices in the underground operation space. These sensors and monitoring devices may include, but are not limited to, video monitoring cameras, sound sensors, environmental monitoring sensors (such as temperature sensors, humidity sensors, gas concentration sensors, dust sensors, etc.), equipment operation parameter monitoring sensors (such as motor current monitoring sensors, vibration monitoring sensors, etc.), equipment positioning devices (such as GPS locators, RFID tags, etc.), and personnel positioning devices (such as positioning bracelets or cards worn by workers, etc.).

[0067] Specifically, in the embodiments of the present application, video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data for reflecting the underground operation status may be collected through the above-mentioned sensors and monitoring devices. Among them, the environmental monitoring data may include temperature and humidity data, dust detection data, and gas concentration data such as methane (the main component of gas), oxygen, and carbon dioxide.

[0068] In some embodiments, the data collected in real time or at regular intervals through the above-mentioned sensors and monitoring devices may be transmitted to the ground monitoring center or the cloud server through a network (such as through wired networks such as Ethernet and industrial buses, or wireless networks such as ZigBee, Bluetooth, and mobile communication networks). Further, in order to determine the dynamic coupling relationship between various data in the multimodal monitoring data, in the embodiments of the present application, the ground monitoring center or the cloud server may process the received multimodal monitoring data according to a preset processing method, so as to obtain the time-series data respectively corresponding to each monitoring data.

[0069] It can be understood that for data such as environmental monitoring data, equipment operation parameters, equipment location data, and personnel location data, they can be directly converted into time-series data according to the time sequence and the collected data content for subsequent data analysis and processing. For multimedia data such as video surveillance data and audio surveillance data, feature extraction needs to be performed through corresponding algorithms, and then converted into time-series data according to the extracted key information and its corresponding time-series information.

[0070] Exemplarily, in some embodiments of the present application, the video surveillance data can be collected by a camera array deployed in the underground roadway, and a multi-object tracking algorithm based on deep learning can be used to extract the attention information from the video surveillance data to obtain the first time-series data corresponding to the video surveillance data, where the attention information can at least include personnel operation specification information, equipment movement trajectory information, and material stacking state information. Specifically, in the embodiments of the present application, multiple cameras can work together to form a monitoring network covering the underground roadway. These cameras can capture the video images of the underground operation scenes in real time to ensure as much as possible a dead-angle-free monitoring. Further, in the embodiments of the present application, a multi-object tracking algorithm based on deep learning can be used to process the video surveillance data collected by these camera arrays, so as to accurately identify and track key targets such as personnel, equipment, and materials underground, and then extract the attention information related to these targets, such as personnel operation specification information, equipment movement trajectory information, material stacking state information, etc. Finally, by combining these key information with timestamps, the first time-series data can be obtained.

[0071] Only as an example, in some embodiments, the first time-series data can be represented as a time series, where each time point is associated with the underground operation state information reflected by the video surveillance data collected by each camera, such as the position of personnel (where the personnel identity can be determined by face recognition, and the personnel position can be determined by the camera number and its installation position), operation specification information (for example, whether it is detected that the operator is not wearing protective equipment, whether there are violations such as leaving the post, sleeping, smoking, etc.), equipment movement trajectory information (for example, determined by tracking the operation trajectory of underground transport vehicles), and material stacking state information (for example, determined by the shape, height, and position information of the material pile whether there are situations such as irregular stacking and exceeding the safety range). In the embodiments of the present application, more technical details about using a multi-object tracking algorithm based on deep learning to extract attention information from the video surveillance data can be regarded as prior art and will not be elaborated in detail in this specification.

[0072] Furthermore, for the sound monitoring data, in some embodiments of the present application, the sound monitoring data can be collected through a distributed acoustic sensor network, and then Mel Frequency Cepstral Coefficients (MFCCs) combined with a Convolutional Recurrent Neural Network (CRNN) are used to extract voiceprint features, obtaining the second time-series data corresponding to the sound monitoring data.

[0073] It can be understood that Mel Frequency Cepstral Coefficients is a representation method that can convert speech signals into a series of parameters reflecting speech characteristics, while a Convolutional Recurrent Neural Network can learn and extract key features from these parameters, thereby realizing effective analysis and processing of sound monitoring data. In this way, abnormal sounds in underground operations can be identified, such as abnormal sounds generated by equipment failures or coal seam collapses, personnel's cries for help, environmental noises (such as sounds generated by normal operation of machinery), etc., providing an important basis for timely response and handling of underground abnormal situations.

[0074] Specifically, in the embodiments of the present application, the Convolutional Recurrent Neural Network may include a time-frequency map convolutional layer and a bidirectional LSTM (Long Short-Term Memory) layer. The process of using Mel Frequency Cepstral Coefficients combined with a Convolutional Recurrent Neural Network to extract voiceprint features and obtain the second time-series data corresponding to the sound monitoring data may include the following steps:

[0075] First, construct an acoustic feature template that at least includes an abnormal sound library (including abnormal sounds generated by equipment failures and coal seam collapses, etc.), a personnel voice library (including personnel's cries for help), and an environmental noise library (including sounds generated by normal operation of machinery); then, extract the short-time spectral features in the sound monitoring data through the time-frequency map convolutional layer in the Convolutional Recurrent Neural Network, and use the bidirectional LSTM layer in the Convolutional Recurrent Neural Network to obtain the long-time temporal dependence features in the sound monitoring data; further, voiceprint matching can be performed on the extracted short-time spectral features and long-time temporal dependence features based on the acoustic feature template, thereby identifying the event information included in the sound monitoring data; finally, the second time-series data corresponding to the sound monitoring data can be obtained according to the event type, occurrence time, and duration status corresponding to the event information. Similarly, in the embodiments of the present application, this second time-series data can also be represented as a time series, where each time point is associated with the event information reflected by the sound monitoring data collected by each acoustic sensor (when the sound monitoring data corresponding to a certain time point reflects the absence of any event, it can be represented in this time series through a specific code).

[0076] In the embodiments of the present application, more details about using Mel Frequency Cepstral Coefficients in combination with a Convolutional Recurrent Neural Network for voiceprint feature extraction to obtain the second time-series data corresponding to the voice monitoring data can also be regarded as the prior art and will not be elaborated in this specification.

[0077] Step S120: Perform time-frequency domain fluctuation analysis on each time-series data in the multi-modal monitoring data respectively to obtain the corresponding fluctuation analysis results for each time-series data. In some embodiments, step S120 can be executed by the fluctuation analysis module 220 mentioned later.

[0078] It can be understood that the fluctuations of the time-series data corresponding to the above multi-modal monitoring data can reflect the changes in the underground operation state (which may be normal changes or abnormal changes). For example, in the second time-series data corresponding to the voice monitoring data, abnormal fluctuations may correspond to events such as mechanical failures and abnormal behaviors of personnel; in the first time-series data corresponding to the video monitoring data, abnormal fluctuations may correspond to situations such as illegal operations of personnel and abnormal movement trajectories of equipment. In the embodiments of the present application, by performing time-frequency domain fluctuation analysis on each time-series data in the multi-modal monitoring data respectively to obtain the corresponding fluctuation analysis results for each time-series data, the change information of the underground operation state contained in each time-series data corresponding to the multi-modal monitoring data can be mined, and then the dynamic coupling relationship between each monitoring data can be analyzed based on the fluctuation analysis results in the subsequent steps.

[0079] Specifically, in some embodiments of the present application, a sliding time window algorithm can be used to determine the fluctuation feature vectors corresponding to each time-series data in the multi-modal monitoring data within a time window of a preset time step, where the fluctuation features at least include the fluctuation amplitude, fluctuation frequency, and phase offset corresponding to each time-series data.

[0080] Exemplarily, in some embodiments, a suitable time window length can be set (the time window length can be adjusted according to the actual situation of the changes in the underground operation state). By sliding the time window on the time-series data, the fluctuation features (such as including the fluctuation amplitude, fluctuation frequency, and phase offset) of the time-series data within each time window can be sequentially extracted, thereby forming a sequence of fluctuation feature vectors. It can be understood that these fluctuation feature vectors can not only reflect the local fluctuation characteristics of the time-series data in time but also provide a basis for analyzing the dynamic coupling relationship between each monitoring data in the subsequent steps.

[0081] In some embodiments of the present application, a corresponding fluctuation feature matrix may be constructed based on the fluctuation analysis results of each piece of time series data obtained through the above processing. The rows of this matrix may correspond to different time series data, and the columns of this matrix may correspond to the fluctuation features (such as fluctuation amplitude, fluctuation frequency, and phase offset) within each time window.

[0082] Step S130, calculate the correlation degree between the fluctuation analysis results corresponding to different time series data based on the grey relational analysis method, and obtain the fluctuation correlation information including the correlation weight matrix. In some embodiments, step S130 may be executed by the fluctuation correlation information determination module 230 mentioned later.

[0083] In the embodiments of the present application, the correlation degree between the fluctuation analysis results corresponding to different time series data may be calculated based on the grey relational analysis method. The higher this correlation degree, the closer the dynamic coupling relationship between the time series data (that is, the stronger the correlation between the monitoring data of different modes). Therefore, in the embodiments of the present application, by calculating the correlation degree between the fluctuation analysis results corresponding to different time series data, more features reflected by the multi-modal monitoring data can be mined, which is conducive to more accurately identifying the abnormal state of the underground operation.

[0084] Specifically, in this specification, the grey relational analysis method is a mathematical method used to evaluate the similarity and correlation degree between different time series data. It can find out the correlation between them and quantify this correlation degree by analyzing the change trend and fluctuation characteristics of the time series data. In the technical solution of the present application, by applying the grey relational analysis method to process and analyze the multi-modal monitoring data collected by the underground operation state monitoring system, the potential connections and rules between these monitoring data can be revealed, so as to more deeply understand the change pattern of the underground operation state and improve the accuracy and sensitivity of abnormal state recognition.

[0085] Figure 2 is an exemplary sub-step flowchart of the intelligent underground operation state monitoring method shown in some embodiments of this specification. Refer to Figure 2 , in some embodiments, step S130 may include the following sub-steps S131 to S135:

[0086] Sub-step S131, perform standardization processing on the fluctuation analysis results corresponding to each time series data to obtain the standardized fluctuation analysis sequence corresponding to each time series data.

[0087] Sub-step S132, determine the target sequence and the comparison sequences from the standardized fluctuation analysis sequences.

[0088] Sub-step S133, calculate the grey correlation coefficients between each comparison sequence and the target sequence.

[0089] Sub-step S134: Perform weighted averaging on the grey correlation coefficients corresponding to each time point to obtain a correlation degree matrix, and determine the weight coefficients corresponding to each fluctuation feature through the entropy weight method to obtain a weight coefficient matrix;

[0090] Sub-step S135: Perform Hadamard product operation on the correlation degree matrix and the weight coefficient matrix to generate a correlation weight matrix reflecting the dynamic coupling relationship between multi-modal monitoring data.

[0091] Specifically, in sub-step S131, since the fluctuation analysis results corresponding to different time-series data may have different dimensions, orders of magnitude, and feature dimensions, direct comparison may lead to inaccurate results. Therefore, it is necessary to perform standardization processing on the fluctuation analysis results corresponding to each time-series data to eliminate the influence caused by differences in dimensions, orders of magnitude, and feature dimensions, so as to obtain the standardized fluctuation analysis sequence corresponding to each time-series data. Only as an example, in some embodiments of the present application, for dimension standardization processing, the Z-score standardization method can be used to convert the fluctuation analysis results corresponding to each time-series data into standardized data with a mean of 0 and a standard deviation of 1. For order-of-magnitude standardization processing, methods such as logarithmic transformation and exponential transformation can be used to convert the fluctuation analysis results of each time-series data into the same order-of-magnitude range. For feature dimension standardization processing, methods such as principal component analysis (PCA) and linear discriminant analysis (LDA) can be used to perform dimensionality reduction processing on the fluctuation analysis results of each time-series data to extract the main features and eliminate redundant information, or the data in other fluctuation analysis results can be dimensionally extended based on the maximum number of feature dimensions to obtain fluctuation analysis results with the same dimension.

[0092] It can be understood that in the embodiments of the present application, by performing standardization processing on the fluctuation analysis results corresponding to different time-series data, the fluctuation analysis results corresponding to each time-series data can be made comparable and consistent, thereby providing a more accurate and reliable data basis for subsequent analysis and comparison.

[0093] In sub-step S132, the target sequence can be understood as the standardized fluctuation analysis sequence corresponding to the target time-series data to be analyzed, and the comparison sequence can be understood as the standardized fluctuation analysis sequence corresponding to other time-series data.

[0094] In sub-step S133, the calculation process of the grey correlation coefficient can be expressed as follows:

[0095] Assume the target sequence is: X0 = {x0(1), x0(2), …, x0(n)}

[0096] The comparison sequence is: X i = {x i (1), xi (2),…,x i (n)}

[0097] Then the absolute difference between the comparison sequence and the target sequence at each time point (k) can be calculated:

[0098] Δ i (k)=|x'0(k)-x i (k)|

[0099] Furthermore, we can get the difference matrix Δ=[Δ i (k)], the dimension is m×n (m is the number of comparison sequences, n is the number of time points). Then determine the global minimum difference Δ min =min i,k Δ i (k), the global maximum difference Δ max =max i,k Δ i (k).

[0100] Furthermore, the grey relational coefficient can be expressed as:

[0101]

[0102] Among them, γ(x0(k),x i (k)) represents the comparison sequence X i The grey correlation coefficient with the target sequence X0 at the kth time point, ρ is the resolution coefficient, usually between 0 and 1 (usually set to 0.5), which is used to adjust the sensitivity of the correlation coefficient difference. The larger the grey correlation coefficient, the higher the similarity between the comparison sequence and the target sequence at this time point, that is, the stronger the correlation.

[0103] Furthermore, in sub-step S134, the grey correlation coefficients corresponding to each time point may be weighted averaged based on the calculation result of sub-step S133 to obtain a correlation matrix for further analyzing the overall correlation between the comparison sequence and the target sequence.

[0104] At the same time, in the embodiment of the present application, the weight coefficient corresponding to each fluctuation feature can be determined by the entropy weight method to obtain a weight coefficient matrix. Specifically, the amount of information and uncertainty of each fluctuation feature can be measured by calculating the entropy value of each fluctuation feature. The smaller the entropy value, the greater the amount of information contained in the fluctuation feature, and the stronger the ability to reflect the downhole operation status, so a larger weight coefficient should be assigned. Conversely, the fluctuation feature with a larger entropy value contains less information, and has a weaker ability to reflect the downhole operation status, and the weight coefficient should be reduced accordingly. After obtaining the entropy value of each fluctuation feature, the weight coefficient corresponding to each fluctuation feature can be obtained by normalization processing, and then a weight coefficient matrix can be constructed.

[0105] The above correlation matrix reflects the set of grey correlation coefficients corresponding to each time point, while the weight coefficient matrix reflects the importance of each fluctuation feature in the overall analysis. Based on this, in sub-step S135, a correlation weight matrix can be obtained by performing a Hadamard product operation on the correlation matrix and the weight coefficient matrix. This matrix can not only reflect the correlation degree between each time series data and the target time series data, but also consider the importance of each fluctuation feature, so it can more comprehensively reflect the dynamic coupling relationship between multi-modal monitoring data. In some embodiments of the present application, the fluctuation correlation relationship between each time series data and other data in the multi-modal monitoring data can be obtained according to the correlation weight matrix.

[0106] Continue to refer to Figure 1 , after the above step S130, the method further includes:

[0107] Step S140, performing information fusion on the fluctuation analysis result and the corresponding time series data in the multi-modal monitoring data in a time window alignment manner to obtain target monitoring data. In some embodiments, step S140 can be executed by the target monitoring data generation module 240 mentioned later.

[0108] Specifically, in some embodiments of the present application, the fluctuation feature vectors in each time window of the fluctuation analysis result can be concatenated with the original data of the corresponding time series data in the multi-modal monitoring data in the same time window to form target monitoring data with extended dimensions and enhanced features.

[0109] It should be noted that in the embodiments of the present application, by concatenating the fluctuation feature vectors in each time window of the fluctuation analysis result obtained through the above process with the original data of the corresponding time series data in the multi-modal monitoring data in the same time window to obtain the target monitoring data, the target monitoring data can contain more features than the original data, which is beneficial to identifying more subtle abnormal underground operation conditions.

[0110] Step S150, using the fluctuation correlation information as the weight parameter of the attention mechanism, and performing multi-modal feature encoding on the target monitoring data through the encoding network of a pre-trained machine learning model, and outputting the operation state representation vector corresponding to the multi-modal monitoring data. In some embodiments, step S150 can be executed by the operation state representation vector determination module 250 mentioned later.

[0111] In the embodiments of the present application, the encoding network of the machine learning model may include a multi-modal parallel encoding module, a cross-modal attention fusion module, a feature extraction module, and an output module. Among them, the multi-modal parallel encoding module may include branch networks respectively used for encoding the target monitoring data corresponding to the video monitoring data, sound monitoring data, environmental monitoring data, device operation parameters, device positioning data, and personnel positioning data. The cross-modal attention fusion module may be used to use the fluctuation correlation information determined in the above steps as attention weights to dynamically weight and fuse the multi-modal features corresponding to the target monitoring data. The feature extraction module may be used to perform deep feature extraction on the fused multi-modal features to obtain the multi-scale deep features corresponding to the target monitoring data. The output module may be used to map the multi-scale deep features to the operation state representation vectors corresponding to the multi-modal monitoring data. In the embodiments of the present application, the operation state representation vectors may at least include index data corresponding to the device operation state, environmental safety state, and personnel state.

[0112] Specifically, in the implementation of the present application, the machine learning model may be regarded as an encoder-decoder model, and the above encoding network may be regarded as the encoder part thereof, which may be used to perform feature extraction and encoding on the target monitoring data obtained by processing various monitoring data for underground operations (such as video monitoring data, sound monitoring data, environmental monitoring data, device operation parameters, device positioning data, and personnel positioning data) in the above process to form multi-scale deep features. Since these multi-scale deep features can reflect the actual state of underground operations, based on this, in the embodiments of the present application, the multi-scale deep features may further be mapped to the operation state representation vectors corresponding to the multi-modal monitoring data. In the embodiments of the present application, the operation state representation vectors may at least include index data corresponding to the device operation state, environmental safety state, and personnel state.

[0113] It should be noted that, in the embodiments of the present application, the above encoding network may be trained to obtain the above processing capabilities. For more content about the training process of the encoding network, reference may be made to the following text (such as part of step S160), and it will not be described here for the time being.

[0114] Step S160: Input the operation state representation vectors into the decoding network of the pre-trained machine learning model for abnormal analysis of the underground operation state to obtain intelligent underground operation state monitoring data. In some embodiments, step S160 may be executed by the operation state monitoring data determination module 260 mentioned later.

[0115] In the embodiments of the present application, the decoding network can be regarded as the decoder part in the encoder-decoder model. Specifically, in some embodiments of the present application, the decoding network can include a deep learning model (such as a convolutional neural network CNN or a recurrent neural network RNN, etc.), which can be used to decode the input job status representation vector and analyze whether there is an abnormality in the downhole operation status according to the decoding result. In the embodiments of the present application, the decoding network can be trained to obtain the above processing capabilities.

[0116] The following Figure 3 briefly introduces the training process of the machine learning model in the embodiments of the present application:

[0117] Figure 3 is an exemplary training step flowchart of the machine learning model shown in some embodiments of this specification. Referring to Figure 3 , in the embodiments of the present application, the training process of the machine learning model can include the following steps S101 to S104:

[0118] Step S101, obtain multiple groups of sample multimodal monitoring data in downhole operation scenarios, and at the same time determine the actual operation status corresponding to each group of sample multimodal monitoring data.

[0119] Step S102, determine the corresponding sample target monitoring data based on the sample multimodal monitoring data, and use the actual operation status corresponding to the sample multimodal monitoring data as the label of the sample target monitoring data to obtain multiple training samples.

[0120] Step S103, input the training samples into the encoding network and decoding network of the machine learning model for processing to obtain the analysis result of the downhole operation status abnormality.

[0121] Step S104, optimize the network parameters of the encoding network and the decoding network based on the difference between the analysis result of the downhole operation status abnormality and the label. When the preset training condition is met, obtain the trained machine learning model.

[0122] It should be noted that in step S102, the sample target monitoring data corresponding to the sample multimodal monitoring data can be determined with reference to the above steps S110 to S140, and details are not described here again.

[0123] In step S104, a gradient descent algorithm or other optimization algorithm can be used to minimize the difference between the analysis result of the downhole operation status abnormality and the label, so as to continuously adjust the network parameters of the encoding network and the decoding network to improve the prediction accuracy of the machine learning model.

[0124] In some embodiments of the present application, when the difference between the analysis result of the abnormal downhole operation state and the label is less than a preset threshold or reaches a preset number of iterations, it is determined that the preset training condition is satisfied, and the trained machine learning model is obtained. More details about the above machine learning model can be regarded as the prior art and will not be elaborated in this specification.

[0125] It should be noted that in the embodiments of the present application, by processing the target monitoring data obtained in the above process through this machine learning model, the changes in the downhole operation state can be captured more sensitively, so as to more accurately monitor the downhole operation state.

[0126] It can be understood that this precise monitoring ability is of great significance for improving the safety and management efficiency of downhole operations. In the embodiments of the present application, by monitoring the downhole operation state in real time through the above solution and timely discovering and handling potential safety hazards, the occurrence of downhole accidents can be effectively avoided, thereby better ensuring the safety of downhole operation personnel and equipment.

[0127] Figure 4 is a schematic diagram of the modules of an intelligent downhole operation state monitoring system shown in some embodiments of this specification. In some embodiments, Figure 4 the shown intelligent downhole operation state monitoring system 200 can be implemented in the form of software and / or hardware. For example, it can be configured in the form of software and / or hardware into a processing device and / or a terminal device to process the collected multi-modal monitoring data, so as to realize the analysis of abnormal downhole operation states.

[0128] Referring to Figure 4 , in some embodiments, the intelligent downhole operation state monitoring system 200 may include a multi-modal monitoring data acquisition module 210, a fluctuation analysis module 220, a fluctuation correlation information determination module 230, a target monitoring data generation module 240, an operation state characterization vector determination module 250, and an operation state monitoring data determination module 260.

[0129] The multi-modal monitoring data acquisition module 210 can be used to acquire multi-modal monitoring data for reflecting the downhole operation state, where the multi-modal monitoring data includes time-series data respectively obtained based on video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data.

[0130] The fluctuation analysis module 220 can be used to perform time-frequency domain fluctuation analysis on each time-series data in the multi-modal monitoring data to obtain the fluctuation analysis result corresponding to each time-series data.

[0131] The fluctuation correlation information determination module 230 can be used to calculate the correlation degree between the fluctuation analysis results corresponding to different time-series data based on the grey correlation analysis method, and obtain the fluctuation correlation information including the correlation weight matrix.

[0132] The target monitoring data generation module 240 can be used to perform information fusion on the fluctuation analysis result and the corresponding time-series data in the multi-modal monitoring data through time window alignment to obtain the target monitoring data.

[0133] The job status representation vector determination module 250 can be used to use the fluctuation correlation information as the weight parameter of the attention mechanism, and perform multi-modal feature encoding on the target monitoring data through the encoding network of the pre-trained machine learning model, and output the job status representation vector corresponding to the multi-modal monitoring data.

[0134] The job status monitoring data determination module 260 can be used to input the job status representation vector into the decoding network of the pre-trained machine learning model to perform abnormal analysis of the underground job status, and obtain the intelligent underground job status monitoring data.

[0135] For more details about each of the above modules, reference can be made to other parts of this specification (such as Figures 1 to 3 the relevant descriptions of the part), which will not be elaborated here.

[0136] It should be understood that Figure 4 the shown intelligent underground job status monitoring system 200 and its modules can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented through hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in the memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art can understand that the above system and method can be implemented using computer-executable instructions and / or included in the processor control code. For example, such code is provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules of this specification can be implemented not only by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but also by software implemented by various types of processors, or by a combination of the above hardware circuits and software (for example, firmware).

[0137] Note that the above description of the intelligent underground operation status monitoring system 200 is provided for illustrative purposes only and is not intended to limit the scope of this specification. It can be understood that for those skilled in the art, various modules can be arbitrarily combined or a subsystem can be formed and connected to other modules without departing from this principle according to the description of this specification. For example, Figure 4 The multi-modal monitoring data acquisition module 210, the fluctuation analysis module 220, the fluctuation correlation information determination module 230, the target monitoring data generation module 240, the operation status representation vector determination module 250, and the operation status monitoring data determination module 260 described in Figure 4 can be different modules in a system or a single module can implement the functions of two or more of the above modules. Such variations are all within the protection scope of this specification. In some embodiments, the foregoing modules can be part of a processing device and / or a terminal device.

[0138] In summary, the beneficial effects that the embodiments of this specification may bring include, but are not limited to: (1) In the intelligent underground operation status monitoring system and method provided in some embodiments of this specification, by performing fluctuation analysis on each time-series data in the multi-modal monitoring data to obtain the corresponding fluctuation analysis results, and then fusing the fluctuation analysis results with the corresponding time-series data in the multi-modal monitoring data to obtain the target monitoring data, it can make the target monitoring data contain more features than the original data, thereby facilitating the identification of more subtle abnormal underground operation conditions; (2) In the intelligent underground operation status monitoring system and method provided in some embodiments of this specification, by calculating the correlation degree between the fluctuation analysis results corresponding to different time-series data based on the grey correlation analysis method to obtain the fluctuation correlation information including the correlation weight matrix, and then using the fluctuation correlation information as the weight parameter of the attention mechanism, performing multi-modal feature encoding on the target monitoring data through the encoding network of the pre-trained machine learning model, and outputting the operation status representation vector corresponding to the multi-modal monitoring data; finally, inputting the operation status representation vector into the decoding network of the pre-trained machine learning model for abnormal analysis of the underground operation status to obtain the intelligent underground operation status monitoring data, it can combine the dynamic coupling relationship between different modalities of monitoring data to achieve the identification of abnormal states, thereby more sensitively capturing the changes in the underground operation status and more accurately realizing the monitoring of the underground operation status.

[0139] It should be noted that the beneficial effects that different embodiments may bring are different. In different embodiments, the beneficial effects that may be produced can be any one or several combinations of the above, or any other beneficial effects that may be obtained.

[0140] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only an example and does not constitute a limitation to this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification, so such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this specification.

[0141] Meanwhile, this specification uses specific terms to describe the embodiments of this specification. Such as "one embodiment", "an embodiment", and / or "some embodiments" mean a certain feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that the "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more at different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0142] In addition, those skilled in the art can understand that various aspects of this specification can be illustrated and described by several patentable types or situations, including any new and useful processes, machines, products, or combinations of substances, or any new and useful improvements to them. Accordingly, various aspects of this specification can be executed entirely by hardware, can be executed entirely by software (including firmware, resident software, microcode, etc.), or can be executed by a combination of hardware and software. The above hardware or software can all be referred to as "data blocks", "modules", "engines", "units", "components", or "systems". In addition, various aspects of this specification may be embodied as a computer product located in one or more computer-readable media, and the product includes computer-readable program codes.

[0143] A computer storage medium may contain a propagated data signal containing computer program codes, such as on a baseband or as part of a carrier wave. This propagated signal may have various forms of manifestation, including electromagnetic form, optical form, etc., or a suitable combination of forms. A computer storage medium can be any computer-readable medium other than a computer-readable storage medium, and this medium can be connected to an instruction execution system, device, or equipment to implement communication, propagation, or transmission for use of a program. The program codes located on the computer storage medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.

[0144] The computer program codes required for the operations of each part of this specification can be written in any one or more programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages such as C, Visual Basic, Fortran2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages such as Python, Ruby, and Groovy, or other programming languages. This program code can run entirely on the user's computer, or run on the user's computer as an independent software package, or run partially on the user's computer and partially on a remote computer, or run entirely on a remote computer or processing device. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., through the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0145] In addition, unless clearly stated in the claims, the order of the processing elements and sequences, the use of numbers and letters, or the use of other names in this specification are not used to limit the order of the processes and methods in this specification. Although some currently considered useful embodiments of the invention are discussed through various examples in the above disclosure, it should be understood that such details only serve the purpose of illustration. The appended claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that conform to the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only through a software solution, such as installing the described system on an existing processing device or mobile device.

[0146] Similarly, it should be noted that, in order to simplify the presentation of the disclosure in this specification and thus help the understanding of one or more embodiments of the invention, in the previous description of the embodiments of this specification, sometimes multiple features are merged into one embodiment, drawing, or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than those mentioned in the claims. In fact, the features of the embodiments are fewer than all the features of the individual embodiments disclosed above.

[0147] In some embodiments, numbers are used to describe components and the quantity of attributes. It should be understood that such numbers used in the description of embodiments are, in some examples, modified by the modifiers "about", "approximately" or "substantially". Unless otherwise specified, "about", "approximately" or "substantially" indicate that the said numbers allow a variation of ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may vary according to the characteristics required by individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the method of retaining the general number of digits. Although the numerical ranges and parameters used in some embodiments of this specification to confirm the breadth of their scope are approximate values, in specific embodiments, such numerical settings are as precise as possible within the feasible range.

[0148] For each patent, patent application, patent application publication and other materials cited in this specification, such as articles, books, specifications, publications, documents, etc., their entire contents are hereby incorporated into this specification by reference. This excludes the application history documents that are inconsistent with or conflict with the content of this specification, and also excludes the documents that limit the broadest scope of the claims of this specification (currently or subsequently appended to this specification). It should be noted that if there are inconsistencies or conflicts between the descriptions, definitions, and / or uses of terms in the supplementary materials of this specification and the content described in this specification, the descriptions, definitions, and / or uses of terms in this specification shall prevail.

[0149] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification may be considered to be consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. An intelligent underground operation status monitoring method, characterized in that, Including: Obtain multimodal monitoring data for reflecting the downhole operation state, where the multimodal monitoring data includes time series data respectively obtained based on video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data; Conduct time-frequency domain fluctuation analysis on each time series data in the multimodal monitoring data to obtain the fluctuation analysis results corresponding to each time series data; Calculate the correlation degree between the fluctuation analysis results corresponding to different time series data based on the grey relational analysis method to obtain the fluctuation correlation information including the correlation weight matrix; Perform information fusion on the fluctuation analysis results and the corresponding time series data in the multimodal monitoring data by means of time window alignment to obtain the target monitoring data; Use the fluctuation correlation information as the weight parameter of the attention mechanism, and perform multimodal feature encoding on the target monitoring data through the encoding network of a pre-trained machine learning model to output the operation state representation vector corresponding to the multimodal monitoring data, where the operation state representation vector at least includes the index data corresponding to the equipment operation state, environmental safety state, and personnel state; Input the operation state representation vector into the decoding network of the pre-trained machine learning model for abnormal analysis of the downhole operation state to obtain the intelligent downhole operation state monitoring data.

2. The method according to claim 1, characterized in that The obtaining of the multimodal monitoring data for reflecting the downhole operation state includes: Collect the video monitoring data through a camera array deployed in the underground roadway, and extract the attention information from the video monitoring data by using a multi-object tracking algorithm based on deep learning to obtain the first time series data corresponding to the video monitoring data, where the attention information at least includes personnel operation specification information, equipment movement trajectory information, and material stacking state information; Collect the sound monitoring data through a distributed acoustic sensor network, and perform voiceprint feature extraction by using Mel frequency cepstral coefficients combined with a convolutional recurrent neural network to obtain the second time series data corresponding to the sound monitoring data.

3. The method according to claim 2, wherein The convolutional recurrent neural network includes a time-frequency graph convolutional layer and a bidirectional LSTM layer. The performing of voiceprint feature extraction by using Mel frequency cepstral coefficients combined with a convolutional recurrent neural network to obtain the second time series data corresponding to the sound monitoring data includes: Construct an acoustic feature template including at least an abnormal sound library, a personnel voice library, and an environmental noise library; Extract the short-time frequency spectrum features in the sound monitoring data through the time-frequency graph convolutional layer; Use the bidirectional LSTM layer to obtain the long-time sequence dependence features in the sound monitoring data; Perform voiceprint matching on the extracted short-time frequency spectrum features and long-time sequence dependence features based on the acoustic feature template to identify the event information included in the sound monitoring data; Obtain the second time series data corresponding to the sound monitoring data according to the event type, occurrence time, and duration state corresponding to the event information.

4. The method according to claim 1, wherein The conducting of time-frequency domain fluctuation analysis on each time series data in the multimodal monitoring data includes: The sliding time window algorithm is used to determine the fluctuation feature vectors corresponding to each time series data in the multi-modal monitoring data within a time window of a preset time step. Among them, the fluctuation features at least include the fluctuation amplitude, fluctuation frequency, and phase offset corresponding to each time series data.

5. The method according to claim 1, characterized in that Calculating the correlation degree between the fluctuation analysis results corresponding to different time series data based on the grey relational analysis method to obtain the fluctuation correlation information including the correlation weight matrix, including: Performing normalization processing on the fluctuation analysis results corresponding to each time series data to obtain the normalized fluctuation analysis sequence corresponding to each time series data; Determining the target sequence and the comparison sequence from the normalized fluctuation analysis sequence; Calculating the grey correlation coefficients between each comparison sequence and the target sequence; Performing weighted average on the grey correlation coefficients corresponding to each time point to obtain the correlation degree matrix, and determining the weight coefficients corresponding to each fluctuation feature through the entropy weight method to obtain the weight coefficient matrix; Performing Hadamard product operation on the correlation degree matrix and the weight coefficient matrix to generate a correlation weight matrix reflecting the dynamic coupling relationship between the multi-modal monitoring data.

6. The method according to claim 4, wherein Fusing the information of the fluctuation analysis results with the corresponding time series data in the multi-modal monitoring data in the way of time window alignment to obtain the target monitoring data, including: Concatenating the fluctuation feature vectors within each time window in the fluctuation analysis results with the original data of the corresponding time series data in the multi-modal monitoring data within the same time window to form the target monitoring data after dimension expansion and feature enhancement.

7. The method according to claim 1, characterized in that The encoding network includes a multi-modal parallel encoding module, a cross-modal attention fusion module, a feature extraction module, and an output module; among them, The multi-modal parallel encoding module includes branch networks respectively used for encoding the target monitoring data corresponding to the video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data; The cross-modal attention fusion module is used to dynamically weight and fuse the multi-modal features corresponding to the target monitoring data by using the fluctuation correlation information as the attention weight; The feature extraction module is used to perform deep feature extraction on the fused multi-modal features to obtain the multi-scale deep features corresponding to the target monitoring data; The output module is used to map the multi-scale deep features into the operation state representation vector corresponding to the multi-modal monitoring data, where the operation state representation vector at least includes the index data corresponding to the equipment operation state, environmental safety state, and personnel state.

8. The method according to claim 7, wherein The decoding network includes a deep learning model, and the deep learning model is used to decode the operation state representation vector to output the abnormal analysis result of the corresponding underground operation state based on the operation state representation vector.

9. The method according to claim 8, wherein The machine learning model is trained based on the following method: Obtaining multiple groups of sample multi-modal monitoring data in the underground operation scenario, and simultaneously determining the actual operation state corresponding to each group of sample multi-modal monitoring data; Determine the corresponding sample target monitoring data based on the sample multi-modal monitoring data, and use the actual operation status corresponding to the sample multi-modal monitoring data as the label of the sample target monitoring data to obtain multiple training samples; Input the training samples into the encoding network and decoding network of the machine learning model for processing to obtain the analysis result of abnormal downhole operation status; Optimize the network parameters of the encoding network and the decoding network based on the difference between the analysis result of abnormal downhole operation status and the label. When the preset training conditions are met, obtain the trained machine learning model.

10. An intelligent underground operation status monitoring system, characterized in that, Include: A multi-modal monitoring data acquisition module for acquiring multi-modal monitoring data used to reflect the downhole operation status, where the multi-modal monitoring data includes time-series data respectively obtained based on video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data; A fluctuation analysis module for performing time-frequency domain fluctuation analysis on each time-series data in the multi-modal monitoring data to obtain the corresponding fluctuation analysis result for each time-series data; A fluctuation correlation information determination module for calculating the correlation degree between the fluctuation analysis results corresponding to different time-series data based on the grey correlation analysis method to obtain the fluctuation correlation information including the correlation weight matrix; A target monitoring data generation module for fusing the fluctuation analysis results with the corresponding time-series data in the multi-modal monitoring data through time window alignment to obtain the target monitoring data; An operation status characterization vector determination module for using the fluctuation correlation information as the weight parameter of the attention mechanism, performing multi-modal feature extraction on the target monitoring data through the encoding network, and outputting the operation status characterization vector corresponding to the multi-modal monitoring data, where the operation status characterization vector at least includes index data corresponding to the equipment operation status, environmental safety status, and personnel status; An operation status monitoring data determination module for inputting the operation status characterization vector into a pre-trained machine learning model for abnormal analysis of the downhole operation status to obtain intelligent downhole operation status monitoring data.

Images