An intelligent downhole operation state monitoring system and method

Abstract

The embodiment of the specification discloses an intelligent downhole operation state monitoring system and method, and relates to the technical field of monitoring data processing. The method comprises the following steps: acquiring multi-modal monitoring data reflecting the downhole operation state; performing fluctuation analysis on each time series data in the multi-modal monitoring data respectively to obtain corresponding fluctuation analysis results; calculating the correlation degree between the fluctuation analysis results corresponding to different time series data based on a grey correlation analysis method to obtain fluctuation correlation information; fusing the fluctuation analysis results and the corresponding time series data through time window alignment to obtain target monitoring data; taking the fluctuation correlation information as a weight parameter of an attention mechanism, performing multi-modal feature coding on the target monitoring data through an encoding network, and outputting a corresponding operation state representation vector; inputting the operation state representation vector into a decoding network for downhole operation state anomaly analysis to obtain intelligent downhole operation state monitoring data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of monitoring data processing, in particular to an intelligent downhole operation state monitoring system and method. BACKGROUND

[0002] With the complication of mine exploitation and downhole engineering operation, real-time monitoring of downhole operation state has become a core link to ensure safety in production. The traditional monitoring system mainly relies on single modal data (such as video monitoring, environmental sensor data, etc.) for local operation state judgment. However, the downhole operation environment has strong dynamics, multiple interference sources, and limited space (limited space may result in many blind areas in video monitoring), etc. Single data source is easily disturbed by noise, and it is difficult to fully reflect the overall situation of the operation state, and it is difficult to capture the collaborative abnormal state under complex conditions.

[0003] For example, when the motion trajectory of the downhole operation personnel is completely synchronized with the motion trajectory of the downhole coal mine transportation equipment, it indicates that the personnel may have the behavior of riding the coal mine transportation equipment (which is not allowed according to the regulations), and this behavior is difficult to find abnormalities based on single modal data (such as monitoring video). For another example, in the process of coal mining, coal seam collapse will cause a large amount of gas release, therefore, downhole gas concentration exceeding the standard is usually accompanied by a variety of phenomena such as coal dust flying and abnormal noise, and these phenomena may appear as isolated events in single modal data, making it difficult to accurately judge the true state reflected based on single modal data.

[0004] Based on this, it is necessary to study an intelligent downhole operation state monitoring method based on multi-modal monitoring data correlation analysis to capture more subtle abnormal operation states and improve the monitoring accuracy and safety management level of downhole operation state. SUMMARY

[0005] To solve the above problems, one aspect of an embodiment of the present specification provides an intelligent downhole operation state monitoring method, which comprises:

[0006] Obtaining multi-modal monitoring data for reflecting downhole operation state, wherein the multi-modal monitoring data comprises time series data obtained based on video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data and personnel positioning data respectively;

[0007] Performing time-frequency domain fluctuation analysis on each time series data in the multi-modal monitoring data respectively to obtain fluctuation analysis results corresponding to each time series data respectively;

[0008] Calculating the correlation degree between the fluctuation analysis results corresponding to different time series data based on the grey correlation analysis method to obtain fluctuation correlation information containing correlation weight matrix;

[0009] fusing the fluctuation analysis result and corresponding time series data in the multi-modal monitoring data through time window alignment to obtain target monitoring data;

[0010] taking the fluctuation correlation information as a weight parameter of an attention mechanism, performing multi-modal feature coding on the target monitoring data through an encoding network of a pre-trained machine learning model, and outputting a job state representation vector corresponding to the multi-modal monitoring data, wherein the job state representation vector at least contains index data corresponding to a device running state, an environment safety state, and a personnel state;

[0011] inputting the job state representation vector into a decoding network of the pre-trained machine learning model to perform downhole job state anomaly analysis, and obtaining intelligent downhole job state monitoring data.

[0012] In some embodiments, the multi-modal monitoring data reflecting the downhole job state is obtained, including:

[0013] the video monitoring data is collected by a camera array deployed in a downhole tunnel, and attention information is extracted from the video monitoring data by using a multi-target tracking algorithm based on deep learning, to obtain first time series data corresponding to the video monitoring data, wherein the attention information at least includes personnel job specification information, device movement trajectory information, and material stacking state information;

[0014] the sound monitoring data is collected by a distributed acoustic sensor network, and a Mel-frequency cepstral coefficient is combined with a convolutional recurrent neural network to extract a voiceprint feature, to obtain second time series data corresponding to the sound monitoring data.

[0015] In some embodiments, the convolutional recurrent neural network includes a time-frequency graph convolutional layer and a bidirectional LSTM layer, and the voiceprint feature is extracted by using the Mel-frequency cepstral coefficient combined with the convolutional recurrent neural network to obtain the second time series data corresponding to the sound monitoring data, including:

[0016] an acoustic feature template including at least an abnormal sound library, a personnel voice library, and an environmental noise library is constructed;

[0017] short-time spectral features in the sound monitoring data are extracted through the time-frequency graph convolutional layer;

[0018] long-time time series dependent features in the sound monitoring data are obtained by using the bidirectional LSTM layer;

[0019] the extracted short-time spectral features and long-time time series dependent features are matched based on the acoustic feature template to identify event information contained in the sound monitoring data;

[0020] According to the event type, occurrence time and duration state corresponding to the event information, second time sequence data corresponding to the sound monitoring data is obtained.

[0021] In some embodiments, the time-frequency domain fluctuation analysis on each time sequence data in the multi-modal monitoring data respectively comprises:

[0022] The sliding time window algorithm is used to determine the fluctuation feature vector corresponding to each time sequence data in the multi-modal monitoring data in a preset time step time window, wherein the fluctuation feature at least includes the fluctuation amplitude, fluctuation frequency and phase offset corresponding to each time sequence data.

[0023] In some embodiments, the correlation degree between the fluctuation analysis results corresponding to different time sequence data is calculated based on the grey correlation analysis method, to obtain fluctuation correlation information including a correlation weight matrix, comprising:

[0024] The fluctuation analysis results corresponding to each time sequence data are standardized to obtain a standardized fluctuation analysis sequence corresponding to each time sequence data;

[0025] A target sequence and a comparison sequence are determined from the standardized fluctuation analysis sequence;

[0026] The grey correlation coefficients between each comparison sequence and the target sequence are calculated;

[0027] The grey correlation coefficients corresponding to each time point are weightedly averaged to obtain a correlation degree matrix, and the weight coefficients corresponding to each fluctuation feature are determined by the entropy weight method to obtain a weight coefficient matrix;

[0028] The correlation degree matrix and the weight coefficient matrix are subjected to Hadamard product operation to generate a correlation weight matrix reflecting the dynamic coupling relationship between the multi-modal monitoring data.

[0029] In some embodiments, the fluctuation analysis result and the corresponding time sequence data in the multi-modal monitoring data are information fused in a time window alignment manner to obtain a target monitoring data, comprising:

[0030] The fluctuation feature vector in each time window in the fluctuation analysis result is spliced with the original data in the same time window in the corresponding time sequence data in the multi-modal monitoring data to form the target monitoring data with expanded dimensions and enhanced features.

[0031] In some embodiments, the encoding network comprises a multi-modal parallel encoding module, a cross-modal attention fusion module, a feature extraction module and an output module; wherein,

[0032] The multi-modal parallel encoding module comprises branch networks respectively for encoding target monitoring data corresponding to the video monitoring data, the sound monitoring data, the environmental monitoring data, the device operation parameter, the device positioning data and the personnel positioning data;

[0033] The cross-modal attention fusion module is configured to use the fluctuation correlation information as attention weights to dynamically weight and fuse multi-modal features corresponding to the target monitoring data.

[0034] The feature extraction module is configured to perform deep feature extraction on the fused multi-modal features to obtain multi-scale deep features corresponding to the target monitoring data.

[0035] The output module is configured to map the multi-scale deep features into a job state representation vector corresponding to the multi-modal monitoring data, wherein the job state representation vector at least contains index data corresponding to a device operation state, an environmental safety state and a personnel state.

[0036] In some embodiments, the decoding network comprises a deep learning model configured to decode the job state representation vector to output a corresponding downhole job state anomaly analysis result based on the job state representation vector.

[0037] In some embodiments, the machine learning model is obtained by training in the following manner:

[0038] A plurality of sets of sample multi-modal monitoring data under downhole job scenarios are obtained, and actual job states corresponding to each set of sample multi-modal monitoring data are determined.

[0039] Sample target monitoring data corresponding to the sample multi-modal monitoring data are determined, and the actual job states corresponding to the sample multi-modal monitoring data are taken as labels of the sample target monitoring data, to obtain a plurality of training samples.

[0040] The training samples are input into the encoding network and the decoding network of the machine learning model for processing, to obtain a downhole job state anomaly analysis result.

[0041] The network parameters of the encoding network and the decoding network are optimized based on differences between the downhole job state anomaly analysis result and the labels, until a preset training condition is met, to obtain the trained machine learning model.

[0042] Another aspect of the embodiments of the present specification also provides an intelligent downhole job state monitoring system, which comprises:

[0043] A multi-modal monitoring data acquisition module is configured to acquire multi-modal monitoring data reflecting a downhole operation state, wherein the multi-modal monitoring data comprises time series data obtained based on video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data respectively;

[0044] A fluctuation analysis module is configured to perform time-frequency domain fluctuation analysis on each time series data in the multi-modal monitoring data respectively to obtain fluctuation analysis results corresponding to each time series data respectively;

[0045] A fluctuation correlation information determination module is configured to calculate correlation degrees between fluctuation analysis results corresponding to different time series data based on a grey correlation analysis method to obtain fluctuation correlation information containing an association weight matrix;

[0046] A target monitoring data generation module is configured to perform information fusion on the fluctuation analysis results and corresponding time series data in the multi-modal monitoring data through time window alignment to obtain target monitoring data;

[0047] An operation state representation vector determination module is configured to take the fluctuation correlation information as a weight parameter of an attention mechanism, perform multi-modal feature extraction on the target monitoring data through an encoding network, and output an operation state representation vector corresponding to the multi-modal monitoring data, wherein the operation state representation vector at least contains index data corresponding to an equipment operation state, an environmental safety state, and a personnel state;

[0048] An operation state monitoring data determination module is configured to input the operation state representation vector into a pre-trained machine learning model to perform downhole operation state anomaly analysis and obtain intelligent downhole operation state monitoring data.

[0049] The intelligent downhole operation state monitoring system and method provided by the embodiments of the present specification can bring at least the following beneficial effects:

[0050] (1) By performing fluctuation analysis on each time series data in the multi-modal monitoring data to obtain respective fluctuation analysis results, and then performing information fusion on the fluctuation analysis results and corresponding time series data in the multi-modal monitoring data to obtain target monitoring data, the target monitoring data can contain more features than the original data, thereby facilitating the identification of more subtle downhole operation anomalies;

[0051] (2) The correlation degree between the fluctuation analysis results corresponding to different time series data is calculated by a grey correlation analysis method to obtain fluctuation correlation information containing a correlation weight matrix, then the fluctuation correlation information is used as a weight parameter of an attention mechanism, a multi-modal feature coding is performed on the target monitoring data through an encoding network of a pre-trained machine learning model, and a job state representation vector corresponding to the multi-modal monitoring data is output; finally, the job state representation vector is input into a decoding network of the pre-trained machine learning model for downhole job state anomaly analysis, and intelligent downhole job state monitoring data is obtained, which can realize the identification of abnormal state in combination with the dynamic coupling relationship between monitoring data of different modalities, so as to more sensitively capture the change of downhole job state and more accurately realize the monitoring of downhole job state.

[0052] Additional features will be set forth in the description that follows, and in part will become apparent to those skilled in the art, which features, along with BRIEF DESCRIPTION OF DRAWINGS

[0053] The present specification will be further described in the manner of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same numbers represent the same structures, in which:

[0054] Figure 1 is an exemplary flowchart of an intelligent downhole job state monitoring method according to some embodiments of the present specification;

[0055] Figure 2 is an exemplary sub-step flowchart of an intelligent downhole job state monitoring method according to some embodiments of the present specification;

[0056] Figure 3 is an exemplary training step flowchart of a machine learning model according to some embodiments of the present specification;

[0057] Figure 4 is an exemplary module diagram of an intelligent downhole job state monitoring system according to some embodiments of the present specification. DETAILED DESCRIPTION

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present specification, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some examples or embodiments of the present specification, and for those skilled in the art, the present specification can also be applied to other similar scenarios without creative labor on the basis of these drawings. Unless it is clear from the language environment or otherwise stated, the same reference numbers in the drawings represent the same structures or operations.

[0059] It should be understood that the "system", "device", "unit" and / or "module" used in the present specification is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0060] As shown in the specification and claims, unless the context clearly indicates otherwise, the words "one", "a", "an", and / or "the" do not mean to specify a single number, but also include a plurality. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0061] Flowcharts are used in the present specification to illustrate the operations performed by the system according to the embodiments of the present specification. It should be understood that the preceding or subsequent operations are not necessarily performed in sequence. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or more steps of the operation can be removed from these processes.

[0062] The intelligent downhole operation state monitoring system and method provided by the embodiments of the present specification will be described in detail below in conjunction with the drawings.

[0063] Figure 1 is an exemplary flowchart of the intelligent downhole operation state monitoring method according to some embodiments of the present specification. In some embodiments, the intelligent downhole operation state monitoring method can be executed by processing logic, which can include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (instructions executed on a processing device to simulate the hardware), etc. or any combination thereof. In some embodiments, Figure 1 One or more operations in the flowchart of the intelligent downhole operation state monitoring method shown can be implemented by processing devices and / or terminal devices. For example, the intelligent downhole operation state monitoring method can be stored in the form of computer programs and / or instructions in the storage device, and called and / or executed by the processing device and / or terminal device.

[0064] Referring toFigure 1 The intelligent downhole operation status monitoring method provided in this application embodiment may include the following steps S110 to S160:

[0065] Step S110 involves acquiring multimodal monitoring data reflecting the downhole operation status. This multimodal monitoring data includes time-series data obtained from video monitoring data, sound monitoring data, environmental monitoring data, equipment operating parameters, equipment positioning data, and personnel positioning data. In some embodiments, step S110 can be performed by the multimodal monitoring data acquisition module 210 mentioned below.

[0066] In some embodiments of this application, multimodal monitoring data can be acquired through various sensors and monitoring devices deployed in the downhole working space. These sensors and monitoring devices may include, but are not limited to, video surveillance cameras, sound sensors, environmental monitoring sensors (such as temperature sensors, humidity sensors, gas concentration sensors, dust sensors, etc.), equipment operating parameter monitoring sensors (such as motor current monitoring sensors, vibration monitoring sensors, etc.), equipment positioning devices (such as GPS locators, RFID tags, etc.), and personnel positioning devices (such as positioning wristbands or cards worn by workers, etc.).

[0067] Specifically, in this embodiment, the aforementioned sensors and monitoring equipment can be used to collect video monitoring data, sound monitoring data, environmental monitoring data, equipment operating parameters, equipment positioning data, and personnel positioning data to reflect the status of downhole operations. The environmental monitoring data may include temperature and humidity data, dust detection data, and concentration data of gases such as methane (a major component of natural gas), oxygen, and carbon dioxide.

[0068] In some embodiments, the data collected in real time or periodically by the aforementioned sensors and monitoring devices can be transmitted to a ground monitoring center or cloud server via a network (e.g., a wired network such as Ethernet or industrial bus, or a wireless network such as ZigBee, Bluetooth, or mobile communication networks). Furthermore, to determine the dynamic coupling relationships between various data points in the multimodal monitoring data, in this embodiment, the ground monitoring center or cloud server can process the received multimodal monitoring data according to a preset processing method to obtain the time-series data corresponding to each monitoring data point.

[0069] It can be understood that for data such as environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data, the data can be directly converted into time series data according to the time sequence order and the collected data content, so as to facilitate subsequent data analysis and processing. For multimedia data such as video monitoring data and sound monitoring data, feature extraction needs to be performed through a corresponding algorithm, and the multimedia data is converted into time series data according to the extracted key information and the corresponding time sequence information.

[0070] Exemplarily, in some embodiments of the present application, the video monitoring data can be collected by a camera array deployed in the underground roadway, and a multi-target tracking algorithm based on deep learning is used to extract attention information from the video monitoring data to obtain first time series data corresponding to the video monitoring data, wherein the attention information can at least include personnel operation specification information, equipment movement trajectory information, and material stacking state information. Specifically, in the embodiments of the present application, multiple cameras can work cooperatively to form a monitoring network covering the underground roadway. These cameras can capture video pictures of the underground operation scene in real time, and can ensure as much as possible no dead angle monitoring. Further, in the embodiments of the present application, a multi-target tracking algorithm based on deep learning can be used to process the video monitoring data collected by the camera array, so as to accurately identify and track key targets such as personnel, equipment, and materials in the underground, and then extract attention information related to these targets, such as personnel operation specification information, equipment movement trajectory information, and material stacking state information, and finally combine the key information with the time stamp to obtain the first time series data.

[0071] For example only, in some embodiments, the first time series data can be represented as a time sequence, wherein each time point is associated with underground operation state information reflected by the video monitoring data collected by each camera, such as the position of personnel (wherein the identity of the personnel can be determined through face recognition, and the position of the personnel can be determined through the camera number and the installation position thereof), operation specification information (for example, whether the operation personnel are detected without wearing protective equipment, whether there is an off-duty, sleeping, smoking, or other irregular behavior), movement trajectory information of equipment (for example, by tracking the running trajectory of the underground transport vehicle), and material stacking state information (for example, by determining whether there is irregular stacking, exceeding the safety range, or the like, through the shape, height, and position information of the material pile). In the embodiments of the present application, more technical details about using a multi-target tracking algorithm based on deep learning to extract attention information from the video monitoring data can be regarded as prior art, and will not be discussed in detail in the present specification.

[0072] Further, for the sound monitoring data, in some embodiments of the present application, the sound monitoring data can be collected by a distributed acoustic sensor network, and then a mel-frequency cepstral coefficient combined with a convolutional recurrent neural network can be used for voiceprint feature extraction to obtain the second time series data corresponding to the sound monitoring data.

[0073] It can be understood that the mel-frequency cepstral coefficient is a representation method that can convert a speech signal into a series of parameters reflecting the characteristics of the speech, and the convolutional recurrent neural network can learn and extract key features from these parameters, thereby realizing effective analysis and processing of the sound monitoring data. In this way, abnormal sounds in underground operations, such as abnormal sounds generated by equipment failure or coal collapse, personnel distress sounds, environmental noise (such as sounds generated by normal operation of machinery), etc. can be identified, thereby providing an important basis for timely response and handling of abnormal situations underground.

[0074] Specifically, in the embodiments of the present application, the convolutional recurrent neural network can include a time-frequency graph convolution layer and a bidirectional LSTM (Long Short-Term Memory) layer. The process of using a mel-frequency cepstral coefficient combined with a convolutional recurrent neural network for voiceprint feature extraction to obtain the second time series data corresponding to the sound monitoring data can include the following steps:

[0075] First, an acoustic feature template including at least an abnormal sound library (including abnormal sounds generated by equipment failure and coal collapse, etc.), a personnel voice library (including personnel distress sounds), and an environmental noise library (including sounds generated by normal operation of machinery) is constructed. Then, the short-time spectral features in the sound monitoring data are extracted by the time-frequency graph convolution layer in the convolutional recurrent neural network, and the long-time time series dependent features in the sound monitoring data are obtained by the bidirectional LSTM layer in the convolutional recurrent neural network. Further, the extracted short-time spectral features and long-time time series dependent features can be matched with the voiceprint based on the acoustic feature template, thereby identifying the event information contained in the sound monitoring data. Finally, the second time series data corresponding to the sound monitoring data can be obtained according to the event type, occurrence time and duration state corresponding to the event information. Similarly, in the embodiments of the present application, the second time series data can also be represented as a time series, in which each time point is associated with event information reflected by the sound monitoring data collected by each acoustic sensor (when the sound monitoring data corresponding to a certain time point reflects the absence of any event, a specific code can be used to represent it in the time series).

[0076] In the embodiments of the present application, more details about using the mel-frequency cepstral coefficient combined with a convolutional recurrent neural network to extract a voiceprint feature and obtain second time-series data corresponding to the sound monitoring data can also be regarded as prior art, and will not be discussed in detail in the present specification.

[0077] In step S120, time-frequency domain fluctuation analysis is performed on each time-series data in the multi-modal monitoring data respectively to obtain fluctuation analysis results corresponding to each time-series data respectively. In some embodiments, step S120 can be performed by the fluctuation analysis module 220 mentioned below.

[0078] It can be understood that the fluctuation of the time-series data corresponding to the multi-modal monitoring data described above can reflect the change of the downhole operation state (which can be normal change or abnormal change). For example, in the second time-series data corresponding to the sound monitoring data, abnormal fluctuation can correspond to events such as mechanical failure and abnormal behavior of personnel; in the first time-series data corresponding to the video monitoring data, abnormal fluctuation can correspond to situations such as personnel rule violation and abnormal equipment movement trajectory. In the embodiments of the present application, by performing time-frequency domain fluctuation analysis on each time-series data in the multi-modal monitoring data respectively to obtain fluctuation analysis results corresponding to each time-series data respectively, the change information of the downhole operation state contained in each time-series data corresponding to the multi-modal monitoring data can be mined, and then the dynamic coupling relationship between the monitoring data can be analyzed based on the fluctuation analysis results in subsequent steps.

[0079] Specifically, in some embodiments of the present application, a sliding time window algorithm can be used to determine the fluctuation feature vector corresponding to each time-series data in the multi-modal monitoring data in a preset time step time window, wherein the fluctuation feature at least includes the fluctuation amplitude, fluctuation frequency and phase shift of each time-series data.

[0080] Exemplarily, in some embodiments, a suitable time window length (which can be adjusted according to the actual situation of the change of the downhole operation state) can be set, and by sliding the time window on the time-series data, the fluctuation features (such as the fluctuation amplitude, fluctuation frequency and phase shift) of the time-series data in each time window can be extracted in turn, thereby forming a fluctuation feature vector sequence. It can be understood that these fluctuation feature vectors can not only reflect the local fluctuation characteristics of the time-series data in time, but also provide a basis for subsequent analysis of the dynamic coupling relationship between the monitoring data.

[0081] In some embodiments of the present application, a fluctuation feature matrix can be constructed based on the fluctuation analysis results of each time series data obtained by the above processing, the rows of the matrix can correspond to different time series data, and the columns of the matrix can correspond to the fluctuation features (such as fluctuation amplitude, fluctuation frequency and phase shift) in each time window.

[0082] In step S130, the correlation degrees between the fluctuation analysis results corresponding to different time series data are calculated based on the grey correlation analysis method to obtain fluctuation correlation information including a correlation weight matrix. In some embodiments, step S130 can be performed by the fluctuation correlation information determination module 230 mentioned below.

[0083] In the embodiments of the present application, the correlation degrees between the fluctuation analysis results corresponding to different time series data can be calculated based on the grey correlation analysis method. The higher the correlation degree is, the closer the dynamic coupling relationship between the time series data is (i.e., the stronger the correlation between the monitoring data of different modalities is). Therefore, in the embodiments of the present application, by calculating the correlation degrees between the fluctuation analysis results corresponding to different time series data, more features reflected by the multi-modal monitoring data can be mined, thereby facilitating more accurate identification of the abnormal state of the downhole operation.

[0084] Specifically, in the present specification, the grey correlation analysis method is a mathematical method for evaluating the similarity and correlation degree between different time series data. It can find out the correlation between them and quantify the correlation degree by analyzing the change trend and fluctuation features of the time series data. In the technical solutions of the present application, by using the grey correlation analysis method to process and analyze the multi-modal monitoring data collected by the downhole operation state monitoring system, the potential relationship and regularity between these monitoring data can be revealed, thereby more deeply understanding the change pattern of the downhole operation state and improving the accuracy and sensitivity of the abnormal state identification.

[0085] Figure 2 is an exemplary sub-step flowchart of the intelligent downhole operation state monitoring method according to some embodiments of the present specification. Referring to Figure 2 In some embodiments, step S130 can include the following sub-steps S131-S135:

[0086] Sub-step S131, standardizing the fluctuation analysis results corresponding to each time series data to obtain a standardized fluctuation analysis sequence corresponding to each time series data.

[0087] Sub-step S132, determining a target sequence and a comparison sequence from the standardized fluctuation analysis sequence.

[0088] Sub-step S133, calculating the grey correlation coefficients between each comparison sequence and the target sequence.

[0089] In sub-step S134, the correlation degree matrix is obtained by weighted average of the grey correlation coefficients corresponding to each time point, and the weight coefficient matrix is obtained by determining the weight coefficients corresponding to each fluctuation feature through the entropy weight method.

[0090] In sub-step S135, the correlation weight matrix reflecting the dynamic coupling relationship between the multi-modal monitoring data is generated by Hadamard product operation of the correlation degree matrix and the weight coefficient matrix.

[0091] Specifically, in sub-step S131, since the fluctuation analysis results corresponding to the time series data at different time sequences may have different dimensions, orders of magnitude and feature dimensions, direct comparison may lead to inaccurate results. Therefore, the fluctuation analysis results corresponding to each time series data need to be standardized to eliminate the influence caused by the differences in dimensions, orders of magnitude and feature dimensions, so as to obtain the standardized fluctuation analysis sequence corresponding to each time series data. For example, in some embodiments of the present application, for dimension standardization, the Z-score standardization method can be used to convert the fluctuation analysis results corresponding to each time series data into standardized data with a mean of 0 and a standard deviation of 1. For order of magnitude standardization, logarithmic transformation, exponential transformation and other methods can be used to convert the fluctuation analysis results of each time series data into the same order of magnitude range. For feature dimension standardization, principal component analysis (PCA), linear discriminant analysis (LDA) and other methods can be used to reduce the dimension of the fluctuation analysis results of each time series data to extract main features and eliminate redundant information, or the dimension of the data in other fluctuation analysis results can be expanded based on the maximum number of feature dimensions to obtain fluctuation analysis results with the same dimension.

[0092] It can be understood that, in the embodiments of the present application, by standardizing the fluctuation analysis results corresponding to the time series data at different time sequences, the fluctuation analysis results corresponding to each time series data can be made comparable and consistent, thereby providing more accurate and reliable data basis for subsequent analysis and comparison.

[0093] In sub-step S132, the target sequence can be understood as the standardized fluctuation analysis sequence corresponding to the target time series data to be analyzed, and the comparison sequence can be understood as the standardized fluctuation analysis sequence corresponding to other time series data.

[0094] In sub-step S133, the calculation process of the grey correlation coefficient can be represented as follows:

[0095] Suppose the target sequence is: X0={x0(1), x0(2), …, x0(n)}

[0096] The comparison sequence is: X i ={x i (1), xi (2),…,x i (n)}

[0097] The absolute difference value of the comparison sequence and the target sequence at each time point (k) can be calculated as follows:

[0098] Δ i (k)=|x'0(k)-x i (k)|

[0099] Further, the difference matrix Δ=[Δ i (k)] with the dimension of m×n (m is the number of comparison sequences, and n is the number of time points) can be obtained. Then, the global minimum difference Δ min =min i,k Δ i (k) and the global maximum difference Δ max =max i,k Δ i (k) are determined.

[0100] Further, the grey correlation coefficient can be expressed as:

[0101]

[0102] wherein γ(x0(k),x i (k)) represents the grey correlation coefficient of the comparison sequence X i at the kth time point and the target sequence X0, and ρ is a resolution coefficient, usually taking a value between 0 and 1 (usually set as 0.5), for adjusting the sensitivity of the difference of the correlation coefficient. The greater the grey correlation coefficient is, the higher the similarity of the comparison sequence to the target sequence at the time point is, i.e., the stronger the correlation is.

[0103] Further, in the sub-step S134, the grey correlation coefficients corresponding to each time point can be weighted and averaged based on the calculation result of the sub-step S133 to obtain a correlation degree matrix, which is used for further analyzing the overall correlation between the comparison sequence and the target sequence.

[0104] Meanwhile, in the embodiments of the present application, the weight coefficient matrix can be obtained by determining the weight coefficients corresponding to each fluctuation feature through the entropy weight method. Specifically, the information amount and uncertainty of each fluctuation feature are measured by calculating the entropy value of each fluctuation feature. The smaller the entropy value is, the greater the information amount contained in the fluctuation feature is, and the stronger the reflecting ability of the downhole operation state is, so a greater weight coefficient should be given. Conversely, the fluctuation feature with a greater entropy value contains less information and has weaker reflecting ability of the downhole operation state, so the weight coefficient should be correspondingly reduced. After obtaining the entropy values of the fluctuation features, the weight coefficients corresponding to each fluctuation feature can be obtained through normalization processing, and then the weight coefficient matrix is constructed.

[0105] The correlation degree matrix reflects the set of grey correlation coefficients corresponding to each time point, and the weight coefficient matrix reflects the importance of each fluctuation feature in the overall analysis. Based on this, in sub-step S135, a correlation weight matrix can be obtained by performing Hadamard product operation on the correlation degree matrix and the weight coefficient matrix. This matrix not only reflects the correlation degree between each time series data and the target time series data, but also takes into account the importance of each fluctuation feature, so it can more comprehensively reflect the dynamic coupling relationship between multi-modal monitoring data. In some embodiments of the present application, the fluctuation correlation relationship between each time series data and other data in the multi-modal monitoring data can be obtained according to the correlation weight matrix.

[0106] With reference to the foregoing Figure 1 After step S130, the method further includes:

[0107] Step S140, information fusion of the fluctuation analysis result and the corresponding time series data in the multi-modal monitoring data in a time window alignment manner to obtain target monitoring data. In some embodiments, step S140 can be performed by the target monitoring data generation module 240 mentioned later.

[0108] Specifically, in some embodiments of the present application, the fluctuation feature vector in each time window in the fluctuation analysis result can be spliced with the original data in the same time window of the corresponding time series data in the multi-modal monitoring data to form target monitoring data with expanded dimensions and enhanced features.

[0109] It should be noted that in the embodiments of the present application, by splicing the fluctuation feature vector in each time window in the fluctuation analysis result obtained by the above process with the original data in the same time window of the corresponding time series data in the multi-modal monitoring data, target monitoring data is obtained, which can contain more features than the original data, thereby facilitating the identification of more subtle downhole operation abnormal conditions.

[0110] Step S150, taking the fluctuation correlation information as the weight parameter of the attention mechanism, performing multi-modal feature coding on the target monitoring data through the encoding network of the pre-trained machine learning model, and outputting the operation state representation vector corresponding to the multi-modal monitoring data. In some embodiments, step S150 can be performed by the operation state representation vector determination module 250 mentioned later.

[0111] In the embodiments of the present application, the encoding network of the machine learning model can include a multi-modal parallel encoding module, a cross-modal attention fusion module, a feature extraction module, and an output module. The multi-modal parallel encoding module can include branch networks for encoding target monitoring data corresponding to the video monitoring data, sound monitoring data, environmental monitoring data, device operation parameters, device positioning data, and personnel positioning data, respectively. The cross-modal attention fusion module can be used to dynamically weight and fuse the multi-modal features corresponding to the target monitoring data by using the fluctuation correlation information determined in the above step as the attention weight. The feature extraction module can be used to perform deep feature extraction on the fused multi-modal features to obtain multi-scale deep features corresponding to the target monitoring data. The output module can be used to map the multi-scale deep features to a job state representation vector corresponding to the multi-modal monitoring data. In the embodiments of the present application, the job state representation vector can include at least index data corresponding to the device operation state, the environmental safety state, and the personnel state.

[0112] Specifically, in the embodiments of the present application, the machine learning model can be regarded as an encoder-decoder model, and the encoding network described above can be regarded as the encoder part thereof. The encoding network can be used to extract and encode features of the target monitoring data obtained by processing various monitoring data (such as video monitoring data, sound monitoring data, environmental monitoring data, device operation parameters, device positioning data, and personnel positioning data) for downhole operations in the above process, to form multi-scale deep features. Since these multi-scale deep features can reflect the actual state of downhole operations, based on this, in the embodiments of the present application, the multi-scale deep features can be further mapped to a job state representation vector corresponding to the multi-modal monitoring data. In the embodiments of the present application, the job state representation vector can include at least index data corresponding to the device operation state, the environmental safety state, and the personnel state.

[0113] It should be noted that in the embodiments of the present application, the encoding network can be trained to obtain the processing capability described above. For more details about the training process of the encoding network, please refer to the following (for example, step S160), which will not be described here.

[0114] Step S160, inputting the job state representation vector into the decoding network of the pre-trained machine learning model to perform downhole operation state anomaly analysis, and obtaining intelligent downhole operation state monitoring data. In some embodiments, step S160 can be performed by the job state monitoring data determination module 260 mentioned below.

[0115] In the embodiments of the present application, the decoding network can be regarded as a decoder part in an encoder-decoder model. Specifically, in some embodiments of the present application, the decoding network can include a deep learning model (such as a convolutional neural network CNN or a recurrent neural network RNN, etc.), which can be used to decode the input job state representation vector and analyze whether there is an abnormality in the downhole job state according to the decoding result. In the embodiments of the present application, the decoding network can be trained to obtain the above processing capability.

[0116] The following will be described in combination with Figure 3 The training process of the machine learning model in the embodiments of the present application will be briefly introduced as follows:

[0117] Figure 3 is an exemplary training step flowchart of the machine learning model according to some embodiments of the present application. Refer to Figure 3 In the embodiments of the present application, the training process of the machine learning model can include the following steps S101-S104:

[0118] Step S101, a plurality of sets of sample multi-modal monitoring data under downhole job scenes are obtained, and the actual job state corresponding to each set of sample multi-modal monitoring data is determined.

[0119] Step S102, the sample target monitoring data corresponding to the sample multi-modal monitoring data is determined based on the sample multi-modal monitoring data, and the actual job state corresponding to the sample multi-modal monitoring data is taken as the label of the sample target monitoring data, thereby obtaining a plurality of training samples.

[0120] Step S103, the training samples are input into the encoding network and the decoding network of the machine learning model for processing, thereby obtaining a downhole job state abnormality analysis result.

[0121] Step S104, the network parameters of the encoding network and the decoding network are optimized based on the difference between the downhole job state abnormality analysis result and the label, until a preset training condition is met, thereby obtaining a trained machine learning model.

[0122] It should be noted that in step S102, the sample target monitoring data corresponding to the sample multi-modal monitoring data can be determined with reference to the above steps S110-S140, which will not be described herein again.

[0123] In step S104, a gradient descent algorithm or other optimization algorithm can be used to minimize the difference between the downhole job state abnormality analysis result and the label, thereby continuously adjusting the network parameters of the encoding network and the decoding network to improve the prediction accuracy of the machine learning model.

[0124] In some embodiments of the present application, the preset training condition can be determined to be met when the difference between the downhole operation state anomaly analysis result and the label is less than a preset threshold or reaches a preset iteration number, and the trained machine learning model is obtained. More details about the machine learning model can be regarded as prior art, and will not be described in detail in the present specification.

[0125] It should be noted that, in the embodiments of the present application, the target monitoring data obtained by the above process can be processed by the machine learning model to more sensitively capture the changes in the downhole operation state, thereby more accurately monitoring the downhole operation state.

[0126] It can be understood that such accurate monitoring capability is of great significance to improve the safety and management efficiency of downhole operation. In the embodiments of the present application, the downhole operation state is monitored in real time by the above scheme, and potential safety hazards are discovered and handled in a timely manner, which can effectively avoid the occurrence of downhole accidents, thereby better protecting the safety of downhole operation personnel and equipment.

[0127] Figure 4 is a schematic diagram of the intelligent downhole operation state monitoring system according to some embodiments of the present specification. In some embodiments, Figure 4 The intelligent downhole operation state monitoring system 200 shown in the figure can be implemented in software and / or hardware, for example, it can be configured in the form of software and / or hardware to a processing device and / or terminal device for processing the collected multi-modal monitoring data, thereby realizing downhole operation state anomaly analysis.

[0128] Referring to Figure 4 In some embodiments, the intelligent downhole operation state monitoring system 200 can include a multi-modal monitoring data acquisition module 210, a fluctuation analysis module 220, a fluctuation correlation information determination module 230, a target monitoring data generation module 240, an operation state representation vector determination module 250, and an operation state monitoring data determination module 260.

[0129] The multi-modal monitoring data acquisition module 210 can be used to acquire multi-modal monitoring data reflecting the downhole operation state, wherein the multi-modal monitoring data includes time series data obtained based on video monitoring data, sound monitoring data, environmental monitoring data, device operating parameters, device positioning data, and personnel positioning data.

[0130] The fluctuation analysis module 220 can be used to perform time-frequency domain fluctuation analysis on each time series data in the multi-modal monitoring data respectively, and obtain a fluctuation analysis result corresponding to each time series data respectively.

[0131] The fluctuation correlation information determination module 230 can be configured to calculate the correlation degree between fluctuation analysis results corresponding to different time series data based on the grey correlation analysis method, and obtain fluctuation correlation information including a correlation weight matrix.

[0132] The target monitoring data generation module 240 can be configured to perform information fusion on the fluctuation analysis results and corresponding time series data in the multi-modal monitoring data in a time window alignment manner, and obtain target monitoring data.

[0133] The job state representation vector determination module 250 can be configured to use the fluctuation correlation information as a weight parameter of an attention mechanism, perform multi-modal feature coding on the target monitoring data through an encoding network of a pre-trained machine learning model, and output a job state representation vector corresponding to the multi-modal monitoring data.

[0134] The job state monitoring data determination module 260 can be configured to input the job state representation vector into a decoding network of the pre-trained machine learning model to perform downhole job state anomaly analysis, and obtain intelligent downhole job state monitoring data.

[0135] Further details about the above-mentioned modules can be referred to other places in this specification (for example Figures 1 to 3 Part and its related description), which will not be repeated here.

[0136] It should be understood that Figure 4 The illustrated intelligent downhole job state monitoring system 200 and its modules can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software or a combination of software and hardware. The hardware part can be implemented by special logic; the software part can be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or a specially designed hardware. Those skilled in the art can understand that the above-mentioned system and method can be implemented by computer executable instructions and / or included in processor control code, such as provided on a carrier medium, such as a magnetic disk, CD or DVD-ROM, a programmable memory, such as a read-only memory (firmware), or a data carrier, such as an optical or electronic signal carrier. The system and its modules of this specification can not only have hardware circuit implementation, such as very large scale integrated circuit or gate array, semiconductor, such as logic chip, transistor, or programmable hardware device, such as field programmable gate array, programmable logic device, etc., but also can be implemented by software, for example, executed by various types of processors, and also can be implemented by a combination of the above-mentioned hardware circuit and software (for example, firmware).

[0137] It should be noted that the above description of the intelligent downhole operation state monitoring system 200 is provided for illustrative purposes only and is not intended to limit the scope of the present specification. It can be understood that, for those skilled in the art, any combination of the various modules or connection of the subsystems can be made without departing from the principles of the present specification according to the description of the present specification. For example, Figure 4 The multi-modal monitoring data acquisition module 210, the fluctuation analysis module 220, the fluctuation correlation information determination module 230, the target monitoring data generation module 240, the operation state representation vector determination module 250, and the operation state monitoring data determination module 260 described in the above can be different modules in a system, or a module can implement the functions of two or more modules described above. Variations such as this are within the scope of protection of the present specification. In some embodiments, the various modules described above can be part of a processing device and / or a terminal device.

[0138] In summary, the beneficial effects that may be brought about by the embodiments of the present specification include but are not limited to: (1) In the intelligent downhole operation state monitoring system and method provided in some embodiments of the present specification, by performing fluctuation analysis on each time series data in multi-modal monitoring data to obtain a respective fluctuation analysis result, and then performing information fusion on the fluctuation analysis result and the corresponding time series data in the multi-modal monitoring data to obtain target monitoring data, the target monitoring data can contain more features than the original data, thereby facilitating the identification of more subtle downhole operation abnormal conditions; (2) In the intelligent downhole operation state monitoring system and method provided in some embodiments of the present specification, by calculating the correlation degree between the fluctuation analysis results corresponding to different time series data based on the grey correlation analysis method to obtain fluctuation correlation information containing a correlation weight matrix, then using the fluctuation correlation information as a weight parameter of an attention mechanism, and performing multi-modal feature encoding on the target monitoring data through the encoding network of a pre-trained machine learning model, outputting an operation state representation vector corresponding to the multi-modal monitoring data; Finally, inputting the operation state representation vector into the decoding network of the pre-trained machine learning model for downhole operation state anomaly analysis to obtain intelligent downhole operation state monitoring data, the dynamic coupling relationship between the monitoring data of different modalities can be combined to realize the identification of abnormal states, thereby more sensitively capturing the changes in the downhole operation state and more accurately monitoring the downhole operation state.

[0139] It should be noted that different embodiments can produce different beneficial effects, and in different embodiments, the beneficial effects that can be produced can be any one or a combination of the above, or any other beneficial effects that can be obtained.

[0140] Having described the basic concepts, it is obvious to those skilled in the art that the foregoing detailed description is merely illustrative and not restrictive of the present specification. Various modifications, improvements, and changes can be made to the present specification by those skilled in the art, although not explicitly described herein. Such modifications, improvements, and changes are suggested by the present specification, and thus still fall within the spirit and scope of the exemplary embodiments of the present specification.

[0141] Meanwhile, specific words are used in the present specification to describe the embodiments of the present specification. As "one embodiment", "an embodiment", and / or "some embodiments" mean a certain feature, structure, or characteristic in connection with at least one embodiment of the present specification. Therefore, it should be emphasized and noted that "an embodiment" or "one embodiment" or "an alternative embodiment" mentioned in different places in the present specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the present specification can be properly combined.

[0142] In addition, those skilled in the art can understand that aspects of the present specification can be described and claimed in a broad form or in a variety of specific forms, including any new and useful processes, machines, products, or compositions of matter, or any new and useful improvements thereof. Accordingly, various aspects of the present specification can be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software can be referred to as "data blocks", "modules", "engines", "units", "components", or "systems". In addition, aspects of the present specification can be embodied as a computer product located in one or more computer readable media, including computer readable program code.

[0143] The computer storage media can include a propagated data signal with computer program code embodied therein, e.g., in baseband or as part of a carrier wave. Such propagated signal can take a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. Computer storage media can be any computer readable medium except for a propagating transitory, signal per se. The computer storage media can be any computer readable medium that can be accessed by a general purpose or special purpose computing system, apparatus, or device to implement the techniques for which that computer storage media contains or stores program code. Program code embodied on a computer storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, or any suitable combination of the foregoing.

[0144] Computer program code for carrying out operations of the aspects of the present specification can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, and conventional procedural programming languages, such as the "C" programming language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, dynamic programming languages, such as Python, Ruby and Groovy, or other programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic program code can be downloaded from an on-demand computing platform, such as Amazon Web Services, Microsoft Azure, or Google Cloud Platform, or a service provider, such as Salesforce.com. In some embodiments, the electronic program code can be downloaded from a cloud computing environment, or used as a service, such as Software as a Service (SaaS).

[0145] In addition, the order of execution or sequence of any of the processes depicted in or comprising the embodiments of the disclosure, and that discussed in the description of the disclosure is not limiting. Like numbers refer to like elements throughout. Although the above description may

[0146] It should also be noted that, as used in the description of the embodiments of the present disclosure and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" or "the component" can include within its scope a plurality of such components, and reference to "an element" or "the element" can include within its scope a plurality of such elements. Similarly, for purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it is to be understood that the present disclosure can be practiced without the specific details. In other instances, well-known methods, procedures, components, and networks have not been described in detail so as not to unnecessarily obscure aspects of the present disclosure.

[0147] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0148] For each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, and documents, referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0149] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. An intelligent downhole operation state monitoring method, characterized in that, The method comprises the following steps: acquiring multi-modal monitoring data reflecting the state of downhole operation, wherein the multi-modal monitoring data comprises time series data obtained based on video monitoring data, sound monitoring data, environmental monitoring data, equipment operating parameters, equipment positioning data and personnel positioning data respectively; performing time-frequency domain fluctuation analysis on each time series data in the multi-modal monitoring data respectively to obtain fluctuation analysis results corresponding to each time series data respectively; calculating the correlation degree between fluctuation analysis results corresponding to different time series data based on a grey correlation analysis method to obtain fluctuation correlation information containing a correlation weight matrix, wherein the different time series data are time series data corresponding to monitoring data of different modalities in the multi-modal monitoring data; fusing the fluctuation analysis results and corresponding time series data in the multi-modal monitoring data through time window alignment to obtain target monitoring data; using the fluctuation correlation information as a weight parameter of an attention mechanism, performing multi-modal feature coding on the target monitoring data through an encoding network of a pre-trained machine learning model, and outputting an operation state representation vector corresponding to the multi-modal monitoring data, wherein the operation state representation vector at least contains index data corresponding to equipment operating state, environmental safety state and personnel state; inputting the operation state representation vector into a decoding network of the pre-trained machine learning model to perform downhole operation state anomaly analysis and obtain intelligent downhole operation state monitoring data; wherein the calculation of the correlation degree between fluctuation analysis results corresponding to different time series data based on the grey correlation analysis method to obtain the fluctuation correlation information containing the correlation weight matrix comprises: performing standardization processing on the fluctuation analysis results corresponding to each time series data to obtain a standardized fluctuation analysis sequence corresponding to each time series data; determining a target sequence and comparison sequences from the standardized fluctuation analysis sequence; calculating the grey correlation coefficients between each comparison sequence and the target sequence; performing weighted average on the grey correlation coefficients corresponding to each time point to obtain a correlation degree matrix, and determining a weight coefficient matrix by an entropy weight method, wherein the weight coefficient matrix corresponds to the weight coefficients of each fluctuation feature; performing Hadamard product operation on the correlation degree matrix and the weight coefficient matrix to generate a correlation weight matrix reflecting the dynamic coupling relationship between the multi-modal monitoring data.

2. The method of claim 1, wherein, The acquisition of the multi-modal monitoring data reflecting the state of downhole operation comprises: acquiring the video monitoring data through a camera array deployed in a downhole tunnel, and extracting attention information from the video monitoring data by using a multi-target tracking algorithm based on deep learning to obtain first time series data corresponding to the video monitoring data, wherein the attention information at least includes personnel operation specification information, equipment movement trajectory information and material stacking state information; acquiring the sound monitoring data through a distributed acoustic sensor network, and extracting voiceprint features by using a mel-frequency cepstrum coefficient combined with a convolutional recurrent neural network to obtain second time series data corresponding to the sound monitoring data.

3. The method of claim 2, wherein, The convolutional recurrent neural network comprises a time-frequency graph convolution layer and a bidirectional LSTM layer, the voiceprint feature extraction is performed by adopting a mel-frequency cepstral coefficient in combination with a convolutional recurrent neural network, second time sequence data corresponding to the sound monitoring data is obtained, and the second time sequence data comprises the following steps: An acoustic feature template including an abnormal sound library, a personnel voice library, and an environmental noise library is constructed; Short-time frequency spectrum features in the sound monitoring data are extracted through the time-frequency graph convolution layer; Long-time time sequence dependent features in the sound monitoring data are obtained by using the bidirectional LSTM layer; Voiceprint matching is performed on the extracted short-time frequency spectrum features and long-time time sequence dependent features based on the acoustic feature template, and event information contained in the sound monitoring data is identified; Second time sequence data corresponding to the sound monitoring data is obtained according to an event type, an occurrence time, and a continuous state corresponding to the event information.

4. The method of claim 1, wherein, The time-frequency domain fluctuation analysis is respectively performed on each time sequence data in the multi-modal monitoring data, and the time-frequency domain fluctuation analysis comprises the following steps: A sliding time window algorithm is used to determine a fluctuation feature vector corresponding to each time sequence data in the multi-modal monitoring data in a preset time step time window, wherein the fluctuation feature at least includes a fluctuation amplitude, a fluctuation frequency, and a phase offset corresponding to each time sequence data.

5. The method of claim 4, wherein, The fluctuation analysis result is fused with the corresponding time sequence data in the multi-modal monitoring data in a time window alignment manner to obtain target monitoring data, and the fusion comprises the following steps: The fluctuation feature vector in each time window in the fluctuation analysis result is spliced with original data in the same time window in the corresponding time sequence data in the multi-modal monitoring data to form target monitoring data with expanded dimensions and enhanced features.

6. The method of claim 1, wherein, The encoding network comprises a multi-modal parallel encoding module, a cross-modal attention fusion module, a feature extraction module, and an output module, wherein The multi-modal parallel encoding module comprises branch networks respectively used for encoding target monitoring data corresponding to the video monitoring data, the sound monitoring data, the environmental monitoring data, the device operation parameter, the device positioning data, and the personnel positioning data; The cross-modal attention fusion module is used for taking the fluctuation correlation information as an attention weight to dynamically weight and fuse multi-modal features corresponding to the target monitoring data; The feature extraction module is used for performing deep feature extraction on the fused multi-modal features to obtain multi-scale deep features corresponding to the target monitoring data; The output module is used for mapping the multi-scale deep features into a job state representation vector corresponding to the multi-modal monitoring data, wherein the job state representation vector at least includes index data corresponding to a device operation state, an environmental safety state, and a personnel state.

7. The method of claim 6, wherein, The decoding network comprises a deep learning model, the deep learning model is used for decoding the job state representation vector to output an actual job state abnormality analysis result corresponding to the job state representation vector.

8. The method of claim 7, wherein, The machine learning model is obtained by training in the following manner: A plurality of groups of sample multi-modal monitoring data in downhole operation scenes are obtained, and actual job states corresponding to each group of sample multi-modal monitoring data are determined. determine corresponding sample target monitoring data based on the sample multi-modal monitoring data, and take an actual operation state corresponding to the sample multi-modal monitoring data as a label of the sample target monitoring data, to obtain a plurality of training samples; input the training samples into an encoding network and a decoding network of the machine learning model for processing, to obtain an underground operation state abnormality analysis result; optimize network parameters of the encoding network and the decoding network based on a difference between the underground operation state abnormality analysis result and the label, until a preset training condition is met, to obtain the trained machine learning model.

9. An intelligent downhole operation state monitoring system, characterized by, Comprise: a multi-modal monitoring data acquisition module configured to acquire multi-modal monitoring data reflecting an underground operation state, wherein the multi-modal monitoring data comprises time series data obtained based on video monitoring data, sound monitoring data, environmental monitoring data, equipment operation parameters, equipment positioning data, and personnel positioning data respectively; a fluctuation analysis module configured to perform time-frequency domain fluctuation analysis on each time series data in the multi-modal monitoring data respectively, to obtain a fluctuation analysis result corresponding to each time series data respectively; a fluctuation correlation information determination module configured to calculate correlation degrees between fluctuation analysis results corresponding to different time series data based on a grey correlation analysis method, to obtain fluctuation correlation information comprising an association weight matrix, wherein the different time series data are time series data corresponding to monitoring data of different modalities in the multi-modal monitoring data; a target monitoring data generation module configured to fuse the fluctuation analysis results and corresponding time series data in the multi-modal monitoring data through time window alignment, to obtain target monitoring data; an operation state representation vector determination module configured to take the fluctuation correlation information as a weight parameter of an attention mechanism, to perform multi-modal feature extraction on the target monitoring data through an encoding network, and to output an operation state representation vector corresponding to the multi-modal monitoring data, wherein the operation state representation vector at least comprises index data corresponding to an equipment operation state, an environmental safety state, and a personnel state; an operation state monitoring data determination module configured to input the operation state representation vector into a pre-trained machine learning model to perform underground operation state abnormality analysis, to obtain intelligent underground operation state monitoring data; The fluctuation correlation information determination module is specifically configured to: perform standardization processing on the fluctuation analysis results corresponding to each time series data, to obtain a standardized fluctuation analysis sequence corresponding to each time series data; determine a target sequence and comparison sequences from the standardized fluctuation analysis sequence; calculate grey correlation coefficients between each comparison sequence and the target sequence; perform weighted average on the grey correlation coefficients corresponding to each time point to obtain a correlation degree matrix, and determine weight coefficients corresponding to each fluctuation feature through an entropy weight method to obtain a weight coefficient matrix; perform Hadamard product operation on the correlation degree matrix and the weight coefficient matrix, to generate an association weight matrix reflecting a dynamic coupling relationship between the multi-modal monitoring data.

Images