A coal mine digital twin cross-modal self-supervised learning method and system based on vision and sound

Abstract

The application discloses a coal mine digital twin cross-modal self-supervised learning method and system based on vision and sound, which comprises the following steps: acquiring a continuous time sequence image sequence and a continuous time sequence acoustic signal, constructing a three-dimensional digital twin model, and mapping the continuous time sequence acoustic signal to the three-dimensional digital twin model; constructing a positive sample pair according to a visual event and an acoustic event at the same time and at the same spatial position, and constructing a negative sample pair according to a visual event and an acoustic event at different times or at different spatial positions; training a cross-modal correlation network with the positive sample pair and the negative sample pair, learning a mapping relationship between a visual feature space and an acoustic feature space; and deploying the trained cross-modal correlation network to the three-dimensional digital twin model. The coal mine intelligent scheduling platform is constructed by fusing the three-dimensional digital twin and the vision-acoustic cross-modal mutual calibration, and multi-dimensional accurate perception of an underground environment, intelligent abnormal early warning and scheduling auxiliary decision-making are realized through self-supervised collaborative learning.

Description

Technical Field

[0001] This invention relates to the field of coal mining technology, and in particular to a cross-modal self-supervised learning method and system for digital twins in coal mines based on vision and sound. Background Technology

[0002] In recent years, machine learning and pre-trained model technologies have made significant progress in fields such as industrial vision and speech recognition. In the intelligent construction of coal mines, deep learning-based visual perception models have been widely applied to tasks such as underground personnel identification and equipment status monitoring. Pre-trained models, learning prior knowledge from large-scale general datasets, have alleviated the problem of insufficient labeled data for coal mine scenarios to some extent. Meanwhile, the rapid development of digital twin technology has provided a technological foundation for the deep integration of the physical and information worlds of coal mines. By constructing a three-dimensional digital twin model of the mine, the visualization and dynamic monitoring of the production environment, equipment status, and personnel activities can be achieved.

[0003] However, existing intelligent coal mine technology solutions still have the following shortcomings: the underground environment of coal mines is complex and changeable, and visual perception is easily affected by factors such as insufficient lighting and dust obstruction. Single-modal perception methods cannot guarantee reliable monitoring in all times and all areas; existing cross-modal perception solutions are mostly static fusion methods, lacking the ability to deeply explore and dynamically align different modal information such as visual and acoustic information in the spatiotemporal dimension, making it difficult to achieve collaborative understanding and mutual calibration of multimodal information; the deployment of machine learning models in coal mine scenarios heavily relies on a large amount of manually labeled data, while underground abnormal working condition samples are scarce and labeling costs are high. The fine-tuning effect of pre-trained models in the coal mine field is limited by domain differences, making it difficult to achieve continuous self-optimization and adaptive updates of the models, thus limiting the generalization ability and long-term service performance of intelligent perception systems.

[0004] To address the aforementioned issues, this invention provides a cross-modal self-supervised learning method and system for digital twins in coal mines based on vision and sound. This method enables self-supervised collaborative learning of visual and acoustic perception in underground coal mines, automatically mining the spatiotemporal correlations of multimodal data without relying on extensive manual annotation, and significantly reducing model deployment and maintenance costs. Summary of the Invention

[0005] In view of the problems existing in the current intelligent coal mine technology, this invention is proposed.

[0006] Therefore, the purpose of this invention is to provide a cross-modal self-supervised learning method and system for coal mine digital twins based on vision and sound. The purpose is to construct a coal mine intelligent scheduling platform that integrates three-dimensional digital twins and vision-acoustic cross-modal mutual calibration, and to achieve multi-dimensional accurate perception of the underground environment, intelligent early warning of anomalies, and scheduling auxiliary decision-making through self-supervised collaborative learning of visual and acoustic information.

[0007] To address the aforementioned technical problems, this invention provides a cross-modal self-supervised learning method for digital twins in coal mines based on vision and sound, including acquiring a continuous temporal sequence of time-aligned images and continuous temporal acoustic signals of a coal mine scene.

[0008] A three-dimensional digital twin model is constructed based on the continuous time-series image sequence, and the continuous time-series acoustic signal is mapped to the coordinate space of the three-dimensional digital twin model and time-frequency domain features are extracted.

[0009] Positive sample pairs are constructed based on visual and acoustic events occurring at the same time and in the same spatial location, while negative sample pairs are constructed based on visual and acoustic events occurring at different times or in different spatial locations.

[0010] The cross-modal association network is trained using the positive and negative sample pairs to learn the mapping relationship between the visual feature space and the acoustic feature space, enabling visual events and acoustic events to achieve mutual representation based on spatiotemporal consistency in the shared embedding space.

[0011] The trained cross-modal association network is deployed on the three-dimensional digital twin model to achieve real-time mutual calibration between the visual and acoustic modalities.

[0012] As a preferred embodiment of the cross-modal self-supervised learning method for coal mine digital twins according to the present invention, the step of constructing a three-dimensional digital twin model based on the continuous temporal image sequence includes:

[0013] Local 3D point clouds are generated frame by frame from the continuous temporal image sequence. The local 3D point clouds are recursively extrapolated to the global coordinate system through inter-frame pose estimation and stitched together to form a global 3D point cloud model. The global 3D point cloud model is then subjected to surface mesh reconstruction and texture mapping to generate the 3D digital twin model. The 3D digital twin model is dynamically updated in response to the streaming input of the continuous temporal image sequence.

[0014] As a preferred embodiment of the cross-modal self-supervised learning method for coal mine digital twins described in this invention, the texture mapping includes:

[0015] The texture feature points of each frame in the continuous temporal image sequence are obtained, and the texture feature points are matched with the spatial points in the global three-dimensional point cloud model to establish the mapping relationship between the pixels of the image and the three-dimensional spatial points. The texture information of multiple frames of images is fused and mapped onto the surface of the three-dimensional model after the surface mesh is reconstructed to generate a three-dimensional digital twin model with real texture.

[0016] As a preferred embodiment of the cross-modal self-supervised learning method for coal mine digital twins according to the present invention, the step of mapping the continuous temporal acoustic signal to the coordinate space of the three-dimensional digital twin model includes:

[0017] An underground-specific acoustic propagation model is established based on the tunnel geometry information in the three-dimensional digital twin model. The underground-specific acoustic propagation model includes the tunnel wall reflection coefficient and the tunnel cross-section attenuation coefficient.

[0018] The downhole-specific acoustic propagation model performs roadway reflection compensation and roadway attenuation compensation on the arrival time difference and arrival intensity difference of the acoustic signal, and calculates the three-dimensional spatial location of the acoustic event in the three-dimensional digital twin model based on the compensated parameters.

[0019] As a preferred embodiment of the cross-modal self-supervised learning method for coal mine digital twins described in this invention, constructing the positive sample pairs and the negative sample pairs includes:

[0020] The image region of the visual event is detected and located from the continuous temporal image sequence using a target detection algorithm, and the image region is mapped to the coordinate space of the three-dimensional digital twin model.

[0021] The time interval and spatial location of the acoustic event are detected and located from the continuous temporal acoustic signal by time-frequency domain feature extraction and sound source localization algorithms.

[0022] The visual events with a time difference less than a time threshold and a spatial distance less than a distance threshold are paired with the acoustic events to form the positive sample pairs.

[0023] The visual event with a time difference greater than or equal to the time threshold or a spatial distance greater than or equal to the distance threshold is paired with the acoustic event to form the negative sample pair.

[0024] As a preferred embodiment of the cross-modal self-supervised learning method for digital twins in coal mines according to the present invention, the cross-modal association network includes a temporal convolutional network and a Transformer network. The temporal convolutional network extracts the local temporal features of the continuous temporal image sequence and the continuous temporal acoustic signal, and the Transformer network captures the global temporal dependencies of the continuous temporal image sequence and the continuous temporal acoustic signal.

[0025] As a preferred embodiment of the cross-modal self-supervised learning method for digital twins in coal mines described in this invention, the real-time mutual calibration of visual and acoustic modalities includes:

[0026] The visual event is positively predicted to acoustic features through the cross-modal association network. The predicted acoustic features are compared with the actual acoustic signal. If the deviation exceeds a first threshold, it is judged as an anomaly.

[0027] The back-prediction of acoustic events to visual features is performed through the cross-modal association network. The predicted visual features are compared with the actual image. If the deviation exceeds a second threshold, it is determined to be a type II anomaly.

[0028] The first type of anomaly and the second type of anomaly are located and displayed in the three-dimensional digital twin model.

[0029] As a preferred embodiment of the cross-modal self-supervised learning method for coal mine digital twins according to the present invention, it further includes: when the deviation between the visual event and the acoustic event through the forward prediction and the backward prediction are within the first threshold and the second threshold respectively, but the spatial distance between the acoustic event and the visual event in the three-dimensional digital twin model exceeds the third threshold, it is determined to be a type three anomaly.

[0030] As a preferred embodiment of the cross-modal self-supervised learning method for digital twins in coal mines according to the present invention, the method further includes: mapping the spatial localization result of the acoustic event to the three-dimensional digital twin model to generate a dynamic acoustic event layer; when the acoustic event is located in a first spatial region, controlling the visual acquisition device to increase the acquisition frequency and resolution of the first spatial region; when the acoustic event localization fails, predicting the compensated localization of the acoustic event based on the visual event through the cross-modal association network.

[0031] A cross-modal self-supervised learning system for coal mine digital twins based on vision and sound, employing the aforementioned cross-modal self-supervised learning method for coal mine digital twins, is characterized by comprising:

[0032] The visual acquisition module is used to acquire continuous time-series image sequences of coal mine scenes in real time.

[0033] An acoustic acquisition module is used to synchronously acquire continuous temporal acoustic signals that are time-aligned with the continuous temporal image sequence.

[0034] A 3D digital twin construction module is used to construct and dynamically update the 3D digital twin model based on the continuous temporal image sequence.

[0035] The cross-modal association learning module is used to construct positive sample pairs based on visual events and acoustic events at the same time and in the same spatial location, and to construct negative sample pairs based on visual events and acoustic events at different times or in different spatial locations, and to train the cross-modal association network with the positive sample pairs and the negative sample pairs.

[0036] The cross-modal mutual calibration and anomaly detection module is used to deploy the trained cross-modal association network on the three-dimensional digital twin model; to perform forward prediction of the visual event to acoustic feature through the cross-modal association network, and to perform reverse prediction of the acoustic event to visual feature; to use the deviation of the forward prediction and the deviation of the reverse prediction as anomaly detection signals, to determine the anomaly type and to locate and display it in the three-dimensional digital twin model.

[0037] The control and verification module is used to generate device control commands based on the anomaly detection signal, and to acquire image sequences and acoustic signals after command execution through the visual acquisition module and the acoustic acquisition module to verify the execution effect of the device control commands. The verification results are then fed back to the cross-modal association learning module to iteratively optimize the cross-modal association network.

[0038] The beneficial effects of this invention are as follows: The technical solution of this invention constructs a spatiotemporally consistent visual-acoustic cross-modal self-supervised learning framework, which can automatically learn the intrinsic correlation of multimodal information without extensive manual annotation. This effectively overcomes the problem of scarce labeled samples in the complex environment of underground coal mines and significantly reduces model deployment and maintenance costs. This method uses an underground-specific acoustic propagation model established based on roadway geometry information to accurately compensate for roadway reflection and attenuation of acoustic signals, greatly improving the spatial positioning accuracy of acoustic events in the 3D digital twin model and providing a reliable foundation for spatial alignment and collaborative perception of multimodal information. This method constructs a multi-level anomaly detection system through a visual-acoustic bidirectional mutual calibration mechanism, which can effectively identify multiple anomaly states where visual and acoustic information are inconsistent, and achieve precise location and visualization of anomalies in the 3D digital twin model. Simultaneously, it actively guides the visual acquisition device to focus on the anomaly area using acoustic events, significantly improving the robustness and all-time, all-domain monitoring capability of multimodal intelligent perception in complex underground environments. This method achieves closed-loop self-learning optimization by feeding anomaly detection results back to a cross-modal correlation network, enabling the system to continuously iterate and update along with the coal mine production process, thus enhancing the model's long-term service performance and adaptability. Through the deep fusion of a 3D digital twin model and multimodal perception, this method realizes the full-element digital presentation and intelligent scheduling auxiliary decision-making of the mine production environment, equipment status, and personnel activities. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1This diagram illustrates the overall process of a cross-modal self-supervised learning method for digital twins in coal mines.

[0041] Figure 2 A schematic diagram illustrating the process of constructing a 3D digital twin model using a cross-modal self-supervised learning method for coal mine digital twins is shown.

[0042] Figure 3 A schematic diagram of the coordinate space mapping and time-frequency domain feature extraction process of acoustic signals in the cross-modal self-supervised learning method of digital twin in coal mine is shown.

[0043] Figure 4 A schematic diagram of the training process for the cross-modal association network of the cross-modal self-supervised learning method for digital twins in coal mines is shown.

[0044] Figure 5 A schematic diagram of the self-supervised training process of the cross-modal association network in the cross-modal self-supervised learning method of digital twin in coal mine is shown.

[0045] Figure 6 A schematic diagram of the overall structure of a digital twin cross-modal self-supervised learning system for coal mines is shown. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0047] The terminology used in this invention is that which is currently widely used in the art in consideration of the function of the invention; however, these terms may vary according to the intent of those skilled in the art, precedent, or new technology in the art. Furthermore, specific terms may be chosen by the applicant, and in such cases, their detailed meanings will be described in the detailed description of the invention. Therefore, the terms used in this specification should not be construed as simple names, but rather based on their meanings and the overall description of the invention.

[0048] Reference Figure 1 This paper presents a cross-modal self-supervised learning method for digital twins in coal mines based on vision and sound. The method includes steps S100-S500.

[0049] Step S100: Multimodal Data Synchronous Acquisition and Time Alignment. Acquire continuous time-series image sequences and continuous time-series acoustic signals for the coal mine scene. Deploy several sets of intrinsically safe image acquisition devices and microphone arrays in underground coal mine roadways, working faces, and key equipment locations. The image acquisition devices acquire continuous time-series image sequences at a sampling rate of 30 frames per second, while the microphone arrays synchronously acquire continuous time-series acoustic signals at a sampling rate of 16 kHz. Both types of signals are aggregated to the edge computing node via an industrial Ethernet switch. The edge computing node performs timestamp alignment processing on the two types of signals according to the calibration parameters of the image acquisition devices and microphone arrays and the clock synchronization protocol.

[0050] Step S200: Construction of a 3D Digital Twin Model. A 3D digital twin model is constructed based on a continuous temporal image sequence. The continuous temporal acoustic signals are mapped to the coordinate space of the 3D digital twin model, and time-frequency domain features are extracted. Edge computing nodes generate local 3D point clouds frame-by-frame using a motion reconstruction algorithm based on the continuous temporal image sequence. The local 3D point clouds are recursively extrapolated to the global coordinate system through inter-frame pose estimation, and stitched together to form a global 3D point cloud model. Surface mesh reconstruction and texture mapping are performed on the global 3D point cloud model to generate a 3D digital twin model. For multi-channel acoustic signals acquired by the microphone array, edge computing nodes perform pre-emphasis, framing, and windowing preprocessing. Short-time Fourier transform is used to convert each channel signal into a time-spectrum image. The generalized cross-correlation method is used to calculate the arrival time difference between each microphone pair. Combined with the corresponding tunnel spatial geometry information in the 3D digital twin model, tunnel reflection compensation is performed on the arrival time difference. The continuous temporal acoustic signals are mapped to the coordinate space of the 3D digital twin model, and time-frequency domain features are extracted.

[0051] Step S300: Coordinate space mapping and time-frequency domain feature extraction of acoustic signals. Positive sample pairs are constructed based on visual events and acoustic events at the same time and spatial location, and negative sample pairs are constructed based on visual events and acoustic events at different times or spatial locations. Edge computing nodes detect and locate image regions of visual events from continuous temporal image sequences using target detection algorithms, and map the image regions to the coordinate space of the 3D digital twin model; they detect and locate the time interval and spatial location of acoustic events from continuous temporal acoustic signals using time-frequency domain feature extraction and sound source localization algorithms; taking advantage of the inherent temporal synchronization and spatial homology of visual and acoustic events in coal mines, visual events with a time difference less than a time threshold and a spatial distance less than a distance threshold are paired with acoustic events as positive sample pairs, and visual events with different times or spatial locations are paired with acoustic events as negative sample pairs.

[0052] Step S400: Automatic construction of positive and negative sample pairs based on spatiotemporal consistency. A cross-modal association network is trained using positive and negative sample pairs to learn the mapping relationship between the visual and acoustic feature spaces, enabling visual and acoustic events to mutually represent each other in a shared embedding space based on spatiotemporal consistency. A contrastive learning loss function is constructed using positive and negative sample pairs to train the cross-modal association network, enabling visual and acoustic features to mutually represent each other in a shared embedding space based on spatiotemporal consistency, and learning the mapping relationship between the visual and acoustic feature spaces.

[0053] Step S500: Self-supervised training of the cross-modal association network. The trained cross-modal association network is deployed on a 3D digital twin model to achieve real-time mutual calibration between the visual and acoustic modalities. For a new input image-acoustic data pair, the network calculates the cross-modal feature distance between visual and acoustic features in the shared embedding space in real time. :

[0054]

[0055] Where D is the dimension of the shared embedding space. Let k be the representation of visual event features in the shared embedding space. Let the acoustic event features be represented in the k-th dimension of the shared embedding space; when When the threshold is exceeded, it is determined to be a modal mismatch, triggering a real-time mutual calibration operation between the visual and acoustic modalities, enabling the two modalities to achieve spatiotemporal consistency-based collaborative perception in the three-dimensional digital twin model.

[0056] In one specific implementation, refer to Figure 2 The construction of a three-dimensional digital twin model based on a continuous temporal image sequence includes: generating local three-dimensional point clouds frame by frame from the continuous temporal image sequence; recursively extrapolating the local three-dimensional point clouds to the global coordinate system through inter-frame pose estimation; stitching them together to form a global three-dimensional point cloud model; and performing surface mesh reconstruction and texture mapping on the global three-dimensional point cloud model to generate a three-dimensional digital twin model. The three-dimensional digital twin model is dynamically updated in response to the streaming input of the continuous temporal image sequence.

[0057] Specifically, for each frame in a continuous temporal image sequence, edge computing nodes extract ORB feature points, estimate inter-frame pose using bundle adjustment, and recursively stitch the local point cloud into the global coordinate system. When a new frame arrives, feature points are extracted and matched with those of the previous frame, the pose of the current frame is calculated using the PnP algorithm, and the incremental new point cloud is stitched into the global model. The system calculates the point cloud registration error between two adjacent updates in real time. The calculation formula is as follows:

[0058]

[0059] in, R represents the number of successfully matched feature point pairs between two adjacent frames, R is the inter-frame rotation matrix, and t is the inter-frame translation vector. The coordinates of the m-th matching feature point in the previous frame are 3D coordinates. These are the 3D coordinates of the corresponding matched feature point in the current frame. The update is considered successful when the value is less than a set threshold; otherwise, loop closure detection and global optimization are triggered. This streaming update mechanism enables the 3D digital twin model to be dynamically updated in real time in response to frame-by-frame input of a continuous temporal image sequence.

[0060] In one specific implementation, refer to Figure 2 Texture mapping includes: acquiring texture feature points of each frame in a continuous temporal image sequence, matching the texture feature points with spatial points in a global 3D point cloud model, establishing a mapping relationship between image pixels and 3D spatial points, fusing and mapping the texture information of multiple frames onto the surface of the 3D model after surface mesh reconstruction, and generating a 3D digital twin model with realistic texture.

[0061] Specifically, SIFT and ORB texture feature points are extracted from each frame of the image, and feature matching is performed with spatial points in the global point cloud model to establish a mapping relationship between image pixels and 3D spatial points. For each triangular facet of the 3D mesh, visibility is determined based on the angle between the facet's normal vector and the camera's viewing direction; a facet is considered visible if the angle is less than 90 degrees. Texture pixels from multiple frames are fused using the following weighting formula:

[0062]

[0063] in, The fusion weights for the f-th frame are... Let f be the normal vector corresponding to the target triangle in the f-th frame image. Let be the unit vector of the camera's line of sight during the acquisition of the f-th frame, α be the viewpoint sensitivity index, and G be the total number of image frames covering the same triangular facet. The position of the camera's optical center. Using the reference camera position as an example, β is the distance attenuation coefficient. This mechanism results in a greater contribution of texture from the frontal viewpoint and a decrease in the contribution of textures farther from the reference position, effectively eliminating texture stitching seams.

[0064] In one specific implementation, refer to Figure 2The process of mapping continuous temporal acoustic signals to the coordinate space of a three-dimensional digital twin model includes: establishing a specific underground acoustic propagation model based on the roadway geometry information in the three-dimensional digital twin model; this model includes the roadway wall reflection coefficient and the roadway cross-section attenuation coefficient; and then compensating for roadway reflection and attenuation based on the arrival time difference and arrival intensity difference of the acoustic signals. Finally, the three-dimensional spatial location of the acoustic event in the three-dimensional digital twin model is calculated based on the compensated parameters.

[0065] Specifically, based on the tunnel cross-sectional shape, size, and wall material in the 3D digital twin model, the wall reflection coefficient and cross-sectional attenuation coefficient are calculated. The microphone array receives signals and obtains the original time difference of arrival (TDOA) through generalized cross-correlation. A dedicated model performs tunnel reflection compensation on the original TDOA to eliminate pseudo-path delay bias caused by wall reflection. Simultaneously, tunnel attenuation compensation is performed on the arrival intensity difference to eliminate sound wave attenuation distortion caused by abrupt changes in cross-section. The sound source location is minimized using the following cost function:

[0066]

[0067] Where M represents the number of microphones. Let be the spatial location vector of the sound source to be estimated. , Let be the spatial locations of the i-th and j-th microphones, and c be the speed of sound in air. To compensate for the time difference of arrival, Let be the effective acoustic propagation distance from the sound source to the i-th microphone in the 3D model. The first term constrains the consistency between the time difference and the geometric distance difference, and the second term constrains the degree of matching between the sound source location and the roadway geometry model.

[0068] In one specific implementation, refer to Figure 3 The construction of positive and negative sample pairs includes: detecting and locating image regions of visual events from a continuous temporal image sequence using object detection algorithms, and mapping these image regions to the coordinate space of a 3D digital twin model; detecting and locating the time interval and spatial location of acoustic events from continuous temporal acoustic signals using time-frequency domain feature extraction and sound source localization algorithms; pairing visual events with a time difference less than a time threshold and a spatial distance less than a distance threshold with acoustic events as positive sample pairs; and pairing visual events with a time difference greater than or equal to a time threshold or a spatial distance greater than or equal to a distance threshold with acoustic events as negative sample pairs.

[0069] Specifically, visual events are detected from images and mapped to a three-dimensional coordinate space using a fast region convolutional neural network. The temporal interval and spatial location of acoustic events are detected from acoustic signals using short-time Fourier transform and time-of-arrival (TOA) source localization. (Time threshold) The calculation formula is:

[0070]

[0071] in, To compensate for the time difference of arrival, To ensure hardware synchronization accuracy between the image acquisition device and the acoustic acquisition module, α is a time window adjustment coefficient, ranging from 0.5 to 1.5, adaptively adjusted according to the tunnel length. Distance threshold. Calculate using the following formula:

[0072]

[0073] Where β is the spatial distance adjustment coefficient. and These are the standard deviations of visual localization and acoustic localization, respectively. For the point cloud density of the 3D model, The volume of a voxel unit. When the time difference is less than... And the spatial distance is less than When a pair is selected, it is considered a positive sample pair; otherwise, it is considered a negative sample pair.

[0074] In one specific implementation, refer to Figure 4 The cross-modal correlation network includes a temporal convolutional network and a Transformer network. The temporal convolutional network extracts local temporal features of continuous temporal image sequences and continuous temporal acoustic signals, while the Transformer network captures the global temporal dependencies of continuous temporal image sequences and continuous temporal acoustic signals.

[0075] Specifically, the temporal convolutional network consists of four one-dimensional convolutional layers, each followed by batch normalization and corrected linear units to extract local temporal features; a six-layer Transformer encoder captures long-range temporal dependencies. The network's temporal receptive field is determined according to the following formula:

[0076]

[0077] in, The kernel size is [size]. To expand the step size, For encoder layers, This represents the effective window length for self-attention. This cascaded architecture utilizes a temporal convolutional network to efficiently extract short-term transient acoustic features, and a Transformer network to model long-term periodic variations; the two are complementary. Image and acoustic modalities are processed by the temporal convolutional network and the Transformer network, respectively, and then mapped to a shared embedding space. Training is performed using a contrastive loss.

[0078] In one specific implementation, refer to Figure 5Real-time mutual calibration between visual and acoustic modalities includes: performing forward prediction from visual events to acoustic features through a cross-modal correlation network; comparing the predicted acoustic features with the actual acoustic signal; and classifying any deviation exceeding a first threshold as a type I anomaly. Conversely, performing backward prediction from acoustic events to visual features through the same cross-modal correlation network; comparing the predicted visual features with the actual image; and classifying any deviation exceeding a second threshold as a type II anomaly. Type I and type II anomalies are located and displayed in the 3D digital twin model.

[0079] Specifically, forward prediction from vision to acoustics and backward prediction from acoustics to vision are performed through a cross-modal correlation network, and the forward prediction bias and backward prediction bias are calculated. The anomaly threshold is dynamically adjusted according to the real-time downhole environment.

[0080]

[0081] Where t is the current time, and Let be the moving mean and moving standard deviation of the positive prediction bias for the most recent N normal samples, respectively. and Let N and 1 be the moving mean and moving standard deviation of the backward prediction bias for the most recent N normal samples, respectively, where N is 100. The background noise power spectral density of the acoustic signal at the current moment. For reference noise power spectral density, This is the blur index of the image at the current moment. For reference ambiguity index, and This is the adjustment coefficient. This dynamic thresholding mechanism allows the system to adaptively increase the threshold to avoid false alarms when environmental noise increases or image quality deteriorates, and decrease the threshold to improve anomaly detection sensitivity when environmental conditions are favorable. Positive prediction bias. and reverse prediction bias The calculation formula is:

[0082]

[0083] Where D is the dimension of the shared embedding space; when > A type of exception is triggered when something is "seen but not heard". > Two types of anomalies are triggered at certain times, characterized by "hearing but not seeing". The two types of anomalies are located to the specific coordinates of the 3D digital twin model through the attention weights of the cross-modal association network and highlighted, which is different from the traditional anomaly detection method that only relies on single-modal threshold judgment.

[0084] In one specific implementation, refer to Figure 5It also includes: when the deviation between visual events and acoustic events through forward and backward prediction is within the first and second thresholds respectively, but the spatial distance between the acoustic events and visual events in the three-dimensional digital twin model exceeds the third threshold, it is judged as a type three anomaly.

[0085] Specifically, when ≤ and ≤ When this is achieved, it indicates that the visual modality and the acoustic modality are consistent at the feature level. At this point, the spatial distance between the acoustic event and the visual event in the 3D digital twin model can be further calculated. The third threshold It integrates two environmental factors unique to underground coal mines—signal-to-noise ratio and geometric precision factor—for dynamic calculations, unlike conventional fixed distance thresholds. Exceeding the third threshold The anomaly is classified into three categories: the first is the "what is seen and heard matches, but the location is inconsistent" anomaly, which has important physical meaning in the coal mine scenario—it may indicate internal equipment failure or hidden fault points. The third threshold... Determined according to the following dynamic formula:

[0086]

[0087] Where L is the number of samples in the validation set. and The verification set is respectively The position vectors of acoustic localization and visual localization in each sample. This represents the real-time signal-to-noise ratio of the signal received by the microphone array at the current moment. For reference signal-to-noise ratio, This is the geometric precision factor for the current positioning scenario. As a reference geometric accuracy factor, η is the global scale coefficient, with a value of 1.5-2.5, and θ is the signal-to-noise ratio impact index, with a value of 0.3-0.5. This dynamic threshold mechanism automatically relaxes the spatial distance tolerance to reduce the false negative rate when the signal-to-noise ratio decreases, and automatically increases the threshold when the geometric accuracy factor increases to avoid misjudging positioning errors caused by the microphone array geometry as abnormalities. The calculation formula is ,in Locating coordinates for acoustic events Locate the coordinates for visual events. After the three types of anomalies are determined, simultaneously locate the acoustic and visual coordinates in the 3D digital twin model, mark them with different colors and indicate the deviation distance with connecting lines, so that dispatchers can quickly identify the source of the positioning deviation.

[0088] In one specific implementation, refer to Figure 5It also includes: mapping the spatial localization results of acoustic events to a three-dimensional digital twin model to generate a dynamic acoustic event layer; when an acoustic event is located in a first spatial region, controlling the visual acquisition device to increase the acquisition frequency and resolution of the first spatial region; when acoustic event localization fails, using a cross-modal association network to predict the acoustic event based on the visual event for compensatory localization.

[0089] Specifically, the spatial localization results of acoustic events are overlaid as a dynamic acoustic event layer onto a 3D digital twin model, displaying the energy distribution as a semi-transparent sound field cloud map. When an acoustic event is located in the first spatial region, the optical zoom module of the visual acquisition device in that region is controlled to focus on the target area, while simultaneously increasing the acquisition frame rate to 2 times and the resolution to 1.5 times. Confidence threshold. The following formula is used to determine:

[0090]

[0091] in, The baseline threshold is set at 0.6. Let KL divergence be the probability density of the visual and acoustic spatial distributions. The normalization constant is Let Variance be the acoustic localization result over the most recent N frames. This is the reference variance. When the confidence level is lower than... When acoustic positioning fails, the acoustic prediction branch of the cross-modal association network is used to predict acoustic compensation positioning coordinates based on visual events to replace the failure result.

[0092] A cross-modal self-supervised learning system for digital twins in coal mines based on vision and sound, referring to Figure 6 The above-mentioned cross-modal self-supervised learning method for digital twins in coal mines is characterized by including:

[0093] The visual acquisition module is used to acquire continuous temporal image sequences of coal mine scenes in real time. It consists of several intrinsically safe explosion-proof cameras for mining and is deployed in underground coal mine roadways, working faces, and key equipment locations to acquire continuous temporal image sequences of coal mine scenes in real time.

[0094] The acoustic acquisition module is used to synchronously acquire continuous temporal acoustic signals aligned with the time sequence of continuous temporal images. It consists of several microphone arrays, each containing no fewer than four microphone units, and is deployed synchronously and spatially adjacent to the visual acquisition module for synchronously acquiring continuous temporal acoustic signals aligned with the time sequence of continuous temporal images.

[0095] The 3D digital twin construction module is used to build and dynamically update a 3D digital twin model based on a continuous temporal image sequence. Deployed on an edge computing server, it generates local 3D point clouds frame by frame using a motion reconstruction algorithm based on a continuous temporal image sequence. These points are then recursively stitched together through inter-frame pose estimation to form a global point cloud model. Finally, surface mesh reconstruction and texture mapping are performed to build and dynamically update the 3D digital twin model.

[0096] The cross-modal association learning module is used to construct positive sample pairs based on visual and acoustic events occurring at the same time and spatial location, and negative sample pairs based on visual and acoustic events occurring at different times or spatial locations. These positive and negative sample pairs are then used to train the cross-modal association network, learning the mapping relationship between the visual feature space and the acoustic feature space.

[0097] The cross-modal mutual calibration and anomaly detection module is used to deploy the trained cross-modal association network onto a 3D digital twin model. It performs forward prediction from visual events to acoustic features and backward prediction from acoustic events to visual features through the cross-modal association network. The deviations in forward and backward predictions are used as anomaly detection signals to determine the anomaly type and locate and display it in the 3D digital twin model. Simultaneously, a dynamic acoustic event layer is generated and overlaid onto the 3D scene.

[0098] The control and verification module generates device control commands based on anomaly detection signals. It then acquires image sequences and acoustic signals after command execution via visual and acoustic acquisition modules to verify the execution effect of the control commands. The verification results are fed back to the cross-modal association learning module for iterative optimization of the cross-modal association network.

[0099]

[0100] Where θ is the set of trainable parameters for the cross-modal correlation network, t is the number of iterations, and α is the learning rate. The contrastive learning loss function is used, where β is the feedback adjustment coefficient. To verify the module's evaluation loss regarding the effect of instruction execution, To establish a baseline for reference loss, all modules are interconnected via an underground industrial Ethernet ring network to enable real-time data transmission and command issuance.

[0101] This invention provides a cross-modal self-supervised learning method and system for digital twins in coal mines based on vision and sound. By constructing a vision-acoustic cross-modal self-supervised learning framework based on spatiotemporal consistency, it automatically builds positive and negative sample pairs by leveraging the inherent temporal synchronization and spatial homology of visual and acoustic events in underground coal mines. This eliminates the need for extensive manual annotation to train the cross-modal association network, significantly reducing model deployment and maintenance costs and overcoming the engineering challenges of scarce and difficult-to-annotate abnormal samples in underground coal mines. The technical solution utilizes an underground-specific acoustic propagation model established based on the geometric information of the roadway in a 3D digital twin model to accurately compensate for acoustic signal reflection from the roadway walls and cross-sectional attenuation. Combined with a localization cost function that integrates arrival time difference constraints and roadway geometric matching constraints, it significantly improves the 3D localization accuracy of acoustic events in the enclosed space of underground coal mines, providing a reliable foundation for the spatial alignment of visual and acoustic information. A multi-level dynamic threshold anomaly detection system is constructed through a visual-acoustic bidirectional mutual calibration mechanism. The anomaly judgment threshold is adaptively adjusted based on environmental parameters such as real-time background noise power spectral density, image blur, signal-to-noise ratio, and geometric accuracy factor, overcoming the poor adaptability of fixed thresholds in complex underground environments. It can effectively identify three types of anomalies where visual and acoustic information are inconsistent and accurately locate and display them in a 3D digital twin model, significantly improving the robustness of multimodal perception in complex environments. Through an acoustic event-driven visual focusing acquisition mechanism, the acoustic positioning results are used to actively guide the visual acquisition device to zoom and focus and improve the acquisition frequency and resolution. When acoustic positioning fails, the cross-modal association network is used to predict acoustic compensation positioning based on visual events, realizing the active guidance and mutual coordination of auditory perception to visual perception. Fifth, by feeding back the anomaly detection verification results to the cross-modal association learning module, a complete intelligent closed loop of "deployment-operation-verification-optimization" is constructed, enabling the cross-modal association network to continuously iterate and self-optimize with the coal mine production process, enhancing the long-term service performance and adaptability of the system.

[0102] It is important to note that the constructions and arrangements of this application shown in several different exemplary embodiments are merely illustrative. Although only a few embodiments are described in detail in this disclosure, those who consult this disclosure will readily understand that many modifications are possible (e.g., changes in the size, dimensions, structure, shape, and proportions of various elements, as well as parameter values ​​(e.g., temperature, pressure, etc.), mounting arrangements, use of materials, color, orientation, etc.) without substantially departing from the novel teachings and advantages of the subject matter described in this application). For example, an element shown as integrally formed may be composed of multiple parts or elements, the position of elements may be inverted or otherwise altered, and the nature or number or position of discrete elements may be changed or altered. Therefore, all such modifications are intended to be included within the scope of the invention. The order or sequence of any process or method steps may be changed or rearranged according to alternative embodiments. In the claims, any "device plus function" clause is intended to cover the structure described herein that performs the function, and not only structurally equivalent but also equivalent in structure. Other substitutions, modifications, alterations, and omissions may be made in the design, operation, and arrangement of the exemplary embodiments without departing from the scope of the invention. Therefore, the present invention is not limited to the specific embodiments, but extends to various modifications that still fall within the scope of the appended claims.

[0103] Furthermore, to provide a concise description of exemplary embodiments, not all features of actual embodiments (i.e., those features not relevant to the currently considered best mode for carrying out the invention, or those features not relevant to implementing the invention) may be omitted. It should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention and not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications and substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A cross-modal self-supervised learning method for digital twins in coal mines based on vision and sound, characterized in that, include: Acquire time-aligned continuous temporal image sequences and continuous temporal acoustic signals of a coal mine scene; A three-dimensional digital twin model is constructed based on the continuous time-series image sequence, and the continuous time-series acoustic signal is mapped to the coordinate space of the three-dimensional digital twin model and time-frequency domain features are extracted. Positive sample pairs are constructed based on visual events and acoustic events that occur at the same time and in the same spatial location, while negative sample pairs are constructed based on visual events and acoustic events that occur at different times or in different spatial locations. The cross-modal association network is trained using the positive sample pairs and the negative sample pairs to learn the mapping relationship between the visual feature space and the acoustic feature space, so that visual events and acoustic events can achieve mutual representation based on spatiotemporal consistency in the shared embedding space. The trained cross-modal association network is deployed on the three-dimensional digital twin model to achieve real-time mutual calibration between the visual and acoustic modalities.

2. The cross-modal self-supervised learning method for digital twins in coal mines according to claim 1, characterized in that, The construction of the three-dimensional digital twin model based on the continuous temporal image sequence includes: Local 3D point clouds are generated frame by frame from the continuous temporal image sequence. The local 3D point clouds are recursively extrapolated to the global coordinate system through inter-frame pose estimation and stitched together to form a global 3D point cloud model. The global 3D point cloud model is then subjected to surface mesh reconstruction and texture mapping to generate the 3D digital twin model. The 3D digital twin model is dynamically updated in response to the streaming input of the continuous temporal image sequence.

3. The cross-modal self-supervised learning method for digital twins in coal mines according to claim 2, characterized in that, The texture mapping includes: The texture feature points of each frame in the continuous temporal image sequence are obtained, and the texture feature points are matched with the spatial points in the global three-dimensional point cloud model to establish the mapping relationship between the pixels of the image and the three-dimensional spatial points. The texture information of multiple frames of images is fused and mapped onto the surface of the three-dimensional model after the surface mesh is reconstructed to generate a three-dimensional digital twin model with real texture.

4. The cross-modal self-supervised learning method for digital twins in coal mines according to claim 1, characterized in that, The step of mapping the continuous temporal acoustic signal to the coordinate space of the three-dimensional digital twin model includes: An underground-specific acoustic propagation model is established based on the tunnel geometry information in the three-dimensional digital twin model. The underground-specific acoustic propagation model includes the tunnel wall reflection coefficient and the tunnel cross-section attenuation coefficient. The downhole-specific acoustic propagation model performs roadway reflection compensation and roadway attenuation compensation on the arrival time difference and arrival intensity difference of the acoustic signal, and calculates the three-dimensional spatial location of the acoustic event in the three-dimensional digital twin model based on the compensated parameters.

5. The cross-modal self-supervised learning method for digital twins in coal mines according to claim 1, characterized in that, Constructing the positive sample pairs and the negative sample pairs includes: The image region of the visual event is detected and located from the continuous temporal image sequence using a target detection algorithm, and the image region is mapped to the coordinate space of the three-dimensional digital twin model; The time interval and spatial location of the acoustic event are detected and located from the continuous temporal acoustic signal by time-frequency domain feature extraction and sound source localization algorithm; The visual events with a time difference less than a time threshold and a spatial distance less than a distance threshold are paired with the acoustic events to form the positive sample pairs; The visual event with a time difference greater than or equal to the time threshold or a spatial distance greater than or equal to the distance threshold is paired with the acoustic event to form the negative sample pair.

6. The cross-modal self-supervised learning method for digital twins in coal mines according to claim 1, characterized in that, The cross-modal association network includes a temporal convolutional network and a Transformer network. The temporal convolutional network extracts local temporal features of the continuous temporal image sequence and the continuous temporal acoustic signal, while the Transformer network captures the global temporal dependencies of the continuous temporal image sequence and the continuous temporal acoustic signal.

7. The cross-modal self-supervised learning method for digital twins in coal mines according to claim 1, characterized in that, The real-time mutual calibration between the visual and acoustic modalities includes: The visual event is positively predicted to acoustic features through the cross-modal association network. The predicted acoustic features are compared with the actual acoustic signal. If the deviation exceeds the first threshold, it is judged as an anomaly. The back prediction of acoustic events to visual features is performed through the cross-modal association network. The predicted visual features are compared with the actual image. If the deviation exceeds the second threshold, it is judged as a type II anomaly. The first type of anomaly and the second type of anomaly are located and displayed in the three-dimensional digital twin model.

8. The cross-modal self-supervised learning method for digital twins in coal mines according to claim 7, characterized in that, Also includes: When the deviations between the visual event and the acoustic event through the forward prediction and the reverse prediction are within the first threshold and the second threshold, respectively, but the spatial distance between the acoustic event and the visual event in the three-dimensional digital twin model exceeds the third threshold, it is determined to be a type three anomaly.

9. The cross-modal self-supervised learning method for digital twins in coal mines according to claim 8, characterized in that, Also includes: The spatial localization results of the acoustic events are mapped to the three-dimensional digital twin model to generate a dynamic acoustic event layer; When the acoustic event is located in the first spatial region, the visual acquisition device is controlled to increase the acquisition frequency and resolution of the first spatial region. When the acoustic event localization fails, the cross-modal association network predicts the compensated localization of the acoustic event based on the visual event.

10. A cross-modal self-supervised learning system for coal mine digital twins based on vision and sound, employing the cross-modal self-supervised learning method for coal mine digital twins as described in claims 1 to 9, characterized in that... include: The visual acquisition module is used to acquire continuous time-series image sequences of coal mine scenes in real time. An acoustic acquisition module is used to synchronously acquire continuous temporal acoustic signals that are time-aligned with the continuous temporal image sequence; A 3D digital twin construction module is used to construct and dynamically update the 3D digital twin model based on the continuous temporal image sequence; The cross-modal association learning module is used to construct positive sample pairs based on visual events and acoustic events at the same time and in the same spatial location, and to construct negative sample pairs based on visual events and acoustic events at different times or in different spatial locations, and to train the cross-modal association network with the positive sample pairs and the negative sample pairs. A cross-modal mutual calibration and anomaly detection module is used to deploy the trained cross-modal association network on the three-dimensional digital twin model; to perform forward prediction of the visual event to acoustic feature through the cross-modal association network, to perform reverse prediction of the acoustic event to visual feature, and to use the deviation of the forward prediction and the deviation of the reverse prediction as anomaly detection signals to determine the anomaly type and locate and display it in the three-dimensional digital twin model; The control and verification module is used to generate device control commands based on the anomaly detection signal, and to acquire image sequences and acoustic signals after command execution through the visual acquisition module and the acoustic acquisition module to verify the execution effect of the device control commands. The verification results are then fed back to the cross-modal association learning module to iteratively optimize the cross-modal association network.

Images