Unmanned mine card operation state voiceprint monitoring system and method and medium

Abstract

The invention discloses an unmanned mine card operation state voiceprint monitoring system, method and device. The system comprises a three-dimensional space mapping module, a time synchronization module and an abnormity positioning module. Voiceprint and image acquisition devices are symmetrically arranged along the two sides of a mine road, and an accurate space model is constructed by utilizing three-dimensional laser scanning, so that full coverage and no blind area of a monitoring area are ensured. Microsecond-level time synchronization is realized by adopting a precise time protocol supporting a hardware timestamp, and time-space alignment of voiceprint and image data is ensured by combining a linear model to compensate clock drift. Voiceprint spatial-temporal characteristics are extracted based on a time-aware bidirectional multi-scale network, anomaly detection is carried out through a mahalanobis distance, and finally, the real-time position, speed and three-dimensional coordinates of a part are fused, so that accurate traceability from abnormal voiceprint to a specific fault part is realized. According to the method, the defects of single dimension, misalignment of space-time correlation and fuzzy positioning of the existing monitoring technology are overcome, and the timeliness and the maintenance efficiency of fault early warning are remarkably improved.

Description

Technical Field

[0001] This document relates to the field of unmanned mining truck monitoring technology, and in particular to an unmanned mining truck operating status acoustic monitoring system, method and medium. Background Technology

[0002] As core equipment for intelligent operations in mining areas, real-time monitoring of the operational status of unmanned mining trucks is crucial for ensuring production safety. Current technologies for monitoring the operational status of unmanned mining trucks largely rely on single-dimensional devices such as vibration sensors and temperature sensors, which have the following technical shortcomings: Firstly, existing solutions mostly focus on physical quantities such as vibration and temperature, and lack effective perception of abnormal sound patterns caused by gear meshing, bearing wear, etc., which makes it easy to miss early potential faults and delay fault warning. Secondly, the equipment deployment lacks systematic spatial planning, the monitoring range of multiple devices is insufficiently overlapping or there are coverage gaps, and there is a lack of accurate three-dimensional spatial mapping mechanism, making it difficult to achieve accurate correlation between monitoring data and the physical location of mining cards, thus affecting the accuracy of anomaly location. Third, the inconsistent time bases of sensors, cameras and other devices result in large deviations in data timestamps, making it difficult to effectively integrate multi-source data and reducing monitoring reliability. Fourth, even if existing technologies detect anomalies, it is difficult to achieve accurate source tracing by combining the movement status of mining trucks and the spatial distribution of components. They mostly remain at the regional level for early warning, and cannot provide specific component guidance for maintenance, thus increasing maintenance costs and time. Therefore, there is an urgent need for a monitoring system that can integrate voiceprint features, spatial mapping, and precise positioning to address the shortcomings of the existing technologies. Summary of the Invention

[0003] According to embodiments of the present invention, an unmanned mining truck operation status acoustic monitoring system, method, and medium are provided to solve the above-mentioned technical problems.

[0004] According to an embodiment of the present invention, an acoustic signature monitoring system for the operating status of an unmanned mining truck is provided, characterized in that it includes a three-dimensional spatial mapping module, a time synchronization module, and an anomaly location module; The three-dimensional spatial mapping module is used to symmetrically deploy multiple sets of poles on both sides of the monitored road section, and install voiceprint acquisition devices and image acquisition devices on the poles; acquire spatial point cloud data containing all poles and acquisition devices through a three-dimensional scanning device, and establish a unified three-dimensional spatial model based on the point cloud data; calculate the effective voiceprint monitoring area of ​​each voiceprint acquisition device and the effective image field of view of each image acquisition device based on the three-dimensional spatial model, and ensure that the overlap rate between the effective voiceprint monitoring area and the effective image field of view is not lower than a preset threshold. The time synchronization module is used to construct a star-shaped synchronization network based on a precision time protocol, using an edge computing terminal as the master clock to synchronize the voiceprint acquisition device and image acquisition device as slave clocks; to compensate for the cumulative clock drift of each slave clock through a linear correction model; and to associate the event of the mining truck passing through a preset location reference point based on the synchronized and corrected timestamp, and to calculate the driving speed of the mining truck. The anomaly localization module is used to process the voiceprint data collected by the voiceprint acquisition device through a time-aware bidirectional multi-scale network to extract a spatiotemporal joint feature vector; based on the comparison between the spatiotemporal joint feature vector and the historical normal voiceprint feature distribution, it identifies abnormal voiceprints; in response to the identification of abnormal voiceprints, it determines the spatial position of the mining truck at the time of the anomaly by combining the time of the anomaly occurrence, the speed of the mining truck, and the three-dimensional spatial model; and it matches the spatial position with the pre-stored three-dimensional component model of the mining truck to locate the specific component that malfunctioned.

[0005] According to an embodiment of the present invention, a method for monitoring the operating status of an unmanned mining truck using acoustic signatures is provided, characterized in that the method includes: S1. Deploy data acquisition equipment in the monitored road section, acquire spatial point cloud data and establish a unified three-dimensional model, determine the effective monitoring area of ​​the soundprint and image acquisition equipment and ensure that the overlap rate meets the standard. S2. Synchronize the clocks of all acquisition devices based on a precision time protocol, timestamp the data based on the event of the mining truck passing through the location reference point, and calculate the speed of the mining truck. S3. Process the voiceprint data and extract features. Identify anomalies through the voiceprint analysis model. Combine the anomaly timestamp, driving speed and three-dimensional model to locate the spatial position of the mining truck, and match it with the component coordinates to determine the specific faulty component.

[0006] According to an embodiment of the present invention, an electronic device is provided, comprising: Processor; and, A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the voiceprint monitoring method for the unmanned mining truck's operating status.

[0007] The unmanned mining truck operation status acoustic monitoring system provided in this application achieves three core beneficial effects by constructing a technical system that integrates precise spatial mapping, time synchronization alignment, and intelligent acoustic analysis: First, it breaks through the limitations of traditional single physical quantity monitoring such as vibration and temperature, utilizing a time-aware bidirectional multi-scale network to deeply mine the spatiotemporal characteristics of acoustics, significantly improving the sensitivity and timeliness of early detection of faults in key components such as gears and bearings; second, through 3D point cloud modeling and precise time protocol synchronization, it achieves precise correlation between acoustics, images, and mining truck location data, fundamentally solving the problem of fusion distortion of multi-source data in spatial and temporal dimensions; third, it innovatively binds abnormal acoustics with the mining truck's motion status and component spatial coordinates dynamically, achieving a leap from regional-level alarms to component-level positioning, providing precise guidance for maintenance and significantly reducing operation and maintenance costs and time. Overall, this system improves the reliability, accuracy, and intelligence level of unmanned mining truck operation status monitoring, providing effective technical support for safe production in mining areas. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a schematic diagram of the voiceprint monitoring system for the unmanned mining truck operation status according to an embodiment of the present invention; Figure 2 This is a flowchart of the voiceprint monitoring method for the operating status of unmanned mining trucks according to an embodiment of the present invention. Detailed Implementation

[0010] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0011] System Implementation Examples According to an embodiment of the present invention, an acoustic signature monitoring system for the operating status of unmanned mining trucks is provided. Figure 1 This is a schematic diagram of the voiceprint monitoring system for the unmanned mining truck operation status according to an embodiment of the present invention. Figure 1As shown, the unmanned mining truck operation status voiceprint monitoring system of this embodiment of the invention specifically includes: The three-dimensional spatial mapping module 10 is used to symmetrically deploy multiple sets of poles on both sides of the monitored road section, and install voiceprint acquisition devices and image acquisition devices on the poles; acquire spatial point cloud data containing all poles and acquisition devices through a three-dimensional scanning device, and establish a unified three-dimensional spatial model based on the point cloud data; calculate the effective voiceprint monitoring area of ​​each voiceprint acquisition device and the effective image field of view of each image acquisition device based on the three-dimensional spatial model, and ensure that the overlap rate between the effective voiceprint monitoring area and the effective image field of view is not lower than a preset threshold. The time synchronization module 12 is used to construct a star-shaped synchronization network based on a precision time protocol, using an edge computing terminal as the master clock to synchronize the voiceprint acquisition device and image acquisition device as slave clocks; to compensate for the cumulative clock drift of each slave clock through a linear correction model; and to associate the event of the mining truck passing through a preset location reference point based on the synchronized and corrected timestamp, and to calculate the driving speed of the mining truck. The anomaly localization module 14 is used to process the voiceprint data collected by the voiceprint acquisition device through a time-aware bidirectional multi-scale network to extract a spatiotemporal joint feature vector; based on the comparison between the spatiotemporal joint feature vector and the historical normal voiceprint feature distribution, to identify abnormal voiceprints; in response to the identification of abnormal voiceprints, combined with the time of the anomaly occurrence, the speed of the mining truck, and the three-dimensional spatial model, to determine the spatial position of the mining truck at the time of the anomaly occurrence; and to match the spatial position with the pre-stored three-dimensional component model of the mining truck to locate the specific component that malfunctioned.

[0012] The equipment deployment method for the 10 monitoring sections in the three-dimensional spatial mapping module is as follows: Multiple sets of poles are symmetrically arranged on both sides of the monitored road section, with the line connecting each set of poles perpendicular to the center line of the road surface; The voiceprint acquisition device is installed at a designated height on the pole, with its sound pickup direction facing the critical component area in the middle of the vehicle body of the mining truck in the opposite lane. The image acquisition device is installed on the pole, and its shooting direction is tilted along the direction of the mining truck's travel, focusing on the mining truck's identification feature area; The three-dimensional scanning equipment includes a three-dimensional laser profile sensor for local fine scanning and a three-dimensional mapping lidar for large-area supplementary scanning.

[0013] To better understand the equipment deployment method of the monitored road section, a specific deployment example of the present invention is given: along the east side of the monitored road section, with the base station as the center, one pole is deployed every 50 meters to the north and south, numbered as East 1, East 2, East 3, ... East n. On the west side of the road, with the pole on the east side as the reference, n poles are symmetrically deployed, numbered as West 1, West 2, West 3, ... West n, thus forming n sets of symmetrical poles: East 1-West 1, East 2-West 2, East 3-West 3, ... East n-West n; In this embodiment of the invention, the 50-meter interval between the two poles is the optimal value calculated based on the speed of the mining truck and the response time of the equipment. If the interval is too large, the time difference between the mining truck passing through the two sets of poles will be too long, which may result in the loss of instantaneous abnormal sound patterns, such as short-term abnormal noises from gears. If the interval is too small, the equipment data will become redundant due to the close distance, resulting in the sound patterns of the same mining truck being collected repeatedly. During pole installation, the centerline of each group of symmetrical poles is calibrated using a total station. The line connecting the symmetrical poles must be perpendicular to the centerline of the road surface to ensure that the monitoring range of the equipment on both sides overlaps at the center of the road surface. Symmetrical centerline calibration can avoid monitoring blind spots caused by the offset of equipment on one side. For example, if the pole on the east side is offset by 1m to the west, the sound pickup range of its soundprint sensor will tilt towards the shoulder, resulting in incomplete soundprint collection of mining trucks in the middle lane. Acoustic sensors are installed on both sides of the uprights. The iFlytek SC-L231 acoustic sensor can be selected, featuring an 8-microphone array, a 120° pickup angle, and an effective distance of 5–20m. It is installed at a height of 4m on the uprights and fixed with an L-shaped bracket. The pickup surface is at a 15° downward angle to the road surface, facing the middle of the mining truck body. Specifically, the sensor on the east upright faces west, covering the engine compartment and front axle area of ​​the mining trucks in the east lane; the sensor on the west upright faces east, covering the electric wheel and gearbox area of ​​the mining trucks in the west lane. An industrial camera is mounted on the pole. The camera can be a night vision type with a 60° field of view, a 30m shooting range, and a resolution of 2560×1440. It is fixed to the end of a horizontal bar 5m high at the top of the pole. The lens is tilted 45° towards the direction of the mining truck's movement, focusing on the truck's front number area. In this embodiment, mounting the camera on the west side of the pole avoids direct sunlight from the east, resulting in higher contrast in the captured image. The 45° tilt angle ensures that the truck's front remains centered in the lens area as it passes, preventing blurring of the number area due to truck yaw. Using each group of symmetrical uprights as a scanning unit, a three-dimensional laser profile sensor (Gocator 2670) is deployed on the road surface in the middle of each group of uprights to scan along a direction perpendicular to the road surface. At the same time, a three-dimensional mapping lidar is used for large-area supplementary scanning, such as the LeiShen Intelligent RS-LIDAR-M1.

[0014] The process of establishing a unified three-dimensional model by the three-dimensional space mapping module 10 also includes: The intersection of point cloud data obtained from multiple scans by a 3D scanning device is used to remove outliers caused by occlusion. The point cloud data is denoised and stitched using a point cloud processing algorithm; Based on the processed point cloud data, the three-dimensional spatial coordinates, horizontal orientation angle, vertical pitch angle and pickup angle parameters of the voiceprint acquisition device are extracted, and the three-dimensional spatial coordinates, horizontal orientation angle, vertical pitch angle and field of view parameters of the image acquisition device are also extracted. Based on the extracted voiceprint acquisition device parameters, the effective voiceprint monitoring area is calculated using a spatial geometric model, and based on the extracted image acquisition device parameters, the effective field of view area of ​​the image is calculated, and then the volume overlap rate of the two is calculated.

[0015] More specifically, the process of establishing a unified 3D model includes: by setting the scanning range of the scanning unit to cover each group of poles (including acoustic sensors and industrial cameras), road surface (mining truck driving trajectory), and road shoulder (to avoid scanning blind spots when mining trucks cross the line), the scanning frequency is 3 scans per group of poles, and the intersection of the three data is taken to reduce occlusion error. The intersection of the three data sets specifically includes: selecting points that can be collected in all three scans from the three generated point cloud data sets, i.e., the point cloud that exists in all three scans, retaining these points as valid data, and removing points that only appear in one or two scans, thereby reducing point cloud data distortion caused by occlusion. After the point cloud data is generated, the Statistical Outlier Removal algorithm can be used to remove outliers, and the data sets can be stitched together by the Iterative Nearest Point ICP algorithm to finally form a complete three-dimensional point cloud model including poles, sensors, cameras and road surfaces. The coordinate system adopts the mining area independent coordinate system, with the X-axis in the north-south direction, the Y-axis in the east-west direction, and the Z-axis perpendicular to the ground. Based on point cloud data, parameters of the voiceprint sensor are extracted, specifically including: Set spatial coordinates as X represents the north-south direction of the road segment, Y represents the east-west direction, and Z represents the elevation. The vertical angle of depression is in the horizontal direction towards the center of the road surface. The pickup angle is ; Effective pickup distance The range is Due to the influence of the sound wave attenuation coefficient, the sound signature arrival intensity must be greater than or equal to the threshold. Based on the parameters of the acoustic sensor, the effective monitoring area of ​​the acoustic sensor is calculated through spatial geometry: Using voiceprint sensor coordinates Starting from the origin, extending along the pickup direction, the horizontal boundary extends from the pickup half-angle. and distance Decide, Calculate the horizontal left boundary using the following formula. ; Calculate the horizontal right boundary using the following formula. ; The range of the horizontal X interval is obtained as follows ; Combined with the angle of depression and the height range of key components of mining trucks To obtain the vertical boundary, calculate the vertical upper boundary using the following formula. ; Calculate the vertical lower boundary using the following formula: ; Pick and The intersection of these points yields the effective range in the vertical direction as follows: ; The effective monitoring area of ​​the voiceprint sensor is: ,in, , ; Based on point cloud data, parameters of industrial cameras are extracted, specifically including: Spatial coordinates are It is usually mounted on the same pole as the voiceprint sensor, but slightly taller; The horizontal direction and vertical angle of depression in the direction the mining truck is traveling are: The field of view is ; Effective shooting distance range is Due to resolution limitations, the mining card features must be clearly identifiable. Based on the parameters of the industrial camera, and following the method described above for determining the effective monitoring area (which will not be repeated here), the effective field of view of the industrial camera is calculated as follows: , , ,in, The horizontal left boundary of the camera. The horizontal right boundary of the camera. The effective range of the camera in the vertical direction; By taking the intersection of the coordinates of the effective area of ​​the voiceprint and the field of view of the camera, we obtain: ; ; ; The overlap rate is calculated as follows: (overlapping area volume / effective voiceprint area volume) × 100%; If the overlap rate is greater than or equal to the threshold T, it can be set to 80% for a valid match; otherwise, the device orientation needs to be adjusted, which can be fine-tuned. , Until the standard is met.

[0016] The time synchronization module 12 is specifically used for: A star topology time synchronization network is constructed based on an industrial-grade precision time protocol that supports hardware timestamps. Using the edge computing terminal in the system as the master clock, the voiceprint acquisition device and the image acquisition device, which are slave clocks, are synchronized. The cumulative clock drift of each slave clock is compensated by a linear correction model. The corrected time is obtained by adding the product of the current device time, the drift coefficient and the time interval since the last calibration to the current device time.

[0017] Specifically, in this embodiment of the invention, the time synchronization module deploys a Precision Time Protocol (PTP) module in the system, which can be IEEE 1588v2. The edge computing terminal is used as the master clock, and the voiceprint sensor and industrial camera are used as slave clocks. A star-shaped synchronization network is constructed through a dedicated synchronization cable. IEEE 1588v2 records the transmission and reception times of synchronization frames through hardware timestamps, which can eliminate the uncertainty of network transmission delay, achieve microsecond-level synchronization, and ensure that the voiceprint and image data are strictly aligned in the time dimension. For example, if the soundprint sensor and camera on the East 1 pole are not synchronized, it may result in a 2ms discrepancy, where "the sensor records the soundprint of the mining truck at t=10:00:00.000, and the camera records the image at the same time at t=10:00:00.002". This could cause the system to misjudge that "the soundprint comes from the previous mining truck". In this case, after synchronization via PTP, the timestamp discrepancy between the two is less than the specified threshold, and they can be accurately associated as data from the same mining truck. The specific method by which the time synchronization module 12 calculates the speed of the mining truck is as follows: The distance between two adjacent sets of position reference points is obtained from the three-dimensional space mapping module; Obtain the first and second timestamps of the same mining card passing through the two sets of reference points, after time synchronization correction; Calculate the time difference between the second time stamp and the first time stamp, and verify whether the time difference is greater than the minimum effective time threshold; The instantaneous speed is calculated based on the ratio of the interval to the time difference, and the effective driving speed is obtained by smoothing multiple consecutively calculated instantaneous speeds using a sliding window averaging algorithm.

[0018] Specifically, when the mining truck enters the monitoring area, each device is simultaneously triggered to record a timestamp: When the acoustic fingerprint sensor detects that the mechanical acoustic fingerprint of the mining truck exceeds a threshold, it records a trigger timestamp. ; Industrial cameras use motion detection algorithms (such as frame difference) to identify the time when the front of a mining truck enters the center of its field of view and record the shooting timestamp. ; Edge computing terminals need to periodically compare the deviation between the timestamps of each device and the master clock, and compensate for the cumulative drift of the device's local clock through a linear correction model; The core task of edge computing terminals is to process voiceprint data acquisition and device control in parallel. With limited computing resources, the low complexity of linear models makes them the optimal choice. The linear correction model is specifically as follows: ,in, For the corrected time, The device is at its current moment, and k is the drift coefficient. The time interval between the last time calibration completion time on the device and the current time; The local clock of a device may drift due to changes in temperature and voltage, such as the MCU clock of a voiceprint sensor and the CMOS clock of an industrial camera. If not calibrated, the time deviation may exceed the specified range after 24 hours, which will destroy the time correlation of long-term data. Based on the synchronized timestamp, associate the mining card's location and time information: Using the central axis of each group of symmetrical uprights as the position reference point, such as the midpoint of the line connecting uprights East 1 and West 1, the coordinates are... It can be extracted from point cloud data; When the mining truck passes the reference point, the trigger time is based on the soundprint sensor. With camera shooting time Since the data acquisition delay of an acoustic fingerprint sensor is typically longer than the image exposure delay of a camera, it is necessary to calculate the response delay. ; All voiceprint timestamps are uniformly corrected to equivalent image timestamps, i.e. This ensures that voiceprints and images are associated on the same time reference. The effective speed of the mining truck is calculated based on the position and time data of two adjacent sets of uprights. The specific operation is as follows: From the preprocessed point cloud data, extract the coordinates of reference points for two adjacent sets of symmetrical pillars (such as East 1-West 1 and East 2-West 2), calculate the reference point spacing L, and store L in the parameter library of the edge computing terminal. For example, if the coordinates of the previous set of reference points are... The latter group is Since the two sets of poles are distributed along the north-south direction of the road section, that is, the Y and Z coordinates are approximately the same, the spacing is... ; When the mining truck passes through the previous set of reference points (e.g., East 1 to West 1), it triggers synchronous sampling by the voiceprint sensor and camera. After correction by the time calibration module, a unified timestamp for that moment is obtained. ; When the mining truck reaches the next set of reference points (e.g., East 2 to West 2), the above process is repeated to obtain the corrected timestamp. ; Edge computing terminals are associated with the same mining card via device ID. and This ensures continuous driving data for the same mining truck; Calculate the time difference between the mining truck passing through two sets of reference points. ,like If the data indicates that the mining truck exceeded its speed limit and passed through in a short time, it is considered invalid data (because a small time difference will amplify the calculation error) and must be discarded while waiting for the next set of data. The time difference threshold; Combining spacing L with time difference Calculate the instantaneous speed of the mining truck. The sliding window algorithm is used to smooth speed fluctuations. The window size is set to M sets of continuous data, such as the speed values ​​of the most recent M passes through adjacent poles. The arithmetic mean of the values ​​within the window is calculated as the current effective speed. M needs to be set according to the actual situation.

[0019] The voiceprint analysis model in the anomaly localization module 14 is a time-aware bidirectional multi-scale network used to extract the spatiotemporal joint feature vector of the voiceprint. The time-aware bidirectional multi-scale network includes a bidirectional long short-term memory network for capturing temporal dynamic features, a group of convolutional layers for parallel extraction of multi-scale frequency domain features, and a multi-head self-attention mechanism for fusing temporal and frequency domain features.

[0020] Specifically, the voiceprints of mining trucks have the characteristics of multi-frequency superposition and temporal dynamic changes. Traditional single-scale models, such as simple CNN or RNN, are difficult to capture frequency details and time trends at the same time. However, TIM-Net's bidirectional LSTM can trace the cause and effect of voiceprints, and multi-scale convolution can accurately match the voiceprint frequency bands of different components. The combination of the two improves the feature extraction accuracy compared with traditional models. The specific steps for training the Time-Aware Bidirectional Multi-Scale Network (TIM-Net) are as follows: The raw audio data collected by the voiceprint sensor is preprocessed, specifically including frame segmentation and windowing, noise reduction, and feature normalization. The continuous audio is segmented into Gms / segment, and the Hanning window is used to reduce spectral leakage. G needs to be set according to the actual type of the original audio. To remove environmental noise in the mining area, wavelet thresholding is used. Specific parameter settings include selecting the db4 wavelet as the wavelet basis, setting the number of decomposition layers to 5, and selecting the soft thresholding function. The voiceprint data of mining trucks suffers from environmental noise superposition and spectral leakage. The Hanning window can reduce the spectral sidelobe energy, and wavelet denoising can preserve the low-frequency details of mechanical voiceprints. The Mc Wiemel coefficients of each audio segment are extracted and Z-score normalized to ensure that the input data distribution conforms to the standard normal distribution, where Mc∈[20,80]. The lower limit of 20 ensures coverage of key low-frequency bands and avoids loss of acoustic features of components such as chassis and gearbox due to insufficient dimensionality; the upper limit of 80 avoids feature redundancy caused by oversampling of high-frequency bands. High-frequency acoustic features are mostly transient signals such as friction and collision, and do not require excessively dense scales. Standardization ensures that the distribution of voiceprint data is consistent across different time periods and devices, avoiding misjudgments by the model due to different input scales, such as misjudging high-energy noise as abnormal voiceprints; The voiceprint of mining cards has a strong time-series dependence. Unidirectional LSTM can only capture past information, while bidirectional LSTM can simultaneously model the relationship between the past, present, and future, which can improve the recognition accuracy of periodic features. A bidirectional LSTM (Long Short-Term Memory) network module is constructed to capture the temporal dynamic features of voiceprints. Specific parameters include: Two-layer bidirectional LSTM, each hidden layer has 128 dimensions, the forward LSTM and the backward LSTM each have 64 dimensions, and the concatenation results in 128 dimensions; Single-layer networks struggle to capture complex temporal sequences, and more than three layers can lead to parameter redundancy, while 128 dimensions can cover most of the temporal feature variance. The input dimension is Mc, which is consistent with the dimension of the preprocessed Mel spectrum features. The dropout rate is 0.2 to prevent overfitting, and the time step is Gms, which is consistent with the frame length. After temporal feature extraction, three sets of parallel convolutional layers are used to extract multi-scale frequency features. The specific parameter configuration is as follows: High-frequency convolutional layers, settings The convolutional kernel has c output channels and a stride of 1. Intermediate frequency convolutional layer, set The convolution kernel has c output channels and a stride of 1. Low-frequency convolutional layers, set The convolutional kernel has c output channels and a stride of 1. in, , , Both 'c' and 'c' need to be set according to the data characteristics to cover different frequency ranges. Each convolutional group is followed by a ReLU activation function and global average pooling, ultimately outputting a 3×c-dimensional frequency feature vector. A multi-head self-attention mechanism is used to fuse temporal features (128 dimensions) and frequency features (3×c dimensions). Specific parameters include: The temporal features and frequency features are concatenated into a (128+3×c) dimensional vector, which is used as the input to the attention mechanism; Attention head set Each head processes Mc-dimensional features, using a weight matrix. , , After processing, we obtain a (128+3×c)-dimensional weighted feature, where, , , ∈ ; The (128+3×c) dimensional features are compressed into a 128-dimensional spatiotemporal joint feature vector through 1×1 convolution, which contains core information such as frequency distribution, temporal trend, and energy intensity. The importance of different features changes dynamically with the scene, and the attention mechanism can automatically assign weights, resulting in higher accuracy than feature fusion by simple concatenation. Collect historical normal and abnormal voiceprint samples (including components such as engines and electric wheels), and divide them into training and validation sets in a 7:3 ratio; The loss function chosen is cross-entropy loss, and the optimizer chosen is Adam. When the feature extraction accuracy on the validation set is ≥95%, training is stopped, and the model parameters are saved to obtain the trained TIM-Net.

[0021] The specific method by which the anomaly localization module 14 identifies abnormal voiceprints is as follows: Using the unique identifier of the mining truck as an index, the spatiotemporal joint feature vector, real-time location, and driving speed are integrated to construct a multi-dimensional data matrix; Calculate the Mahalanobis distance between the real-time voiceprint feature vector and the normal feature distribution obtained based on historical normal voiceprint samples; If the Mahalanobis distance exceeds the threshold set according to the standard deviation of normal samples, the current voiceprint is determined to be abnormal.

[0022] Specifically, the abnormal location module 14 identifies abnormal voiceprints by using the unique identifier of the mining truck (such as the number "K012" identified by the camera) as an index, integrating voiceprint features, real-time location, and driving speed to construct a three-dimensional matrix. Where N1 is the time step, 128 dimensions are the voiceprint features output by TIM-Net, and 3 dimensions are the real-time location of the mining truck. and driving speed v; Effective speed based on mining cards and the time to pass the previous set of poles Real-time calculation of the mining card's position at any time t: , The x-coordinate of the previous set of pole reference points, and y and z are the lateral offset of the mining card and the height of the sound source; The voiceprint feature vector at each time step is compared with the corresponding time (x(t), y, z, ...). Binding; Calculate the mean value of the historical voiceprint feature vectors of several mining trucks during normal operation. Covariance matrix ; According to the formula Calculate the Mahalanobis distance D between the real-time feature vector and the normal distribution. If the voiceprint is abnormal, the time of occurrence of the abnormality will be recorded simultaneously. Where e is the spatiotemporal joint feature vector, This represents the standard deviation of the Mahalanobis distance for normal samples.

[0023] The specific method by which the anomaly location module 14 locates the faulty component is as follows: A pre-stored abnormal voiceprint feature library for key components is provided, which includes the frequency peak range, periodicity, and energy percentage characteristics of the abnormal voiceprints of each component. When an abnormal voiceprint is detected, its key features are extracted, and its cosine similarity with the features of each component in the feature library is calculated. One or more candidate faulty components with the highest similarity are then selected. The spatial location of the mining card at the moment the anomaly occurred is compared with the coordinate range of each candidate component in the pre-stored 3D model of the mining card to finally determine the unique faulty component.

[0024] In a specific implementation of the abnormal location module 14 of this invention for locating faulty components, abnormal acoustic samples of key components of the mining truck, such as electric wheels, gearboxes, and engines, are collected in advance, and their key features are extracted and stored. The key features include frequency peak range, period, and energy percentage. For detected abnormal voiceprints, extract their key features and calculate their cosine similarity with the abnormal features of each component in the feature library, and output the top 3 candidate components with the highest similarity. according to Query the coordinates of the mine card location at the corresponding time from the three-dimensional matrix M1. The 3D model of the mining truck is called, including the coordinate range of each component, such as the z-coordinate of the electric wheel (0.7-0.9m) and the z-coordinate of the gearbox (1.2-1.5m). By comparing with the model, the specific component corresponding to the coordinates is determined from the candidate components, such as z=0.8m falling within the height range of the electric wheel; The final analysis identified the abnormal components and locations of the mining truck, such as abnormal noise from the electric wheel bearing, located at (150.2, 3.2, 0.8).

[0025] By employing the embodiments of the present invention, the following beneficial effects are achieved: The unmanned mining truck operation status acoustic monitoring system provided in this application achieves three core beneficial effects by constructing a technical system that integrates precise spatial mapping, time synchronization alignment, and intelligent acoustic analysis: First, it breaks through the limitations of traditional single physical quantity monitoring such as vibration and temperature, utilizing a time-aware bidirectional multi-scale network to deeply mine the spatiotemporal characteristics of acoustics, significantly improving the sensitivity and timeliness of early detection of faults in key components such as gears and bearings; second, through 3D point cloud modeling and precise time protocol synchronization, it achieves precise correlation between acoustics, images, and mining truck location data, fundamentally solving the problem of fusion distortion of multi-source data in spatial and temporal dimensions; third, it innovatively binds abnormal acoustics with the mining truck's motion status and component spatial coordinates dynamically, achieving a leap from regional-level alarms to component-level positioning, providing precise guidance for maintenance and significantly reducing operation and maintenance costs and time. Overall, this system improves the reliability, accuracy, and intelligence level of unmanned mining truck operation status monitoring, providing effective technical support for safe production in mining areas.

[0026] Method Implementation Examples According to an embodiment of the present invention, a method for monitoring the operating status of unmanned mining trucks using voiceprints is provided. Figure 2 This is a flowchart of the voiceprint monitoring method for the operating status of unmanned mining trucks according to an embodiment of the present invention. Figure 2 As shown, the voiceprint monitoring method for the operating status of unmanned mining trucks in this embodiment of the invention specifically includes: S1. Deploy data acquisition equipment in the monitored road section, acquire spatial point cloud data and establish a unified three-dimensional model, determine the effective monitoring area of ​​the soundprint and image acquisition equipment and ensure that the overlap rate meets the standard. S2. Synchronize the clocks of all acquisition devices based on a precision time protocol, timestamp the data based on the event of the mining truck passing through the location reference point, and calculate the speed of the mining truck. S3. Process the voiceprint data and extract features. Identify anomalies through the voiceprint analysis model. Combine the anomaly timestamp, driving speed and three-dimensional model to locate the spatial position of the mining truck, and match it with the component coordinates to determine the specific faulty component.

[0027] Device Examples According to an embodiment of the present invention, an electronic device includes: Processor; and, A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the voiceprint monitoring method for the unmanned mining truck's operating status.

[0028] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voiceprint monitoring system for the operating status of unmanned mining trucks, characterized in that, It includes a 3D spatial mapping module, a time synchronization module, and an anomaly localization module; The three-dimensional spatial mapping module is used to symmetrically deploy multiple sets of poles on both sides of the monitored road section, and install voiceprint acquisition devices and image acquisition devices on the poles; acquire spatial point cloud data containing all poles and acquisition devices through a three-dimensional scanning device, and establish a unified three-dimensional spatial model based on the point cloud data; calculate the effective voiceprint monitoring area of ​​each voiceprint acquisition device and the effective image field of view of each image acquisition device based on the three-dimensional spatial model, and ensure that the overlap rate between the effective voiceprint monitoring area and the effective image field of view is not lower than a preset threshold. The time synchronization module is used to construct a star-shaped synchronization network based on a precision time protocol, with the edge computing terminal as the master clock, and to synchronize the voiceprint acquisition device and the image acquisition device as slave clocks. The cumulative clock drift of each slave clock is compensated by a linear correction model; based on the synchronized and corrected timestamp, the event of the mining truck passing through the preset location reference point is associated, and the speed of the mining truck is calculated. The anomaly localization module is used to process the voiceprint data collected by the voiceprint acquisition device through a time-aware bidirectional multi-scale network, extract a spatiotemporal joint feature vector, and identify abnormal voiceprints based on the comparison between the spatiotemporal joint feature vector and the historical normal voiceprint feature distribution. In response to the detection of abnormal voiceprints, the spatial location of the mining truck at the time of the abnormality is determined by combining the time of the abnormality, the speed of the mining truck, and the three-dimensional spatial model; the spatial location is matched with the pre-stored three-dimensional component model of the mining truck to locate the specific component that malfunctioned.

2. The system according to claim 1, characterized in that, The symmetrical arrangement on both sides specifically includes: Multiple sets of poles are symmetrically arranged on both sides of the monitored road section, with the line connecting each set of poles perpendicular to the center line of the road surface; The voiceprint acquisition device is installed at a designated height on the pole, with its sound pickup direction facing the critical component area in the middle of the vehicle body of the mining truck in the opposite lane. The image acquisition device is installed on one of the poles, and its shooting direction is tilted along the direction of the mining truck's travel, focusing on the mining truck's identification feature area; The three-dimensional scanning equipment includes a three-dimensional laser profile sensor for local fine scanning and a three-dimensional mapping lidar for supplementary scanning.

3. The system according to claim 2, characterized in that, The process of establishing a unified 3D model by the 3D spatial mapping module also includes: The intersection of point cloud data obtained from multiple scans by a 3D scanning device is used to remove outliers caused by occlusion. The point cloud data is denoised and stitched using a point cloud processing algorithm; Based on the processed point cloud data, the three-dimensional spatial coordinates, horizontal orientation angle, vertical pitch angle and pickup angle parameters of the voiceprint acquisition device are extracted, and the three-dimensional spatial coordinates, horizontal orientation angle, vertical pitch angle and field of view parameters of the image acquisition device are also extracted. Based on the extracted voiceprint acquisition device parameters, the effective voiceprint monitoring area is calculated using a spatial geometric model, and based on the extracted image acquisition device parameters, the effective field of view area of ​​the image is calculated, and then the volume overlap rate of the two is calculated.

4. The system according to claim 1, characterized in that, The time synchronization module is specifically used for A star topology time synchronization network is constructed based on an industrial-grade precision time protocol that supports hardware timestamps. Using the edge computing terminal in the system as the master clock, the voiceprint acquisition device and the image acquisition device, which are slave clocks, are synchronized. The cumulative clock drift of each slave clock is compensated by a linear correction model. The corrected time is obtained by adding the product of the current device time, the drift coefficient and the time interval since the last calibration to the current device time.

5. The system according to claim 1, characterized in that, The specific method by which the time synchronization module calculates the speed of the mining truck is as follows: The distance between two adjacent sets of position reference points is obtained from the three-dimensional space mapping module; Obtain the first and second timestamps of the same mining card passing through the two sets of reference points, after time synchronization correction; Calculate the time difference between the second time stamp and the first time stamp, and verify whether the time difference is greater than the minimum effective time threshold; The instantaneous speed is calculated based on the ratio of the interval to the time difference, and the effective driving speed is obtained by smoothing multiple consecutively calculated instantaneous speeds using a sliding window averaging algorithm.

6. The system according to claim 1, characterized in that, The voiceprint analysis model in the anomaly localization module is a time-aware bidirectional multi-scale network used to extract the spatiotemporal joint feature vector of the voiceprint. The time-aware bidirectional multi-scale network includes a bidirectional long short-term memory network for capturing temporal dynamic features, a group of convolutional layers for parallel extraction of multi-scale frequency domain features, and a multi-head self-attention mechanism for fusing temporal and frequency domain features.

7. The system according to claim 6, characterized in that, The specific method by which the anomaly localization module identifies abnormal voiceprints is as follows: Using the unique identifier of the mining truck as an index, the spatiotemporal joint feature vector, real-time location, and driving speed are integrated to construct a multi-dimensional data matrix; Calculate the Mahalanobis distance between the real-time voiceprint feature vector and the normal feature distribution obtained based on historical normal voiceprint samples; If the Mahalanobis distance exceeds the threshold set according to the standard deviation of normal samples, the current voiceprint is determined to be abnormal.

8. The system according to claim 1, characterized in that, The specific method by which the anomaly location module locates faulty components is as follows: A pre-stored abnormal voiceprint feature library for key components is provided, which includes the frequency peak range, periodicity, and energy percentage characteristics of the abnormal voiceprints of each component. When an abnormal voiceprint is detected, its key features are extracted, and its cosine similarity with the features of each component in the feature library is calculated. One or more candidate faulty components with the highest similarity are then selected. The spatial location of the mining card at the moment the anomaly occurred is compared with the coordinate range of each candidate component in the pre-stored 3D model of the mining card to finally determine the unique faulty component.

9. A monitoring method based on the voiceprint monitoring system for the operating status of an unmanned mining truck according to any one of claims 1-8, characterized in that, The method includes: S1. Deploy data acquisition equipment in the monitored road section, acquire spatial point cloud data and establish a unified three-dimensional model, determine the effective monitoring area of ​​the soundprint and image acquisition equipment and ensure that the overlap rate meets the standard. S2. Synchronize the clocks of all acquisition devices based on a precision time protocol, timestamp the data based on the event of the mining truck passing through the location reference point, and calculate the speed of the mining truck. S3. Process the voiceprint data and extract features. Identify anomalies through the voiceprint analysis model. Combine the anomaly timestamp, driving speed and three-dimensional model to locate the spatial position of the mining truck, and match it with the component coordinates to determine the specific faulty component.

10. An electronic device, comprising: processor; as well as, A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the voiceprint monitoring method for the operating status of the unmanned mining truck as described in claim 9.

Images