A Mining Intelligent Voice and Video Interactive Communication Method and System

Abstract

This application relates to the field of communication technology and discloses a method and system for intelligent voice and video interactive communication in mining, aiming to solve the problems of poor audio and video signal quality, low bandwidth utilization, lack of semantic understanding, and interaction lag in the complex environment of mines. The method includes: acquiring raw signals through an intrinsically safe audio and video terminal; performing voice enhancement using sound source orientation estimation and a depth complex network; reconstructing high dynamic range video based on a physical imaging model; achieving audio and video semantic alignment and key event extraction using a lightweight spatiotemporal feature fusion network; dynamically allocating bandwidth based on event confidence and channel state, and transmitting high / ordinary priority data in a tiered manner; and achieving real-time communication between the ground and underground via a two-way interactive channel. This application significantly improves signal availability, bandwidth efficiency, and emergency response capabilities, ensuring highly stable, low-latency, and high-fidelity communication.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, specifically relating to a mining intelligent voice and video interactive communication method and system. Background Technology

[0002] With the deepening of intelligent construction in mines, the underground working environment has placed higher demands on efficient, reliable, and real-time communication methods. The mine environment is characterized by complex spatial structure, strong electromagnetic interference, poor lighting conditions, high dust concentration, and frequent movement of personnel and equipment, which leads to a series of problems for traditional communication systems in audio and video signal transmission, such as severe signal attenuation, significant quality degradation, limited bandwidth resources, and delayed interactive response.

[0003] Currently, mining communication systems mostly employ traditional solutions such as wired telephones, analog intercoms, or narrowband wireless voice transmission. Their architecture is typically based on fixed channels and static protocols, making it difficult to adapt to the dynamically changing underground working environment. In terms of voice communication, due to high environmental noise and severe acoustic reverberation, voice signals are prone to distortion, interruption, or delay, severely affecting communication clarity and reliability. Regarding video surveillance, limited by low illumination, high-contrast lighting conditions, and limited network bandwidth underground, video images often suffer from details loss, motion blur, and localized overexposure or underexposure, making it difficult to achieve high-quality, real-time video transmission and recognition.

[0004] Furthermore, existing systems typically process voice and video as independent transport streams, lacking cross-modal semantic association and intelligent collaborative capabilities. When abnormal events occur underground (such as personnel falls, equipment failures, or sudden changes in environmental parameters), the system cannot automatically identify and integrate audio and video information for comprehensive analysis, nor can it adaptively adjust transmission strategies according to the urgency of the event. This results in fragmented information transmission and slow emergency response, failing to meet the urgent needs of modern mines for real-time monitoring, rapid early warning, and collaborative command.

[0005] Therefore, there is an urgent need for a communication method and system that can adapt to the complex environment of mines, achieve high-quality audio and video acquisition, intelligent semantic recognition, adaptive bandwidth scheduling, and real-time two-way interaction, so as to improve the level of intelligence in mine safety production and emergency response capabilities. Summary of the Invention

[0006] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a mining intelligent voice and video interactive communication method and system. It deploys intrinsically safe audio and video terminal nodes with multimodal perception capabilities underground, combining an acoustic scene-adaptive voice enhancement algorithm with a low-light, high dynamic range video reconstruction model to preprocess the original audio and video signals. Simultaneously, it constructs a localized communication scheduling center centered on an edge computing gateway, utilizing a lightweight spatiotemporal feature fusion network to perform semantic-level alignment and key event extraction on the audio and video streams. Based on this, a dynamic bandwidth allocation mechanism based on joint decision-making of task priority and channel state is adopted to transmit the processed multimedia data to the ground command center according to urgency, and simultaneously establishes a two-way real-time interactive channel, achieving highly stable, low-latency, and high-fidelity voice and video collaborative communication between underground workers and the ground control terminal.

[0007] According to one aspect of this application, a mining intelligent voice and video interactive communication method is provided, comprising:

[0008] By deploying intrinsically safe audio and video acquisition terminals in key locations of mine roadways and on mobile devices, the original audio and video signals of the underground working area are acquired simultaneously. The original audio signal contains environmental noise, mechanical vibration interference and multiple voices mixed together. The original video signal has brightness imbalance, blurred details and motion blur under low light, high dust and strong reflective conditions.

[0009] The original audio signal is subjected to speech enhancement processing based on sound source location estimation and spectrum masking. Specifically, it includes: using multi-channel audio data received by the microphone array, calculating the direction of arrival of the sound source through a generalized cross-correlation phase transformation algorithm, and filtering effective speech sources by combining the preset spatial prior information of the personnel activity area; then using a deep complex convolutional neural network to perform complex domain masking on the spectrogram corresponding to the effective speech source, suppressing background noise and non-target speaker speech, and outputting an enhanced single-channel clean speech signal.

[0010] The original video signal is subjected to high dynamic range reconstruction processing guided by a physical imaging model. Specifically, this includes: performing sub-pixel level motion compensation registration on multiple consecutive low dynamic range video images, constructing a local exposure response function model, using this model to reverse map and restore the pixel values ​​of overexposed and underexposed areas, and generating a high dynamic range video frame sequence with complete brightness and darkness details.

[0011] The enhanced single-channel clean voice signal and high dynamic range video frame sequence are input into a lightweight spatiotemporal feature fusion network deployed on the downhole edge computing gateway. The lightweight spatiotemporal feature fusion network adopts an architecture that combines three-dimensional convolution and attention mechanisms to perform cross-modal temporal alignment of audio and video streams and extract key event feature vectors that characterize the safety status of the operation. Key events include personnel falls, abnormal equipment operation, signs of gas leaks, and illegal intrusion.

[0012] Based on the confidence score of the key event feature vector, the channel quality index of the current communication link, and the preset task urgency level threshold, a dynamic bandwidth allocation decision is made to divide the audio and video data to be transmitted into a high-priority stream and a normal-priority stream. The high-priority stream contains key event segments with a confidence score higher than the first threshold and the context data before and after them for five seconds, while the normal-priority stream contains the remaining regular monitoring data.

[0013] The high-priority stream is transmitted losslessly through a reserved bandwidth channel via a mining industrial ring network, while the ordinary-priority stream is transmitted lossily through variable bit rate compression. Both adopt forward error correction coding and interleaving retransmission mechanism to ensure transmission reliability.

[0014] At the ground command center receiving end, the received high-priority stream and ordinary-priority stream are decoded and reassembled to restore a continuous audio and video playback sequence. At the same time, a downlink voice command channel initiated by the ground operator is established. The downlink voice command is transmitted in reverse through the same path to the speaker unit of the underground audio and video acquisition terminal to realize two-way real-time interaction.

[0015] According to another aspect of this application, a mining intelligent voice and video interactive communication system is provided, comprising:

[0016] The intrinsically safe audio and video acquisition terminal module for mining is used to be deployed in key locations in mine roadways and on mobile devices to simultaneously acquire the original audio and video signals of the underground working area.

[0017] The speech enhancement processing module is used to perform speech enhancement processing on the original audio signal based on sound source location estimation and spectrum masking, and output the enhanced single-channel clean speech signal.

[0018] The high dynamic range video reconstruction module is used to perform high dynamic range reconstruction processing on the original video signal based on a physical imaging model to generate a high dynamic range video frame sequence.

[0019] The edge computing gateway module integrates a lightweight spatiotemporal feature fusion network, which is used to perform cross-modal temporal alignment between the enhanced single-channel clean speech signal and the high dynamic range video frame sequence, and extract key event feature vectors that characterize the safety status of the operation.

[0020] The dynamic bandwidth allocation decision module is used to divide the audio and video data to be transmitted into high-priority streams and ordinary-priority streams based on the confidence score of the key event feature vector, the channel quality index of the current communication link, and the preset task urgency level threshold.

[0021] The hierarchical transmission module is used to transmit high-priority streams with lossless compression through reserved bandwidth channels via the mining industrial ring network, and transmit ordinary-priority streams with lossy compression in a variable bit rate manner.

[0022] The two-way interaction module is used to decode and reassemble the received data at the ground command center receiving end, and to establish a downlink voice command channel initiated by the ground operator to achieve two-way real-time interaction.

[0023] As one embodiment of the present invention, the intrinsically safe audio and video acquisition terminal module for mining includes an explosion-proof shell, an intrinsically safe power management unit, a quaternary linear microphone array, a wide dynamic range image sensor, an infrared illumination unit, and an industrial-grade embedded processor; the quaternary linear microphone array has a spacing of 10 centimeters between adjacent microphones and an operating frequency range of 100 Hz to 8 kHz; the wide dynamic range image sensor has a photosensitive area of ​​one inch, a maximum resolution of 1,920 x 1,080, and a minimum illumination of 0.001 lux; the infrared illumination unit has a center wavelength of 850 nanometers, an illumination angle of 90 degrees, and a power of no more than 1.5 watts.

[0024] As one embodiment of the present invention, the deep complex convolutional neural network used in the speech enhancement processing module includes three encoder stages, one bottleneck layer and three decoder stages. Each encoder stage consists of a two-dimensional convolutional layer, a batch normalization layer and a complex gated recurrent unit. The bottleneck layer adopts a dilated convolutional structure to expand the receptive field. The decoder stage fuses with the feature map of the corresponding encoder stage through skip connections, and finally outputs a complex masking matrix composed of real and imaginary parts.

[0025] As one embodiment of the present invention, the local exposure response function model constructed in the high dynamic range video reconstruction module adopts a piecewise linear fitting method, dividing the pixel response curve into three segments: dark area, linear area and saturation area, and fitting the slope parameter and offset parameter respectively. The fitting process is achieved by minimizing the weighted sum of the luminance consistency error between adjacent frames and the gradient smoothing constraint term.

[0026] As one embodiment of the present invention, the input of the lightweight spatiotemporal feature fusion network is a synchronously aligned audio Mel spectrogram sequence and a video frame sequence. First, audio temporal features are extracted through a one-dimensional convolutional layer, and video spatial features are extracted through a two-dimensional convolutional residual block. Then, the two are concatenated and input into a three-dimensional convolutional layer for spatiotemporal joint modeling. Finally, the correlation weights between audio and video features are calculated through a multi-head cross-attention mechanism, and the weighted fusion is fed into a fully connected classification head to output key event categories and confidence scores.

[0027] As one embodiment of the present invention, the preset task urgency level thresholds in the dynamic bandwidth allocation decision module include three levels: the first level threshold corresponds to events that directly threaten personnel life safety, and the confidence score must be greater than 0.9; the second level threshold corresponds to major equipment failures or sudden environmental changes, and the confidence score must be greater than 0.75; the third level threshold corresponds to general violations of regulations, and the confidence score must be greater than 0.6; the channel quality index is calculated by weighting the bit error rate and signal-to-noise ratio fed back from the physical layer, with weighting coefficients of 0.6 and 0.4, respectively.

[0028] In one embodiment of the present invention, the high-priority stream in the hierarchical transmission module adopts a lossless compression algorithm of improved context-adaptive binary arithmetic coding, with a compression ratio of not less than 2:1; the ordinary priority stream adopts variable bitrate H.265 coding based on region of interest, with a higher bitrate allocated to the face and device operation panel areas, and a lower bitrate allocated to the remaining background areas; the forward error correction coding adopts Lisso code, with a bitrate of two-thirds and an interleaving depth of sixteen.

[0029] As one embodiment of the present invention, the two-way interaction module is equipped with a voice activation detection unit at the receiving end of the ground command center, which is used to identify the start and end boundaries of the operator's voice commands and only open the downlink transmission channel during the period when valid voice is detected, so as to avoid occupying bandwidth during the silent period. The downlink voice commands are output after echo cancellation processing at the downhole terminal side. The echo cancellation adopts the normalized least mean square adaptive filtering algorithm with a filter order of 512.

[0030] In summary, this application includes at least one of the following beneficial technical effects:

[0031] This invention effectively overcomes the severe degradation of the original signal quality caused by the complex acoustic and optical environment of the mine by deploying an intrinsically safe terminal with joint acoustic and optical sensing capabilities at the underground front end, and combines targeted voice enhancement and video reconstruction algorithms, thus significantly improving the availability of audio and video input.

[0032] By introducing a lightweight cross-modal feature fusion network at the edge, localized real-time identification and semantic extraction of safety-critical events are achieved, avoiding the bandwidth pressure caused by uploading massive amounts of raw data to the ground. By establishing a dynamic hierarchical transmission mechanism based on the urgency of the event and the channel state, reliable and low-latency delivery of high-risk event data is prioritized under limited bandwidth resources.

[0033] By constructing a complete two-way interactive closed loop, ground command personnel can intervene in emergencies underground in a timely manner. The overall solution significantly improves the intelligence, stability, and emergency response capabilities of the mining audio-visual communication system without increasing underground communication infrastructure, and solves the inherent defects of existing systems such as poor signal quality, low bandwidth utilization, lack of semantic understanding, and interaction lag. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall technical solution architecture of a mining intelligent voice and video interactive communication method and system proposed in this invention;

[0035] Figure 2 This is a schematic diagram of the core principle framework of the acoustic scene-adaptive speech enhancement and low-light high dynamic range video reconstruction in this invention. Detailed Implementation

[0036] This invention provides a method and system for intelligent voice and video interactive communication in mines, aiming to solve the technical problems of existing mine communication systems, such as severe signal quality degradation, low bandwidth resource utilization efficiency, lack of semantic recognition capability for safety events, and delayed interactive response in complex acoustic and optical environments. By deploying intrinsically safe audio and video terminal nodes with multimodal perception capabilities underground, combined with front-end signal enhancement and reconstruction algorithms, a lightweight cross-modal feature fusion network at the edge, and a dynamic bandwidth allocation mechanism based on joint decision-making of task priority and channel state, highly stable, low-latency, and high-fidelity two-way voice and video collaborative communication is achieved.

[0037] The first aspect is the intelligent voice and video interactive communication method for mining disclosed in this application, which specifically includes the following steps:

[0038] S1, through mining intrinsically safe audio and video acquisition terminals deployed at key locations in mine roadways and on mobile devices, synchronously acquires the original audio and video signals of the underground working area;

[0039] S2 performs speech enhancement processing on the original audio signal based on sound source location estimation and spectral masking;

[0040] S3 performs high dynamic range reconstruction processing on the original video signal based on a physical imaging model;

[0041] S4 inputs the enhanced single-channel clean speech signal and high dynamic range video frame sequence into a lightweight spatiotemporal feature fusion network deployed on the downhole edge computing gateway to perform cross-modal temporal alignment and extract key event feature vectors.

[0042] S5 performs dynamic bandwidth allocation decisions based on the confidence score of key event feature vectors, the channel quality index of the current communication link, and the preset task urgency level threshold.

[0043] S6 transmits high-priority streams with lossless compression through a reserved bandwidth channel via the mining industrial ring network, and transmits ordinary-priority streams with lossy compression using a variable bit rate.

[0044] The S7 decodes and reassembles the received data at the ground command center receiving end, and establishes a downlink voice command channel initiated by the ground operator to achieve two-way real-time interaction.

[0045] In step S1, the original audio and video signals of the underground working area are simultaneously acquired by the intrinsically safe audio and video acquisition terminals deployed at key locations in the mine roadways and on mobile devices. This includes the following sub-steps:

[0046] S101: Deploy intrinsically safe audio and video acquisition terminals for mining at mine roadway intersections, working face entrances, along conveyor belts, in mining equipment control rooms, and on personnel positioning card carrying devices to ensure full coverage of high-risk areas and key operational nodes underground.

[0047] S102: The terminal includes an explosion-proof housing, an intrinsically safe power management unit, a quad linear microphone array, a wide dynamic range image sensor, an infrared fill light unit, and an industrial-grade embedded processor.

[0048] Through the above sub-steps S101-S102, the physical deployment strategy and core hardware composition of the terminal are clarified, laying a physical foundation for subsequent high-quality signal acquisition.

[0049] S103: A four-element linear microphone array is arranged equidistantly in the horizontal direction, with an adjacent microphone spacing of 10 cm. The operating frequency range covers 100 Hz to 8000 Hz to meet the requirements for complete capture of the fundamental frequency of human voice and its harmonic components.

[0050] S104: The wide dynamic range image sensor adopts a back-illuminated CMOS structure with a photosensitive area of ​​1 inch, a maximum resolution of 1920 x 1080, and a minimum illumination sensitivity of 0.001 lux, thus enabling imaging with infrared illumination in the absence of visible light.

[0051] S105: The center wavelength of the infrared supplementary lighting unit is 850 nanometers, the illumination angle is 90 degrees, and the output power is strictly limited to within 1.5 watts to ensure compliance with the explosion-proof standards for intrinsically safe electrical equipment.

[0052] S106: An industrial-grade embedded processor running a real-time operating system, responsible for synchronously acquiring, timestamping, and locally buffering raw audio and video signals. The sampling frequency is uniformly set to 48 kHz for audio and 30 frames per second for video. All acquired data is temporarily stored in a double buffer in its uncompressed raw format, awaiting use by subsequent processing modules.

[0053] After completing the terminal deployment and synchronous acquisition of the original audio and video signals in step S1, the system obtains initial data containing information about the complex downhole environment. The original audio signal is typically mixed with strong environmental noise, mechanical vibration interference, and multiple voices overlapping, directly affecting speech clarity and the accuracy of subsequent semantic analysis. Therefore, this method proceeds to step S2, which aims to perform targeted enhancement processing on the original audio signal to extract clean, identifiable speech components, providing high-quality audio input for subsequent cross-modal feature fusion and key event recognition.

[0054] In step S2, speech enhancement processing based on sound source location estimation and spectral masking is performed on the original audio signal, specifically including the following sub-steps:

[0055] S201: The original audio signal contains broadband steady-state noise generated by large mechanical equipment such as fans, crushers, and conveyor belts, transient impact interference caused by drilling, blasting, and rockfall, and speech aliasing caused by multiple workers speaking at the same time.

[0056] S202: Sound source arrival direction estimation based on microphone array.

[0057] Using four-channel audio data received by a quaternary linear microphone array, a generalized cross-correlation phase transform algorithm is performed to estimate the direction of arrival of the sound source, specifically including:

[0058] S202a: Perform a short-time Fourier transform on each frame of the four-channel audio signal to obtain its complex spectrum X. i (f), where i = 1, 2, 3, 4 represent channel indices.

[0059] S202b: For each pair of microphone channels i and f, calculate its generalized cross-correlation function using the following formula:

[0060]

[0061] in, For X i The complex conjugate of (f), where τ is the time delay variable.

[0062] S202c: R calculated across all possible channel pairs (e.g., adjacent channel pairs) ij In (τ), find the time delay value τ corresponding to the peak value of its absolute value. peak .

[0063] S202d: Based on the geometric layout of the quaternary linear microphone array, the time delay estimate τ is... peak Converting this to the incident angle θ of the sound source, for a uniform linear array with a spacing of d (d = 0.1 meters in this embodiment), the angle θ between the sound source direction and the array normal can be expressed by the formula... Estimate the velocity of sound, where c is the speed of sound (a typical value can be taken in a mining environment, such as c = 340 m / s). Solving this equation will yield an estimate of the direction angle θ of the sound source.

[0064] S203: Based on the preset prior information about the personnel activity area, determine the validity of the sound source incident angle θ estimated in step S202, specifically including:

[0065] S203a: Acquire spatial prior information and terminal pose. Import the predefined three-dimensional coordinate boundaries of legal work areas, equipment restricted areas, and escape routes from the mine digital twin model. Simultaneously, acquire the known installation position coordinates (x_t, y_t, z_t) of the audio / video acquisition terminal in the digital twin model and the direction vector of its array axis.

[0066] S203b: Estimate the spatial location of the sound source. A beamforming algorithm based on a near-field model is used. Based on the sound source incident angle θ obtained in step S202, the multi-channel signals received by the four-element linear microphone array are delayed and weighted in that direction (beamforming) to form a beam focused in the θ direction. By scanning different distances along the beam focus along this ray, the beam output energy corresponding to each focus distance is calculated. The distance corresponding to the focus with the highest energy is taken as the estimated value of the sound source distance R. Combining the known terminal position (x_t, y_t, z_t), array direction and angle θ, and distance R, the three-dimensional position coordinates (x_s, y_s, z_s) of the sound source in the digital twin coordinate system can be determined through geometric calculations.

[0067] S203c: Validity determination. Perform the following two checks:

[0068] Spatial region check: Determine whether the sound source location (x_s, y_s, z_s) estimated in step S203b is within the three-dimensional boundary of the "legal work area" defined by the digital twin model.

[0069] Distance check: Calculate the Euclidean distance D between the sound source location (x_s, y_s, z_s) and the terminal location (x_t, y_t, z_t).

[0070] A sound source is considered a valid speech source only if both conditions are met: "the sound source is located within a legal work area" and "the distance to D is no more than 5 meters".

[0071] S203d: Output the judgment result. Output the judgment result (valid / invalid) and the corresponding estimated location and distance information of the sound source to guide subsequent steps.

[0072] S204: Based on the validity determination result output in step S203d, perform the following operations:

[0073] S204a: Determine the time range of the audio segment. When a sound source is determined to be valid, based on the determination time t_0, extract multi-channel audio data with a time range of [t_0-T_pre, t_0+T_post] from the original audio data buffer as an audio segment.

[0074] Where T_pre is the pre-buffer time and T_post is the post-buffer time, the sum of which, T_total, should at least cover the minimum analysis time required for subsequent network processing (for example, if T_pre=0.5 seconds and T_post=1.5 seconds, then T_total=2 seconds).

[0075] S204b: Perform segment preprocessing. The extracted multi-channel audio segment (with the same number of channels as the microphone array, e.g., 4 channels) is sampled at 48kHz, pre-emphasized, and high-pass filtered to boost high-frequency components and suppress low-frequency noise. This is then used as input to a subsequent deep complex convolutional neural network.

[0076] S205: Construct and apply a deep complex convolutional neural network for spectral masking.

[0077] The multi-channel audio segments extracted and preprocessed in step S204 are converted into complex-form short-time Fourier transform spectrograms with dimensions of 321 frequency points multiplied by 200 time frames, and then input into a deep complex convolutional neural network for spectral masking. The network structure includes:

[0078] The encoder consists of three stages: each stage contains a two-dimensional convolutional layer (3x3 kernel, stride 2), a batch normalization layer, a complex activation function (such as CReLU), and a complex gated recurrent unit.

[0079] A bottleneck layer: a dilated convolutional structure with a dilation rate of 4 is used, followed by batch normalization and complex activation functions to expand the receptive field without increasing the number of parameters.

[0080] The three decoder stages are as follows: Each stage first upsamples the input feature map, then concatenates it with the feature map output from the corresponding encoder stage through skip connections, and then passes it through a two-dimensional convolutional layer (3x3 kernel, stride 1), a batch normalization layer, and a complex activation function.

[0081] The network ultimately outputs a complex masking matrix consisting of real and imaginary parts.

[0082] S206: The complex masking matrix obtained in step S205 is multiplied element-wise with the original complex spectrum corresponding to the audio segment extracted in step S204 to suppress background noise and non-target speaker components, resulting in an enhanced multi-channel complex spectrum. Subsequently, the enhanced multi-channel complex spectra are summed along the channel dimension and merged into a single-channel complex spectrum.

[0083] The inverse short-time Fourier transform (ISTFT) is performed on the single-channel complex spectrum to restore it to the time domain signal, and the output is a single-channel clean speech signal. The sampling rate of the output signal is maintained at 48 kHz, and its signal-to-noise ratio (SNR) is improved by no less than 15 dB.

[0084] In the unique environment of a mine, the integrity of communication quality depends not only on clear voice but also on the availability and detail rendering capabilities of video surveillance footage. Uneven lighting, dust scattering, and strong reflections from equipment underground often lead to problems such as localized overexposure and severe underexposure, loss of detail, and motion blur in video images, severely hindering surface personnel's visual judgment and analysis of the operational status. Therefore, this method proceeds to step S3, which aims to perform high dynamic range reconstruction processing on the original video signal to restore and enhance the brightness and darkness details of the image, providing high-quality, information-complete video input for subsequent cross-modal fusion with the enhanced audio stream.

[0085] In step S3, high dynamic range reconstruction processing based on a physical imaging model is performed on the original video signal, specifically including the following sub-steps:

[0086] S301: Due to uneven lighting, dust scattering, and strong reflection from metal surfaces, the original video signal generally suffers from both local overexposure (such as areas directly illuminated by lighting) and severe underexposure (such as deep in the tunnel), resulting in the loss of key detail information.

[0087] S302: Select five consecutive frames of low dynamic range (LDR) video images. Using the first frame as the reference frame, perform the following operations on each subsequent frame:

[0088] S302a: A dense optical flow algorithm based on gradient consistency constraints is adopted. This algorithm estimates the two-dimensional displacement vector (i.e. optical flow) of each pixel between the reference frame and the current frame by assuming that the image brightness and gradient of corresponding points between adjacent frames remain constant and minimizing the weighted sum of brightness error and gradient difference.

[0089] S302b: Using the displacement vector field calculated in step S302a, each pixel in the current frame is mapped back to a sub-pixel position in the reference frame coordinate system. Then, using bicubic interpolation, the interpolated gray value is calculated based on the gray values ​​of the 16 neighboring integer pixels around the sub-pixel position in the reference frame, and this is used as the new gray value of that pixel in the current frame after motion compensation.

[0090] After the above processing, a five-frame image sequence is obtained that is aligned with the first frame at sub-pixel precision, effectively eliminating the blur caused by camera shake or scene motion.

[0091] S303: Establish a local exposure response function model to describe the relationship between image sensor pixel values ​​and the radiance of a real scene. This model divides the pixel response curve into three intervals:

[0092] Dark areas: pixel values ​​less than 30;

[0093] Linear region: pixel values ​​between 30 and 220;

[0094] Saturation zone: Pixel value greater than 220.

[0095] For each interval, a linear function is fitted with slope and offset parameters.

[0096] S304: Solve for the parameters of the model in step S303 by minimizing an objective function. This objective function is a weighted sum of the photometric consistency error between adjacent frames and the gradient smoothing constraint term of the response function, and its expression is as follows:

[0097]

[0098] Z p,t Let g be the observation value of pixel p in frame t. k Let E be the response function for the k-th segment. p Let Δt be the scene radiance. t Let be the exposure time, w(·) be the weighting function (used to reduce the influence of saturation and noise-prone areas), and λ be the smoothing regularization coefficient. This objective function is solved using an iterative optimization algorithm (such as the Gauss-Newton method) to obtain a complete and smooth exposure response curve g(z).

[0099] S305: Using the exposure response curve g(z) obtained in step S304, calculate the pixel observation value Z for the overexposed (pixel value close to 255) and underexposed (pixel value close to 0) regions in the original LDR video frame. p,t Perform inverse mapping to recover the corresponding true logarithmic radiance lnEp, calculated using the following formula: , where Δt is the exposure time of the t-th frame.

[0100] Subsequently, multiple lnE values ​​were recovered for each pixel p in all frames (t=1,...,5). p The estimated values ​​are fused to synthesize the final HDR radiosity map. The fusion uses a weighted average method, with a weight function w(Z) used to evaluate the reliability of different pixel values ​​(typically, medium-brightness areas are the most reliable). One specific implementation uses a Gaussian weight function:

[0101] Then the logarithmic radiance at pixel p after fusion for: By performing the above calculations on all pixels in the image, a high dynamic range (HDR) video frame sequence with full brightness and darkness details is finally synthesized.

[0102] S306: The synthesized HDR video frame sequence has its dynamic range extended to 16 bits per channel, which can effectively preserve key visual information such as the texture of the anchor bolts on the top of the underground roadway, the readings of equipment instruments, and the facial features of personnel, and output it as a high-quality video stream.

[0103] After completing the high dynamic range video reconstruction processing in step S3, in order to achieve the leap from raw signal to intelligent understanding and support event-priority-based decision-making, it is necessary to deeply fuse and understand the information from these two modalities in terms of time and semantics. Based on this, step S4 is performed.

[0104] In step S4, the enhanced single-channel clean speech signal and high dynamic range video frame sequence are input into a lightweight spatiotemporal feature fusion network deployed on the downhole edge computing gateway to perform cross-modal temporal alignment and extract key event feature vectors. This includes the following sub-steps:

[0105] S401: A lightweight spatiotemporal feature fusion network is deployed in an edge computing gateway located in a mining area substation or central pump room, or other chamber with stable power supply and heat dissipation. This gateway connects to each audio / video acquisition terminal via Gigabit Ethernet. The network input consists of time-synchronized aligned audio Mel-spectrum sequences and video frame sequences, with each input sample having a 2-second time window.

[0106] The above steps clarify the deployment environment and connection method of the core computing unit (edge ​​gateway), and define the form and specifications of network input data, ensuring the stability of the processing flow and the standardization of input data.

[0107] S402: Extract temporal features through audio branches.

[0108] The input audio Mel spectrogram sequence (with a tensor shape of [batch size, T, M], where T is the number of time frames and M is the number of Mel bands, e.g., M=64) is input to the audio branch. This branch first passes through a one-dimensional convolutional layer, configured as follows:

[0109] Input channel count: M (same as Mel band count). Output channel count / feature dimension: 128. Kernel size: 64. Stride: 2. Convolution direction: Convolution along the time dimension (i.e., a dimension of length T) to extract local temporal patterns. Other components: After this one-dimensional convolutional layer, batch normalization and rectified linear unit activation functions are applied sequentially.

[0110] S403: Repeatedly expand the features (dimension 128) output by the audio branch along the time dimension to align them with the video feature map in the spatial dimension, and then concatenate them with the video features (7x7x256) in the channel dimension to form a fusion tensor of size 7x7x384.

[0111] The fused tensor is input into three 3D convolutional layers for spatiotemporal joint modeling. The kernel size of each 3D convolutional layer is 2x3x3, and the stride is 1.

[0112] S404: Apply a multi-head cross-attention mechanism for refined feature interaction.

[0113] A multi-head cross-attention module is introduced to perform refined interaction on the features after spatiotemporal modeling, specifically including the following operations:

[0114] S404a: Feature Projection and Multi-Head Partitioning. The video features (as contextual information) and audio features (as the sequences of interest) after spatiotemporal modeling are input into three independent linear transformation layers (fully connected layers) to generate query, key, and value vectors. Subsequently, Q, K, and V are evenly divided into h heads along the channel dimension (e.g., setting the number of heads h=8), and each head is processed independently.

[0115] S404b: Perform scaled dot product attention calculation. For each head i, calculate the attention weights of the audio features relative to the video content, and then perform weighted aggregation of the audio features. The calculation formula is as follows:

[0116]

[0117] Among them, Q i K is a query subvector derived from video features. i and V i d represents the key and value sub-vectors derived from audio features. k The dimension of the key vector (i.e., the feature dimension of each head).

[0118] S404c: Merge multi-head outputs and fuse them with the original features. The outputs of all heads i are concatenated along the channel dimension, and then a linear transformation layer is used to project the feature dimensions back to their original size. The features processed by the attention mechanism and projected are then added element-wise along the channel dimension to the original visual features (or features processed by residual connections) input in step S404a, achieving deep information fusion.

[0119] S405: The features fused in step S404 (assuming their dimensions are [batch size, number of channels, height, width]) are first compressed into a one-dimensional feature vector by passing it through a global average pooling layer. This vector is then fed into a fully connected classification head.

[0120] This classification header contains the following layers in sequence:

[0121] The first fully connected hidden layer has an input dimension equal to the length of the aforementioned one-dimensional feature vector and an output dimension of 512, followed by a ReLU activation function.

[0122] The second fully connected hidden layer has an input dimension of 512 and an output dimension of 256, followed by a ReLU activation function.

[0123] The output layer is a fully connected layer with an input dimension of 256 and an output dimension of 4 (corresponding to four key event categories). It is followed by a Softmax activation function to normalize the output into a confidence score representing the probability of each category.

[0124] The four key incident categories include: falls, abnormal equipment operation, signs of gas leaks, and unauthorized entry.

[0125] In step S4, a lightweight spatiotemporal feature fusion network is used to perform in-depth analysis on the enhanced audio and video streams. The system is now able to identify and quantify key events and their confidence levels that characterize the underground safety status. However, identifying the events themselves is insufficient to ensure that critical information can be transmitted back to the ground command center in a timely and reliable manner. Under the limited network bandwidth and dynamically changing channel conditions in the mine, intelligent scheduling of transmission resources is necessary to ensure low-latency and high-reliability transmission of high-priority event data. Therefore, this method proceeds to step S5, which aims to perform dynamic bandwidth allocation decisions based on the event confidence levels, real-time channel status, and preset emergency level rules output in step S4. This intelligently divides the audio and video data to be transmitted into streams of different priorities, laying the decision-making foundation for subsequent efficient hierarchical transmission.

[0126] In step S5, a dynamic bandwidth allocation decision is made based on the key event identification results and communication status, dividing the audio and video data to be transmitted into high-priority streams and ordinary-priority streams. This includes the following sub-steps:

[0127] S501: Critical Event Confidence Score: Output by the lightweight spatiotemporal feature fusion network in step S4, a scalar value ranging from 0 to 1, used to quantify the reliability of the identified event (such as a person falling).

[0128] Channel Quality Index (CQI): A metric used to comprehensively evaluate the transmission quality of a current communication link; a higher value indicates better channel conditions. This metric is calculated based on physical layer measurement parameters (such as bit error rate and signal-to-noise ratio).

[0129] Preset task urgency level threshold: A pre-set confidence score threshold value corresponding to different event types, used to classify the urgency of events (such as high, medium, low), thereby associating different transmission priorities.

[0130] S502: Set the task urgency level threshold.

[0131] Preset three levels of task urgency and their corresponding thresholds:

[0132] Level 1 threshold: 0.9, corresponding to events that directly threaten the life safety of personnel (e.g., a person falls and remains still for a long time).

[0133] Secondary threshold: 0.75, corresponding to major equipment failures or sudden environmental events (e.g., main ventilation fan shutdown, sudden increase in methane concentration).

[0134] Level 3 threshold: 0.6, corresponding to general violation of operation events (e.g., entering the work area without wearing a safety helmet).

[0135] S503: Calculate the Channel Quality Index (CQI). This includes the following operations:

[0136] S503a: Obtain real-time physical layer parameters. Obtain the bit error rate (BER) and signal-to-noise ratio (SNR, in decibels) of the current link from the physical layer of the communication system in real time.

[0137] S503b: Applying a weighted calculation formula. Substituting the obtained BER and SNR into the following formula, calculate the Channel Quality Index (CQI):

[0138]

[0139] S503c: Outputs CQI value. The calculated CQI value is output for subsequent bandwidth allocation decisions.

[0140] S504: Based on the critical event type identified in step S4, determine its corresponding urgency level threshold (as defined in step S502). When the confidence score of any critical event exceeds the preset urgency level threshold for that type of event, and the channel quality index (CQI) of the current link is greater than 0.5, it is determined that high-priority transmission needs to be initiated for that event.

[0141] As one specific implementation method, the mapping relationship between critical event types and emergency level thresholds is as follows:

[0142] For incidents involving falls, the Level 1 threshold (0.9) applies.

[0143] For events involving abnormal equipment operation or signs of gas leakage, a secondary threshold (0.75) applies.

[0144] For unauthorized entry incidents, a Level 3 threshold (0.6) applies.

[0145] S505: Divide the data according to the determination result of step S504:

[0146] High-priority stream: contains the key event segment that triggers the high-priority transmission decision, and the contextual audio and video data 5 seconds before and after the time of occurrence, ensuring that the total duration of this data segment is not less than 12 seconds.

[0147] Normal priority stream: Contains all regular monitoring audio and video data that have not been classified as high priority streams.

[0148] Step S5 completes the priority division of audio and video data streams based on events and channels. To implement this hierarchical strategy during transmission, the method proceeds to step S6, which aims to achieve differentiated and efficient utilization of bandwidth resources by implementing lossless compression with reserved bandwidth transmission and lossy compression with variable bit rate transmission for high-priority and ordinary-priority streams respectively through the mining industrial ring network.

[0149] In step S6, the prioritized data streams are transmitted hierarchically via the mining industrial ring network, specifically including the following sub-steps:

[0150] S601: Hierarchical transmission is performed through a mining industrial ring network. This ring network uses a redundant ring topology composed of gigabit fiber optic cables and supports IEEE 802.1Q VLAN (Virtual Local Area Network) segmentation to provide logically isolated network channels.

[0151] The above steps clarify the underlying network architecture for carrying out hierarchical transmission tasks. Its redundant ring topology ensures reliability, while the VLAN function provides a technical foundation for allocating independent logical channels for data of different priorities.

[0152] S602: Establishes a lossless compressed transmission channel for high-priority streams.

[0153] The high-priority flow identified in step S5 is assigned to a separate VLAN, and a guaranteed bandwidth is configured for this VLAN on the network switch (e.g., by configuring a fixed bandwidth policy or setting a high IEEE 802.1Q priority label), thus forming a logical transmission channel with reserved bandwidth. Before transmission, the high-priority flow is losslessly compressed using an optimized context-adaptive binary arithmetic coding method.

[0154] As a specific implementation method, the optimization is reflected in: leveraging the strong correlation between frames before and after the high-priority stream, the coding parameters (such as initial probability distribution and context model selection) of multiple consecutive frames are jointly estimated and reused, reducing the overhead of independent initialization for each frame, thereby improving compression efficiency.

[0155] The lossless compression scheme has a compression ratio of no less than 2:1, thereby effectively saving bandwidth while ensuring the integrity of critical event data.

[0156] S603: Assign the ordinary priority stream defined in step S5 to another independent VLAN. Encode this data stream using variable bitrate H.265 based on the region of interest (ROI).

[0157] By using face detection algorithms and device control panel template matching technology, the face region and key device control interface region in the video frame are identified.

[0158] These regions of interest are assigned higher coding quality and lower quantization parameters (QP) with values ​​ranging from 20 to 25.

[0159] For the remaining background areas in the video frame, a lower coding quality is assigned, using a higher quantization parameter (QP) with a value ranging from 35 to 40.

[0160] By employing the aforementioned regionally differentiated coding strategy, the overall average bit rate is controlled to within 2 megabits per second (Mbps).

[0161] S604: To improve transmission reliability in harsh mine channel environments, forward error correction (FEC) coding and interleaving retransmission mechanisms are added to both high-priority and normal-priority data streams processed in S602 and S603. Specifically, this includes:

[0162] S604a: Performs forward error correction coding. Reed-Solomon Code (RS code) is used for encoding.

[0163] As a specific implementation, RS(255, 191) code is used, where the codeword length n = 255 bytes and the information bit length k = 191 bytes. Its encoding rate is approximately 191 / 255 ≈ 0.75, close to 2 / 3. This encoding can correct errors in up to 32 consecutive bytes (corresponding to 256 bits) within each 255-byte codeword. To meet the precise requirement of a 2 / 3 code rate, parameters such as RS(30, 20) can be achieved through shortening code techniques.

[0164] S604b: Performs interleaving. Block interleaving is performed on the FEC-coded data stream to resist burst fading in the channel. The interleaving depth is set to 16, constructing a 16-row × N-column block interleaver. The encoded data is written row by row and then read out column by column for transmission, thus distributing the originally continuous data packets into multiple different transmission slots. At the receiving end, corresponding deinterleaving operations are performed to restore the original order.

[0165] After completing the hierarchical transmission processing in step S6, the high-priority data stream ensures the complete and reliable delivery of evidence for critical events through lossless compression and reserved bandwidth channels. The ordinary-priority data stream maintains usable monitoring continuity within limited bandwidth through intelligent coding based on regions of interest. All data undergoes error-resistant enhancement processing to adapt to the harsh channel conditions in the mine. At this point, the optimized audio and video data has been transmitted to the surface via the mine's industrial ring network. However, to enable the ground command center to effectively perceive and intervene in the underground situation in real time, the received hierarchical data stream must be restored to understandable and interactive audio and video information. Therefore, this method proceeds to the final step S7, which aims to decode, reassemble, and synchronize the data received on the ground, and establish a low-latency downlink voice command channel, thereby forming a complete two-way real-time interactive closed loop between the underground and the surface.

[0166] In step S7, data decoding and reconstruction are completed at the ground command center receiving end, and a two-way real-time interactive channel is established. This includes the following sub-steps:

[0167] S701: Runs dedicated decoding software on a high-performance server cluster configured in the ground command center. Decodes high-priority streams and ordinary-priority streams received through the mining industrial ring network (corresponding to the lossless compression of S602 and the H.265 encoding of S603).

[0168] After decoding, based on the precise timestamps carried in the data packets, the two types of data streams are time-aligned and interleaved to reconstruct a continuous, synchronized audio and video playback sequence, which is then displayed on the monitoring screen. Time periods containing key events are highlighted on the timeline.

[0169] S702: The system is equipped with a voice activation detection unit that continuously monitors the input from the ground operator's microphone. This unit uses a dual-threshold energy detection method combined with zero-crossing rate analysis to accurately identify the start and end boundaries of the operator's voice commands. A voice command is considered valid and the downlink transmission channel is activated only when the following conditions are met simultaneously: the duration of the voice segment is greater than 300 milliseconds, and its signal energy exceeds a preset first-level threshold.

[0170] When a valid voice command is detected, the system immediately activates the downlink voice command channel. This command is transmitted in reverse through the same mining industrial ring network and sent to the speaker unit of the target underground audio / video acquisition terminal for playback.

[0171] S703: To prevent downlink voice commands played by the terminal speaker from being picked up again by its own microphone, thus causing echo interference to the uplink audio, an echo cancellation module is deployed on the downhole audio and video acquisition terminal side.

[0172] This module employs a normalized least mean square adaptive filtering algorithm with a filter order of 512. The algorithm uses the downlink voice command signal to be played as the reference signal and the input from the terminal microphone as the mixed signal including echo.

[0173] By continuously calculating the error between the microphone input (mixed signal) and the echo signal predicted by the filter for the reference signal, and iteratively updating the filter coefficients based on this error, the echo components are estimated and canceled in real time, ultimately ensuring the purity of the uplink transmitted voice signal.

[0174] In summary, in the intelligent voice and video interactive communication method for mining applications of this application, an intrinsically safe multimodal acquisition terminal is first deployed at key underground locations (S1). This method addresses the harsh acoustic and optical environments by employing voice enhancement based on sound source location estimation and depth complex networks (S2) and high dynamic range video reconstruction based on physical imaging models (S3), which significantly improves the availability and fidelity of the original audio and video signals from the source.

[0175] Secondly, the enhanced audio and video streams are localized at the downhole edge computing gateway. A lightweight spatiotemporal feature fusion network is used to achieve cross-modal alignment and joint analysis, automatically extracting and quantifying key events that characterize the safety status (such as personnel falls and equipment malfunctions). This achieves a leap from "signal transmission" to "semantic understanding" and avoids the bandwidth pressure of massive raw data backhaul.

[0176] Furthermore, based on a joint decision-making process using event semantics (confidence level) and channel quality index (CQI), data priorities are dynamically assigned (S5). Then, differentiated transmission is implemented for data of different priorities through the mining industrial ring network: high-priority event data uses lossless compression and reserved bandwidth to ensure its reliability and low latency; ordinary monitoring data uses intelligent lossy compression based on regions of interest to optimize bandwidth utilization (S6). This mechanism achieves optimal resource allocation under limited bandwidth.

[0177] Finally, the audio and video monitoring footage was fully restored and the events were labeled at the ground command center (S7 section), while a downlink voice command channel driven by the ground was established. Through voice activation detection and echo cancellation on the underground terminal side, clear and interference-free uplink and downlink voice communication was ensured, forming a complete interactive closed loop of "visible, audible, and commandable on the ground, and callable and responsive underground".

[0178] Additionally, this application discloses a mining intelligent voice and video interactive communication system, including a mining intrinsically safe audio and video acquisition terminal module, a voice enhancement processing module, a high dynamic range video reconstruction module, an edge computing gateway module, a dynamic bandwidth allocation decision module, a hierarchical transmission module, and a two-way interaction module.

[0179] The intrinsically safe audio and video acquisition terminal module for mining achieves synchronous acquisition and local preprocessing of raw signals; the voice enhancement processing module and the high dynamic range video reconstruction module can be integrated into the terminal embedded processor or edge computing gateway to improve signal quality.

[0180] The edge computing gateway module serves as a local communication scheduling hub, carrying a lightweight spatiotemporal feature fusion network to achieve real-time identification of critical events.

[0181] The dynamic bandwidth allocation decision module runs at the application layer of the gateway and generates transmission strategies based on the urgency of events and channel conditions.

[0182] The hierarchical transmission module is implemented by the network interface controller of the gateway and the mining industrial ring network switch, and performs the classification, encoding and scheduling of data streams.

[0183] The two-way interaction module covers voice activation detection in the ground command center, downlink command encoding, and echo cancellation and audio playback functions in the downhole terminal, forming a complete closed-loop interaction link.

[0184] During deployment, the number of edge computing gateways is configured according to the size of the mining area, with at least one gateway deployed in each area to ensure that the network hop count from all audio / video terminals to the gateway does not exceed three hops. The ground command center connects to the mine's backbone network via a 10 Gigabit uplink port, establishing secure encrypted tunnels with each edge gateway. All software modules are containerized, supporting remote firmware upgrades and configuration distribution. Through the coordinated operation of the above methods and the system, the intelligence level, stability, and emergency response capabilities of mine audio / video communication are significantly improved without adding new underground communication infrastructure.

[0185] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0186] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A mine intelligent voice and video interactive communication method, characterized in that, The method comprises the following steps: By deploying mine intrinsic safety audio and video acquisition terminals at key positions in the mine tunnel and on mobile devices, the original audio signals and the original video signals in the underground operation area are synchronously acquired; Performing speech enhancement processing on the original audio signals based on sound source direction estimation and spectral masking; Performing high dynamic range reconstruction processing on the original video signals based on a physical imaging model; Inputting the single-channel pure speech signal after speech enhancement processing and the high dynamic range video frame sequence after high dynamic range reconstruction processing into a lightweight space-time feature fusion network deployed in the underground edge computing gateway to perform cross-modal time alignment and extract a key event feature vector; Based on the confidence score of the key event feature vector, the channel quality indicator of the current communication link, and the preset task emergency level threshold, a dynamic bandwidth allocation decision is made; Through the mine industrial ring network, high-priority streams are transmitted in a lossless compressed manner through a reserved bandwidth channel, and ordinary priority streams are transmitted in a lossy compressed manner through a variable bit rate; The received data is decoded and reorganized at the ground command center receiver, and a downlink voice instruction channel initiated by the ground operator is established to realize two-way real-time interaction; Performing speech enhancement processing on the original audio signals based on sound source direction estimation and spectral masking, comprising: Using the four-channel audio data received by the four-element linear microphone array, the direction of arrival of the sound source is calculated by the generalized cross-correlation phase transform algorithm; Combined with the spatial prior information of the personnel activity area introduced by the mine digital twin model, the validity of the sound source direction is determined, and only when the sound source is located in the legal operation area and the distance from the terminal is not more than five meters, it is determined as an effective speech source; The multi-channel audio segment corresponding to the effective speech source is input into a deep complex convolutional neural network, and a complex masking matrix composed of real and imaginary parts is output; After multiplying the complex masking matrix and the original complex spectrum element by element, the enhanced single-channel pure speech signal is output after inverse short-time Fourier transform.

2. The intelligent voice and video interactive communication method for mine according to claim 1, characterized in that, Performing high dynamic range reconstruction processing on the original video signals based on a physical imaging model, comprising: Using a gradient-oriented optical flow estimation algorithm to perform sub-pixel level motion compensation registration on five consecutive low dynamic range video images; Constructing a local exposure response function model, dividing the pixel response curve into three segments of dark, linear and saturated regions, fitting the slope parameters and offset parameters respectively, and implementing the fitting process by minimizing the weighted sum of the adjacent frame luminosity consistency error and the gradient smoothing constraint term; Using the local exposure response function model to perform inverse mapping on the pixel values in the overexposed and underexposed regions to restore the true radiance and generate a high dynamic range video frame sequence.

3. The intelligent voice and video interactive communication method for mine according to claim 1, characterized in that, Inputting the enhanced single-channel pure speech signal and the high dynamic range video frame sequence into the lightweight space-time feature fusion network deployed in the underground edge computing gateway, comprising: Inputting the synchronously aligned audio mel-spectrogram sequence and the video frame sequence into the audio branch and the video branch respectively, the audio branch extracts time sequence features through one-dimensional convolution layers, and the video branch extracts spatial features through two-dimensional convolution residual blocks; After repeating and expanding the audio features along the time dimension, the video features are spliced in the channel dimension to form a fusion tensor; The fusion tensor is input into three three-dimensional convolution layers for spatio-temporal joint modeling; The correlation weight between the audio and video features is calculated by the multi-head cross-attention mechanism, the audio features are weighted and aggregated, and then added to the original video features to output the key event category and confidence score.

4. The intelligent voice and video interactive communication method for mine according to claim 1, characterized in that, Based on the confidence score of the key event feature vector, the channel quality indicator of the current communication link, and the preset task urgency level threshold, a dynamic bandwidth allocation decision is made, including: The preset three-level task urgency level threshold: the first threshold is 0.9, corresponding to a direct threat to personnel life safety event; the second threshold is 0.75, corresponding to a major equipment failure or environmental mutation event; the third threshold is 0.6, corresponding to a general violation operation event; The channel quality indicator CQI = 0.6 × (1 - BER) + 0.4 × SNR / (SNR + 10), where BER is the bit error rate and SNR is the signal-to-noise ratio; When the confidence score of any key event exceeds the corresponding threshold and CQI is greater than 0.5, the key event segment and its context data five seconds before and after are divided into high-priority streams, and the rest of the data is divided into ordinary priority streams.

5. The intelligent voice and video interactive communication method for mine according to claim 1, characterized in that, The high-priority stream is transmitted through the reserved bandwidth channel for lossless compression transmission, and the ordinary priority stream is transmitted through the variable bit rate for lossy compression transmission, including: The high-priority stream is compressed losslessly using the improved context adaptive binary arithmetic coding, with a compression ratio of no less than two to one; The ordinary priority stream is encoded using the variable bit rate H.265 based on the region of interest, with higher bit rates allocated to the face and device operation panel regions, and lower bit rates allocated to the rest of the background regions; Both types of data streams are attached with forward error correction coding with a code rate of two-thirds and an interleaving depth of sixteen.

6. The intelligent voice and video interactive communication method for mine according to claim 1, characterized in that, The received high-priority stream and ordinary priority stream are decoded and reorganized at the ground command center receiver, and a downlink voice command channel initiated by the ground operator is established simultaneously, including: The high-priority stream and ordinary priority stream are accurately aligned according to the timestamp to restore the continuous audio and video playback sequence; A voice activity detection unit is configured to identify the start and end boundaries of the operator's voice command using double-threshold energy detection combined with zero-crossing rate analysis; The downlink voice command channel is only opened during the detection of valid voice periods, and the downlink voice command is transmitted in reverse through the mine industrial ring network to the loudspeaker unit of the audio and video acquisition terminal in the mine.

7. A mine intelligent voice video interactive communication system, characterized in that, Including: A mine intrinsically safe audio and video acquisition terminal module is used to deploy in key locations and mobile devices in mine tunnels to synchronously acquire the original audio and video signals of the underground operation area; A speech enhancement processing module is used to perform speech enhancement processing based on sound source direction estimation and spectral masking on the original audio signal to output an enhanced single-channel pure speech signal; A high dynamic range video reconstruction module is used to perform high dynamic range reconstruction processing based on a physical imaging model guide on the original video signal to generate a high dynamic range video frame sequence; The edge computing gateway module is integrated with a lightweight space-time feature fusion network, and is used for cross-modal time sequence alignment of the enhanced single-channel pure voice signal and the high dynamic range video frame sequence, and extraction of a key event feature vector representing a work safety state; The dynamic bandwidth allocation decision module is used for dividing the audio and video data to be transmitted into a high-priority flow and an ordinary-priority flow based on a confidence score of the key event feature vector, a channel quality index of a current communication link, and a preset task emergency level threshold; The hierarchical transmission module is used for lossless compression transmission of the high-priority flow through a reserved bandwidth channel and lossy compression transmission of the ordinary-priority flow through a variable code rate mode through a mine industrial ring network; The bidirectional interaction module is used for decoding and recombining the received data at a receiving end of a ground command center, establishing a downlink voice instruction channel initiated by a ground operator, and realizing bidirectional real-time interaction; The voice enhancement processing module is used for: calculating a sound source direction of arrival through a generalized cross-correlation phase transform algorithm by using four-channel audio data received by a four-element linear microphone array; determining the validity of the sound source direction in combination with spatial prior information of a personnel activity area imported by a mine digital twin model; inputting a multi-channel audio segment corresponding to an effective voice source into a deep complex convolutional neural network to output a complex masking matrix; multiplying the complex masking matrix and an original complex spectrum element by element, and then performing inverse short-time Fourier transform to output an enhanced single-channel pure voice signal.

Images