A multi-spectrum fusion underground rubber-tired vehicle intelligent sensing and safety warning system

Abstract

The application discloses a kind of multispectral fusion underground rubber-tired vehicle intelligent perception and safety warning system, it is related to unmanned field, including: infrared camera module, for obtaining vehicle infrared image;Visible light camera module, for obtaining the visible light image of vehicle;Data processing module receives the infrared image and visible light image obtained, and target detection is carried out to received image;Warning module, for according to target detection result, through sound and light prompt user;Display module, for displaying the infrared image and visible light image obtained, and target detection result;For the insufficient environment perception ability of underground rubber-tired vehicle in prior art, the infrared camera module and visible light camera module of the present application respectively collect the infrared image and visible light image of vehicle, after time synchronization, image preprocessing, image fusion and target detection etc. Step, accurate identification and detection to obstacle, personnel and other vehicles in underground environment are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned driving, and in particular to a multi-spectrum fusion underground rubber-tyred vehicle intelligent sensing and safety warning system. BACKGROUND

[0002] With the continuous extension of coal and other mineral resources mining to the deep part, the underground operation environment is becoming increasingly complex, and the safety risk is increasing. The underground rubber-tyred vehicle is the most widely used transportation tool in coal mines and other underground engineering, and its driving process faces many disaster hidden dangers such as fault dislocation, roof collapse, and gas leakage. The traditional underground rubber-tyred vehicle mainly relies on manual driving and visual observation to perceive the surrounding environment, and has many problems such as many blind areas, incomplete information, and untimely response, which can easily cause accidents such as collision, scratching, and hitting people, and seriously threatens the safety of vehicles and personnel. Therefore, an intelligent underground rubber-tyred vehicle environment sensing and safety warning system is urgently needed to improve the sensing ability and safety guarantee level of underground vehicles in complex environments.

[0003] At present, the research on intelligent sensing of underground rubber-tyred vehicles mainly focuses on machine vision. Most of the existing technologies use visible light cameras to collect underground scene images, and use traditional image processing and pattern recognition methods for target detection and safety warning. However, in the underground environment with insufficient light and smoke interference, the quality of visible light images is poor, the noise is large, and the contrast is low, which seriously affects the detection accuracy. In addition, the visible light camera is difficult to effectively capture the radiation information of heat source targets, and lacks sensitivity to high-temperature targets such as personnel and animals. SUMMARY

[0004] In view of the insufficient environment sensing ability of the underground rubber-tyred vehicle in the prior art, the present application provides a multi-spectrum fusion underground rubber-tyred vehicle intelligent sensing and safety warning system, which collects infrared images and visible light images of the vehicle through infrared camera modules and visible light camera modules respectively, and realizes accurate identification and detection of obstacles, personnel and other vehicles in the underground environment through time synchronization, image preprocessing, image fusion and target detection and other steps.

[0005] The purpose of the present application is achieved by the following technical solutions.

[0006] The present specification provides a multispectral fusion underground rubber-tyred vehicle intelligent perception and safety warning system, comprising: an infrared camera module for acquiring vehicle infrared images; a visible light camera module for acquiring visible light images of the vehicle; a data processing module receiving the acquired infrared images and visible light images and performing target detection on the received images; a warning module for prompting the user through sound and light according to the target detection result; a display module for displaying the acquired infrared images and visible light images and the target detection result; an installation in the cab of the underground rubber-tyred vehicle connected with the data processing module for displaying the infrared images acquired by the infrared camera module, the visible light images acquired by the visible light camera module, and the fusion images and target detection results output by the data processing module. Further comprising: a data storage module connected with the data processing module for storing the infrared environment images acquired by the infrared camera module and the visible light environment images acquired by the visible light camera module and performing compression processing on the image data.

[0007] Further, the data processing module comprises: an image acquisition unit for receiving the acquired infrared images and visible light images and performing time synchronization on the received images; an image preprocessing unit for performing denoising and enhancement preprocessing on the time-synchronized images; an image fusion unit for performing spatial registration and fusion on the preprocessed infrared images and visible light images to generate fusion images; a target detection unit for performing target detection on the fusion images to identify obstacles, personnel and other vehicles in the images.

[0008] Further, the time synchronization on the received images comprises: extracting timestamp information in the infrared images and visible light images; comparing the extracted timestamp information of the infrared images and visible light images to determine the infrared images and visible light images with a time difference less than a preset threshold as synchronized images; when the time difference is greater than the preset threshold, generating synchronized images using an adaptive time interpolation method; pairing the synchronized infrared images and visible light images as time-synchronized images.

[0009] Further, when the time difference is greater than the preset threshold, the adaptive time interpolation method is used to generate synchronized images, comprising: calculating the compensation frame number of the infrared images and visible light images according to the frame rate and time difference of the infrared images and visible light images; wherein the compensation frame number is the quotient of the time difference divided by the frame interval time rounded up; selecting one frame of image before and after the original image with the smallest timestamp difference as compensation frame images from the infrared images and visible light images according to the compensation frame number; for each pair of compensation frames of the infrared images and visible light images, calculating the difference weight according to the difference between the corresponding timestamp and the timestamp of the original image; wherein the smaller the timestamp difference, the heavier the weight, and vice versa; weighting and averaging the infrared compensation frame and the original infrared image, and the visible light compensation frame and the original visible light image respectively to obtain the interpolated infrared image and visible light image as synchronized images.

[0010] Further, the time-synchronized images are pre-processed by denoising and enhancing, including: using wavelet transform to perform multi-scale decomposition on the time-synchronized infrared image and visible light image, decomposing the images into low-frequency sub-band coefficients and high-frequency sub-band coefficients; wherein the low-frequency sub-band coefficients represent the main structure and texture information of the image, and the high-frequency sub-band coefficients represent the details and noise information of the image; calculating the statistical features of the high-frequency sub-band coefficients including mean, variance and kurtosis, and calculating the noise variance in the infrared image and the visible light image using the noise variance estimation algorithm according to the statistical features; comparing the calculated noise variance with a preset threshold value, calculating the filter window size of the median filter, and setting the median filter according to the calculated filter window size to perform median filtering on the infrared image and the visible light image; preferably, the noise variance is compared with the preset noise threshold value, if the noise variance is greater than the preset noise threshold value, the noise level is set to a high noise level, and if the noise variance is less than or equal to the preset noise threshold value, the noise level is set to a low noise level; the set noise level selects the corresponding filter window size from the preset window size database, wherein the preset window size database stores the optimal filter window size for different noise levels; according to the filter window size selected by the window size determination sub-module, the filter parameters of the median filter are set, including the kernel function, the filter step and the filling method, etc.; the infrared image and the visible light image are filtered by the median filter with the set parameters to remove the noise points in the images.

[0011] According to the median-filtered infrared image and visible light image, the local histogram of the image is calculated using the histogram equalization algorithm, the local histogram is enhanced according to the gray scale mapping function of the image, and the enhanced infrared image and visible light image are obtained; preferably, the denoised infrared image and visible light image are divided into several sub-regions of equal size; the local histogram of each sub-region is calculated respectively, and the frequency distribution of the pixels of each gray level in the sub-region is counted; according to the local histogram of each sub-region calculated by the histogram calculation sub-module, the gray scale mapping function of each sub-region is adaptively determined by using the adaptive threshold selection method, and the gray scale values in the sub-region are mapped to the target gray scale range; the gray scale mapping function of each sub-region is determined, the pixel gray scale values in each sub-region are mapped and transformed, and the transformed sub-regions are recombined into complete infrared image and visible light image, and the contrast and brightness enhanced infrared image and visible light image are obtained.

[0012] whereby the infrared image and the visible light image are multi-scale decomposed by wavelet transform, the image is decomposed into low-frequency sub-band coefficients and high-frequency sub-band coefficients. The purpose of this step is to separate the main structural information and detail noise information of the image, so as to process the high-frequency sub-band coefficients in the subsequent step. The low-frequency sub-band coefficients represent the main structure and texture information of the image, usually containing the main content and background of the image. This part of the coefficient is not directly processed in the subsequent step, but it retains the main information of the image, which is very important for the final image enhancement and fusion. The high-frequency sub-band coefficients represent the detail and noise information of the image, and the subsequent step mainly processes the high-frequency sub-band coefficients. By extracting the statistical features (mean, variance and kurtosis) of the high-frequency sub-band coefficients, and using the noise variance estimation algorithm to calculate the noise variance in the image. According to the estimated noise variance, set the filter window size of the median filter, and perform median filtering on the infrared image and the visible light image to remove the noise in the image. After median filtering, the noise of the infrared image and the visible light image is suppressed, and the image quality is improved. Next, the histogram equalization algorithm is used to calculate the local histogram of the image. The local histogram here is calculated based on the median filtered image, not directly using the high-frequency sub-band coefficients. Finally, according to the gray scale mapping function, the local histogram is enhanced to obtain the enhanced infrared image and visible light image. This step adjusts the contrast and brightness distribution of the image, so that the visual quality of the image is improved.

[0013] Further, the gray scale mapping function is as follows:

[0014]

[0015] wherein, represents the new gray value of the gray level i in the sub-region j after mapping, represents the total number of pixels in the sub-region j, represents the pixel frequency of the gray level k in the sub-region j, represents the gray mean value of the sub-region j, represents the average temperature of the sub-region of the infrared image corresponding to the sub-region j, and T represents the average temperature of the entire infrared image; represents a parameter for controlling the shape of the mapping function.

[0016] is a Sigmoid function based on the gray cumulative histogram, used to adjust the contrast of the mapping.

[0017] is a Sigmoid function based on the current gray value and the gray mean value, used to adjust the gray center of the mapping.

[0018] is a Sigmoid function based on the current gray value, the gray mean value and the temperature difference, which is used to adjust the temperature dependence of the mapping.

[0019] wherein, represents an adaptive gain coefficient, which is adaptively adjusted according to the local contrast and the brightness level of the sub-region j, when the local contrast is lower or the brightness level is lower, the value of is larger, and the local contrast and the brightness are enhanced. represents an adaptive threshold, which is dynamically adjusted according to the cumulative distribution function and the local histogram feature of the sub-region j, when the cumulative distribution function value is greater than , the mapping function tends to be saturated, and the brightness is improved. represents a local contrast enhancement factor, which is used to adjust the mapping function according to the difference between the gray value and the mean value of the pixels in the sub-region j, and the greater the difference, the more obvious the local contrast enhancement. represents an image fusion factor, which is used to adjust the mapping function according to the difference between the average temperature of the sub-region j corresponding to the infrared image sub-region and the average temperature of the entire infrared image, and when the difference is greater, it means that the temperature information of the sub-region j in the infrared image is more significant, the adjustment range of the mapping function is larger, and the hotspot area of the infrared image is highlighted. The gray mapping function comprehensively considers the characteristics of the infrared image and the visible light image collected by the underground rubber-tired vehicle, and through the introduction of the adaptive gain coefficient, the adaptive threshold and the local contrast enhancement factor, the gray mapping function can be adaptively adjusted according to the local features of the image, and the contrast and brightness of the image are improved. At the same time, by introducing the image fusion factor, the temperature information of the infrared image can be used to highlight the hotspot area and enhance the target information in the image. This targeted gray mapping function can better adapt to the underground environment and improve the image enhancement effect.

[0020] Further, the pre-processed infrared image and the visible light image are spatially registered and fused to generate a fused image, including: using an Oriented FAST and Rotated BRIEF (ORB) algorithm to respectively detect feature points of the pre-processed infrared image and the visible light image, extract scale invariant feature transform (SIFT) features and binary robust independent elementary features (BRIEF) descriptors of the images, and obtain a feature point set representing local features of the infrared image and the visible light image; respectively matching the SIFT feature descriptors of the infrared image and the visible light image to obtain SIFT feature matching pairs; respectively extracting BRIEF descriptors corresponding to the matching pairs from the infrared image and the visible light image according to the SIFT feature matching pairs to obtain matched BRIEF descriptor pairs; using a Hamming window distance to measure the similarity of the matched BRIEF descriptor pairs to obtain final feature point matching pairs; using a random sample consensus (RANSAC) algorithm according to the final feature point matching pairs, calculating a homography matrix between the infrared image and the visible light image through random sampling and iterative fitting; using the homography matrix to perform geometric transformation on the infrared image to spatially register the infrared image to the visible light image, and obtaining a registered infrared image; and fusing the registered infrared image and the visible light image to obtain a fused image.

[0021] Specifically, the ORB (Oriented FAST and Rotated BRIEF) algorithm is a fast and efficient local feature extraction and description algorithm that combines FAST feature point detection and BRIEF feature description and improves and optimizes them. The FAST (Features from Accelerated Segment Test) algorithm is used to detect corner points in different scale spaces of the image to obtain candidate feature points. The Harris response value is calculated for the candidate feature points, and the feature points are sorted and filtered according to the response value to obtain a stable set of feature points. A scale space is constructed using an image pyramid to achieve multi-scale detection of feature points and improve the scale invariance of feature points. The main direction of the feature point is obtained by calculating the gray centroid in its neighborhood. The feature point neighborhood is aligned according to the main direction to achieve rotation invariance. In the feature point neighborhood, a series of random point pairs are selected according to the idea of the BRIEF (Binary Robust Independent Elementary Features) algorithm. The gray value of each point pair is compared to generate a binary descriptor, and the BRIEF descriptor is obtained. The BRIEF descriptor is rotated and compensated according to the main direction of the feature point to obtain a rotation-invariant descriptor, i.e., the RotatedBRIEF descriptor. Through the ORB algorithm, we can extract a series of feature points with scale invariance and rotation invariance from infrared images and visible light images, and describe them with binary BRIEF descriptors.

[0022] Binary Robust Independent Elementary Features (BRIEF) descriptor: A binary feature descriptor that generates a compact binary feature vector by comparing the brightness values of pixels around a key point. Local feature point set: A set of local feature points extracted from an image that have distinguishing and stable characteristics, such as SIFT features or BRIEF descriptors. Thermal radiation information: The thermal radiation energy information of an object in an infrared image, reflecting the temperature distribution characteristics of the object. Temperature attribute description: Adding temperature attributes to feature points based on the thermal radiation information of the infrared image for subsequent feature matching and fusion. Hamming distance: The Hamming distance between two strings of the same length is the number of different characters at corresponding positions, commonly used to measure the similarity between binary feature vectors. Random Sample Consensus (RANSAC) algorithm: A robust parameter estimation method that estimates the optimal model parameters from data containing outliers through random sampling and iterative fitting. Homography matrix: A matrix that describes the projection transformation relationship between two planes, commonly used for image registration and transformation. Spatial registration: Aligning images acquired by different sensors to the same coordinate system through geometric transformation to eliminate the positional and angular differences between sensors.

[0023] Further, the registered infrared image and the visible light image are fused to obtain a fused image, including: fusing the registered infrared image and the visible light image to obtain a fused image, using a directional filter bank and a scale filter bank to perform multi-scale and multi-directional decomposition on the infrared image and the visible light image to obtain infrared image sub-band coefficients and visible light image sub-band coefficients in different scales and directions; wherein, using LP decomposition in the non-subsampled Contourlet transform to perform multi-scale decomposition on the infrared image to obtain infrared image low-frequency sub-bands and high-frequency sub-bands in different scales; using DFB decomposition in the non-subsampled Contourlet transform to perform multi-directional decomposition on the high-frequency sub-bands of the infrared image to obtain infrared image band-pass sub-bands in different directions, which together constitute Contourlet coefficients of the infrared image; using LP decomposition in the non-subsampled Contourlet transform to perform multi-scale decomposition on the visible light image to obtain visible light image low-frequency sub-bands and high-frequency sub-bands in different scales; using DFB decomposition in the non-subsampled Contourlet transform to perform multi-directional decomposition on the high-frequency sub-bands of the visible light image to obtain visible light image band-pass sub-bands in different directions, which together constitute Contourlet coefficients of the visible light image; wherein, initializing a connection weight matrix of a PCNN model, inputting the infrared image Contourlet coefficients and the visible light image Contourlet coefficients into the PCNN model respectively to construct infrared PCNN and visible light PCNN; using dynamic response characteristics of the infrared PCNN and the visible light PCNN to adaptively capture significant features in the infrared image Contourlet coefficients and the visible light image Contourlet coefficients to obtain significant feature coefficients of the infrared image and the visible light image; using a weighted average strategy to generate fused Contourlet coefficients according to the significant feature coefficients of the infrared image and the visible light image; performing non-subsampled inverse Contourlet transform on the fused Contourlet coefficients to reconstruct a fused image containing infrared radiation information and visible light texture details. The image fusion unit adaptively extracts and fuses thermal radiation features of the registered infrared image and texture detail information of the visible light image using the non-subsampled Contourlet transform and the PCNN model, and generates a fused image containing infrared target information and having visible light high resolution, which provides more comprehensive and accurate image expression for subsequent target detection.

[0024] The input neurons of the coupled neural network are initialized, the infrared image sub-band coefficients and the visible light image sub-band coefficients are respectively input to different input neurons, and a mapping relationship between the input neurons and the link matrix is established; through iterative learning of the pulse neural network, the weight coefficients of the link matrix are updated by using the interconnection and coupling relationship between the neurons, the infrared image sub-band coefficients and the visible light image sub-band coefficients are weighted and fused, and the sub-band coefficients of the fused image are obtained; the sub-band coefficients of the fused image are subjected to non-subsampled inverse Contourlet transform, the sub-band coefficients of the fused image are subjected to inverse transform reconstruction by using an interpolation filter bank, and a time-frequency domain fused fused image is obtained; a thermal radiation pseudo-color mapping algorithm is used to perform pseudo-color mapping on a thermal radiation target region in the time-frequency domain fused fused image, and a thermal radiation enhanced fused image is obtained as the final fused image output.

[0025] The non-subsampled Contourlet transform can more effectively extract the multi-scale and multi-direction structural features of an image. Meanwhile, the pulse coupled neural network is introduced for adaptive fusion, and through self-learning of the link weight between the neurons, the adaptive weighted fusion of the infrared image sub-band coefficients and the visible light image sub-band coefficients can be realized according to the dynamic changes of the downhole environment. In addition, the fused image is subjected to adaptive illumination compensation and thermal radiation pseudo-color mapping, which can further enhance the visual quality of the image and the saliency of the thermal radiation target. The scale invariance, direction selectivity and adaptability to the complex downhole environment of the fused image are improved, thereby providing a clearer and more accurate target detection and hidden danger identification basis for the intelligent perception and safety warning system of the downhole rubber-tyred vehicle.

[0026] Preferably, the thermal radiation pseudo-color mapping algorithm is used to perform pseudo-color mapping on a thermal radiation target region in the time-frequency domain fused fused image to obtain a thermal radiation enhanced fused image, including: performing thermal radiation threshold segmentation on the time-frequency domain fused fused image, and extracting a pixel region with a gray value greater than a threshold value in the fused image as a candidate thermal radiation target region; performing morphological filtering on the candidate thermal radiation target region to remove regions with an area less than a threshold value, and obtaining a thermal radiation target mask; extracting a thermal radiation target region in the time-frequency domain fused image according to the thermal radiation target mask, using a temperature inversion algorithm based on a radiation transfer equation, and calculating the temperature distribution of the surface of the thermal radiation target by using the gray value of the thermal radiation target region; using a color mapping function to map the temperature value of the thermal radiation target to a pseudo-color color space according to the temperature distribution of the surface of the thermal radiation target, and generating a pseudo-color image of the thermal radiation target; and performing weighted fusion on the pseudo-color image and the time-frequency domain fused fused image to obtain the thermal radiation enhanced fused image.

[0027] The mathematical expression of the color mapping function is as follows:

[0028] The temperature value range is which is linearly mapped to the interval [0, 1]:

[0029]

[0030] Then, the normalized temperature values are mapped to the RGB color space using a piecewise function represents the red channel. represents the green channel. represents the blue channel.

[0031]

[0032] Finally, the RGB values are mapped to the interval [0, 255] to obtain the color values of the pseudo-color image:

[0033]

[0034]

[0035]

[0036] where round(x) represents the rounding function.

[0037] where the non-subsampled Contourlet transform is a multi-scale, multi-directional image representation method that decomposes an image into sparse representations using directional and scale filter banks. Image sub-band coefficients are the coefficients obtained after multi-scale, multi-directional decomposition of an image, reflecting the characteristics of the image at different scales and directions. Link matrix is a matrix in an impulse-coupled neural network that represents the connection weights between neurons, used to control the interaction and information transmission between neurons.

[0038] Further, the fused image is subjected to target detection to identify obstacles, personnel, and other vehicles in the image, including: using a pre-trained convolutional neural network to extract features from the fused image, through multiple convolutional and pooling layers of the convolutional neural network, to extract multi-scale and multi-level features of the image and obtain a feature representation of the image; using the YOLO algorithm to detect and classify the extracted feature representation, by dividing the image into fixed-size grids and predicting the bounding boxes and class probabilities of targets in each grid, to detect and identify obstacles, personnel, and other vehicles in the image.

[0039] ​Further, the network result of the YOLO algorithm contains multiple convolutional layers and fully connected layers, the convolutional layers are used to extract local features of the image, and the fully connected layers are used to predict the bounding box and class probability of the target, and the final target detection result is obtained by merging the overlapping bounding boxes through the non-maximum suppression algorithm.

[0040] Compared with the prior art, the application has the advantages that:

[0041] Through the dual-mode imaging of the infrared camera and the visible light camera, the thermal radiation and optical feature information of the underground environment are obtained, the perception range of the vehicle-mounted sensor is expanded, the shortcomings of single imaging mode are made up, and a data foundation is provided for comprehensive and accurate environment perception.

[0042] The image time synchronization method combining time stamp comparison and adaptive interpolation is adopted, the problem that the infrared and visible light images are not synchronized is effectively solved, the time registration of the image frame level precision is realized, and the effects of subsequent image fusion and target detection are guaranteed.

[0043] In view of the image quality degradation problem caused by the weak light and dust environment in the underground, an adaptive image enhancement method in the wavelet transform domain is proposed. Through multi-scale decomposition and noise estimation, the image smoothness and contrast are adaptively adjusted, and the visual quality of the infrared and visible light images is effectively improved, so that a high-quality image source is provided for subsequent processing.

[0044] The image registration method based on ORB features is introduced, and the registration problem caused by the few texture features and large light changes in the underground scene is overcome. Through robust feature extraction and random sample consensus estimation, the sub-pixel level registration of the infrared and visible light images is realized, and the registration precision is greatly improved, thereby laying a geometric foundation for image fusion.

[0045] An adaptive multi-scale infrared / visible light image fusion method is innovatively proposed. The method uses non-subsampled Contourlet transform to adaptively extract infrared radiation features and visible light texture details, and realizes optimized fusion of the features through a pulse coupled neural network, so that the thermal radiation information of the infrared image is retained, the geometric structure of the visible light image is enhanced, and a clearer and more comprehensive scene representation is obtained than single imaging.

[0046] The deep convolutional neural network and the improved YOLO algorithm are applied to the fused image, real-time detection and position regression of typical targets in the underground are realized. Compared with the traditional method, the detection model fully utilizes the multispectral features of the fused image, and through end-to-end feature learning and multi-scale prediction, the detection precision of obstacles, personnel, vehicles and other targets in the underground environment is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] The present specification will be further described in the manner of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same numbers represent the same structures, in which:

[0048] Figure 1 is an exemplary block diagram of a multispectral fusion underground rubber-tyred vehicle intelligent perception and safety warning system according to some embodiments of the present specification;

[0049] Figure 2 is an exemplary flowchart of acquiring a synchronized image according to some embodiments of the present specification;

[0050] Figure 3 is an exemplary flowchart of acquiring a denoised and enhanced image according to some embodiments of the present specification;

[0051] Figure 4 is an exemplary flowchart of acquiring a registered infrared image according to some embodiments of the present specification;

[0052] Figure 5 is an exemplary flowchart of acquiring a final fused image according to some embodiments of the present specification. DETAILED DESCRIPTION

[0053] The method and system provided by the embodiments of the present specification will be described in detail below with reference to the accompanying drawings.

[0054] Figure 1 is an exemplary block diagram of a multispectral fusion underground rubber-tyred vehicle intelligent perception and safety warning system according to some embodiments of the present specification, which is an infrared camera module, a visible light camera module, a data processing module, a warning module, a display module, and a data storage module. The infrared camera module is installed at the front and rear of the underground rubber-tyred vehicle, aiming to capture real-time environmental images in front and rear. The infrared camera adopts advanced sensing technology, which can work normally under complex conditions such as low light, dust, and smoke, providing clear and stable image output. The visible light camera module is installed at the front and rear of the underground rubber-tyred vehicle, aiming to capture visible light images of the environment in front and rear. This module provides detailed environmental information by capturing high-resolution and color images. The visible light camera performs well under normal lighting conditions, ensuring high-quality image output in well-lit environments.

[0055] The data processing module is the core part of the entire system, responsible for receiving image data from the infrared camera and the visible light camera, preprocessing and fusing these data, and using advanced algorithms for target detection and danger assessment. The specific composition and functions are as follows: The image acquisition unit receives real-time image data from the infrared camera and the visible light camera. Its main function is to ensure that high-quality image data can be stably and accurately captured in complex underground environments, and these image data are time-synchronized, and the synchronized image data are transmitted to the image preprocessing unit for further processing to ensure the effectiveness and accuracy of subsequent processing steps. The image preprocessing unit performs denoising and enhancement processing on the received image data to improve image quality, and transmits the preprocessed image data to the image fusion unit. The main processing steps include: image denoising: using median filtering, mean filtering and other techniques to remove noise points in the image to ensure image quality. Image enhancement: using histogram equalization and other methods to enhance the contrast and brightness of the image, and improve the detail performance of the image.

[0056] The image fusion unit is a key part of the data processing module, responsible for aligning and fusing the pre-processed infrared and visible light images to generate a high-quality fusion image containing more environmental information. This unit provides a more comprehensive and clear environmental perception capability by combining the advantages of both images. Image fusion, the infrared and visible light images after alignment are fused to generate a high-quality fusion image, providing more rich environmental information. Common algorithms include multi-scale fusion and weighted average, among which multi-scale fusion refers to using multi-scale image fusion technology to fuse image features at different scales to improve image details and contrast. Weighted average refers to calculating the weighted average value according to the characteristics of the infrared and visible light images to generate a fusion image. The target detection unit uses deep learning algorithms to analyze the fusion image and detect and identify obstacles, personnel and other vehicles in the image. Common target detection networks include YOLO, SSD and Faster, R-CNN. The data processing module uses deep learning and computer vision technology to ensure high precision and real-time of the detection results. The warning system includes an audible and visual alarm device connected to the data processing module. When the data processing module detects potential hazards such as obstacles in front, personnel approaching or other vehicles approaching, the warning system will immediately trigger the alarm device to alert the operator to take necessary safety measures. This multi-warning method can ensure that the operator can receive warning information in time in different environments, improving safety. The display system is installed in the cab to display real-time images captured by the visible light camera and infrared camera and the results after data processing. The display system includes a high-definition display screen with a friendly user interface that can clearly display camera images, detected obstacle locations and their category information. The operator can fully understand the environment around the vehicle through this system and make timely and accurate driving decisions. The data storage module efficiently stores real-time image data captured by the infrared and visible light cameras and compresses them to reduce storage space and transmission bandwidth requirements for future processing and analysis.

[0057] The infrared camera module uses a non-cooled focal plane array infrared sensor with a detection wavelength range of 8-14 μm in the long-wave infrared range, a pixel count of 640x480, and a pixel size of 17 μm. The thermal sensitivity (NETD) is less than 50 mK, and it can work stably in an environment temperature range of -20°C to +60°C. The infrared camera is equipped with a 25mm fixed-focus lens with a field of view of 18°x14°, and the focal length matches the size of the image sensor, allowing it to capture clear images from 12m to infinity. The infrared camera module is installed in the special protective cover on the front and rear of the underground rubber-tyred vehicle, the protective cover is made of aluminum alloy and filled with nitrogen, and the protection level reaches IP67, which can effectively prevent dust, water vapor and vibration from affecting the infrared camera. The infrared camera is connected to the central processor through the RS485 interface to transmit digital video streams with a frame rate of 25fps, ensuring real-time and stability.

[0058] The visible light camera module uses a 1 / 1.8-inch CMOS color image sensor with a pixel count of 1920×1080, a pixel size of 2.9μm, a minimum illumination of 0.01Lux, and a signal-to-noise ratio greater than 55dB. It can operate stably in ambient temperatures ranging from -30℃ to +70℃. The visible light camera is equipped with an 8mm fixed-focus lens with a field of view of 42°×23°. The focal length matches the image sensor size, enabling it to acquire clear color images from 0.5m to infinity. The visible light camera module is installed in dedicated protective covers at the front and rear of the underground rubber-tired vehicle. These covers are made of stainless steel and filled with nitrogen, achieving an IP69K protection rating, effectively preventing the effects of dust, moisture, vibration, and high-pressure water jets on the visible light camera. The visible light camera connects to the central processing unit via a gigabit Ethernet interface, transmitting H.264 compressed digital video streams at a frame rate of 30fps. The encoding bit rate is adjustable, supporting up to 8Mbps, ensuring image quality and real-time performance. The image signals from the infrared camera module and the visible light camera module are preprocessed, spatiotemporally aligned, and enhanced by the central processing unit before being sent to the image fusion module for multispectral fusion to obtain a fused and enhanced environmental perception image, providing reliable visual information for the automatic navigation and safety early warning of underground rubber-tired vehicles.

[0059] Figure 2 This is an exemplary flowchart illustrating the acquisition of synchronized images according to some embodiments of this specification. The data processing module is the core of the entire system, responsible for receiving image data from infrared and visible light cameras, preprocessing and fusing this data, and using advanced algorithms for target detection and hazard assessment. The image acquisition unit uses a high-speed digital video interface, such as GigE Vision or Camera Link, to connect to the infrared and visible light cameras and receive their real-time digital video streams. The image acquisition unit is equipped with a dedicated image acquisition card, such as an FPGA-based image acquisition card, supporting multi-channel, high-resolution, and high-frame-rate image acquisition to ensure stable and reliable acquisition of high-quality image data in complex downhole environments. The image acquisition card has a precise time synchronization function, achieving time synchronization between the infrared and visible light cameras through an external trigger signal or the IEEE 1588 Precision Time Protocol (PTP), with a time error of less than 100μs. The synchronized infrared and visible light images are transmitted to the industrial computer's memory via the PCIe bus for subsequent processing by the image preprocessing unit.

[0060] Figure 3is an exemplary flowchart of obtaining denoised and enhanced images according to some embodiments of the present specification. The image preprocessing unit, image fusion unit and target detection unit can be concentrated in a high-performance industrial computer, such as an Intel Core i7 processor, 16 GB DDR4 memory and NVIDIA GTX 1080 graphics card, running image processing algorithms developed based on C++ and CUDA parallel computing framework, or distributed in different industrial computers to run, denoising and enhancing the synchronous image data transmitted by the image acquisition unit to improve image quality. For time synchronization, the image preprocessing unit compares the timestamp information of the infrared image and the visible light image. If the time difference is less than a preset threshold (such as 10 ms), they are determined as synchronous images; if the time difference is greater than the preset threshold, an adaptive time interpolation method is used to generate synchronous images.

[0061] Figure 4 is an exemplary flowchart of obtaining registered infrared images according to some embodiments of the present specification. The specific steps of the adaptive time interpolation method are as follows: according to the frame rate (such as 25 fps) and time difference of the infrared image and the visible light image, the compensation frame number is calculated, that is, the quotient of the time difference divided by the frame interval time (40 ms) is rounded up; select one frame before and after the original image with the smallest timestamp difference from the infrared image and the visible light image sequence as the compensation frame; for each pair of infrared compensation frame and visible light compensation frame, calculate the difference weight according to the difference between their timestamps and the timestamp of the original image, that is, the ratio of the timestamp difference to the frame interval time; the infrared compensation frame and the original infrared image, and the visible light compensation frame and the original visible light image are weighted and averaged respectively to obtain the interpolated infrared image and visible light image as the synchronous image. For denoising and enhancement, the image preprocessing unit uses wavelet transform to decompose the synchronous image for 5 levels, extracts the mean, variance and kurtosis of the high-frequency sub-band coefficients, calculates the image noise variance using the noise variance estimation algorithm, and adaptively sets the median filter window size to filter the image. Then, the image preprocessing unit performs histogram equalization on the filtered image to enhance the local contrast and brightness. The gray scale mapping function of histogram equalization is: wherein, represents the new gray value of gray level i in sub-region j after mapping, represents the total number of pixels in sub-region j, represents the pixel frequency of gray level k in sub-region j, represents the average gray value of sub-region j, represents the average temperature of the sub-region of the infrared image corresponding to sub-region j, and T represents the average temperature of the entire infrared image. The adaptive parameter is dynamically adjusted according to the local statistical characteristics of the image to obtain the best enhancement effect. Finally, the image preprocessing unit transmits the infrared image after denoising and enhancement and the visible light image to the image fusion unit through the PCIe bus for multispectral fusion processing.

[0062] Figure 5 is an exemplary flowchart for obtaining a final fused image according to some embodiments of the present specification, realizing fast and accurate registration and fusion of infrared images and visible light images. For spatial registration, the image fusion unit for spatial registration and fusion of the preprocessed infrared image and visible light image generates a fused image, including: using the ORB algorithm to respectively detect feature points of the preprocessed infrared image and visible light image, extract scale invariant feature transform (SIFT) features and binary robust independent elementary features (BRIEF) descriptors of the images, and obtain a feature point set representing local features of the infrared image and visible light image; respectively matching the SIFT feature descriptors of the infrared image and visible light image to obtain SIFT feature matching pairs; respectively extracting BRIEF descriptors corresponding to the matching pairs in the infrared image and visible light image according to the SIFT feature matching pairs to obtain matched BRIEF descriptor pairs; using Hamming window distance to measure the similarity of the matched BRIEF descriptor pairs to obtain final feature point matching pairs; using the random sample consensus (RANSAC) algorithm according to the final feature point matching pairs, calculating a homography matrix between the infrared image and visible light image through random sampling and iterative fitting; using the homography matrix to perform geometric transformation on the infrared image to spatially register the infrared image to the visible light image to obtain a registered infrared image; and fusing the registered infrared image and visible light image to obtain a fused image.

[0063] To eliminate false matches, the image fusion unit uses the random sample consensus (RANSAC) algorithm to perform geometric verification on the initial matching pairs. Randomly extract 4 pairs of feature points from the initial matching pairs, calculate the homography matrix therebetween, then calculate the re-projection error of all matching pairs under the matrix, and take the matching pairs with an error less than a threshold as inliers. Repeat the iteration multiple times, find the homography matrix with the most inliers, and use all inliers to perform least squares optimization on the matrix parameters to obtain the final homography transformation matrix of the infrared image to the visible light image. Using the homography matrix, perform projection transformation on the infrared image to register it to the visible light image coordinate system, completing the spatial registration of the infrared image and visible light image. The registered infrared image and visible light image have consistent pixel sizes and field of view ranges, preparing for subsequent fusion processing.

[0064] For image fusion, the image fusion unit adopts a fusion algorithm based on pulse coupled neural network (PCNN) and nonsubsampled contourlet transform (NSCT). First, the registered infrared image and the preprocessed visible light image are subjected to nonsubsampled contourlet transform, and the images are subjected to 4-layer directional decomposition and 3-layer scale decomposition by using a directional filter bank (DFB) and a scale filter bank (LP) to obtain infrared image subband coefficients and visible light image subband coefficients in different scales and directions. Then, the infrared image subband coefficients and the visible light image subband coefficients are input into different input neurons of the PCNN to establish a mapping relationship between the input neurons and the linking matrix. The PCNN is a biological neural network model simulating the mammalian visual cortex neurons, and can adaptively extract the salient features of the images through the interconnection and coupling between the neurons. In the iterative learning process of the PCNN, the weight coefficients of the linking matrix are constantly updated, so that the infrared image subband coefficients and the visible light image subband coefficients are adaptively weighted according to their respective salient features, the coefficients with high saliency obtain greater weights, and the coefficients with low saliency have reduced weights, thereby realizing selective fusion of the image details. After multiple iterations, the linking matrix converges to a stable state, and the output neurons generate the fused subband coefficients. The fused subband coefficients are subjected to NSCT inverse transform, and the subband coefficients are subjected to inverse transform reconstruction by using a scaling filter bank (SR), so as to realize fusion of the infrared image and the visible light image in the spatial domain and the frequency domain at the same time, and obtain a fused image containing rich details and textures. Finally, in order to highlight the thermal radiation target, the image fusion unit performs pseudo-color mapping on the thermal radiation region in the fused image.

[0065] The thermal radiation pseudo-color mapping algorithm is adopted to map the temperature value of the target region to the RGB color space to generate an intuitive and eye-catching thermal radiation pseudo-color image as the final fused image output. The pseudo-color mapping adopts a cubic Hermite interpolation function to map the temperature value T to the coordinate value (r, g, b) in the RGB color space: , , wherein, represents the normalized temperature value. and are the lowest temperature and the highest temperature of the target region, respectively. By adjusting the control points of the mapping function, different pseudo-color mapping effects such as red-yellow-white and blue-green-red can be generated. In summary, the image fusion unit realizes rapid registration and fusion of the infrared image and the visible light image by a series of algorithms such as ORB feature extraction, RANSAC registration, NSCT transform, PCNN fusion and pseudo-color mapping, to generate a clear, stable and information-rich fused image, thereby providing reliable environmental perception information for target detection and safety warning.

[0066] The target detection unit realizes real-time detection and recognition of obstacles, persons and vehicles in the fusion image. For feature extraction, the target detection unit uses a pre-trained convolutional neural network (CNN) to perform multi-scale and multi-level feature extraction on the fusion image. Specifically, a YOLOv5 convolutional neural network pre-trained on the COCO dataset is used, which consists of three parts: a training hard forward backbone network, a multi-scale feature pyramid (FPN), and an output detection head (Head). First, the fusion image is input to the backbone network of YOLOv5, which uses an optimized DarkNet-53 network structure containing multiple serial residual components (Residual Block), which can increase the depth of the network while avoiding gradient disappearance.

[0067] The backbone network extracts feature representations of the image at different semantic levels through 5 convolutional layers and pooling layers. Then, the multi-scale feature maps extracted by the backbone network are input to the FPN, which uses a Top-Down and Lateral Connection structure to fuse high-level semantic features and low-level detail features, generating a more accurate and comprehensive feature pyramid. Next, the feature pyramid is input to the output detection head, which uses a Mixed Depth-wise Separable Convolution and a Mish activation function to reduce network parameters while improving target detection accuracy. The detection head divides each feature map into SxS cells, each of which predicts B bounding boxes, each containing 4 position parameters (center coordinates, width and height), 1 confidence score and C class probabilities. For the COCO dataset, S=19, B=3, and C=80. Finally, the prediction results output by the detection head are subjected to non-maximum suppression (NMS), merging bounding boxes with an overlap higher than a threshold (e.g. 0.5) to eliminate duplicate detections of the same target, resulting in the final target detection results. Each detection result contains the class label, confidence score and bounding box position of the target.

[0068] For target detection, the target detection unit employs an optimized YOLO (You Only Look Once) algorithm to achieve real-time target detection on the fused image. The YOLO algorithm converts the target detection problem into a regression problem, directly predicts the bounding box and class probability of the target in the image through a single convolutional neural network forward inference, avoiding time-consuming intermediate steps such as traditional sliding window and candidate region generation. In order to further improve the detection speed, the target detection unit uses TensorRT to optimize the YOLOv5 network. TensorRT is a high-performance deep learning inference optimization library provided by NVIDIA, which can convert the trained network model into an optimized inference engine. Through kernel fusion, inter-layer optimization, mixed precision quantization and other technologies, the computing efficiency of GPU is maximized. Deploying the YOLOv5 network optimized by TensorRT on Jetson AGX Xavier can achieve real-time target detection of more than 30 frames per second. The target detection unit continuously detects the fused image and identifies obstacles (such as underground equipment, pipelines), personnel (such as workers, technicians) and other vehicles (such as transport vehicles, inspection vehicles) in the image, and labels their categories, positions and confidence. When an emergency situation (such as personnel intrusion, vehicle out of control) is detected, the target detection unit sends the event type, target ID, detection result and other information to the application server in JSON format, and the application server triggers an alarm prompt to inform the on-site personnel to take emergency measures in time. At the same time, the target detection unit also renders the detection result on the fused image, labeling the target in the form of a box and a label, which is convenient for operators to intuitively monitor the underground scene. In summary, the target detection unit realizes high-precision, low-latency, real-time detection and identification of obstacles, personnel and vehicles in the fused image through the optimized YOLOv5 convolutional neural network and TensorRT acceleration engine, which can timely discover underground safety hazards and provide reliable environmental perception support for intelligent decision-making and emergency disposal.

[0069] The warning system includes an audible and visual alarm device connected to the data processing module. When the data processing module detects potential hazards (such as obstacles ahead, approaching personnel, or other vehicles approaching), the warning system will immediately trigger the alarm device to alert the operator to take necessary safety measures through sound and light. This multi-warning method can ensure that the operator receives timely warning information in different environments, improving safety. The display system is installed in the cab and displays real-time images captured by the infrared camera and the results of data processing. The display system includes a high-definition display screen with a friendly user interface that clearly displays camera images, detected obstacle locations, and category information. The operator can make timely and accurate driving decisions by comprehensively understanding the vehicle's environment through this system. The data storage module efficiently stores real-time image data captured by the infrared and visible light cameras, and compresses them to reduce storage space and transmission bandwidth requirements, facilitating later processing and analysis.

Claims

1. A multi-spectrum fusion underground rubber-tyred vehicle intelligent sensing and safety warning system, comprising: an infrared camera module for acquiring infrared images of the vehicle; a visible light camera module for acquiring visible light images of the vehicle; a data processing module for receiving the acquired infrared images and visible light images and performing target detection on the received images; a warning module for prompting a user through sound and light according to the target detection results; a display module for displaying the acquired infrared images and visible light images and the target detection results; preprocessing of the time-synchronized images for denoising and enhancement, comprising: performing multi-scale decomposition of the time-synchronized infrared images and visible light images using wavelet transform to decompose the images into low-frequency sub-band coefficients and high-frequency sub-band coefficients; wherein the low-frequency sub-band coefficients represent the main structure and texture information of the images, and the high-frequency sub-band coefficients represent the details and noise information of the images; extracting statistical features including mean, variance and kurtosis of the high-frequency sub-band coefficients, and calculating the noise variance in the infrared images and visible light images using a noise variance estimation algorithm according to the statistical features; comparing the calculated noise variance with a preset threshold value to calculate the filter window size of a median filter, and setting the median filter according to the calculated filter window size to perform median filtering on the infrared images and visible light images; calculating the local histogram of the infrared images and visible light images after median filtering using a histogram equalization algorithm, and performing enhancement processing on the local histogram according to a gray scale mapping function to obtain enhanced infrared images and visible light images; the gray scale mapping function is as follows: ； wherein, represents the new gray value of gray level i in sub-region j after mapping, represents the total number of pixels in sub-region j, represents the frequency of pixels of gray level k in sub-region j, represents the average gray value of sub-region j, represents the average temperature of the sub-region of the infrared image corresponding to sub-region j, and T represents the average temperature of the entire infrared image; represents a parameter for controlling the shape of the mapping function; wherein a fusion image is generated, comprising: performing feature point detection on the preprocessed infrared images and visible light images using an ORB algorithm to extract scale-invariant feature transform (SIFT) features and binary robust independent elementary features (BRIEF) descriptors of the images to obtain a feature point set representing the local features of the infrared images and visible light images; performing matching on the SIFT feature descriptors of the infrared images and visible light images respectively to obtain SIFT feature matching pairs; extracting the BRIEF descriptors corresponding to the matching pairs from the infrared images and visible light images respectively according to the SIFT feature matching pairs to obtain matched BRIEF descriptor pairs; performing similarity measurement on the matched BRIEF descriptor pairs using a Hamming window distance to obtain final feature point matching pairs; calculating the homography matrix between the infrared images and visible light images using a random sample consensus (RANSAC) algorithm through random sampling and iterative fitting according to the final feature point matching pairs; performing geometric transformation on the infrared images using the homography matrix to spatially register the infrared images to the visible light images to obtain registered infrared images; performing fusion on the registered infrared images and visible light images to obtain a fusion image; wherein a temperature attribute is added to the feature points according to the thermal radiation information of the infrared images for subsequent feature matching and fusion.

2. The multi-spectrum fusion underground rubber-tyred vehicle intelligent sensing and safety warning system according to claim 1, characterized in that: the data processing module comprises: The image acquisition unit is configured to receive the acquired infrared image and the visible light image, and to time synchronize the received images. The image preprocessing unit is configured to perform denoising and enhancement preprocessing on the time-synchronized images. The image fusion unit is configured to perform spatial registration and fusion on the preprocessed infrared image and the visible light image to generate a fused image. The target detection unit is configured to perform target detection on the fused image to identify obstacles, persons, and other vehicles in the image.

3. The multi-spectrum fusion underground rubber-tired vehicle intelligent perception and safety warning system according to claim 2, wherein: time synchronizing the received images comprises: extracting timestamp information from the infrared image and the visible light image; comparing the extracted timestamp information of the infrared image and the visible light image, and determining the infrared image and the visible light image with a time difference less than a preset threshold as synchronized images; when the time difference is greater than the preset threshold, generating synchronized images using an adaptive time interpolation method; pairing the synchronized infrared image and the visible light image as the time-synchronized images.

4. The multi-spectrum fusion underground rubber-tired vehicle intelligent perception and safety warning system according to claim 3, wherein: when the time difference is greater than the preset threshold, generating synchronized images using an adaptive time interpolation method comprises: calculating compensation frame numbers of the infrared image and the visible light image according to frame rates and the time difference of the infrared image and the visible light image, wherein the compensation frame number is the quotient of the time difference divided by the frame interval time rounded up; selecting, according to the compensation frame numbers, one frame of image before and after the original image with the smallest timestamp difference as compensation frame images from the infrared image and the visible light image; for each pair of compensation frames of the infrared image and the visible light image, calculating a difference weight according to the difference between the corresponding timestamp and the timestamp of the original image; performing weighted averaging on the infrared compensation frame and the original infrared image, and the visible light compensation frame and the original visible light image, respectively, to obtain interpolated infrared images and visible light images as synchronized images.

5. The multi-spectrum fusion underground rubber-tired vehicle intelligent perception and safety warning system according to claim 1, wherein: fusing the registered infrared image and the visible light image to obtain a fused image comprises: performing non-subsampled Contourlet transform on the registered infrared image and the preprocessed visible light image, respectively, to perform multi-scale and multi-directional decomposition on the infrared image and the visible light image using a directional filter bank and a scale filter bank to obtain infrared image sub-band coefficients and visible light image sub-band coefficients at different scales and directions; initializing input neurons of a coupled neural network, inputting the infrared image sub-band coefficients and the visible light image sub-band coefficients into different input neurons, respectively, and establishing a mapping relationship between the input neurons and a link matrix; updating weight coefficients of the link matrix through iterative learning of the pulse neural network, using the interconnection and coupling relationship between the neurons, and performing weighted fusion on the infrared image sub-band coefficients and the visible light image sub-band coefficients to obtain sub-band coefficients of the fused image. The non-subsampled contourlet inverse transform is performed on the subband coefficients of the fusion image, and an interpolation filter bank is used to perform inverse transform reconstruction on the subband coefficients of the fusion image to obtain a fusion image fused in the time-frequency domain; A thermal radiation pseudo-color mapping algorithm is used to perform pseudo-color mapping on a thermal radiation target region in the fusion image fused in the time-frequency domain, to obtain a thermal radiation enhanced fusion image as the final output fusion image.

6. The multispectral fusion intelligent perception and safety warning system for downhole rubber-tired vehicles according to claim 1, characterized in that: Target detection is performed on the fusion image to identify obstacles, personnel and other vehicles in the image, including: A pre-trained convolutional neural network is used to extract features from the fusion image, and multi-scale and multi-level features of the image are extracted through multiple convolutional layers and pooling layers of the convolutional neural network to obtain a feature representation of the image; A YOLO algorithm is used to perform target detection and classification on the extracted feature representation, and the image is divided into fixed-size grids, the bounding boxes and class probabilities of the targets are predicted in each grid, and obstacles, personnel and other vehicles in the image are detected and identified.

7. The multispectral fusion intelligent perception and safety warning system for downhole rubber-tired vehicles according to claim 6, characterized in that: The network result of the YOLO algorithm contains multiple convolutional layers and fully connected layers, the convolutional layers are used to extract local features of the image, and the fully connected layers are used to predict the bounding boxes and class probabilities of the targets, and the non-maximum suppression algorithm is used to merge overlapping bounding boxes to obtain the final target detection result.

Images