Coal mining machine positioning and visual analysis system under multi-algorithm optimized fisheye camera

Abstract

The invention discloses a coal mining machine positioning and visual analysis system under a multi-algorithm optimized fisheye camera, and relates to the field of coal mining machine positioning and visual analysis. The deployment acquisition module is installed according to an optimization algorithm, has an intelligent function and acquires images in a multispectral manner; the preprocessing module corrects distortion by using an improved model and enhances an image in combination with multiple algorithms; the identification calculation module integrates multiple technologies to identify features of the coal mining machine and accurately position the coal mining machine; the tracking and positioning module fuses multi-algorithm tracking, and deep learning is used for predicting a trajectory during shielding; the visual analysis module analyzes the state of the coal mining machine through multiple technologies; and the transmission interaction module transmits data by using a fusion technology and provides multiple interaction modes. The method is accurate in positioning, the fisheye camera and the correction technology improve the positioning accuracy, and accurate tracking can be achieved when shielding is carried out; visual analysis is deep, and the state of the coal mining machine and environmental hidden dangers are monitored; data transmission is safe and efficient, and interaction is convenient; the system has strong anti-interference and self-optimization capabilities, can adapt to an underground environment, and improves the coal mining operation efficiency and safety.

Description

Technical Field

[0001] The present invention relates to the technical field of coal mining machine positioning and visual analysis, and in particular to a coal mining machine positioning and visual analysis system using a multi-algorithm optimized fisheye camera. Background Art

[0002] In the coal mining industry, accurate positioning and condition monitoring of shearers are crucial to improving production efficiency and ensuring safe production. However, the extremely complex underground environment of coal mines presents numerous challenges for shearer positioning and visual analysis.

[0003] The cramped underground space, cluttered with various equipment and supports, and the presence of numerous metal structures renders traditional positioning technologies like GPS and UWB inaccurate due to severe signal shielding and strong electromagnetic interference. Conventional cameras, due to their limited field of view, struggle to fully capture the shearer's large-scale movements across the coal face, making it impossible to obtain comprehensive, real-time information on the shearer's location and operating status. Even with fisheye cameras, the images they capture are severely distorted. Directly using these for target recognition would result in significant errors in the coordinate mapping process, affecting positioning accuracy.

[0004] At the same time, the harsh underground environment significantly impacts image quality. High concentrations of dust permeating the air not only obscure parts of the shearer's structure but also reduce the quality of light transmission, blurring the captured images. Furthermore, underground lighting conditions are complex and uneven illumination is common. When the shearer operates in different areas, the brightness and contrast of the image vary significantly, further increasing the difficulty of image processing and target recognition. Existing image recognition algorithms fail to fully account for the distortion characteristics of fisheye cameras, resulting in insufficient accuracy in feature extraction and difficulty in accurately identifying the shearer's key features. Dynamic tracking methods also fail to effectively integrate the shearer's motion model. When the target is obscured or subject to noise interference, tracking errors or target loss are likely to occur, failing to meet the requirements of safe production and efficient mining in coal mines. Therefore, the development of a shearer positioning and visual analysis system that can adapt to the complex underground environment is urgent. Summary of the Invention

[0005] The present invention proposes a multi-algorithm optimization fisheye camera coal mining machine positioning and visual analysis system to solve the problems mentioned in the above-mentioned prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a multi-algorithm optimized fisheye camera coal mining machine positioning and visual analysis system, comprising: Fisheye camera deployment and image acquisition module: Fisheye cameras are installed at intervals on the top of the coal face based on a spatial layout algorithm. The focal length is dynamically adjusted, the fill light intensity is adjusted according to the ambient light intensity, and images are captured through a synchronous trigger mechanism. The frequency is adaptively adjusted according to the status. Image preprocessing module: Fisheye images use pre-calibrated distortion parameters and a hyperbolic tangent equidistant projection model to correct distortion. Deep learning calibration is combined with particle swarm optimization for optimization. A multi-scale Retinex image enhancement algorithm is used to enhance illumination. An adaptive contrast stretching algorithm is introduced to adaptively stretch the image based on image distribution. Feature Recognition and Coordinate Calculation Module: This module processes images using an improved lightweight YOLOv5s model that integrates the Transformer and attention mechanisms. It also introduces a curriculum learning strategy to extract improved SIFT feature points. Based on binocular vision principles and the geometry of underground tunnels, it establishes equations for coordinate conversion. Dynamic tracking and positioning module: This module integrates Kalman filtering, optical flow, and particle filtering algorithms to track the dynamics of the coal mining machine. Kalman filtering is combined with dynamic noise covariance prediction, assisted by optical flow, and particle filtering to estimate the state. The long short-term memory network (LSTM) model is used to predict the trajectory. Visual Analysis Module: This module analyzes shearer images and positioning data, uses multi-view geometry and deep learning algorithms combined with fisheye cameras to capture 3D models of images, analyzes motion trajectory and speed, uses deep learning anomaly detection models to determine operating status, and uses image segmentation and semantic understanding to monitor components and warn of anomalies. Data transmission and interaction module: Upload coal mining machine location data and visual analysis results through the integration of 5G and blockchain technology, receive control instructions from the mine monitoring system, set up a local human-computer interaction interface, use virtual reality VR and augmented reality AR technology to view the work scene, and support gesture recognition and voice interaction.

[0007] Furthermore, it also includes: Anti-interference enhancement module: It adopts deep learning multimodal data fusion anti-interference strategy, integrates camera image data, inertial measurement unit data, lidar data and environmental sensor data, uses multimodal attention fusion network to automatically assign weights according to sensor data, adopts deep learning defogging and denoising algorithms to process affected images, adopts outlier removal strategy to process image and positioning data, sets time window and spatial neighborhood range, and judges data points as abnormal and removes them if the difference exceeds the set threshold.

[0008] Furthermore, it also includes: System self-calibration and optimization module: Regularly self-calibrate the fisheye camera, set calibration markers, adjust the camera distortion correction parameters and coordinate mapping parameters based on the markers and image position information combined with the deep learning parameter estimation model, use the reinforcement learning optimization algorithm to optimize the system parameters, learn the parameter adjustment strategy through interaction with the system environment, and compress the deep learning model using model compression technology.

[0009] Furthermore, the focal length f in the fisheye camera deployment and image acquisition module is dynamically adjusted according to the distance d between the coal mining machine and the camera. The adjustment formula is: , f0 is the initial focal length, d0 is the initial distance, Δv is the change in shearer speed, k is the speed influence coefficient, and the fill light intensity L is adjusted according to the ambient light intensity I. The formula is , L0 is the initial fill light intensity, I0 is the standard light intensity; The camera uses intelligent obstacle avoidance and adaptive adjustment technology. The built-in micro-electromechanical system (MEMS) sensor monitors the vibration and tilt of the camera, automatically adjusts the installation angle and position when it detects position changes, and uses ultrasonic vibration combined with electrostatic adsorption to regularly remove dust from the lens.

[0010] Furthermore, the isometric projection model formula in the image preprocessing module is: , r d is the radial distance after correction, r is the radial distance before distortion, k1, k2, k3, k4 are the distortion correction coefficients; The wavelet transform detail enhancement algorithm is introduced. The wavelet transform is used to decompose the image into different frequency sub-bands, and the high-frequency sub-bands are enhanced to highlight the detail features of the coal mining machine. The image quality evaluation index is used to guide the adaptive adjustment method of image enhancement parameters, and the image enhancement algorithm parameters are automatically adjusted according to the image clarity and contrast.

[0011] Furthermore, the improved polar geometry constraint equation in the feature recognition and coordinate calculation module is , x1, x2 are the homogeneous coordinates of the corresponding points in the left and right images respectively, F is the basic matrix, λ is the geometric structure influencing factor, and g(x1, x2) is the geometric structure function considering the curvature and slope of the roadway; Knowledge distillation technology is introduced to consider the dynamic changes of underground tunnels. The coordinate transformation formula is corrected in real time by establishing a dynamic tunnel geometric model. A deep learning feature point matching optimization algorithm is adopted, and a convolutional neural network is used to learn the matching relationship between feature points.

[0012] Furthermore, the Kalman filter state prediction equation in the dynamic tracking and positioning module is: , the update equation is , and ν k are process noise and measurement noise, respectively; A deep learning target re-identification algorithm and a trajectory association algorithm are introduced. The target re-identification algorithm uses a deep learning model to extract and match the appearance features of the coal mining machine. The trajectory association algorithm associates the reappearing target with the previous trajectory by learning the historical trajectory and motion pattern of the coal mining machine. Multi-sensor fusion technology is used to combine inertial measurement unit and lidar sensor data to predict and track the motion trajectory of the coal mining machine.

[0013] Furthermore, the visual analysis module uses deep learning semantic segmentation and instance segmentation technology to analyze the coal mining machine and its surrounding environment, segment the coal mining machine components, identify obstacles and coal walls, predict safety hazards and issue warnings by analyzing the spatial relationship between the coal mining machine and surrounding objects, and use behavioral analysis models to analyze and judge the behavior of coal mining machine operators.

[0014] Furthermore, the data transmission and interaction module adopts a combined architecture of edge computing and cloud computing, uses edge computing devices to process and analyze data, and uploads computing tasks to the cloud computing platform for analysis and processing. The human-computer interaction interface supports intelligent recommendation functions, provides operation suggestions and decision support based on the working status, historical data and operating habits of the coal mining machine, and adopts multi-language interaction technology.

[0015] Furthermore, the anti-interference enhancement module adopts a deep learning electromagnetic interference suppression algorithm, learns the characteristics of electromagnetic interference signals by training a deep learning model, processes the interfered data, adopts redundant sensor deployment and data backup strategies, uses redundant sensors to supplement data, and uses adaptive modulation and coding technology in the communication process to dynamically adjust communication parameters according to channel quality.

[0016] Compared with the existing technology, the beneficial effects of the present invention are: In terms of positioning, the improved fisheye camera deployment method and high-precision distortion correction technology enable precise positioning of the shearer. The specially designed fisheye camera, combined with intelligent zoom, autofocus, and multispectral imaging technologies, captures a more comprehensive and clear image of the shearer. This, combined with an optimized distortion correction algorithm, significantly improves coordinate mapping accuracy. The dynamic tracking algorithm, incorporating multiple advanced algorithms, accurately predicts the shearer's position even when obscured, with a position error of less than 0.2 meters. This effectively solves the problem of shearer positioning in complex underground environments and provides strong support for the precise control of coal mining operations.

[0017] In terms of visual analysis, the system utilizes advanced deep learning technology to conduct in-depth analysis of the shearer and its working environment. A high-precision model constructed using 3D reconstruction technology visually displays the shearer's structure and operating status. Anomaly detection models promptly identify abnormalities within the shearer. Semantic segmentation and instance segmentation technologies accurately analyze the wear and assembly relationships of shearer components, as well as potential safety hazards in the surrounding environment, ensuring the safe and stable operation of the shearer. Furthermore, analysis of operator behavior helps standardize operational procedures and reduce safety incidents caused by human error.

[0018] In terms of data transmission and interaction, the convergence of 5G and blockchain communication technologies ensures secure and reliable data transmission, while the combination of edge computing and cloud computing improves data processing efficiency. VR and AR technologies, along with various interactive methods, provide operators with a convenient and intuitive experience. Intelligent recommendation functions also assist in decision-making, enhancing the intelligence of coal mining operations. The anti-interference enhancement module and system self-calibration optimization module enhance the system's adaptability and stability in harsh underground environments, reducing equipment failures and maintenance costs, and significantly improving the overall efficiency and safety of coal mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic block diagram of a multi-algorithm optimized fisheye camera positioning and visual analysis system for coal mining machines proposed by the present invention; Figure 2 This is a schematic diagram comparing the positioning errors of different positioning systems in various areas; Figure 3 Schematic diagram of the change of anomaly detection accuracy of different systems over time; Figure 4 A schematic diagram comparing data transmission delays of different systems in different network environments. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0022] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The present invention will be further described in detail below with reference to the accompanying drawings.

[0023] Reference Figures 1 to 4 : A multi-algorithm optimized fisheye camera positioning and visual analysis system for coal mining machines, including: Fisheye camera deployment and image acquisition module: At the top of the coal mining face, a special fisheye camera is installed every 40-60 meters based on the spatial layout optimization algorithm. The camera uses a model with intelligent zoom and autofocus functions, such as [specific model]. It has a built-in 850nm wavelength infrared fill light with automatic dimming function. In actual operation, according to the light intensity I measured by the ambient light intensity sensor, according to the formula To adjust the fill light intensity L, L0 is set to [initial fill light intensity value], and I0 is set to [standard light intensity value]. The camera utilizes multispectral imaging technology, capable of simultaneously capturing visible, near-infrared, and thermal infrared images. To ensure synchronization and accuracy of image acquisition, the camera uses a high-precision synchronized trigger mechanism for image acquisition, maintaining a frame rate between 25-30 fps. This frame rate is adaptively adjusted based on the shearer's velocity v using the formula f=f0+k×v, where f0 is the initial frame rate [specific value] and k is the adjustment factor [specific value]. Furthermore, the camera incorporates a built-in microelectromechanical system (MEMS) sensor to monitor its vibration, tilt, and other conditions in real time. If it detects a possible collision or a change in position due to roof deformation, it automatically adjusts its mounting angle and position to ensure accurate capture of the shearer. The camera also features an automatic cleaning function, using a combination of ultrasonic vibration and electrostatic adsorption to periodically remove dust from the lens at [specific duration] intervals to ensure image clarity.

[0024] Image preprocessing module: After collecting the fisheye image, the distortion correction parameters obtained by the pre-optimization of the checkerboard calibration method and the particle swarm optimization algorithm are used, and the improved hyperbolic tangent equidistant projection model is used for distortion correction. The formula is: , where k1, k2, k3, and k4 are the determined distortion correction coefficients. After distortion correction, the multi-scale Retinex image enhancement algorithm is first used to enhance the image illumination. An adaptive contrast stretching algorithm is then introduced to adaptively stretch the image based on the local contrast distribution of the image, highlighting the detailed features of the shearer. Subsequently, a wavelet transform is used to decompose the image into sub-bands of different frequencies, and the high-frequency sub-bands are enhanced to further improve the recognition of the shearer features. Furthermore, an adaptive image enhancement parameter adjustment method guided by image quality evaluation indicators is used to automatically adjust the parameters of the aforementioned image enhancement algorithms based on quality evaluation indicators such as image clarity and contrast, thereby optimizing the image enhancement effect.

[0025] Feature recognition and coordinate calculation module: Run the improved lightweight YOLOv5s model that integrates Transformer and attention mechanism to process the preprocessed images. During the model training process, a curriculum learning strategy is introduced. Training starts with simple samples. As the training progresses, the difficulty of the samples is gradually increased to improve the convergence speed and accuracy of the model. After the bounding box of the coal mining machine is detected by the model, the improved SIFT feature point extraction method is used. In the traditional SIFT feature point extraction process, directional consistency constraints are added to improve the stability of the feature points, and the positions of these feature points in the image coordinate system are calculated. Based on the principle of binocular vision and combined with the complex geometric structure of the underground tunnel, the improved polar geometric constraint equation is established. Coordinate transformation is performed. λ is determined based on factors such as the actual curvature and slope of the roadway, and the g(x1, x2) function is constructed based on the roadway's geometric model. Using pre-calibrated parameters, the image coordinates are converted to positions in the 3D roadway coordinate system, enabling precise positioning of the shearer underground.

[0026] The dynamic tracking and positioning module uses a fusion of Kalman filtering, optical flow, and particle filtering algorithms to dynamically track the shearer. During the Kalman filtering process, the noise covariance is dynamically adjusted based on the shearer's motion state and the degree of environmental interference. The optical flow method is used to calculate the shearer's motion vector between adjacent frames, assisting the Kalman filter in more accurately predicting the shearer's trajectory. The particle filter, by sampling and weighting a large number of particles, more accurately estimates the shearer's state when its motion state changes dramatically or when strong interference is present. When the shearer is occluded, a deep learning-based trajectory prediction model is used to generate candidate trajectories and perform trajectory interpolation. This model, based on a long short-term memory (LSTM) network, learns from historical shearer motion data to predict the shearer's position during occlusion. Furthermore, by combining data from multiple sensors, such as an inertial measurement unit and lidar, the shearer's trajectory is more accurately predicted and tracked, improving positioning accuracy and stability.

[0027] Visual analysis module: Using 3D reconstruction technology, based on multi-view geometry and deep learning algorithms, combined with images collected by multiple fisheye cameras, a high-precision 3D model of the coal mining machine is constructed. By analyzing the motion trajectory, speed changes and other information of the coal mining machine, combined with the working status of the coal mining machine (such as coal mining power and cutting speed), an anomaly detection model based on the generative adversarial network (GAN) is used to determine whether the coal mining machine is working normally. Semantic segmentation and instance segmentation technology are used to segment the components of the coal mining machine. Combined with semantic understanding technology, it can not only observe the wear of the components, but also determine whether the assembly relationship between the components is correct. If the area of the wear area exceeds the set threshold S threshold (Set to [Specific Area Value]) or if any abnormalities occur in the assembly relationship, an early warning signal will be issued. Simultaneously, by analyzing the spatial relationship between the shearer and surrounding objects, potential safety hazards such as the risk of collision between the shearer and obstacles or the possibility of coal wall spalling can be predicted, and timely warnings can be issued to ensure safe mining operations.

[0028] Data Transmission and Interaction Module: Utilizing 5G and blockchain-integrated communication technology, the shearer's location data, visual analysis results, and other information are uploaded to the mine monitoring system, while simultaneously receiving control commands from the system. The decentralized and tamper-proof nature of blockchain ensures data security and reliability. A local human-machine interaction interface, utilizing virtual reality (VR) and augmented reality (AR) technologies, allows operators to immersively view the shearer's operating scene through VR devices and access real-time shearer-related information and operating instructions on-site through AR devices. Gesture recognition and voice interaction technologies are introduced to facilitate natural interaction between operators and the system, improving operational convenience and efficiency. Furthermore, an intelligent recommendation function is added to provide operators with operational suggestions and decision support based on the shearer's operating status, historical data, and operator habits.

[0029] The present invention also includes the following modules: Anti-interference enhancement module: adopts a multimodal data fusion anti-interference strategy based on deep learning. It fuses the image data of the camera, the data of the inertial measurement unit, the lidar data and the data of other environmental sensors, and uses the multimodal attention fusion network to automatically assign weights according to the reliability of different sensor data in different environments, thereby improving the system's perception of the state of the coal mining machine. The defogging and denoising algorithms of deep learning are used to process images affected by dust and noise. By training the deep learning model, the mapping relationship between dust images and clear images, noisy images and noise-free images is learned, and the collected contaminated images are defogged and denoised to restore the clarity of the image. At the same time, an outlier removal strategy based on temporal consistency and spatial correlation is used to process the collected image data and positioning data. Set the time window T and the spatial neighborhood range S. Within the time window and the spatial neighborhood, if the difference between a data point and other data points exceeds the set threshold D threshold , then the data point is determined to be an outlier and removed.

[0030] The present invention also includes the following modules: System Self-Calibration and Optimization Module: The fisheye camera performs regular self-calibration. This module uses multiple high-precision calibration markers placed on the coal mining face and captures images of these markers. Based on the known locations of the markers and the positional information in the images, the system automatically adjusts the camera's distortion correction parameters, coordinate mapping parameters, and other related parameters using a deep learning parameter estimation model. A reinforcement learning-based optimization algorithm is used to optimize system parameters (such as the noise covariance of the Kalman filter, hyperparameters of the YOLOv5s model, and parameters of the 3D reconstruction algorithm). This reinforcement learning algorithm uses the system's positioning accuracy, visual analysis accuracy, and real-time performance as reward functions. Through continuous interaction with the system environment, it learns the optimal parameter adjustment strategy to improve system performance. Furthermore, model compression technology is used to compress the deep learning model, reducing the computational workload and storage requirements without compromising model performance, thereby improving system efficiency.

[0031] In this invention, the fisheye camera deployment and image acquisition module utilizes intelligent obstacle avoidance and adaptive adjustment technology. The camera incorporates a built-in microelectromechanical system (MEMS) sensor, boasting high sensitivity and precision. This sensor monitors the camera's vibration, tilt, and other conditions in real time at high frequencies. When conditions such as roof deformation or falling rocks in a coal mine could cause the camera to shift or be impacted, the MEMS sensor quickly detects these subtle changes and transmits the data to the camera's intelligent control system. This system, through complex algorithmic analysis, precisely controls the camera's multi-angle adjustment mechanism, automatically adjusting its mounting angle and position. For example, a stepper motor precisely controls the rotation and elevation of the pan / tilt head, ensuring the camera always captures the coal mining machine at the optimal viewing angle. The automatic cleaning function utilizes a combination of ultrasonic vibration and electrostatic adsorption. The ultrasonic generator periodically activates, generating high-frequency vibrations that dislodge dust particles from the lens surface. Simultaneously, an electrostatic adsorption device creates an electrostatic field around the lens, rapidly attracting the dislodged dust particles and preventing them from reattaching to the lens. In this way, the clarity of image acquisition can be effectively guaranteed, providing high-quality image data for subsequent coal mining machine positioning and visual analysis, thereby ensuring the stable and efficient operation of the entire system.

[0032] In this invention, the image preprocessing module, based on the multi-scale Retinex image enhancement algorithm and the adaptive contrast stretching algorithm, introduces a detail enhancement algorithm based on the wavelet transform. First, the multi-scale Retinex image enhancement algorithm is applied. This algorithm, based on the characteristics of the human visual system's perception of color and brightness, simulates the retina's adaptive response to light, effectively compressing the image's dynamic range and enhancing its local contrast. This allows the coal mining machine image to appear more natural under various lighting conditions. Simultaneously, in conjunction with the adaptive contrast stretching algorithm, the grayscale range is dynamically adjusted based on the image's grayscale distribution, further enhancing the overall image contrast and making the coal mining machine's outline more clearly discernible. Furthermore, a detail enhancement algorithm based on the wavelet transform is introduced. The wavelet transform acts as a sophisticated "filter," decomposing an image into subbands of different frequencies: low-frequency approximate subbands and high-frequency detail subbands. When enhancing the high-frequency subbands, the characteristics of the wavelet coefficients are exploited to adjust their amplitudes, specifically highlighting the coal mining machine's edges, textures, and other detailed features. For example, details such as the shearer's gear structure and bolted connections become more pronounced after enhancement, greatly improving the accuracy of feature recognition. Furthermore, an adaptive adjustment method for image enhancement parameters is employed, guided by image quality evaluation indicators. A feedback mechanism is established by calculating quality evaluation indicators such as image clarity and contrast. Based on the real-time values of these indicators, an intelligent algorithm automatically adjusts the parameters of image enhancement algorithms such as multi-scale Retinex and adaptive contrast stretching, dynamically optimizing the image enhancement effect and ensuring that the shearer images input into subsequent analysis are always in optimal condition.

[0033] In this paper, the feature recognition and coordinate calculation module incorporates knowledge distillation technology based on an improved lightweight YOLOv5s model that integrates the Transformer and attention mechanisms. For feature recognition, the core model is an improved lightweight YOLOv5s model that integrates the Transformer and attention mechanisms. The Transformer architecture empowers the model with powerful global modeling capabilities, enabling it to better capture long-range dependencies within the coal mining machine in the image; the attention mechanism focuses on key areas, improving feature extraction efficiency. Furthermore, knowledge distillation technology is introduced. The rich knowledge learned from massive data training in a large, complex model, such as feature distribution and category discrimination, is transferred to the lightweight model through methods such as soft labeling. This allows the lightweight model to absorb the essence of the large model while maintaining low computational complexity, significantly improving performance and enabling rapid and accurate identification of coal mining machine features. When converting image coordinates to a three-dimensional roadway coordinate system, the complex dynamic changes in underground roadways, including frequent roof subsidence and roadway deformation, are taken into account. To this end, a dynamic roadway geometry model is established, using high-precision sensors to collect real-time deformation data such as displacement and angular changes. Based on this data, mathematical algorithms are used to make real-time corrections to the coordinate transformation formula. For example, when roof subsidence causes changes in roadway height, the vertical coordinate transformation parameters are promptly adjusted to ensure accurate positioning of the shearer. A deep learning-based feature point matching optimization algorithm is employed. Leveraging the powerful feature extraction capabilities of convolutional neural networks, in-depth feature mining is performed on shearer images from different perspectives. The network is trained with a large amount of annotated data, enabling it to learn the accurate matching relationships between feature points. In practical applications, this method can quickly identify reliable matching feature point pairs, improving matching accuracy and efficiency and providing a solid foundation for subsequent positioning and analysis.

[0034] In this invention, the dynamic tracking and positioning module integrates Kalman filtering, optical flow, and particle filtering algorithms, and introduces a deep learning-based target re-identification (RID) algorithm and a trajectory association algorithm. The Kalman filter utilizes a linear dynamic model of the system state, effectively handling noise interference during the shearer's motion through prediction and update steps, and provides an optimal estimate. The optical flow method uses the motion information of pixels in the image to calculate the displacement vector between pixels in adjacent frames, capturing the shearer's motion trends in real time. The particle filter is suitable for complex, non-linear, and non-Gaussian scenarios, simulating the shearer's possible states using a large number of random samples (particles) to improve tracking robustness. Furthermore, a deep learning-based RID and trajectory association algorithm are introduced. When the shearer reappears after being obscured by equipment or coal walls within the tunnel, the RID algorithm leverages a deep convolutional neural network to deeply extract the shearer's unique appearance features, such as its color, shape, and iconic components. Rapid matching and retrieval within a massive feature database ensures rapid and accurate target identification even when conditions such as the shearer's posture and lighting vary. The trajectory association algorithm uses models such as recurrent neural networks to learn the shearer's historical trajectory data and analyze its motion patterns, such as speed, acceleration, and turning radius. After target re-identification, the reappearing shearer is accurately associated with its previous trajectory based on these patterns, seamlessly resuming tracking. Furthermore, multi-sensor fusion technology is utilized. The inertial measurement unit senses the shearer's acceleration, angular velocity, and other information in real time, while the lidar provides high-precision distance and position data. Through a data fusion algorithm, data from different sensors is deeply integrated, combining the strengths of each sensor to more accurately predict and track the shearer's motion trajectory, ensuring stable and reliable operation of the system in complex underground environments.

[0035] In this invention, the visual analysis module utilizes deep learning-based semantic segmentation and instance segmentation techniques to perform a more detailed analysis of the shearer and its surroundings. Semantic segmentation uses a trained deep neural network to classify each pixel in the input image, accurately distinguishing various objects within the shearer and its surroundings according to their categories. For example, components such as the shearer's drum, cutting arm, and fuselage, as well as environmental elements such as the coal wall, tunnel supports, and transport equipment, are precisely labeled with their respective categories. Instance segmentation not only identifies object categories but also accurately segments and locates each specific object instance. It clearly defines the boundaries and spatial position of each shearer component and accurately distinguishes each instance of surrounding obstacles, such as scattered coal and temporarily placed tools. By performing in-depth analysis of the segmented image information, the system accurately determines the spatial relationship between the shearer and surrounding objects. Using spatial geometry algorithms, it calculates parameters such as the distance and angle between the shearer and obstacles, predicting collision risk based on preset safety thresholds. By analyzing the texture and shape changes of the coal wall, combined with geological data and historical monitoring information, a machine learning model is used to assess the possibility of spalling. Once a potential safety hazard is detected, the system quickly issues an early warning signal, prompting operators to take timely action. The visual analysis module also incorporates a deep learning behavioral analysis model. This model, based on a convolutional neural network, is trained using a large amount of labeled video data of operator behavior. It can capture operators' movements and postures in real time and analyze their operational processes and behavioral patterns. By comparing against standard operating procedures, it can determine whether operators have violated operational regulations, such as not wearing safety equipment correctly or operating in the wrong order. Unsafe behavior can be promptly identified and corrected, ensuring the safe and orderly conduct of coal mining operations.

[0036] In this invention, the data transmission and interaction module utilizes an architecture that combines edge computing with cloud computing. Edge computing devices are deployed near the coal mining site and feature high-performance processors and specialized data processing algorithms. For image data captured by fisheye cameras and coal mining machine operating status data collected by various sensors, the edge computing devices can rapidly perform preliminary local processing. For example, image compression algorithms are used to compress image data, and filtering algorithms are used to remove noise from sensor data, significantly reducing data transmission volume. Simultaneously, feature extraction and simple analysis are performed on the data, such as identifying the basic operating status of the coal mining machine and detecting obvious anomalies. These data are promptly fed back to the local control system, improving the real-time response of the system. Complex computing tasks, such as deep learning-based coal mining machine failure prediction and in-depth mining and analysis of large-scale historical data, are uploaded to the cloud computing platform. Leveraging its powerful cluster computing capabilities and massive storage resources, the cloud computing platform utilizes a distributed computing framework and advanced data analysis algorithms to conduct in-depth data analysis, uncovering underlying patterns and valuable insights. Regarding human-computer interaction, the interface has added an intelligent recommendation function. The system monitors the operating parameters of the coal mining machine in real time, such as speed, traction force, and cutting power. It combines historical data on optimal operating records under different working conditions with learning and analysis of operators' past operating habits, and uses intelligent algorithms to provide operators with precise operational recommendations. For example, when the coal mining machine encounters coal seams of different hardness, it recommends a suitable combination of cutting speed and traction speed to assist operators in making more scientific decisions. In addition, multilingual interaction technology is used. The integrated multi-language speech recognition and synthesis engine can automatically identify the language of the operator's voice commands and provide feedback in the corresponding language. Whether it is operators from different dialect regions in China or operators with different language backgrounds in international cooperation projects, they can interact with the system conveniently, greatly expanding the scope of application of the system and improving its convenience and friendliness.

[0037] In this invention, the data transmission and interaction module utilizes an architecture that combines edge computing and cloud computing. First, a large amount of sample data containing various electromagnetic interference signals is collected. This data covers interference signals of varying frequencies, intensities, and waveforms. These samples are then trained using deep learning models such as convolutional neural networks. Through continuous learning, the model accurately identifies the unique characteristics of electromagnetic interference signals, such as specific spectral features and time-domain waveform characteristics. When the system's collected data is subject to electromagnetic interference, the trained model analyzes and separates the interference signals, applying filtering, noise reduction, and other techniques to process the affected data, maximizing the data's authenticity and accuracy, ensuring the reliability of subsequent analysis and decision-making. To address potential sensor interference or damage, redundant sensor deployment and data backup strategies are implemented. Additional redundant sensors of the same type or with complementary functions are deployed at key locations in and around the coal mining machine. These sensors collect data synchronously in real time and back up and store the data. If a primary sensor experiences severe electromagnetic interference or physical damage and ceases to function properly, the system can quickly switch to the redundant sensor, using its collected data to supplement the system's operations, ensuring uninterrupted data collection and monitoring and maintaining normal system operation. Adaptive modulation and coding technologies are employed in the communication link. The system monitors channel quality parameters such as signal strength, signal-to-noise ratio, and bit error rate (BER) in real time. Based on the dynamic changes in these parameters, intelligent algorithms automatically adjust the communication modulation and coding schemes. For example, when channel quality is good, higher-order modulation and efficient coding schemes are used to increase data transmission rates. When channel quality degrades due to interference, lower-order modulation and coding schemes with stronger error correction capabilities are promptly switched to reduce the BER, ensuring communication reliability and anti-interference capabilities, and guaranteeing stable data transmission and interaction between system modules.

[0038] The above are only preferred specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multi-algorithm optimized fisheye camera coal mining machine positioning and visual analysis system, characterized by: Includes the following modules: Fisheye camera deployment and image acquisition module: Fisheye cameras are installed at intervals on the top of the coal face based on a spatial layout algorithm. The focal length is dynamically adjusted, the fill light intensity is adjusted according to the ambient light intensity, and images are captured through a synchronous trigger mechanism. The frequency is adaptively adjusted according to the status. Image preprocessing module: Fisheye images use pre-calibrated distortion parameters and a hyperbolic tangent equidistant projection model to correct distortion. Deep learning calibration is combined with particle swarm optimization for optimization. A multi-scale Retinex image enhancement algorithm is used to enhance illumination. An adaptive contrast stretching algorithm is introduced to adaptively stretch the image based on image distribution. Feature Recognition and Coordinate Calculation Module: This module processes images using an improved lightweight YOLOv5s model that integrates the Transformer and attention mechanisms. It also introduces a curriculum learning strategy to extract improved SIFT feature points. Based on binocular vision principles and the geometry of underground tunnels, it establishes equations for coordinate conversion. Dynamic tracking and positioning module: This module integrates Kalman filtering, optical flow, and particle filtering algorithms to track the dynamics of the coal mining machine. Kalman filtering is combined with dynamic noise covariance prediction, assisted by optical flow, and particle filtering to estimate the state. The long short-term memory network (LSTM) model is used to predict the trajectory. Visual Analysis Module: This module analyzes shearer images and positioning data, uses multi-view geometry and deep learning algorithms combined with fisheye cameras to capture 3D models of images, analyzes motion trajectory and speed, uses deep learning anomaly detection models to determine operating status, and uses image segmentation and semantic understanding to monitor components and warn of anomalies. Data transmission and interaction module: Upload coal mining machine location data and visual analysis results through the integration of 5G and blockchain technology, receive control instructions from the mine monitoring system, set up a local human-computer interaction interface, use virtual reality VR and augmented reality AR technology to view the work scene, and support gesture recognition and voice interaction.

2. The multi-algorithm optimized fisheye camera shearer positioning and visual analysis system according to claim 1 is characterized in that: Also includes: Anti-interference enhancement module: It adopts deep learning multimodal data fusion anti-interference strategy, integrates camera image data, inertial measurement unit data, lidar data and environmental sensor data, uses multimodal attention fusion network to automatically assign weights according to sensor data, adopts deep learning defogging and denoising algorithms to process affected images, adopts outlier removal strategy to process image and positioning data, sets time window and spatial neighborhood range, and judges data points as abnormal and removes them if the difference exceeds the set threshold.

3. The multi-algorithm optimized fisheye camera shearer positioning and visual analysis system according to claim 1 is characterized in that: Also includes: System self-calibration and optimization module: Regularly self-calibrate the fisheye camera, set calibration markers, adjust the camera distortion correction parameters and coordinate mapping parameters based on the markers and image position information combined with the deep learning parameter estimation model, use the reinforcement learning optimization algorithm to optimize the system parameters, learn the parameter adjustment strategy through interaction with the system environment, and compress the deep learning model using model compression technology.

4. The multi-algorithm optimized fisheye camera shearer positioning and visual analysis system according to claim 1 is characterized in that: The focal length f in the fisheye camera deployment and image acquisition module is dynamically adjusted according to the distance d between the coal mining machine and the camera. The adjustment formula is: , f0 is the initial focal length, d0 is the initial distance, Δv is the change in shearer speed, k is the speed influence coefficient, and the fill light intensity L is adjusted according to the ambient light intensity I. The formula is , L0 is the initial fill light intensity, I0 is the standard light intensity; The camera uses intelligent obstacle avoidance and adaptive adjustment technology. The built-in micro-electromechanical system (MEMS) sensor monitors the vibration and tilt of the camera, automatically adjusts the installation angle and position when it detects position changes, and uses ultrasonic vibration combined with electrostatic adsorption to regularly remove dust from the lens.

5. The multi-algorithm optimized fisheye camera shearer positioning and visual analysis system according to claim 1 is characterized in that: The formula of the equidistant projection model in the image preprocessing module is: , r d is the radial distance after correction, r is the radial distance before distortion, k1, k2, k3, k4 are the distortion correction coefficients; The wavelet transform detail enhancement algorithm is introduced. The wavelet transform is used to decompose the image into different frequency sub-bands, and the high-frequency sub-bands are enhanced to highlight the detail features of the coal mining machine. The image quality evaluation index is used to guide the adaptive adjustment method of image enhancement parameters, and the image enhancement algorithm parameters are automatically adjusted according to the image clarity and contrast.

6. The multi-algorithm optimized fisheye camera shearer positioning and visual analysis system according to claim 1 is characterized in that: The improved polar geometry constraint equation in the feature recognition and coordinate calculation module is: , x1, x2 are the homogeneous coordinates of the corresponding points in the left and right images respectively, F is the basic matrix, λ is the geometric structure influencing factor, and g(x1, x2) is the geometric structure function considering the curvature and slope of the roadway; Knowledge distillation technology is introduced to consider the dynamic changes of underground tunnels. The coordinate transformation formula is corrected in real time by establishing a dynamic tunnel geometric model. A deep learning feature point matching optimization algorithm is adopted, and a convolutional neural network is used to learn the matching relationship between feature points.

7. The multi-algorithm optimized fisheye camera shearer positioning and visual analysis system according to claim 1 is characterized in that: The Kalman filter state prediction equation in the dynamic tracking and positioning module is: , the update equation is , and ν k are process noise and measurement noise, respectively; A deep learning target re-identification algorithm and a trajectory association algorithm are introduced. The target re-identification algorithm uses a deep learning model to extract and match the appearance features of the coal mining machine. The trajectory association algorithm associates the reappearing target with the previous trajectory by learning the historical trajectory and motion pattern of the coal mining machine. Multi-sensor fusion technology is used to combine inertial measurement unit and lidar sensor data to predict and track the motion trajectory of the coal mining machine.

8. The multi-algorithm optimized fisheye camera shearer positioning and visual analysis system according to claim 1 is characterized in that: The visual analysis module uses deep learning semantic segmentation and instance segmentation technology to analyze the coal mining machine and its surrounding environment, segment the coal mining machine components, identify obstacles and coal walls, predict safety hazards and issue warnings by analyzing the spatial relationship between the coal mining machine and surrounding objects, and use behavioral analysis models to analyze and judge the behavior of coal mining machine operators.

9. The multi-algorithm optimized fisheye camera shearer positioning and visual analysis system according to claim 1 is characterized in that: The data transmission and interaction module adopts a combination of edge computing and cloud computing architecture, uses edge computing devices to process and analyze data, and uploads computing tasks to the cloud computing platform for analysis and processing. The human-computer interaction interface supports intelligent recommendation functions, provides operation suggestions and decision support based on the working status, historical data and operating habits of the coal mining machine, and adopts multi-language interaction technology.

10. The multi-algorithm optimized fisheye camera shearer positioning and visual analysis system according to claim 1, characterized in that: The anti-interference enhancement module adopts a deep learning electromagnetic interference suppression algorithm, learns the electromagnetic interference signal characteristics by training the deep learning model, processes the interfered data, adopts redundant sensor deployment and data backup strategies, uses redundant sensors to supplement data, and uses adaptive modulation and coding technology in the communication process to dynamically adjust communication parameters according to channel quality.

Images