Working face cut-through position cutter return control system and method based on multi-modal feature fusion

Abstract

The invention discloses a working face cut-through position cutter return control system and method based on multi-modal feature fusion. The system comprises a dual-light fusion camera, a coal mining machine control mechanism, an AI analysis server, a data analysis server and a dust sensor. The method comprises the following steps that S1, when a reset sensor corresponds to a reset magnet, an encoder is reset, and meanwhile, the numerical value of the encoder is transmitted to a data acquisition module; s2, the dust sensor transmits a dust concentration value to an AI analysis server; s3, transmitting the shot original image to an AI analysis server by the dual-light fusion camera, and identifying the position relation between the roller of the coal mining machine and the cut-through position; s4, the AI analysis server transmits an identification result to a data acquisition module; s5, the data acquisition module operates a process driving engine according to the numerical value of the encoder and the identification result, and the process driving engine controls the working state of the coal mining machine; and S6, after the coal mining machine is shut down, coal mining machine roller height adjustment and cutter returning operation are carried out.

Description

Technical Field

[0001] This invention relates to the field of coal mining, and in particular to a working face cutting position return control system and method based on multimodal feature fusion. Background Technology

[0002] The intelligent and routine operation of fully mechanized mining faces is a development trend in the coal mining industry. In situations where fully mechanized mining faces have few or no personnel, sensing equipment posture through video and physical sensors will become one of the important guarantees for safe and efficient coal mine production. In particular, the identification and automatic calibration of the cutting position during the coal cutting process of the coal mining machine will have a significant impact on whether the cutting position can achieve fully automated coal cutting in the triangular coal face.

[0003] In the automated cutting process of triangular coal seams, the coal mining machine moves towards the cut-through position. Without human intervention, the machine needs to stop precisely at the cut-through position, adjust the height of the left and right drums, and then perform a reverse cut. Currently, the main means of ensuring the accuracy of the cut-through position is the encoder. However, due to factors such as slippage and sliding on the working face, the cut-through position needs to be manually calibrated for each cut. The encoder also has cumulative errors, requiring periodic manual calibration. The operation process is quite cumbersome, making it difficult to achieve full automation of the triangular coal seam process and hindering the intelligent and routine advancement of the working face. Summary of the Invention

[0004] The purpose of this invention is to address the above-mentioned problems by providing a multimodal feature fusion working face cutting position return control system and method that improves the accuracy of cutting position recognition, reduces manual intervention, and ensures the automated and stable operation of triangular coal seams.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A multimodal feature fusion working face cutting position return control system includes a dual-light fusion camera, a coal mining machine control mechanism, an AI analysis server, a data analysis server, and a dust sensor. The dual-light fusion camera is located at the end of the working face, fixed on a hydraulic support and facing the cutting position of the coal mining machine. The signal output terminal of the dual-light fusion camera is connected to the signal input terminal of the AI ​​analysis server. The signal terminals of the AI ​​analysis server and the coal mining machine control mechanism are bidirectionally connected to the signal terminal of the data analysis server. The dust sensor is located at the end of the working face and fixed on the hydraulic support. The signal output terminal of the dust sensor is connected to the signal input terminal of the data analysis server.

[0006] Furthermore, the AI ​​analysis server includes an image enhancement module for enhancing the original image captured by the dual-light fusion camera, and a video recognition module for determining the positional relationship between the coal mining machine drum and the cutting point based on the enhanced image. The signal input terminal of the image enhancement module is communicatively connected to the signal output terminal of the dual-light fusion camera, and the signal output terminal of the image enhancement module is connected to the signal input terminal of the video recognition module. The signal output terminal of the video recognition module is connected to the signal input terminal of the data analysis server via MQTT broadcast, and the signal output terminal of the data analysis server is connected to the signal input terminal of the image enhancement module.

[0007] Furthermore, the data analysis server includes a data acquisition module for collecting and analyzing various data, and a process drive engine for driving the coal mining machine. The signal input terminal of the data acquisition module is connected to the signal output terminal of the frequency identification module, the signal output terminal of the dust sensor, and the signal output terminal of the coal mining machine control mechanism. The signal output terminal of the data acquisition module is connected to the signal input terminal of the image enhancement module and the signal input terminal of the process drive engine. The signal output terminal of the process drive engine is connected to the signal input terminal of the coal mining machine control mechanism.

[0008] Furthermore, the coal mining machine control mechanism includes a coal mining machine controller for controlling the movement of the coal mining machine, an encoder for determining the displacement distance of the coal mining machine, a reset magnet and a reset sensor for performing encoder reset operations; there are two reset magnets arranged on a cable groove at the end of the working face, the reset sensor is fixed on the coal mining machine and corresponds to the position of the reset magnet, the signal output terminal of the reset sensor is connected to the signal input terminal of the coal mining machine controller, the signal output terminal of the coal mining machine controller is connected to the signal input terminal of the encoder, the signal output terminal of the encoder is connected to the signal input terminal of the data acquisition module, and the signal output terminal of the process drive engine is connected to the signal input terminal of the coal mining machine controller.

[0009] A method for controlling the cut-through position and return cut of a working face through multimodal feature fusion includes the following steps: S1. When the coal mining machine moves to a position where the reset sensor corresponds to the reset magnet, the reset sensor will generate a signal and send the signal to the coal mining machine controller. The coal mining machine controller will reset the encoder, and at the same time, the encoder will transmit the encoder value to the data acquisition module. S2. The dust sensor transmits the dust concentration value to the data acquisition module, which then transmits it to the AI ​​analysis server. The S3 dual-light fusion camera transmits the captured raw images to the AI ​​analysis server, which performs image processing and identifies the relationship between the coal mining machine drum and the cutting position based on the dust concentration value. S4. The AI ​​analysis server transmits the identification results of the relationship between the coal mining machine drum and the cutting position to the data acquisition module. S5. The data acquisition module runs the process drive engine based on the encoder values ​​and the identification results of the relationship between the coal mining machine drum and the cutting position. The process drive engine sends instructions to the coal mining machine controller and controls the working status of the coal mining machine through the coal mining machine controller. S6. After the coal mining machine stops, adjust the height of the coal mining machine drum and perform the reverse cutter operation.

[0010] Furthermore, step S3 specifically includes the following steps: S31. The dual-light fusion camera transmits the captured images to the image enhancement module. The image enhancement module performs image processing operations on the images captured by the dual-light fusion camera and transmits the processed images to the video recognition module. S32. The video recognition module judges and identifies the positional relationship between the coal mining machine drum and the cutting penetration position, and sends the recognition result to the data acquisition module.

[0011] Furthermore, in step S31, the image enhancement module performs image processing operations on the original image captured by the dual-light fusion camera, specifically including the following steps: S311, For the original image A large-scale low-pass filter is performed to obtain the guiding image G(x,y); its calculation formula is as follows: ; in, For anisotropic diffusion operators, It is the global mean of the illumination component, W and H are the width and height of the original image, respectively, and N is the total number of pixels; S312. Normalize the guide image based on the dust concentration value to obtain the dust index, the calculation formula of which is: , ; S313. Obtain the window radius r and regularization term based on the dust index. The calculation formula is: ; ; S314, Setting [ , ] is [7,20], set [ , ]for[ ], thus obtaining a denoised image that varies with dust concentration. .

[0012] S315. Denoising the image The formula for calculating the atmospheric scattering model is as follows: ; in, The image is a fog-free image, where t(x, y) is the transmittance and A is the atmospheric light component. ; ; in, For transmission rate, It is a moderating factor related to the dust index d; S316. After comprehensive calculation, a fog-free image is obtained, and the calculation formula is as follows: .

[0013] Furthermore, step S32 specifically includes the following steps: S321. Input the haze-free image Scale proportionally to a uniform size of 640×640 and normalize to [0,1]; S322. Detect and output the target bounding box of the coal mining machine drum using the YOLOv8-Det model. ,,in, The coordinates of the center of the target bounding box of the coal mining machine drum are: The width of the target bounding box of the coal mining machine drum. The height of the target bounding box of the coal mining machine drum; S323. Identify coal wall boundary points, output multiple sets of coal wall boundary points, remove isolated points according to horizontal position X, and set a confidence filtering threshold. After removing low-confidence coal wall boundary points, the filtered set of boundary points is obtained. ; S324, Set of Boundary Points By fitting the equation, a straight line describing the coal wall boundary is obtained. ax + by + c =0; S325. Calculate the normal distance from the center point of the coal mining machine drum to the coal wall boundary line. The calculation formula is as follows: ; when If the value is ≤0, then the center of the coal mining machine drum has reached or exceeded the coal wall boundary line, and the coal mining machine drum is judged to have reached the cutting point; when... If the value is greater than 0, the drum has not reached the coal wall boundary line, and it is determined that the coal mining machine drum has not reached the cutting position. When it is determined that the coal mining machine drum has reached the cutting position, the video recognition module will send the recognition judgment result to the data acquisition module.

[0014] Furthermore, in step S5, the process drive engine controls the working status of the coal mining machine through the coal mining machine controller, specifically including the following steps: S51. When the data acquisition module determines that the distance between the coal mining machine and the cut-through position is twice the width of the support through the encoder value, the process drive engine controls the coal mining machine to decelerate to 5m / min through the coal mining machine controller. S52. When the data acquisition module determines that the distance between the coal mining machine and the cut-through position is the same as the support width by using the encoder value, the process drive engine controls the coal mining machine controller to reduce the traveling speed of the coal mining machine to 3m / min and issues a voice alarm. S53, The data acquisition module receives the recognition result from the video recognition module. If the value is ≤0 and the encoder value has been reset, the process drive engine will control the coal mining machine to stop via the coal mining machine controller.

[0015] Furthermore, in step S5, an error range is set for the process drive engine, and... When the value of the encoder is ≤0, it is compared with the value of the cut-through position in the encoder, and the value of the cut-through position in the encoder is adjusted according to the comparison result.

[0016] Compared with the prior art, the advantages and positive effects of this invention are: 1. Robust recognition: Dual-light fusion + adaptive enhancement (guided filtering + DCP) is more stable in low-light and dusty scenes, and significantly improves edge / contour visibility.

[0017] 2. More accurate positioning: Dual-channel confirmation by vision (YOLOv8+RANSAC) and encoder, using normal distance and position window constraints together, reduces the probability of misjudgment and overshoot, and improves its positioning accuracy.

[0018] 3. Accumulated error clearing: The end reset magnet and reset sensor periodically eliminate the encoder's accumulated error, and the long-term accuracy can be maintained.

[0019] 4. One calibration per cut: The cutting position is automatically corrected based on the visual arrival time, suppressing the movement / drift of the working face, and the endpoint is always "aligned with the coal wall".

[0020] 5. Enhanced security: Dual-channel confirmation, graded deceleration, and voice broadcast operation effectively improve security and usability.

[0021] 6. Full-process automation: Forming a vision-sensor-encoder-control closed loop, executing according to the closed loop of "graded deceleration → shutdown → tool change → tool return", reducing manual intervention, improving the consistency of cutting through, and realizing full-process automation of triangular coal. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the system structure of the present invention; Figure 2 This is a flowchart illustrating the method framework of the present invention; Figure 3 This is a main view structural diagram illustrating the implementation operation of the present invention; Figure 4 This is a left-side view of the implementation operation of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art to all other embodiments obtained without creative effort should be included within the protection scope of the present invention.

[0025] This invention provides a multimodal feature fusion working face cutting position return control system, such as... Figure 1 As shown, it includes a dual-light fusion camera, a reset device, an AI analysis server, a data analysis server, and a process drive engine.

[0026] The dual-light fusion camera is installed at each end of the working face, facing the cut-through position, and monitors the cut-through position in real time. It always outputs images with clear texture and outline in low light and dusty / foggy environments, reducing the difficulty of subsequent video recognition of the cut-through position. At the same time, the dual-light fusion camera provides SDK and standard video streaming protocols such as RTSP and ONVIF.

[0027] The reset device includes a reset magnet and a reset sensor. The reset magnet is installed on the cable groove about two frames inward from the cut-through position at both ends of the working face, so that the coal mining machine passes the reset magnet before reaching the cut-through position. The reset sensor is installed on the coal mining machine body and connected to the coal mining machine control system. When the coal mining machine reaches the reset magnet, the reset sensor sends a reset signal to the coal mining machine control system. The coal mining machine control system resets the encoder value at this position, so that the encoder value is the same every time the coal mining machine passes the reset magnet, eliminating the cumulative error generated by the encoder.

[0028] The dust sensors are mounted on brackets corresponding to the cut-through positions at both ends to monitor the dust concentration near the cut-through positions in real time. The analysis server reads the data via Modbus TCP through the data acquisition module and then broadcasts it to the AI ​​analysis server via MQTT.

[0029] The AI ​​analysis server has built-in optimized image enhancement and video recognition algorithms. The AI ​​analysis server pulls a video stream with a resolution of 1920×1080 via RTSP, and the optimized image enhancement algorithm enhances the texture and edge contour information of the video stream image. The video recognition algorithm identifies the positional relationship of the coal wall edge of the coal mining machine drum. When the cutting position is reached, the recognition result is sent to the data analysis server to complete the cutting position identification.

[0030] The image enhancement algorithm first employs a guided filtering-based joint denoising method for preprocessing to ensure real-time performance and edge-preserving effect. A guiding image is first obtained. G(x,y) For the original image Large-scale low-pass filtering was performed using cv2.GaussianBlur() in OpenCV, with a kernel size (ksize) of 0×0 and a standard deviation (sigma) of 50, to extract the estimated illumination components. The illumination field is then subtracted from the original image, corrected, and then the average brightness is added back to generate the guide image. G(x,y) This corrects uneven lighting and enhances structural information. The calculation method is as follows: ; in For anisotropic diffusion operators, clip Ensure the value is within the valid range of 0-1. It is the global mean of the illumination component. W , H These are the image width and height, respectively. N It represents the total number of pixels.

[0031] Further denoising is achieved using guided filtering. OpenCV provides the function `cv2.ximgproc.guidedFilte()`, which takes the original image as input. Guide Image G(x, y) Window radius r Regular terms . r and The noise reduction effect of the guided filter is determined. To enhance the adaptability of the guided filter, parameter mapping is performed in conjunction with the dust sensor concentration. The concentration range of the dust concentration sensor is set to [ The dust concentration in the area during the time the coal mining machine reaches the cutting point is measured. C (mg / ), regarding dust concentration C Take the average value to get Normalization yields the dust index: , ; Obtain the window radius r and the regularization term. With dust concentration d The mapping is: ; ; Through experimental calibration of images of samples with different dust concentrations in the well, [the following was determined] , The value is [7,20]. set up[ , ]for[ Thus, we can obtain an adaptively denoised image that varies with dust concentration. .

[0032] After joint denoising, this invention further employs an improved dark channel prior (DCP) method for dehazing restoration. Traditional dark channel priors use fixed parameters for atmospheric light estimation and transmittance calculation, which can easily lead to over- or under-dehazing under different dust environments. This invention achieves adaptive optimization of the dark channel prior by introducing real-time concentration data from a dust sensor, specifically including the following steps, where the input image satisfies an atmospheric scattering model: ; in The image after denoising. For fog-free images, t(x, y) Transmittance, A This refers to the atmospheric light component.

[0033] Traditional methods directly select atmospheric light from the brightest region of the dark channel. A This invention introduces a dust concentration weight during the candidate point selection process to avoid misclassifying strong underground light sources or reflection points as atmospheric light. Specifically: ; in, , For preliminary transmission rate estimation, In order to match the dust index d The relevant regulatory factors, the greater the dust, The larger the value, the higher the transmission rate. Two key parameters: window size Regulatory factors With dust index d Dynamic adjustment. The window size range was determined through experimental calibration of sample images of different dust concentrations in the well. =11, =25, adjustment factor range =0.9, =0.98. To avoid over-recovery, this invention defines a lower bound for transmittance: ; The heavier the dust, The smaller the value, the stronger the dehazing is allowed, ultimately resulting in a haze-free image.

[0034] The video recognition algorithm described in this invention employs YOLOv8-Det, which is lightweight, fast, and suitable for real-time detection. Specifically, this involves inputting a haze-free image. Scaled proportionally to a uniform size of 640×640 and normalized to [0,1]. YOLOv8-Det model detects and outputs the bounding box of the roller target. ,in With the center coordinates, For width, To determine the height, identify the boundary points of the coal face, label them with a narrow rectangular frame (small in width, covering the coal face in height), and output multiple point sets to obtain the center coordinate set. Remove outliers based on horizontal position X, and set a confidence filtering threshold. After removing low-confidence boundary points, the filtered set is obtained. .

[0035] Boundary points are evaluated using the Random Sample Consensus (RANSAC) algorithm. By fitting the set of equations, an optimal straight line equation describing the coal wall boundary is obtained. ax + by + c =0, and output that the deviation from the straight line is less than a predetermined threshold. The points of each pixel are used as the set of interior points, which are equivalent to the effective boundary points that are consistent with the line, thus ensuring the stability and robustness of the fitting results.

[0036] Calculation of the positional relationship between the drum and the coal wall boundary at the cut-through point, including the signed normal distance from the drum center point to the coal wall boundary line: ; when A value ≤ 0 indicates that the center of the drum has reached or exceeded the coal wall boundary line, and is considered to have reached the cutting point visually. When dis>0, the drum has not reached the coal face. When it is determined that the collecting drum has reached the cutting position, a notification is sent to the data analysis server via MQTT.

[0037] The data analysis server mainly consists of a process-driven engine for the data acquisition module.

[0038] The data acquisition module obtains encoder values, dust sensor data, and video recognition results of the cutting position from the coal mining machine via MQTT and ModbusTcp protocols.

[0039] The process drive engine dynamically adjusts the coal mining machine's attitude based on planned data, such as speed, rolling height, and direction of travel, and performs graded deceleration and voice announcements. It decelerates the machine in stages as it approaches the cut-through position, and further decelerates it as it approaches the set cut-through position. There are 2 more When the distance (support width) is reached, the speed of the decelerated coal mining machine is 5m / min. The distance between the coal mining machine and the cutting point is then broadcast in real-time via the working face intercom. For example, if the distance is 3m, the underground and surface workers are alerted that the coal mining machine is approaching the cutting point and should pay attention to production safety. The distance between the coal mining machine and the set cutting point is then announced. besides At this time, reduce the speed of the coal mining machine to 3m / min to ensure that the coal mining machine reaches the cutting position at a low speed, and prevent the coal mining machine from being too fast and unable to stop in time or exceeding the cutting position, so that the staff can intervene in time.

[0040] Dual-channel confirmation, recording video analysis identifies when the coal mining machine reaches the cutting point. The real-time encoder value for the coal mining machine's movement is P. This is achieved upon visual arrival. ≤ 0, the position of the coal mining machine is also within the range. Inside( ,in , (Where the width of a single bracket is in meters), and both conditions must be met simultaneously to trigger the return tool control.

[0041] The return cutter control is initiated by the process drive engine sending commands to the coal mining machine control system, causing the coal mining machine to decelerate to 0 m / min, then starting the cutter change and adjusting the left and right drums of the coal mining machine to the planned position, and finally starting the return cutter traction to complete the automatic return cutter control of the triangular coal.

[0042] Each cutter is calibrated; any deviation exceeding this limit is acceptable. This indicates that the scraper conveyor has shifted, and the cutting position within the process section range set in the process drive engine will be affected. Modified to Achieve one-cut-one-update at the process endpoint.

[0043] Repeating the above steps enables fully automated control of the cut-through position return process on the working face. It also automatically updates the process segment range to the endpoints of the cut-through position, ensuring the accuracy of the planned process segment range. It is also valuable for judging the upward and downward movement of scraper conveyors.

[0044] This invention also discloses a method for controlling the cut-through position and return cut of the working face through multimodal feature fusion, which is described below in conjunction with... Figure 1 , Figure 2 , Figure 3 , Figure 4 The specific operation process of this control method is described; Step 1: System Setup Process; like Figure 3 As shown, the coal mining machine moves to the left to the left cutting position. Based on the relative positions of the equipment, a reset device is installed. The reset sensor 6 is installed on the coal mining machine body 4, and the reset magnets 7 and 9 are installed at the position of the coal mining machine during the cutting phase. Inward 2 The cable trough at the location ensures reliable triggering when passing the reset magnet; the scraper conveyor 1 is located at the bottom, the center position 2 of the coal mining machine drum is located at the left end of the coal mining machine body, and is supported and controlled by the coal mining machine rocker arm 3. The coal wall 8 is located behind the coal mining machine, and the left end cutting position 5 and the right end cutting position 10 are located on the left and right sides of the coal wall 8, respectively. like Figure 4 As shown, the dust sensor 11 is mounted on the top beam of the bracket 13 at both the beginning and end of the machine, avoiding direct exposure to the wind; the dual-light fusion camera 12 is similarly mounted on the top beam of the bracket 13 at both the beginning and end of the machine, directly opposite the cut-through position. Outputs a 1920×1080 RTSP / ONVIF video stream.

[0045] When reset sensor 6 on the coal mining machine body is triggered by reset magnets 7 and 9, it resets the coal mining machine encoder. The encoder value is reported to the data analysis server via Modbus TCP through the coal mining machine control system. Dust sensor 11 also reports dust concentration to the data analysis server via Modbus TCP. The AI ​​analysis server pulls the video stream from dual-light fusion camera 12 via RTSP for analysis. The data analysis server and AI analysis server interact via MQTT. The AI ​​analysis server reports the recognition results to the data analysis server. The data analysis server controls the coal mining machine's deceleration, cutter changing, and cutter retraction via the process drive engine. The system interaction flow is as follows: Figure 1 As shown.

[0046] Step 2, Image Enhancement and Recognition Process, as follows: Figure 2 As shown; 1. Dust index calculation: Take the average dust concentration when near the cut-through position. ,calculate , ; 2. Joint denoising (guided filtering): Adaptively mapping the window radius with d. [7, 20], regular direction [ ], to obtain the denoised image .

[0047] 3. Improved DCP defogging: (The last part is incomplete and likely refers to a separate feature or feature.) d Adaptive parameters [11, 25]、 [0.9, 0.98]、 [0.12, 0.35], output the restored, haze-free image. .

[0048] 4. Target detection: Scaling to 640×640, using YOLOv8-Det to detect bounding boxes. The set of boundary points with the coal face.

[0049] 5. Boundary fitting: Fit the line ax + by + c = 0 using RANSAC (interior point threshold 10px).

[0050] 6. Visual arrival judgment: Calculate the normal distance , ≤0 visual acuity achieved.

[0051] Step 3, Automatic Return Tool Control Process, such as Figure 2 As shown; 1. Setting parameters: Width of a single support frame and cut-through position setting value The process-driven engine, configured on-site, sends data to the data analysis service and sets the error range. .

[0052] 2. Reset encoder: The coal mining machine passes through Figure 3 When magnet 7 or 9 is reset, sensor 6 is triggered, and the encoder value of the coal mining machine control system is reset.

[0053] 3. Staged deceleration and voice broadcast: The process drive engine detects that the coal mining machine has reached its designated position. = 2 At that time, the coal mining machine was controlled to decelerate to 5 m / min, and the coal mining machine was detected to have reached the target speed. = The speed of the coal mining machine is controlled to decrease to 3m / min, and the speed and distance from the cutting position are broadcast through the voice telephone on the working face to remind personnel to pay attention to production safety.

[0054] 4. Dual-channel confirmation: visual arrival satisfies requirements. ≤0, the coal mining machine position has reached the required level. < If both conditions are met simultaneously, the coal mining machine will decelerate to 0 m / min.

[0055] 5. Development of return cutter: After the dual-channel confirmation is met, the process drive engine controls the coal mining machine to decelerate to 0m / min and stop, completing the process section switch. The left and right drums are adjusted to the planned height according to the new process section, and the return cutter traction is started to complete the automatic return cutter.

[0056] 6. Calibration after each cut: Record encoder values ​​upon visual arrival. ,contrast ,if ≤ Then keep If unchanged, ,Will Updated to .

[0057] 7. Cyclic Steps: After completing the reverse cut, proceed to the next cut and repeat the above steps.

[0058] The present invention has the following beneficial effects: 1. Robust recognition: Dual-light fusion + adaptive enhancement (guided filtering + DCP) is more stable in low-light and dusty scenes, and significantly improves edge / contour visibility.

[0059] 2. More accurate positioning: Dual-channel confirmation by vision (YOLOv8+RANSAC) and encoder, using normal distance and position window constraints together, reduces the probability of misjudgment and overshoot, and improves its positioning accuracy.

[0060] 3. Accumulated error clearing: The end reset magnet and reset sensor periodically eliminate the encoder's accumulated error, and the long-term accuracy can be maintained.

[0061] 4. One calibration per cut: The cutting position is automatically corrected based on the visual arrival time, suppressing the movement / drift of the working face, and the endpoint is always "aligned with the coal wall".

[0062] 5. Enhanced security: Dual-channel confirmation, graded deceleration, and voice broadcast operation effectively improve security and usability.

[0063] 6. Full-process automation: Forming a vision-sensor-encoder-control closed loop, executing according to the closed loop of "graded deceleration → shutdown → tool change → tool return", reducing manual intervention, improving the consistency of cutting through, and realizing full-process automation of triangular coal.

Claims

1. A multimodal feature fusion working face cutting position return control system, characterized in that: The system includes a dual-light fusion camera, a coal mining machine control mechanism, an AI analysis server, a data analysis server, and a dust sensor. The dual-light fusion camera is located at the end of the working face, fixed on a hydraulic support and facing the cutting position of the coal mining machine. The signal output terminal of the dual-light fusion camera is connected to the signal input terminal of the AI ​​analysis server. The signal terminals of the AI ​​analysis server and the coal mining machine control mechanism are bidirectionally connected to the signal terminal of the data analysis server. The dust sensor is located at the end of the working face and fixed on a hydraulic support. The signal output terminal of the dust sensor is connected to the signal input terminal of the data analysis server.

2. The working face cutting position return control system as described in claim 1, characterized in that: The AI ​​analysis server includes an image enhancement module for enhancing the original images captured by the dual-light fusion camera, and a video recognition module for determining the positional relationship between the coal mining machine drum and the cutting point based on the enhanced image. The signal input terminal of the image enhancement module is communicatively connected to the signal output terminal of the dual-light fusion camera, and the signal output terminal of the image enhancement module is connected to the signal input terminal of the video recognition module. The signal output terminal of the video recognition module is connected to the signal input terminal of the data analysis server via MQTT broadcast, and the signal output terminal of the data analysis server is connected to the signal input terminal of the image enhancement module.

3. The working face cutting position return control system based on multimodal feature fusion as described in claim 2, characterized in that: The data analysis server includes a data acquisition module for collecting and analyzing various data, and a process drive engine for driving the coal mining machine. The signal input terminal of the data acquisition module is connected to the signal output terminal of the frequency identification module, the signal output terminal of the dust sensor, and the signal output terminal of the coal mining machine control mechanism. The signal output terminal of the data acquisition module is connected to the signal input terminal of the image enhancement module and the signal input terminal of the process drive engine. The signal output terminal of the process drive engine is connected to the signal input terminal of the coal mining machine control mechanism.

4. The working face cutting position return control system based on multimodal feature fusion as described in claim 3, characterized in that: The coal mining machine control mechanism includes a coal mining machine controller for controlling the movement of the coal mining machine, an encoder for determining the displacement distance of the coal mining machine, a reset magnet and a reset sensor for performing encoder reset operations; there are two reset magnets arranged in a cable groove at the end of the working face, the reset sensor is fixed on the coal mining machine and corresponds to the position of the reset magnet, the signal output terminal of the reset sensor is connected to the signal input terminal of the coal mining machine controller, the signal output terminal of the coal mining machine controller is connected to the signal input terminal of the encoder, the signal output terminal of the encoder is connected to the signal input terminal of the data acquisition module, and the signal output terminal of the process drive engine is connected to the signal input terminal of the coal mining machine controller.

5. A method for controlling the cut-through position and return cut of a working face using multimodal feature fusion, implemented by the multimodal feature fusion working face cut-through position and return cut control system described in claim 4, characterized in that: Includes the following steps: S1. When the coal mining machine moves to a position where the reset sensor corresponds to the reset magnet, the reset sensor will generate a signal and send the signal to the coal mining machine controller. The coal mining machine controller will reset the encoder, and at the same time, the encoder will transmit the encoder value to the data acquisition module. S2. The dust sensor transmits the dust concentration value to the data acquisition module, which then transmits it to the AI ​​analysis server. The S3 dual-light fusion camera transmits the captured raw images to the AI ​​analysis server, which performs image processing and identifies the relationship between the coal mining machine drum and the cutting position based on the dust concentration value. S4. The AI ​​analysis server transmits the identification results of the relationship between the coal mining machine drum and the cutting position to the data acquisition module. S5. The data acquisition module runs the process drive engine based on the encoder values ​​and the identification results of the relationship between the coal mining machine drum and the cutting position. The process drive engine sends instructions to the coal mining machine controller and controls the working status of the coal mining machine through the coal mining machine controller. S6. After the coal mining machine stops, adjust the height of the coal mining machine drum and perform the reverse cutter operation.

6. The method for controlling the cut-through position and return cut of the working face by multimodal feature fusion as described in claim 5, characterized in that: Step S3 specifically includes the following steps: S31. The dual-light fusion camera transmits the captured images to the image enhancement module. The image enhancement module performs image processing operations on the images captured by the dual-light fusion camera and transmits the processed images to the video recognition module. S32. The video recognition module judges and identifies the positional relationship between the coal mining machine drum and the cutting penetration position, and sends the recognition result to the data acquisition module.

7. The multimodal feature fusion working face cutting position return control method as described in claim 6, characterized in that: In step S31, the image enhancement module performs image processing operations on the original image captured by the dual-light fusion camera, specifically including the following steps: S311, For the original image A large-scale low-pass filter is performed to obtain the guiding image G(x,y); its calculation formula is as follows: ； in, For anisotropic diffusion operators, It is the global mean of the illumination component, W and H are the width and height of the original image, respectively, and N is the total number of pixels; S312. Normalize the guide image based on the dust concentration value to obtain the dust index, the calculation formula of which is: ， ； S313. Obtain the window radius r and regularization term based on the dust index. The calculation formula is: ； ； S314, Setting [ , ] is [7,20], set [ , ]for[ ], thus obtaining a denoised image that varies with dust concentration. ; S315. Denoising the image The formula for calculating the atmospheric scattering model is as follows: ； in, The image is a fog-free image, where t(x, y) is the transmittance and A is the atmospheric light component. ； ； in, For transmission rate, It is a moderating factor related to the dust index d; S316. After comprehensive calculation, a fog-free image is obtained, and the calculation formula is as follows: 。 8. The method for controlling the cut-through position and return cut of the working face by multimodal feature fusion as described in claim 7, characterized in that: Step S32 specifically includes the following steps: S321. Input the haze-free image Scale proportionally to a uniform size of 640×640 and normalize to [0,1]; S322. Detect and output the target bounding box of the coal mining machine drum using the YOLOv8-Det model. ,in, The coordinates of the center of the target bounding box of the coal mining machine drum are: The width of the target bounding box of the coal mining machine drum. The height of the target bounding box of the coal mining machine drum; S323. Identify coal wall boundary points, output multiple sets of coal wall boundary points, remove isolated points according to horizontal position X, and set a confidence filtering threshold. After removing low-confidence coal wall boundary points, the filtered set of boundary points is obtained. ; S324, Set of Boundary Points By fitting the equation, a straight line describing the coal wall boundary is obtained. ax + by + c =0; S325. Calculate the normal distance from the center point of the coal mining machine drum to the coal wall boundary line. The calculation formula is as follows: ； when If the value is ≤0, then the center of the coal mining machine drum has reached or exceeded the coal wall boundary line, and the coal mining machine drum is judged to have reached the cutting point; when... If the value is > 0, the drum has not reached the coal wall boundary line, and it is determined that the coal mining machine drum has not reached the cutting position; when it is determined that the coal mining machine drum has reached the cutting position, the video recognition module will send the recognition judgment result to the data acquisition module.

9. The method for controlling the cut-through position and return cut of the working face by multimodal feature fusion as described in claim 8, characterized in that: In step S5, the process drive engine controls the working status of the coal mining machine through the coal mining machine controller, specifically including the following steps: S51. When the data acquisition module determines that the distance between the coal mining machine and the cut-through position is twice the width of the support through the encoder value, the process drive engine controls the coal mining machine to decelerate to 5m / min through the coal mining machine controller. S52. When the data acquisition module determines that the distance between the coal mining machine and the cut-through position is the same as the support width by using the encoder value, the process drive engine controls the coal mining machine controller to reduce the traveling speed of the coal mining machine to 3m / min and issues a voice alarm. S53, The data acquisition module receives the recognition result from the video recognition module. If the value is ≤0 and the encoder value has been reset, the process drive engine will control the coal mining machine to stop via the coal mining machine controller.

10. The multimodal feature fusion working face cutting position return control method as described in claim 9, characterized in that: In step S5, an error range is set for the process drive engine, and... When the value of the encoder is ≤0, it is compared with the value of the cut-through position in the encoder, and the value of the cut-through position in the encoder is adjusted according to the comparison result.

Images