Support pushing state detection method and system based on attention mechanism

Abstract

The invention relates to the technical field of coal mine underground video AI intelligent analysis, in particular to a support pushing state detection method and system based on an attention mechanism, and the method comprises the steps: collecting the working video data of a hydraulic support in real time through a high-definition intrinsic safety type holder camera, and carrying out the processing through a video analysis reasoning server, a target detection algorithm based on an attention mechanism is utilized, the hydraulic support pushing rod can be rapidly and accurately recognized, the functions of calculating the number of adjacent supports and judging the pushing state of each support are further included, and alarm information can be sent out to inform workers when abnormity is detected. The invention aims to detect and early warn the abnormality of the pushing and sliding state of the hydraulic support in real time through AI video analysis, and the safety and efficiency of coal mine production are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of AI intelligent analysis of underground coal mine videos, and in particular to a method and system for detecting the movement state of a support based on an attention mechanism. Background Art

[0002] With the continuous advancement of coal mining technology and the continuous improvement of safety and environmental protection standards, the importance of hydraulic support push-slide status detection has become increasingly prominent. Traditional manual inspection and fixed sensor monitoring methods are difficult to meet the requirements of modern mines for refined and intelligent operations, especially in the monitoring of the push-slide action, fault warning and automatic control of hydraulic supports. For example, uncoordinated push-slide of hydraulic supports may cause support position deviation and affect the normal operation of the coal mining machine; and uneven push-slide action may cause jamming or damage to the scraper conveyor, affecting the overall production efficiency and equipment life. Therefore, the hydraulic support push-slide status detection system based on advanced visual technology and intelligent algorithms can largely make up for the shortcomings of traditional means, accurately grasp the working status of the hydraulic support in real time, and provide strong guarantees for safe production.

[0003] At present, the progress in artificial intelligence, especially in the field of deep learning, has greatly promoted the development of image recognition and processing technology, making it perform better in industrial applications. The target detection model can collect data on low-power, low-cost front-end devices and perform high-performance real-time reasoning processing through the back-end server, which creates conditions for the deployment and implementation of the hydraulic support push-slide state detection system in the coal mine field environment. In addition, the Internet of Things technology enables a large number of front-end sensing devices (such as high-definition cameras) to be seamlessly connected to the back-end data processing platform, realizing high-speed transmission and instant analysis of hydraulic support image data.

[0004] Based on the above technology, the hydraulic support push-slide state detection system can extract the image features of the hydraulic support in real time, make accurate state judgments, and further convert them into control instructions for the hydraulic support, realizing intelligent closed-loop management of the entire process. Therefore, this system obtains the working image data of the hydraulic support through a high-definition camera, uses the target detection model to perform real-time analysis on the back-end server, and judges the state of the push rod of the hydraulic support by analyzing the real-time image. When the hydraulic support is in an abnormal state, the system will issue an abnormal alarm, and the alarm result will be displayed in the form of voice reminders or sound and light alarms. Summary of the invention

[0005] In response to the above-mentioned defects and problems, the present invention provides a method and system for detecting the sliding state of a support based on an attention mechanism, which uses AI video analysis to detect and warn of abnormalities in the sliding state of a hydraulic support in real time, thereby improving the safety and efficiency of coal mine production.

[0006] The solution adopted by the present invention to solve the technical problem is: a method for detecting the displacement state of a support based on an attention mechanism, which is implemented by the following steps:

[0007] Step 1: The intrinsically safe PTZ camera collects video data of the hydraulic support working area and transmits the data to the video analysis and inference server deployed in the ground computer room of the coal mine through the underground intrinsically safe switch and the ground switch;

[0008] Step 2: After the video analysis and reasoning server obtains the video data, it uses the stand push rod target detection algorithm based on the attention mechanism to infer the video data and obtain the target detection result of the stand push rod;

[0009] Step 3: Calculate the frame numbers of adjacent brackets based on the frame number information of the PTZ camera and the target detection result obtained in step 2;

[0010] Step 4: According to the adjacent bracket number information obtained in step 3 and the target detection result obtained by inference, the displacement state of each bracket is determined by a set displacement state calculation method;

[0011] Step 5: Based on the displacement status of each bracket calculated in step 4, the bracket displacement status information is published through the video analysis and reasoning server, and the status information is displayed on the PC at the same time.

[0012] Furthermore, in the step one, the frame number of the gimbal camera and the coordinate values ​​of the pan P, tilt T, and zoom Z of the gimbal camera are obtained; the gimbal coordinate value of the camera is obtained every 30 seconds. If the gimbal coordinates are inconsistent with the preset point information, the gimbal coordinates are adjusted to be consistent with the preset points. The setting of the preset points should follow the principle of the same direction, and all gimbal lenses should be uniformly facing the nose or tail direction.

[0013] Furthermore, in the step 2, the attention mechanism-based support push rod target detection algorithm uses Yolov8 as the basic framework, uses the C2f_DC module to replace the original C2f module, uses the mixed depth separable pyramid pooling to replace the original spatial pyramid pooling, and adds the attention mechanism module ShuffleAttention at the joint between the Backbone end and the Neck end.

[0014] Furthermore, in step 3, the method for calculating the frame number of each bracket is implemented by the following steps:

[0015] S31, obtaining the bracket number of the PTZ camera through step 1;

[0016] S32, calculating the vertical coordinate of the center point of each detection frame;

[0017] S33, calculating the frame number of each bracket;

[0018] The calculation method of step S32 is as follows:

[0019] Among them, y l is the vertical coordinate of the upper left corner of the detection box, y r is the vertical coordinate of the lower right corner of the detection box;

[0020] The calculation method of step S33 is as follows:

[0021] ① First, calculate the difference Δy between the ordinate of the center point of each detection frame and the ordinate of the center point of the detection frame of the PTZ camera bracket. i :

[0022] Δy i =y i -y 0 ,

[0023] Where i belongs to [0,I], I is the total number of target boxes detected in the image, and y 0 is the vertical coordinate of the center point of the bracket detection frame where the gimbal is located, y i The vertical coordinate of the center point of the other bracket detection frame identified in the picture;

[0024] ②Then for Δy i Sort from small to large as follows:

[0025] Δy i '=sort(Δy i ),

[0026] Define the bracket number of the gimbal as N, and further define the function f(Δy i ')as follows:

[0027]

[0028] N is the bracket number of the gimbal, and N+k and Nk are the bracket numbers of other brackets identified in the picture.

[0029] Furthermore, in step 4, the method for determining the displacement state of each bracket is implemented by the following steps:

[0030] S41, initialize the loop, set the number of frames n to 0, and set the area of ​​the detection frame of the previous frame to A prev Equal to the current frame detection box area A curr :

[0031] n=0,

[0032] A prev =A curr ;

[0033] S42: In each cycle, calculate the detection frame area A of the current frame curr , and record the difference ΔA between the area of ​​the previous frame detection box and the area of ​​the current frame detection box:

[0034] ΔA=A prev -A curr ;

[0035] Then update the detection box area of ​​the previous frame to the detection box area of ​​the current frame:

[0036] A prev =A curr ;

[0037] S43: Increase the number of frames n and check whether it reaches 10 frames:

[0038] n=n+1,

[0039] If n<10, return to S42; if n≥10, proceed to the next step of judgment;

[0040] S44: Count the number of frames N where the area difference ΔA is a positive value + And negative frame number N-:

[0041]

[0042] Here [·] represents the indicator function;

[0043] S45: Determine the state of the stent according to the statistical results:

[0044] If N + >7, it is determined to be in the pull-out state; otherwise, if N->7, it is determined to be in the push-slide state; otherwise, it is determined that the bracket is not in motion.

[0045] A support displacement state detection system based on an attention mechanism. The hardware required to realize the system functions includes a mining intrinsically safe pan-tilt camera for front-end image data collection, a video analysis and inference server, a mining switch and a ground switch for image transmission, and a PC host and a corresponding display for ground display.

[0046] Furthermore, the video analysis and inference server includes a video inference card required to support the inference of the support displacement state detection method based on the attention mechanism, and the model of the video inference card is Cambrian MLU370-S4.

[0047] Beneficial effects of the present invention: (1) Real-time and high-precision detection: The present invention uses an intrinsically safe pan-tilt camera to collect working video data of the hydraulic support in real time, and processes it through a video analysis and inference server. The target detection algorithm based on the attention mechanism can quickly and accurately identify the state of the push rod, thereby providing real-time feedback on the movement of the hydraulic support and ensuring the accuracy of the detection results;

[0048] (2) Intelligence and automation: By introducing the attention mechanism and the optimized convolution module, the present invention can effectively improve the detection capability and computational efficiency of the model while maintaining a low model complexity. This enables the system to not only automatically identify the state of the bracket, but also automatically adjust the position of the gimbal camera according to the detection results to ensure that the image data at the best viewing angle can always be captured;

[0049] (3) Real-time monitoring and intelligent early warning: The system has real-time image analysis capabilities, can determine the sliding status of the hydraulic support based on the detection results, and immediately issue an alarm when an abnormality is detected. Through voice reminders or sound and light alarms, the staff is notified in time to take corresponding measures to prevent production safety accidents caused by abnormal sliding status of the hydraulic support;

[0050] (4) Integration and visualization: The system integrates multiple functional modules such as data acquisition, image analysis, status judgment and information release, forming a complete closed-loop solution. By displaying real-time bracket movement status information on the PC, it is convenient for staff to understand the working status of the equipment at any time and respond in time.

[0051] (5) Flexible adaptability and reliability: This system can adapt to the needs of hydraulic support movement status monitoring in different working environments. By dynamically adjusting the position and angle of the pan-tilt camera, the quality of image acquisition can be ensured. At the same time, the system can accurately judge a variety of movement states (such as pulling the frame, pushing it, and no movement), effectively avoiding equipment misoperation and other potential risks, and improving the reliability and adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the detection method of the present invention;

[0053] Figure 2 The Yolov8 network framework diagram of the present invention;

[0054] Figure 3 It is a block diagram of the target detection algorithm of the support push rod of the present invention;

[0055] Figure 4 It is a structural diagram of a dynamic convolutional network of the present invention;

[0056] Figure 5It is the structural diagram of ShuffleAttention of the present invention;

[0057] Figure 6 A diagram of the hybrid depth-separable pyramid pooling structure of the present invention;

[0058] Figure 7 This is a flow chart of the method for calculating the displacement state of each bracket according to the present invention. DETAILED DESCRIPTION

[0059] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0060] See also Figure 1-7 The present invention provides a technical solution of a method and system for detecting the displacement state of a support based on an attention mechanism:

[0061] Embodiment 1: This embodiment provides a technical solution of a method for detecting the displacement state of a support based on an attention mechanism, which is implemented by the following steps:

[0062] Step 1: The intrinsically safe pan-tilt camera collects video data of the hydraulic support working area and transmits the data to the video analysis and inference server deployed in the ground computer room of the coal mine through the underground intrinsically safe switch and the ground switch.

[0063] In the above steps, obtain the frame number of the PTZ camera and the coordinate values ​​of P (translation), T (tilt), and Z (zoom) of the PTZ camera; obtain the PTZ coordinate value of the camera every 30 seconds. If the PTZ coordinates are inconsistent with the preset point information, adjust the PTZ coordinates to be consistent with the preset points. The setting of the preset points should follow the same direction principle, and all PTZ lenses should be uniformly facing the nose or tail of the aircraft. The specific method is:

[0064] S11, get the current position: get the current position coordinate value of the PTZ camera every 30 seconds.

[0065] (P t ,T t ,Z t )←GetCameraPosition(j);

[0066] S12, compare the current position with the preset point: check whether the current position is consistent with the preset point,

[0067] if(P t ,T t ,Z t )≠(P preset ,T preset ,Z preset );

[0068] S13, adjust the PTZ coordinates: If they are inconsistent, adjust the PTZ coordinates to be consistent with the preset points.

[0069] AdjustCamera(j,P preset ,T preset ,Z preset );

[0070] Where j represents the frame number of the PTZ camera. t ,T t ,Z t ) represent the translation, tilt and zoom coordinate values ​​of the PTZ camera at time t, respectively. preset ,T preset ,Z preset ) represents the position coordinates of the preset point. The GetCameraPosition function obtains the current position coordinate value from the gimbal camera. The AdjustCamera function is responsible for adjusting the position of the gimbal to make it the same as the preset point.

[0071] Step 2: After the video analysis and reasoning server obtains the video data, it uses the bracket push rod target detection algorithm based on the attention mechanism to infer the video data and obtain the target detection result of the bracket push rod.

[0072] In step 2, the attention mechanism-based support push rod target detection algorithm uses Yolov8 as the basic framework, uses the C2f_DC module to replace the original C2f module, uses the hybrid depth-separable pyramid pooling to replace the original spatial pyramid pooling, and adds the attention mechanism module ShuffleAttention at the junction between the Backbone end and the Neck end.

[0073] The C2f_DC module is constructed using the dynamic convolution method. This new design can increase the detection capability of the model without increasing the number of network convolution layers or channels, and can also reduce the model complexity. The specific process of the dynamic convolution method is to first give the input feature X and a set of convolution kernels W. 1 ,W 2 ,…,W i , each core corresponds to an expert, and the contribution of each expert is determined by a dynamic coefficient α i Control, these coefficients are generated dynamically for each sample as shown in the following formula:

[0074]

[0075] Where Y is the output, which is the weighted sum of all dynamically selected convolution kernel operations, * represents the convolution operation, α i It is dynamically calculated through a small network, the input of which is the feature after global average pooling.

[0076] Hybrid Depth Separable Pyramid Pooling can improve computational efficiency and model lightweight while enhancing the model's detection performance and multi-scale feature capture capabilities. The attention mechanism module Shuffle Attention can improve YOLOv8's feature selection capabilities and computational efficiency in the feature fusion stage, thereby enhancing the overall performance and generalization capabilities of the model.

[0077] Step 3: Calculate the frame numbers of adjacent brackets based on the frame number information of the PTZ camera and the target detection result obtained in step 2.

[0078] The method for calculating the frame number of each bracket is implemented by the following steps:

[0079] S31, obtaining the bracket number of the PTZ camera through step 1;

[0080] S32, calculating the vertical coordinate of the center point of each detection frame;

[0081] S33, calculating the frame number of each bracket;

[0082] The calculation method of step S32 is as follows:

[0083] Among them, y l is the vertical coordinate of the upper left corner of the detection box, y r is the vertical coordinate of the lower right corner of the detection box;

[0084] The calculation method of step S33 is as follows:

[0085] ① First, calculate the difference Δy between the ordinate of the center point of each detection frame and the ordinate of the center point of the detection frame of the PTZ camera bracket. i :

[0086] Δy i =y i -y 0 ,

[0087] Where i belongs to [0,I], I is the total number of target boxes detected in the image, and y 0 is the vertical coordinate of the center point of the bracket detection frame where the gimbal is located, y i The vertical coordinate of the center point of the other bracket detection frame identified in the picture;

[0088] ②Then for Δy i Sort from small to large as follows:

[0089] Δy i '=sort(Δy i ),

[0090] Define the bracket number of the gimbal as N, and further define the function f(Δy i ')as follows:

[0091]

[0092] N is the bracket number of the gimbal, and N+k and Nk are the bracket numbers of other brackets identified in the picture.

[0093] Step 4: According to the adjacent bracket frame number information obtained in step 3 and the target detection result obtained by inference, the displacement state of each bracket is determined by the set displacement state calculation method.

[0094] The method for determining the displacement status of each bracket is implemented by the following steps:

[0095] S41, initialize the loop, set the number of frames n to 0, and set the area of ​​the detection frame of the previous frame to A prev Equal to the current frame detection box area A curr :

[0096] n=0,

[0097] A prev =A curr ;

[0098] S42: In each cycle, calculate the detection frame area A of the current frame curr , and record the difference ΔA between the area of ​​the previous frame detection box and the area of ​​the current frame detection box:

[0099] ΔA=A prev -A curr ;

[0100] Then update the detection box area of ​​the previous frame to the detection box area of ​​the current frame:

[0101] A prev =A curr ;

[0102] S43: Increase the number of frames n and check whether it reaches 10 frames:

[0103] n=n+1,

[0104] If n<10, return to S42; if n≥10, proceed to the next step of judgment;

[0105] S44: Count the number of frames N where the area difference ΔA is a positive value + And negative frame number N-:

[0106]

[0107] Here [·] represents the indicator function, that is, the Boolean value is converted to a numeric value, True is converted to 1, and False is converted to 0;

[0108] S45: Determine the state of the bracket according to the statistical results: If N + >7, it is determined to be in the pull-out state; otherwise, if N->7, it is determined to be in the push-slide state; otherwise, it is determined that the bracket is not in motion.

[0109] Step 5: Based on the displacement status of each bracket calculated in step 4, the bracket displacement status information is published through the video analysis and reasoning server, and the status information is displayed on the PC at the same time.

[0110] Embodiment 2: This embodiment provides a technical solution of a bracket movement state detection system based on an attention mechanism.

[0111] The hardware required to realize the system functions should include but not be limited to mining intrinsically safe pan-tilt cameras installed on hydraulic supports for front-end image data collection, and video analysis and inference servers deployed in the ground computer room of the coal mine. In addition to the above equipment, it should also include other hardware necessary to realize the system functions, including but not limited to mining switches and ground switches used for image transmission, and PC hosts and corresponding displays used for ground display.

[0112] The video analysis inference server should include a video inference card that can support the inference of the bracket displacement state detection method based on the attention mechanism; the model of the video inference card is Cambrian MLU370-S4.

[0113] The above description is only a preferred embodiment of the present invention and does not limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting the displacement state of a stent based on an attention mechanism, characterized in that: This is accomplished by following these steps: Step 1: The intrinsically safe PTZ camera collects video data of the hydraulic support working area and transmits the data to the video analysis and inference server deployed in the ground computer room of the coal mine through the underground intrinsically safe switch and the ground switch; Step 2: After the video analysis and reasoning server obtains the video data, it uses the stand push rod target detection algorithm based on the attention mechanism to infer the video data and obtain the target detection result of the stand push rod; Step 3: Calculate the frame numbers of adjacent brackets based on the frame number information of the PTZ camera and the target detection result obtained in step 2; Step 4: According to the adjacent bracket number information obtained in step 3 and the target detection result obtained by inference, the displacement state of each bracket is determined by a set displacement state calculation method; Step 5: Based on the displacement status of each bracket calculated in step 4, the bracket displacement status information is published through the video analysis and reasoning server, and the status information is displayed on the PC at the same time.

2. According to the method for detecting the displacement state of a support based on an attention mechanism according to claim 1, it is characterized in that: In the step 1, the frame number of the pan / tilt camera and the coordinate values ​​of the pan / tilt P, tilt T, and zoom Z of the pan / tilt camera are obtained; the pan / tilt coordinate value of the camera is obtained every 30 seconds. If the pan / tilt coordinate and the preset point information are inconsistent, the pan / tilt coordinate is adjusted to be consistent with the preset point. The setting of the preset point should follow the same direction principle, and all pan / tilt lenses should be uniformly facing the nose or tail direction.

3. According to the method for detecting the displacement state of a support based on an attention mechanism according to claim 1, it is characterized in that: In the step 2, the attention mechanism-based support push rod target detection algorithm uses Yolov8 as the basic framework, uses the C2f_DC module to replace the original C2f module, uses the mixed depth separable pyramid pooling to replace the original spatial pyramid pooling, and adds the attention mechanism module ShuffleAttention at the joint between the Backbone end and the Neck end.

4. A method and system for detecting the displacement state of a support based on an attention mechanism according to claim 1 or 2, characterized in that: In step 3, the method for calculating the frame number of each bracket is implemented by the following steps: S31, obtaining the bracket number of the PTZ camera through step 1; S32, calculating the vertical coordinate of the center point of each detection frame; S33, calculating the frame number of each bracket; The calculation method of step S32 is as follows: Among them, y l is the vertical coordinate of the upper left corner of the detection box, y r is the vertical coordinate of the lower right corner of the detection box; The calculation method of step S33 is as follows: ① First, calculate the difference Δy between the ordinate of the center point of each detection frame and the ordinate of the center point of the detection frame of the PTZ camera bracket. i : Δy i =y i -y0, Where i belongs to [0,I], I is the total number of target frames detected in the image, y0 is the ordinate of the center point of the detection frame of the bracket where the gimbal is located, and y i The vertical coordinate of the center point of the other bracket detection frame identified in the picture; ②Then for Δy i Sort from small to large as follows: Δy i '=sort(Δy i ), Define the bracket number of the gimbal as N, and further define the function f(Δy i ')as follows: N is the bracket number of the gimbal, and N+k and Nk are the bracket numbers of other brackets identified in the picture.

5. According to the method for detecting the displacement state of a support based on an attention mechanism according to claim 1, it is characterized in that: In step 4, the method for determining the displacement state of each bracket is implemented by the following steps: S41, initialize the loop, set the number of frames n to 0, and set the area of ​​the detection frame of the previous frame to A prev Equal to the current frame detection box area A curr : n＝0， A prev =A curr ; S42: In each cycle, calculate the detection frame area A of the current frame curr , and record the difference ΔA between the area of ​​the previous frame detection box and the area of ​​the current frame detection box: ΔA=A prev -IN curr ; Then update the detection box area of ​​the previous frame to the detection box area of ​​the current frame: A prev =A curr ; S43: Increase the number of frames n and check whether it reaches 10 frames: n=n+1, If n<10, return to S42; if n≥10, proceed to the next step of judgment; S44: Count the number of frames N where the area difference ΔA is a positive value + and negative frame number N - : Here [·] represents the indicator function; S45: Determine the state of the stent according to the statistical results: If N + >7, it is considered to be in the state of pulling; otherwise, if N - >7, it is judged as the push-slide state; otherwise, it is judged that the bracket is not moving.

6. A stent displacement state detection system based on attention mechanism, characterized in that: The hardware required to realize the system functions includes mining intrinsically safe pan-tilt camera for front-end image data collection, video analysis and inference server, mining switch and ground switch for image transmission, PC host and corresponding display for ground display.

7. A method and system for detecting stent displacement state based on attention mechanism according to claim 6, characterized in that: The video analysis and reasoning server includes a video reasoning card required to support the reasoning of the support displacement state detection method based on the attention mechanism, and the model of the video reasoning card is Cambrian MLU370-S4.

Images