Lifting rope defect positioning and real-time capturing method and system based on image recognition

Abstract

The invention discloses a lifting rope defect positioning and real-time capturing method and system based on image recognition, and aims to solve the problems that traditional detection is low in efficiency, large in error and difficult to adapt to dynamic complex scenes. A multi-scale fusion recognition model based on YOLOv8 is input to output defect types and pixel coordinates, then the defect types and the pixel coordinates are mapped to physical space coordinates through a homography matrix, the defect authenticity is dynamically tracked and verified in combination with a KLT optical flow method, and finally a three-level linkage response mechanism is started. The system comprises an image acquisition module, a preprocessing module, a multi-scale fusion identification module, a space positioning and dynamic tracking module, a real-time capturing module and a linkage response module, and is also provided with a working condition adaptation and function expansion interface. The method supports real-time detection of the running speed of the lifting rope larger than or equal to 2 m / s, can accurately recognize the defects of wire breakage, corrosion and the like, is accurate in positioning and high in robustness, is adaptive to multiple scenes such as mines and elevators, and remarkably improves the intelligence and reliability of safety monitoring of the lifting rope.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of hoisting rope defect positioning, and particularly relates to a hoisting rope defect positioning and real-time capturing method and system based on image recognition. BACKGROUND

[0002] As a core component in an industrial vertical transportation system, the state of a hoisting rope directly determines the reliability and safety of the overall system operation. In the long-term service process, affected by factors such as load fluctuation, environmental corrosion and mechanical friction, the hoisting rope is prone to defects such as broken wires, rust and wear. If these defects are not identified and treated in time, safety accidents are likely to occur. The current detection means mainly rely on manual visual inspection or contact-type equipment, which has obvious limitations in high-speed operation or closed scenes. With the development of image recognition and intelligent sensing technology, a feasible path is provided for the non-contact and automatic detection of the in-service state of the hoisting rope, and the detection method is transformed towards high efficiency and intelligence.

[0003] The hoisting rope bears the core bearing task in mine hoisting, elevator systems and large-scale hoisting equipment, and its safety performance directly affects the stable operation of the equipment and the safety of the operating personnel. The traditional detection methods mainly rely on manual visual inspection, magnetic powder flaw detection or ultrasonic detection, which have certain effects in the early stage, but have problems such as low efficiency, large error and high missed detection rate in actual operation. Especially in long-period continuous operation and dynamic scenes, the traditional methods are difficult to realize real-time continuous monitoring. With the development of intelligent manufacturing, machine vision and image recognition technology gradually shows advantages such as high precision, non-contact and traceability in industrial detection, and becomes an important means for structural defect identification. At present, there are related researches on applying image recognition to the fields of steel wire ropes and conveyor belts, but for the hoisting rope which has high-frequency vibration, complex surface texture and variable running environment, there is still a lack of an intelligent detection method with strong adaptability, high recognition accuracy and timely positioning. The fusion system realizing defect identification, spatial positioning and real-time response has become a key problem to be solved in the field of hoisting rope intelligent monitoring. SUMMARY

[0004] In order to solve the defects in the prior art, the application provides a hoisting rope defect positioning and real-time capturing method and system based on image recognition.

[0005] The technical scheme adopted by the application is a hoisting rope defect positioning and real-time capturing method based on image recognition, which comprises the following steps:

[0006] S1: continuously collecting images of the hoisting rope in dynamic operation to obtain continuous image frames covering the cross section and surface of the hoisting rope;

[0007] S2: Preprocess the continuous image frames to remove image noise caused by high-frequency vibration and environmental interference, and enhance the recognition of surface defect features of the hoisting rope;

[0008] S3: Input the preprocessed image frames into a multi-scale fusion recognition model, and output the defect type of the hoisting rope, including broken wire, rust, wear, loose strand and defect position information in the image pixel coordinate system, through model inference;

[0009] S4: Based on the preset camera imaging model, the mapping relationship between the image pixel coordinates and the hoisting rope physical space coordinates is established, the defect position in the image output by S3 is converted into the defect coordinates in the actual physical space of the hoisting rope, and the defect spatial positioning is realized;

[0010] S5: Multi-frame dynamic tracking is performed on the defects positioned by S4, and the authenticity and stability of the defects are judged by combining the displacement change and position overlap of the defects in the continuous image frames;

[0011] S6: When it is determined that there is a stable defect, the real-time capture and linkage response mechanism of the defect is started, the type, physical coordinates and timestamp information of the defect are output, and the preset response action is executed according to the defect risk level.

[0012] Preferably, the preprocessing in S2 is specifically: based on a 3x3 pixel window, a bilateral filtering operation is performed, image data is processed in parallel in an FPGA chip using a pipeline structure, a sliding window is constructed by two levels of FIFO1 and FIFO2 cache, the absolute value of the difference between the current pixel and the center point of the neighborhood is calculated pixel by pixel, and the image noise reduction and clarity recovery are completed by combining the proximity weight lookup table and the pixel intensity Gaussian weighted fusion operation. The time delay in the preprocessing process meets the continuous processing requirements of high-frequency image frames.

[0013] Preferably, the multi-scale fusion recognition model in S3 is constructed based on the YOLOv8 backbone network, and specifically includes:

[0014] An initial feature map F0 is extracted through a CSP convolution structure, and three scale branches F1, F2 and F3 corresponding to different defect size ranges are constructed using a feature pyramid structure;

[0015] An SE attention mechanism module is introduced in the feature extraction process to weight the intermediate feature channels to focus on the defect texture boundary features;

[0016] In the training stage, a CIoU loss function is used to optimize the defect position prediction accuracy, a FocalLoss is used to suppress the weight deviation of easy classification samples, and steel wire rope field transfer learning parameters are loaded to improve the model generalization ability.

[0017] Preferably, the mapping of image pixel coordinates and physical space coordinates in S4 is achieved by a two-dimensional homography matrix H, assuming that the center point of the defect in the image is (xi, yi), and the corresponding physical position is (Xi, Yi), the mapping formula is: (Xi, Yi) = H x (xi, yi), wherein the homography matrix H is calibrated by the camera intrinsic parameters and the installation parameters during system deployment; the dynamic tracking in S5 uses KLT optical flow method to obtain the velocity vector of the defect in the adjacent frame, by setting a time sequence window, if the displacement vector of the defect in the continuous frames is stable and the position overlap degree is > 0.5, it is determined that it is a real defect and the position cache is updated, otherwise the defect information is excluded.

[0018] Preferably, the real-time capture and linkage response in S6 specifically includes:

[0019] Based on GPU deployment of multi-scale fusion recognition model, the model is quantized and accelerated by TensorRT engine, so that the single frame image inference time is ≤40ms, which meets the identification frequency requirement of improving the rope running speed ≥2m / s;

[0020] Set the defect confidence threshold θ ≥ 0.75, only the stable defects with confidence higher than the threshold are captured in real time;

[0021] The linkage response mechanism includes three levels of action: the first level is image highlighting marking, drawing a bounding box in the defect area and encoding the image frame number; the second level is data communication, sending defect type, physical coordinates and timestamp data packet to the upper control system through industrial communication interface; the third level is risk linkage, when the defect type is high-risk defect such as broken wire, send electrical interruption instruction to trigger the hoisting equipment to pause or the warning device to start.

[0022] A hoisting rope defect positioning and real-time capture system based on image recognition, comprising:

[0023] An image acquisition module for continuously acquiring images of the hoisting rope in dynamic operation to obtain continuous image frames covering the cross section and surface of the hoisting rope;

[0024] A preprocessing module connected with the image acquisition module for receiving the continuous image frames and performing denoising and defect feature enhancement processing;

[0025] A multi-scale fusion recognition module connected with the preprocessing module for receiving the preprocessed image frames and outputting the defect type of the hoisting rope and the defect position in the image pixel coordinate system through model inference;

[0026] A spatial positioning and dynamic tracking module connected with the multi-scale fusion recognition module for establishing the mapping relationship between image pixel coordinates and physical space coordinates, completing the spatial positioning of the defect, and dynamically tracking the defect in multiple frames to determine its authenticity and stability.

[0027] The real-time capture and linkage response module is signal-connected to the spatial positioning and dynamic tracking module. It is used to capture stable defects and output defect information, and execute preset response actions according to the defect risk level.

[0028] Preferably, the image acquisition module includes an HTG8518 optical module, an FPGA control unit, a data buffer unit, and a communication interface unit; the HTG8518 optical module is compatible with various video signal inputs and transmits high-frequency image data through the Aurora high-speed communication protocol; the FPGA control unit integrates a bilateral filtering algorithm to control the synchronization of the image acquisition process; the data buffer unit adopts a FIFO and DDR3 dual-path buffer structure to ensure continuous writing of image data without frame loss; the communication interface unit includes an RS-422 interface and an RGMII interface.

[0029] Preferably, the preprocessing module is integrated into the FPGA chip and is specifically configured to: support bilateral filtering operations with a 3×3 pixel window, construct a sliding window through two-level buffers FIFO1 and FIFO2, transmit image data in parallel pixel by pixel and perform convolution scanning, and combine proximity weight lookup table and pixel intensity Gaussian weighted operation to achieve image noise removal and defect feature enhancement.

[0030] Preferably, the multi-scale fusion recognition module includes a GPU computing unit and a model storage unit; the model storage unit stores a multi-scale fusion recognition model based on the YOLOv8 backbone network, which includes an FPN feature pyramid structure and an SE attention mechanism module; the GPU computing unit deploys the TensorRT engine to quantize and accelerate the model, making the inference time of a single frame image ≤40ms, and supports loading wire rope domain transfer learning parameters to optimize the model's generalization ability.

[0031] Preferably, the system further includes a working condition adaptation unit and a function expansion interface; the working condition adaptation unit includes a 5500K high-brightness LED array, an oleophobic coating on the acquisition end, and a jitter compensation algorithm; the function expansion interface includes a multimodal sensing interface and an edge-cloud switching interface.

[0032] The beneficial effects of this invention are:

[0033] (1) Through a multi-scale feature fusion model and attention mechanism (SE module), the system can simultaneously detect multiple defect types such as broken wires, corrosion, wear, and loose strands, and has excellent recognition ability, especially for small defects. The introduction of ClOU loss function and Focal Loss optimization effectively alleviates the recognition bias caused by sample imbalance.

[0034] (2) The system adopts GPU acceleration and TensorRT inference optimization, with a single frame processing time of less than 40ms, which meets the real-time detection requirements for hoisting rope running speeds higher than 2m / s. Combined with multi-frame buffering and timing verification mechanisms, it can maintain stable output in high-frequency vibration environments.

[0035] (3) Based on camera calibration and homography matrix mapping, the conversion from image pixel coordinates to physical space coordinates is realized. Combined with KLT optical flow method, the dynamic tracking of defects and trajectory filtering are completed, which significantly improves the accuracy of position calibration.

[0036] (4) Strong robustness and adaptability: The system integrates real-time bilateral filtering and image enhancement algorithms on the FPGA side to effectively suppress interference such as dust, oil stains, and changes in lighting.

[0037] (5) A three-layer response mechanism (image marking, communication transmission, and interlocking control) is designed, which supports integration with PLC systems through industrial interfaces such as RS-422 and RGMII, and has plug-and-play deployment capabilities. The system supports an edge-cloud collaborative architecture and can flexibly select local or remote processing modes according to scenario requirements.

[0038] (6) The system adopts a modular design, decoupling hardware and algorithms, and can be adapted to various scenarios such as elevator ropes, bridge cables, and port cranes. It reserves multi-sensor interfaces and online learning functions to support subsequent function expansion and model iteration. Attached Figure Description

[0039] Figure 1 This is a system structure diagram of the present invention;

[0040] Figure 2 This is the ablation image of the loss balance coefficient in an embodiment of the present invention. Detailed Implementation

[0041] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] A method for locating and capturing defects in hoisting ropes in real time based on image recognition includes the following steps:

[0043] S1: Continuous image acquisition is performed on the hoisting rope in dynamic operation to obtain continuous image frames covering the cross-section and surface of the hoisting rope;

[0044] S2: Preprocess the continuous image frames to remove image noise caused by high-frequency vibration and environmental interference, and enhance the recognizability of surface defect features of the rope.

[0045] S3: Input the preprocessed image frame into the multi-scale fusion recognition model, and output the defect type of the lifting rope through model inference, including broken wire, corrosion, wear, loose strands and defect location information in the image pixel coordinate system;

[0046] S4: Based on the preset camera imaging model, establish the mapping relationship between image pixel coordinates and the physical space coordinates of the lifting rope, and convert the defect position in the image output by S3 into the defect coordinates in the actual physical space of the lifting rope to realize defect spatial positioning.

[0047] S5: Perform multi-frame dynamic tracking on the defect located in S4, and combine the displacement change and position overlap of the defect in continuous image frames to determine the authenticity and stability of the defect.

[0048] S6: When a stable defect is determined to exist, the real-time defect capture and linkage response mechanism is activated, the defect type, physical coordinates, and timestamp information are output, and preset response actions are executed according to the defect risk level.

[0049] In this embodiment, the preprocessing in S2 specifically involves: performing bilateral filtering based on a 3×3 pixel window; using a pipelined structure to process image data in parallel in the FPGA chip; constructing a sliding window through two-level buffers FIFO1 and FIFO2; calculating the absolute value of the difference between the current pixel and the neighborhood center point pixel by pixel; and combining proximity weight lookup table and pixel intensity Gaussian weighted fusion operation to complete image noise reduction and sharpness restoration. Furthermore, the time delay during the preprocessing process meets the requirements for continuous processing of high-frequency image frames.

[0050] The multi-scale fusion recognition model in S3 is built based on the YOLOv8 backbone network and specifically includes:

[0051] The initial feature map F0 is extracted using a CSP convolutional structure, and three scale branches F1, F2, and F3 corresponding to different defect size ranges are constructed using a feature pyramid structure.

[0052] In the feature extraction process, an SE attention mechanism module is introduced to weight the intermediate feature channels in order to focus on the boundary features of defect textures;

[0053] During the training phase, the CIoU loss function is used to optimize the accuracy of defect location prediction. FocalLoss is combined to suppress the weight shift of easily classified samples, and wire rope domain transfer learning parameters are loaded to improve the model's generalization ability.

[0054] In step S4, the mapping between image pixel coordinates and physical space coordinates is achieved through a two-dimensional homography matrix H. Let the center point of the defect in the image be (xi,yi), and its corresponding physical position be (Xi,Yi). The mapping formula is: (Xi,Yi) = H × (xi,yi), where the homography matrix H is obtained by calibration of the camera intrinsic parameters and installation parameters during system deployment. In step S5, the dynamic tracking uses the KLT optical flow method to obtain the velocity vector of the defect in adjacent frames. By setting a time series window, if the displacement vector of the defect in consecutive frames is stable and the position overlap is >0.5, it is determined to be a real defect and the position cache is updated; otherwise, the defect information is removed.

[0055] The real-time capture and linkage response in S6 specifically include:

[0056] A multi-scale fusion recognition model is deployed based on GPU, and the model is accelerated by quantization using the TensorRT engine, so that the inference time of a single frame image is ≤40ms, which meets the recognition frequency requirement of the lifting rope running speed ≥2m / s.

[0057] Set a defect confidence threshold θ≥0.75, and only capture stable defects with a confidence level higher than this threshold in real time;

[0058] The linkage response mechanism includes three levels of action: Level 1 is image highlighting and marking, drawing a bounding box in the defect area and encoding the image frame number; Level 2 is data communication, sending the defect type, physical coordinates, and timestamp data packet to the upper control system through the industrial communication interface; Level 3 is risk linkage, when the defect type is a high-risk defect such as wire breakage, an electrical interruption command is sent to trigger the hoisting equipment to stop or the early warning device to start.

[0059] A hoisting rope defect location and real-time capture system based on image recognition, comprising:

[0060] The image acquisition module is used to continuously acquire images of the hoisting rope in dynamic operation, and obtain continuous image frames covering the cross-section and surface of the hoisting rope.

[0061] The preprocessing module is signal-connected to the image acquisition module and is used to receive the continuous image frames and perform noise reduction and defect feature enhancement processing.

[0062] The multi-scale fusion recognition module is signal-connected to the preprocessing module and is used to receive the preprocessed image frames and output the defect type of the lifting rope and the defect location in the image pixel coordinate system through model inference.

[0063] The spatial positioning and dynamic tracking module is signal-connected to the multi-scale fusion recognition module. It is used to establish the mapping relationship between image pixel coordinates and physical space coordinates, complete the spatial positioning of defects, and perform multi-frame dynamic tracking of defects to determine their authenticity and stability.

[0064] The real-time capture and linkage response module is signal-connected to the spatial positioning and dynamic tracking module. It is used to capture stable defects and output defect information, and execute preset response actions according to the defect risk level.

[0065] The image acquisition module includes an HTG8518 optical module, an FPGA control unit, a data buffer unit, and a communication interface unit. The HTG8518 optical module is compatible with various video signal inputs and transmits high-frequency image data via the Aurora high-speed communication protocol. The FPGA control unit integrates a bilateral filtering algorithm to control the synchronization of the image acquisition process. The data buffer unit adopts a dual-path buffer structure of FIFO and DDR3 to ensure continuous writing of image data without frame loss. The communication interface unit includes an RS-422 interface and an RGMII interface.

[0066] The preprocessing module is integrated into the FPGA chip and is specifically configured to support bilateral filtering operations with a 3×3 pixel window. It constructs a sliding window through two-level buffers, FIFO1 and FIFO2, transmits image data in parallel pixel by pixel and performs convolutional scanning, and combines proximity weight lookup table and pixel intensity Gaussian weighted operation to achieve image noise removal and defect feature enhancement.

[0067] The multi-scale fusion recognition module includes a GPU computing unit and a model storage unit. The model storage unit stores a multi-scale fusion recognition model based on the YOLOv8 backbone network. This model includes an FPN feature pyramid structure and an SE attention mechanism module. The GPU computing unit deploys the TensorRT engine to quantize and accelerate the model, making the inference time of a single frame image ≤40ms. It also supports loading wire rope domain transfer learning parameters to optimize the model's generalization ability.

[0068] The system also includes a working condition adaptation unit and a function expansion interface; the working condition adaptation unit includes a 5500K high-brightness LED array, an oleophobic coating on the acquisition end, and a jitter compensation algorithm; the function expansion interface includes a multimodal sensing interface and an edge-cloud switching interface.

[0069] Example 1

[0070] To meet the image acquisition requirements during the dynamic operation of the hoisting rope, the system is designed with high bandwidth and high stability in mind, employing a modular architecture to achieve high-speed acquisition and real-time processing of image data. The image acquisition module is based on the HTG8518 optical module, capable of adapting to various video signal input conditions in industrial environments, and achieves high-speed, lossless transmission of high-frequency image data via the Aurora high-speed communication protocol. This module is controlled and managed by an FPGA to ensure a synchronous and stable acquisition process. The FPGA integrates a bilateral filtering algorithm to perform real-time noise reduction on the raw images, improving image clarity and reducing the risk of misjudgment at the backend. The processed image data is relayed and stored through a dual-path FIFO and DDR3 buffer mechanism, ensuring continuous data writing without frame loss and providing a stable data stream input for subsequent recognition models. The system uses RS-422 and RGMII dual communication interfaces for command interaction and data feedback, respectively, ensuring low-latency linkage between the acquisition end and the PC host computer. Figure 1 The diagram shows the overall structure of the acquisition system, demonstrating the functional division and interface collaboration among the image acquisition, caching, processing, and communication modules, thus laying the hardware foundation for algorithm deployment and coordinated control.

[0071] After image data acquisition, to improve the perception and stability of subsequent recognition models for detailed features, the system deploys an image preprocessing module in the FPGA to perform real-time bilateral filtering based on a 3×3 window. This method uses each frame of image data as input, constructs a sliding window using two-level buffers (FIFO1 and FIFO2), transmits image data pixel-by-pixel in parallel, and scans neighboring pixels using convolution. The system calculates the difference between the current pixel and the center point of its neighborhood, takes the absolute value, accesses a lookup table for proximity weights, and performs Gaussian weighted fusion calculations based on pixel intensity weights. The final filtered result is then output to the DDR3 buffer. Through parallel processing via the FPGA pipeline structure, the system can complete eight additions and integer normalization operations with extremely low latency, effectively restoring image clarity under high-frequency vibration and shock interference, and improving image quality stability in practical deployment scenarios.

[0072] To further optimize model training performance, the system incorporates various image enhancement techniques to expand the original samples after preprocessing. These include common enhancement methods such as random rotation, color perturbation, noise simulation, and occlusion simulation. Simultaneously, a defect texture image overlay mechanism, combining broken wires and corrosion with application requirements, is constructed to enhance the model's generalization ability to multiple defect features. Preprocessing and sample enhancement work synergistically to improve model training quality and robustness to in-situ inference, establishing a stable and realistic defect recognition image data channel.

[0073] On the basis of completing image acquisition and preprocessing, the core task of the system is to accurately identify, spatially locate, and dynamically capture surface defects of the hoisting rope. To achieve a technical closed-loop, the recognition target should not be limited to single defect judgment, but should include three levels: defect type recognition, position calibration, and state change perception, which support the response logic of subsequent positioning and alarm mechanisms. Combining engineering measured samples and professional experience, the system divides common defects of the hoisting rope into four categories: broken wires, corrosion, wear, and loose strands, and establishes image feature templates for each type of defect as the basis for model training and annotation. Defining the target system and defect classification helps to construct an identification model with multi-task collaboration capabilities and promotes the spatial mapping and联动决策 of subsequent inference results.

[0074] To achieve accurate recognition of multiple types of defects, the system constructs a multi-scale fusion detection model based on the YOLOv8 backbone network to balance high-speed inference and detail perception capabilities. The model input is the preprocessed RGB image , first extracts the initial feature map F0 through the CSP convolutional structure, and then uses the Feature Pyramid Network (FPN) to construct three scale branches F1, F2, F3, corresponding to different defect size ranges respectively. Each scale feature map outputs the class probability pc, the position box coordinates (x, y, w, h)), and the confidence s through the detection head, forming a detection output vector .

[0075] To improve the response ability to tiny broken wires and fine corrosion lines, an attention mechanism module SE (Squeeze-and-Excitation) is introduced to weight the intermediate feature channels, adjust the channel importance, and make the model more focused on texture boundary changes. In the training stage, the CIoU loss function is used to optimize the position prediction accuracy, defined as: .

[0076] Among them, ρ is the distance between the center points, c is the diagonal length, and v represents the aspect ratio consistency measure. Combining with Focal Loss to suppress the weight shift of easily classified samples and enhance the discriminative ability of the model for unbalanced defect categories.

[0077] Under the condition of limited training data volume, the model parameters of adjacent fields such as steel wire ropes are loaded through transfer learning, and combined with the multi-scale image enhancement strategy, to improve the recognition robustness and generalization performance of the model for complex defects in actual working conditions.

[0078] After completing multi-scale defect recognition, the system needs to map the detection results in the image to the actual hoisting rope structure to complete the precise positioning from pixel coordinates to physical space. Based on the camera imaging model, assuming that the center point of the defect in the image is (xi, yi), its corresponding physical position (Xi, Yi) can be expressed by the plane coordinate mapping formula as: .

[0079] Here, H is a two-dimensional homography matrix composed of camera intrinsic parameters and installation parameters, which needs to be calibrated during system deployment to ensure that each pixel has physical meaning. During the operation of the lifting rope, to ensure that defects are not missed or mistracked, the system introduces a target tracking mechanism based on the KLT optical flow method to obtain the velocity vector of the detected target in adjacent frames. .

[0080] A time-series window is established to verify the stability of the identification results. If the defect displacement vector is stable and the position overlap is high in consecutive frames, it is determined to be a real defect and the position cache is updated; otherwise, it is automatically rejected to prevent false alarms. This mechanism combines single-frame identification with multi-frame trajectory tracking, enabling defect localization to have not only spatial accuracy but also temporal continuity, providing a reliable foundation for real-time capture and alarm linkage.

[0081] After completing the spatial localization of defects, the system needs to possess the ability to capture dynamic targets in real time and a linkage response mechanism to achieve immediate handling after defect discovery. The system design adopts a high-concurrency image stream processing architecture, deploys a YOLOv8 inference model based on GPU, and uses the TensorRT engine for model quantization and acceleration, keeping the inference time of a single frame image within 40ms, meeting the recognition frequency requirement of rope running speeds above 2m / s. The inference module writes the target box information output in each frame into a queue cache, while the scheduling module compares the recognition results of consecutive frames and extracts targets with a confidence level greater than the set threshold θ=0.75 for real-time capture. When a stable target is confirmed, the system immediately activates three linkage mechanisms: first, the image highlighting module draws the boundary of the target defect area and encodes the image frame number; second, the communication response module sends the defect location and type data packet to the control center through the RS-422 interface; and third, the linkage triggering module activates preset alarms and automatic shutdown commands when the identified category belongs to high-risk defects (such as broken wires). To support deployment requirements for different application scenarios, the system also reserves interfaces for edge computing and the cloud, which can flexibly achieve local rapid processing or remote unified monitoring based on the distribution of computing resources.

[0082] Example 2

[0083] In a mine hoist system, the identification device is deployed above the tensioning wheel at the wire rope exit. Equipped with dual cameras arranged obliquely to cover the entire cross-section of the rope, it also incorporates a high-brightness LED array to enhance image contrast under low-light conditions. The image acquisition frequency is set to 25 frames per second, forming a frame-to-meter mapping mechanism based on the operating speed to ensure no defects are missed. The image stream is transmitted in real-time to the edge processing unit for preprocessing, identification, and location, and is linked with the main control PLC system to form an integrated "identification-location-response" mechanism. Since its deployment, the system has been running continuously for over six months, automatically identifying and recording multiple instances of wire breakage and corrosion defects, demonstrating engineering-grade stability and practicality.

[0084] In the actual deployment of the system, the defect detection and response process takes the edge node as the core control unit and unfolds around the three stages of identification, judgment and linkage to ensure that the image recognition results can be quickly and accurately converted into engineering control commands. The system performs secondary screening on the recognition results output by the YOLOv8 model, and uses the confidence threshold θ≥0.80 and the IoU overlap calculation model (2) mentioned above to judge the persistence of the defect target. The specific judgment criteria are: the detection box overlap IoU>0.5 in three consecutive frames and the categories are consistent before entering the linkage judgment process.

[0085] The response module is designed based on an image sequence buffering mechanism. By setting a frame delay window (default 8 frames), it maintains the synchronous stability of the recognition results in terms of timing. Each image frame in the buffer queue is bound to the recognition result, including defect type, pixel position, confidence value, timestamp, and number. After determining that the response conditions are met, the system packages the data into a structured message, which consists of five fields: defect type code, image frame number, detection position index, frame timestamp, and suggested action instruction. This message is synchronously sent to the upper PLC system via the RS-422 industrial communication bus. At the physical linkage execution level, the system sets up a three-level response strategy: the first level is the image marking feedback layer, which directly highlights the target area on the operation terminal interface and numbers and archives it in real time; the second level is the communication transmission layer, which builds a standard message template and frame recognition log recording mechanism to achieve rapid information exchange; the third level is the interlocking trigger layer, which sends an electrical interrupt command with a 1-second delay when the defect is determined to be a high-level structural risk (such as wire breakage extension), triggering the lifting mechanism to pause or the early warning device to start, reserving time for operation intervention to ensure that the response action is controllable. Each detection and response stage of the system has an independent recording path, and relevant parameters are automatically recorded in the log, including trigger time, processing time, frame number, and location coordinates, providing a complete data foundation for adaptive analysis and stability assessment. This linkage response strategy has been embedded in the industrial control chain, forming a closed chain from image recognition to behavior execution, and is fully capable of process integration.

[0086] In hoisting environments such as mines and tunnels, image acquisition systems face a variety of complex factors, including high dust levels, low light, oil contamination, and vibration interference. To verify the system's recognition stability under non-ideal conditions, adaptive design and specialized testing were conducted on the acquisition link and recognition model at the engineering site. For lighting, a 5500K high-brightness LED array was used in conjunction with a diffuser to suppress overexposure; for jitter compensation, multi-frame fusion was performed using image buffering and IoU overlap criteria. An oleophobic coating was applied to the surface of the acquisition end to reduce adhesive interference.

[0087] In addition, to enhance the robustness of the model under complex image inputs, a loss balance coefficient λ is introduced during the training phase to adjust the loss weight allocation between object detection and feature learning. Figure 2 The trend of the Top-1 error rate of the model on CIFAR-100 is shown when λ∈{1 / 50, 1 / 55, 1 / 60, 1 / 65, 1 / 70}. The results show that λ = 1 / 60 can obtain the best convergence performance, and this value is used as the training configuration in subsequent deployment versions.

[0088] The following are the system's operational stability data under typical interference conditions: (frame-level continuous identification success rate and response retention rate):

[0089] Table 1. Operational stability data under typical disturbance conditions

[0090] In testing, the system maintained high recognition consistency even with image sharpness below 0.80, thanks to bilateral filtering and a multi-scale fusion structure of the model. By introducing a buffer decision mechanism and frame overlap analysis, target drift and response jitter caused by fluctuations in operating conditions were effectively suppressed. Robust design is integrated across multiple levels of acquisition, processing, and decision-making, ensuring the system's continuous operational capability in complex industrial environments and providing a technical foundation for large-scale application.

[0091] Based on the robustness test results under various operating conditions, the system adopts a modular and decoupled architecture. The three main functional modules—acquisition, recognition, and linkage—operate independently through standard hardware interfaces and protocols, demonstrating excellent structural adaptability. Depending on the target environment, the acquisition unit can be configured with a single camera for frontal or dual-camera oblique viewing, and the image area and light source angle can be flexibly adjusted to adapt to different structural scenarios such as elevator wire ropes, bridge cables, and cableways. For space-constrained areas, the system supports customized installation templates, reducing the amount of modification work and construction risks. The platform architecture supports switching between edge deployment and cloud processing modes. In closed environments, the system compresses and embeds the recognition model into the local terminal, achieving a closed loop of acquisition, preprocessing, recognition, and control. In open industrial systems, image data is uploaded to the cloud platform for centralized processing and integrated with the main control system through a RESTful interface, adapting to the remote monitoring needs of the Internet of Things (IoT) architecture. The system communication layer supports MQTT and RS-422 protocols, ensuring stable data transmission in high-interference environments.

[0092] To enhance functional scalability, the system reserves multimodal sensing interfaces, supporting the fusion of state parameters such as vibration and temperature to build a trend prediction mechanism for the entire equipment lifecycle. The model supports online fine-tuning and incremental learning to adapt to the ever-expanding needs of defect types; the hardware solution has customization capabilities, and a lightweight version can be deployed to meet cost-sensitive application scenarios.

[0093] In terms of deployment, the system can be pre-installed and integrated into equipment manufacturing, or retrofitted for upgrading existing equipment, supporting rapid embedded deployment. With its flexible structure, communication compatibility, and stable recognition capabilities, the system has a solid foundation for widespread application in various industries such as coal mining, power, rail transportation, and ports, and possesses the potential to extend into multi-condition intelligent sensing systems.

[0094] In summary, this invention addresses the problem of intelligent identification and real-time capture of defects in hoisting ropes during operation, constructing a systematic technical solution integrating image acquisition, defect detection, spatial positioning, and coordinated response. Through practical engineering deployment and multi-condition verification, the system demonstrates excellent identification accuracy, response efficiency, and operational stability. This method not only improves the level of equipment safety monitoring but also provides a feasible path for the application and promotion of intelligent sensing technology in complex scenarios, possessing broad industry replication value and evolution potential.

[0095] Furthermore, the parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration. The preset parameters or preset thresholds mentioned above have all been set by those skilled in the art based on actual conditions or obtained through simulation with a large amount of data.

[0096] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image recognition-based hoisting rope defect positioning and real-time capturing method, characterized in that, The method comprises the following steps: S1: continuously image acquisition is performed on the hoisting rope in dynamic operation, and continuous image frames covering the cross section and surface of the hoisting rope are acquired; S2: the continuous image frames are preprocessed, image noise caused by high-frequency vibration and environmental interference is removed, and the recognition degree of the defect features of the surface of the hoisting rope is enhanced; S3: the preprocessed image frames are input into a multi-scale fusion recognition model, and the defect type of the hoisting rope is output through model inference, including broken wire, rust, wear, loose strand and defect position information in the image pixel coordinate system; S4: based on a preset camera imaging model, a mapping relationship between the image pixel coordinates and the physical space coordinates of the hoisting rope is established, the defect position in the image output by S3 is converted into the defect coordinates in the actual physical space of the hoisting rope, and spatial positioning of the defect is realized; S5: the defect positioned by S4 is tracked dynamically for multiple frames, the displacement change and position overlap degree of the defect in the continuous image frames are combined, and the authenticity and stability of the defect are judged; S6: when it is judged that there is a stable defect, a real-time capture and linkage response mechanism is started, the type, physical coordinates and timestamp information of the defect are output, and a preset response action is performed according to the defect risk level.

2. The image recognition based hoisting rope defect positioning and real-time capturing method according to claim 1, characterized in that, The preprocessing in S2 is specifically: a bilateral filtering operation is performed based on a 3*3 pixel window, image data is processed in parallel in a pipeline structure in an FPGA chip, a sliding window is constructed through two-stage buffering of FIFO1 and FIFO2, the absolute value of the difference between the current pixel and the center point of the neighborhood is calculated pixel by pixel, and the image noise reduction and definition recovery are completed through the fusion operation of the proximity weight lookup table and the pixel intensity Gaussian weighting, and the time delay in the preprocessing process meets the continuous processing requirement of high-frequency image frames.

3. The image recognition based hoisting rope defect positioning and real-time capturing method according to claim 1, characterized in that, The multi-scale fusion recognition model in S3 is constructed based on a YOLOv8 backbone network, and specifically comprises: initial feature maps F0 are extracted through a CSP convolution structure, and three scale branches F1, F2 and F3 corresponding to different defect size ranges are constructed by using a feature pyramid structure; an SE attention mechanism module is introduced in the feature extraction process, and the intermediate feature channels are weighted to focus on the defect texture boundary features; in the training stage, a CIoU loss function is used to optimize the defect position prediction accuracy, a FocalLoss is combined to suppress the weight deviation of easy classification samples, and transfer learning parameters in the field of steel wire ropes are loaded to improve the model generalization ability.

4. The image recognition based hoisting rope defect positioning and real-time capturing method according to claim 1, characterized in that, The mapping of the image pixel coordinates and the physical space coordinates in S4 is realized through a two-dimensional homography matrix H, the defect center point in the image is (xi, yi), and the corresponding physical position is (Xi, Yi), the mapping formula is (Xi, Yi) = H * (xi, yi), wherein the homography matrix H is obtained by calibrating the camera intrinsic parameters and the installation parameters during system deployment; the dynamic tracking in S5 uses a KLT optical flow method to obtain the velocity vector of the defect in adjacent frames, a time sequence window is set, if the displacement vector of the defect in the continuous frames is stable and the position overlap degree is greater than 0.5, it is judged that the defect is real and the position cache is updated, otherwise the defect information is removed.

5. The image recognition based hoisting rope defect positioning and real-time capturing method according to claim 1, characterized in that, The real-time capture and linkage response in S6 specifically comprises: A multi-scale fusion recognition model is deployed based on a GPU, and the model is quantized and accelerated by a TensorRT engine, so that the single-frame image inference time is ≤40 ms, meeting the identification frequency requirement of improving the rope running speed ≥2 m / s; A defect confidence threshold θ≥0.75 is set, and only stable defects with a confidence higher than the threshold are captured in real time; The linkage response mechanism includes three levels of actions: the first level is image highlighting marking, a boundary box is drawn in the defect area, and the image frame number is encoded; the second level is data communication, and a defect type, physical coordinates, and timestamp data packet are sent to the upper control system through an industrial communication interface; the third level is risk linkage, and when the defect type is a high-risk defect such as a broken wire, an electrical interruption instruction is sent to trigger the hoisting equipment to pause or the warning device to start.

6. An image recognition-based hoisting rope defect positioning and real-time capturing system, characterized by, It comprises: An image acquisition module for continuously acquiring images of the hoisting rope in dynamic operation, obtaining continuous image frames covering the cross section and surface of the hoisting rope; A preprocessing module connected to the image acquisition module for receiving the continuous image frames and performing denoising and defect feature enhancement processing; A multi-scale fusion recognition module connected to the preprocessing module for receiving the preprocessed image frames and outputting the defect type of the hoisting rope and the defect position in the image pixel coordinate system through model inference; A spatial positioning and dynamic tracking module connected to the multi-scale fusion recognition module for establishing a mapping relationship between the image pixel coordinates and the physical space coordinates, completing defect spatial positioning, and dynamically tracking defects for multiple frames to determine their authenticity and stability; A real-time capture and linkage response module connected to the spatial positioning and dynamic tracking module for capturing stable defects and outputting defect information, and executing preset response actions according to the defect risk level.

7. The image recognition based hoisting rope defect positioning and real-time capturing system according to claim 6, characterized in that, The image acquisition module includes an HTG8518 optical module, an FPGA control unit, a data buffer unit, and a communication interface unit; the HTG8518 optical module is adapted to multiple video signal inputs and transmits high-frequency image data through an Aurora high-speed communication protocol; the FPGA control unit integrates a bilateral filtering algorithm for controlling the synchronization of the image acquisition process; the data buffer unit uses a FIFO and DDR3 dual-path buffer structure to ensure continuous image data writing without frame loss; and the communication interface unit includes an RS-422 interface and an RGMII interface.

8. The image recognition based hoisting rope defect positioning and real-time capturing system according to claim 6, characterized in that, The preprocessing module is integrated in an FPGA chip and is specifically configured to support a 3×3 pixel window bilateral filtering operation, build a sliding window through two levels of FIFO1 and FIFO2 cache, transmit image data pixel by pixel in parallel and perform convolution scanning, and realize image noise removal and defect feature enhancement through proximity weight lookup table and pixel intensity Gaussian weighting operation.

9. The image recognition based hoisting rope defect positioning and real-time capturing system according to claim 6, characterized in that, The multi-scale fusion recognition module includes a GPU computing unit and a model storage unit; the model storage unit stores a multi-scale fusion recognition model based on a YOLOv8 backbone network, and the model contains an FPN feature pyramid structure and an SE attention mechanism module; The GPU computing unit deploys a TensorRT engine, quantizes and accelerates the model, makes the single-frame image reasoning time ≤ 40 ms, and supports loading the steel wire rope field migration learning parameters to optimize the model generalization ability.

10. The image recognition based hoisting rope defect positioning and real-time capturing system according to claim 6, characterized in that, The system also comprises a working condition adaptation unit and a function expansion interface; the working condition adaptation unit comprises a 5500K highlight LED array, an oil-repellent coating at the acquisition end and a jitter compensation algorithm; the function expansion interface comprises a multi-modal sensing interface and an edge-cloud switching interface.

Images