Steel wire rope defect recognition and positioning system and method based on real-time object detection

Abstract

The present invention relates to the technical field of wire rope defect identification and positioning, and particularly to a wire rope defect identification and positioning system and method based on real-time object detection. Its technical solution includes a sensor array, a synchronous detection execution mechanism, an NPU neural computing unit, a multi-modal weight management unit, a model library, a multi-core acquisition main board, a communication module, and a power supply module. Based on the multi-modal real-time object detection method, the present invention collects the key features on the surface and inside of the wire rope through a multi-sensor array to achieve multi-modal collaborative training. It not only improves the data representation ability, optimizes the model lightweight and boundary blur problems, but also designs a custom dynamic allocation method, effectively improving the accuracy and precision of defect identification and positioning, realizing an effective closed-loop of model training, and providing a more efficient and reliable technical solution for wire rope operation and maintenance enterprises.

Description

Technical Field

[0001] The present invention relates to the technical field of steel wire rope defect identification and positioning, and particularly to a steel wire rope defect identification and positioning system and method based on real-time object detection. Background Art

[0002] At present, the application of steel wire ropes is crucial in fields such as mines, ports, and construction, involving multiple aspects such as the lifting, transportation, support, shock absorption, and safety guarantee of equipment and personnel. Due to its high strength and wear resistance, the steel wire rope has become an indispensable key material in the field of equipment and personnel lifting. Considering that steel wire ropes are very prone to problems such as wear, corrosion, fatigue, and broken wires during use, in severe cases, it may lead to the fracture of the steel wire rope, causing losses to personnel and equipment. In the existing mode, the detection of steel wire rope defects mostly adopts the manual method of feeling the rope, which not only requires production stoppage, is time-consuming and laborious, but also has a low accuracy rate. To solve the problems existing in the prior art, the present invention provides a steel wire rope defect identification and positioning method, system, and device based on multi-modal real-time object detection.

[0003] Therefore, the present application proposes a steel wire rope defect identification and positioning system and method based on real-time object detection. Summary of the Invention

[0004] The object of the present invention is to address the problem of low efficiency in detecting steel wire rope defects mostly by manual rope feeling in the background art, and to propose a steel wire rope defect identification and positioning system and method based on real-time object detection.

[0005] In a first aspect, the present application provides a steel wire rope defect identification and positioning system based on real-time object detection, including:

[0006] A sensor array, including a four-channel camera and a two-channel magnetic probe. The four-channel camera is used to collect planar images of four directions of the steel wire rope, and the two-channel magnetic probe is used to obtain leakage magnetic field signals and magnetic flux signals;

[0007] A synchronous detection actuator, used to drive the sensor array to move and clamp the steel wire rope for detection, and to release the steel wire rope and return to the initial position after the detection is completed;

[0008] A neural computing unit, whose full English name is Neural Network Processing Unit, hereinafter referred to as NPU for short, is used to perform accelerated operations on the collected data using a deep learning model;

[0009] A multi-modal weight management unit, used to implement a multi-modal dynamic allocation algorithm to achieve dynamic allocation of weights and dynamic fusion of decisions;

[0010] A model library for storing the trained wire rope target detection model structure and parameter files, and supporting online training and remote update;

[0011] A multi-core acquisition mainboard for driving the acquisition program and detection program, and implementing instruction operations and model training tasks at the operating system level;

[0012] A communication module supporting multiple communication methods;

[0013] A power module for powering the entire system.

[0014] Optionally, the sensor array further includes:

[0015] An image acquisition module for driving the four-channel camera to take pictures and screenshots;

[0016] A video acquisition module for continuously acquiring video image information of the moving wire rope;

[0017] An image processing unit for optimizing the image resolution and display effect acquired by the four-channel camera;

[0018] An excitation mechanism for magnetizing the wire rope;

[0019] A high-speed acquisition module for acquiring the data acquired by the two-channel magnetic probe;

[0020] A signal conditioning circuit for denoising the data acquired by the high-speed acquisition module.

[0021] Optionally, the NPU neural computing unit is externally attached to the multi-core acquisition mainboard for real-time synchronous inference analysis of camera images and magnetic probe data.

[0022] Optionally, the communication methods supported by the communication module include low-power wide-area network communication (Lora) based on spread spectrum technology, low-power wide-area network wireless communication technology (LPWAN), low-speed short-distance transmission technology Zigbee, narrow-band Internet of Things technology (NB - IOT), 4G, and WIFI wireless communication mode.

[0023] In a second aspect, the present application provides a wire rope defect identification and positioning method based on real-time target detection, including the following steps:

[0024] S1. Obtain videos and pictures of the wire rope in four directions, namely the front view, rear view, left view, and right view, through four high-definition cameras, and perform image preprocessing and feature extraction;

[0025] S2. Obtain the leakage magnetic field signal and magnetic flux signal of the wire rope through two magnetic probes, and perform signal conditioning and image conversion to obtain the leakage magnetic signal modal diagram and magnetic flux modal diagram;

[0026] S3. Feature fusion is performed on the camera image and the magnetic probe signal image at the same moment to obtain a multi-channel image sample;

[0027] S4. The multi-channel image sample is input into the NPU neural computing unit for training to obtain detection models A, B, and C;

[0028] S5. The multi-modal weight management unit is used to allocate modal weights to detection models A, B, and C;

[0029] S6. Based on the modal weight allocation, dynamic decision fusion is performed on the multi-channel image sample, and the steel wire rope defect location and recognition results are output.

[0030] Optionally, in S2, it specifically includes:

[0031] The Hall and coil sensors are used to obtain the leakage magnetic field signal and magnetic flux signal of the steel wire rope;

[0032] The obtained signals are conditioned to remove noise;

[0033] The one-dimensional data is plotted and cut to obtain the magnetic field distribution, and then the leakage magnetic signal modal diagram and magnetic flux modal diagram are obtained.

[0034] Optionally, in S4, the multi-channel image sample is input into the NPU neural computing unit and trained using a lightweight improvement algorithm based on the yolov11 series object detection algorithm to obtain detection models A, B, and C.

[0035] Optionally, in S5, it specifically includes:

[0036] According to the multi-modal dynamic allocation algorithm, based on a custom dynamic allocation method, modal weights are allocated to detection models A, B, and C; the contribution degrees of the first ten principal components of the fusion modal diagram of the steel wire rope camera are , the confidence level is , and the corresponding multi-modal model weights are , then , the contribution degrees of the first ten principal components of the modal diagram of the steel wire rope leakage magnetic probe are , the confidence level is , the contribution degrees of the first ten principal components of the modal diagram of the steel wire rope magnetic flux probe are , the confidence level is , and the corresponding multi-modal model weights are respectively , , then

[0037] ,

[0038] ,

[0039] The finally obtained normalized weights are respectively , , , where is the weight of the camera, is the normalized weight of the camera, is the weight of the magnetic flux leakage probe signal, is the normalized weight of the magnetic flux leakage probe signal, is the weight of the magnetic flux probe signal, is the normalized weight of the magnetic flux probe signal.

[0040] Optionally, in S6, it specifically includes:

[0041] Perform dynamic decision fusion on the multi-channel image samples according to the modal weight distribution result;

[0042] Output the steel wire rope defect location and recognition result;

[0043] Based on the operation personnel's auxiliary verification of the data of the two types of sensors and the call of the synchronous detection actuator for back-check control, the secondary confirmation of the result is realized.

[0044] Optionally, it further includes model optimization, which specifically includes the following steps:

[0045] The intermediate position of the steel wire rope defect is the original prediction position , by converting the prediction result into a sequence, it becomes: , where , is the predicted label sequence, is the total number of sequences; by expanding and probabilities, the network can quickly focus on the values near the label, and the loss of the steel wire rope target detection model is defined as: , where is the complete cross-entropy value, is the predicted value, is the value before prediction, is the value after prediction, by expanding the probabilities of the values on both sides of the boundary, that is, expanding , ; , , where is the entropy value before prediction, is the entropy value after prediction, ensuring that the regression target is infinitely close to the true label of the corresponding steel wire rope defect position ;According to the sample weight update formula, update the weights of the training samples, so that the model in the next round of training will pay more attention to the samples misclassified previously, thereby gradually improving the model performance. The sample weight update formula is as follows: , where is the weight of the -th round of samples , is the weight of the -th round of samples , is the number of training rounds, is the modality allocation weight, is the true label, is the predicted label of the sample , and exp is the exponential function.

[0046] Compared with the prior art, the present application includes at least one of the following beneficial technical effects:

[0047] Based on the multi-modal real-time object detection method, the present invention solves the problems existing in the field of steel wire rope defect identification and positioning, and achieves the following remarkable effects:

[0048] Fully obtain the key features on the surface and inside of the steel wire rope through a multi-sensor array. The fusion of multi-modal data enables the data to have a more comprehensive and accurate representation, providing a solid foundation for defect positioning and identification.

[0049] Adopt the multi-modal collaborative training idea to optimize the lightweight and boundary blur problems of the real-time detection model. This not only enables the model to process data more efficiently during training, but also reduces the training time while ensuring the model accuracy.

[0050] A customized dynamic allocation method is designed for the weight allocation problem in multi-modal data training. This method can reasonably allocate weights according to the characteristics and contribution degrees of different modality data, improving the accuracy and reliability of the model.

[0051] The multi-modal joint analysis method overcomes the limitations of single-modal data analysis. By comprehensively processing multiple modality data, the accuracy and precision of steel wire rope defect identification and positioning are significantly improved.

[0052] Through dynamic optimization of the model structure and training weights, the present system realizes effective feature learning of the steel wire rope defect identification and positioning model, forms a complete closed-loop system, and ensures that the model can be continuously optimized and adapted to the actual application requirements.

[0053] In summary, based on the multi-modal real-time object detection method, the present invention collects the key features on the surface and inside of the wire rope through a multi-sensor array, realizing multi-modal collaborative training. It not only improves the data representation ability, optimizes the problems of model lightweight and boundary blurring, but also designs a custom dynamic allocation method, effectively improving the accuracy and precision of defect recognition and positioning, realizing an effective closed-loop of model training, and providing a more efficient and reliable technical solution for wire rope operation and maintenance enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is the principle block diagram of the wire rope defect recognition and positioning system based on real-time object detection;

[0055] Figure 2 is the flowchart of the wire rope defect recognition and positioning method based on real-time object detection. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The technical solutions of the present invention will be further described below with reference to the drawings and specific embodiments.

[0057] See Figure 1 : The present application provides a wire rope defect recognition and positioning system based on real-time object detection, which is composed of a sensor array, a synchronous detection actuator, an NPU neural computing unit, a multi-modal weight management unit, a model library, a multi-core acquisition main board, a communication module, and a power module.

[0058] In this embodiment, the sensing array mainly includes a four-channel camera and a two-channel magnetic probe. Among them, the four-channel sensor is used to collect the planar images of the wire rope in four directions, and the wire rope defect information can be obtained through surface texture analysis. The image acquisition module is used to drive the camera to take pictures and screenshots, and the video acquisition module is used to continuously collect the video image information of the moving wire rope. Considering the light environment and surface oil pollution problems of the wire rope, the image processing unit is used to optimize the image resolution and display effect. The two-channel magnetic probe is divided into a Hall probe and a coil probe, which are used to obtain the leakage magnetic field signal and the magnetic flux signal respectively. Before the wire rope conducts magnetic probe data acquisition, it is necessary to use an excitation mechanism to magnetize the wire rope, and then use a high-speed acquisition module to collect data. The collected data has noise and needs to be denoised using a signal conditioning circuit.

[0059] It is worth noting that the synchronous detection actuator is the front-end detection actuator of the device. After the actuator is started, it drives the four-channel camera and the two-channel magnetic probe to move and clamp the wire rope through a motor, so that the wire rope runs through the inside of the device. After stopping the acquisition, the actuator releases the wire rope and moves back to the initial position to ensure the safe operation of the device.

[0060] Among them, the multi-core acquisition mainboard is the main device for driving the acquisition program and the detection program. It has a multi-core CPU and related general interfaces, such as the core boards of the Raspberry Pi series or the Rockchip series. It can implement instruction operations and model training tasks at the operating system level. By externally connecting multiple NPU neural computing units, it can achieve accelerated operation of deep learning models and be used for real-time synchronous inference analysis of camera images and magnetic probe data. In this embodiment, the multi-core acquisition mainboard has a multi-core central processing unit (CPU) and related general interfaces, such as the core boards of the Raspberry Pi series or the Rockchip series, and can implement instruction operations and model training tasks at the operating system level. It provides powerful computing support for the operation of the entire system, ensuring that data acquisition, processing, and analysis can be carried out efficiently.

[0061] In addition, the multi-modal weight management unit incorporates a multi-modal dynamic allocation algorithm to achieve dynamic weight allocation and dynamic fusion of decisions.

[0062] It should be noted that the model library, as the storage unit of the real-time target detection model, is used to store the trained wire rope target detection model structure and parameter files. At the same time, it supports functions such as online training of models and remote update of the model library. The communication module supports multiple communication mode selections, such as Lora, Zigbee, NB-IOT, 4G, and WIFI wireless communication modes. It facilitates data transmission and remote monitoring of the system. The power module supplies power to the entire device to ensure the stable operation of the system. This complete hardware integration makes the system have good scalability and practicability, and the power module is used to supply power to the entire device.

[0063] See Figure 2, this application provides a steel wire rope defect localization and recognition method based on multi-modal real-time object detection. The specific process is as follows. The steel wire rope passes through four high-definition cameras to obtain videos and pictures in four directions: the front view, the rear view, the left view, and the right view. After image preprocessing, high-quality images are obtained. Feature extraction marks the key parts of the images, etc., so as to obtain modal feature maps in four different directions. At the same time, the steel wire rope also passes through two magnetic probes, and leakage magnetic field signals and magnetic flux signals are obtained by using Hall and coil sensors. After signal conditioning, denoised signals are obtained, and one-dimensional data is plotted and cut to obtain the magnetic field distribution, and leakage magnetic signal modal maps and magnetic flux modal maps are obtained. The two types of sensor arrays obtain images in four different directions and images drawn by two different magnetic signals. The camera images at the same moment are feature-fused to obtain multi-channel image samples, and a detection model A is obtained through training by an NPU neural computing unit. At the same time, the images drawn by the magnetic signals at the same moment are subjected to feature extraction, and the NPU is used for training to obtain detection models B and C respectively. Using the allocation method built into the multi-modal weight management unit, based on the custom dynamic allocation method, the dynamic allocation of the real-time inference weights of the three models A, B, and C is realized, and the decision-making is dynamically fused, and finally the steel wire rope defect localization and recognition results are output. The operation and maintenance personnel can perform auxiliary verification based on the data of the two types of sensors, and finally determine the reliability of the steel wire rope defect type and location. In order to ensure the accuracy of the steel wire rope defect types and positions, the result is rechecked by calling the synchronous detection actuator to realize the secondary confirmation of the result.

[0064] As the core of the multi-modal real-time object detection of steel wire ropes, the model structure determines the accuracy of the steel wire rope defect recognition and localization. Here, based on the lightweight improvement network (yolov11) series of object detection algorithms, lightweight improvement is carried out, and GhostNetV3 is used to replace the original backbone network, thereby reducing the parameter complexity. At the same time, a neighborhood-aware self-attention mechanism is added to mine the key information of the feature map. Considering the problem of blurred boundaries in the steel wire rope object detection, the problem of defect boundary regression is converted into a classification problem, and label prediction is converted into sequence prediction. The middle position of the steel wire rope defect is the original prediction position , by converting the prediction result into a sequence, it becomes , where, , is the predicted label sequence, is the total number of sequences.

[0065] By expanding and probability, the network can quickly focus on the values near the label. Since the learning of the bounding box is only applicable to positive samples and there is no problem of class imbalance, the loss of the steel wire rope object detection model is defined as , where, is the complete cross - entropy value, is the predicted value, is the value before prediction, is the value after prediction. By expanding the probabilities of the values on both sides of the boundary, that is, by expanding , ; , , where is the entropy value before prediction, is the entropy value after prediction, so as to ensure that the regression target is infinitely close to the true label of the corresponding wire rope defect position .

[0066] As the core of the dynamic allocation of wire rope multi - modal weights, the custom dynamic allocation method determines the reliability of the wire rope defect recognition and positioning results. The four modal images generated by the four - channel camera, after feature fusion, are put into the detection model for inference to obtain the inference results. At the same time, after the leakage magnetic flux modal image and the magnetic flux modal image generated by the two - channel magnetic probe are respectively subjected to feature extraction, they are put into the detection model for inference to obtain their respective inference results. Considering that the contribution degrees of the multi - modal feature maps to the results are different, the custom dynamic allocation method determines the weight coefficients based on the principal component feature ratio of the multi - modal maps and the iterative training of the multi - modal maps.

[0067] The contribution degrees of the first ten principal components of the fused modal image of the wire rope camera are , and the confidence level is , and the corresponding multi - modal model weight is , then , the contribution degrees of the first ten principal components of the modal image of the wire rope leakage magnetic probe are , and the confidence level is , the contribution degrees of the first ten principal components of the modal image of the wire rope magnetic flux probe are , and the confidence level is ; the corresponding multi - modal model weights are respectively , , then

[0068] ,

[0069] ,

[0070] The finally obtained normalized weights are respectively , , , where is the camera weight, is the normalized camera weight, is the signal weight of the leakage magnetic probe, is the normalized signal weight of the leakage magnetic probe, is the signal weight of the magnetic flux probe, is the normalized weight of the magnetic flux probe signal;

[0071] Subsequently, through iterative training, the weights of the training samples are updated. The weights of the correctly classified samples decrease, while the weights of the misclassified samples increase. In this way, in the next round of training, the model will pay more attention to the samples that were misclassified previously, thereby gradually improving the performance of the model. The sample weight update formula is as follows: , where is the weight of the sample in the -th round , is the weight of the sample in the -th round , is the number of training rounds, is the modal assignment weight, is the true label, is the predicted label of the sample , and exp is the exponential function.

[0072] At the same time, dataset-assisted verification is added. The result can be intervened by manually verifying the labels and retrieving the steel wire rope defect signals, realizing decision fusion, and finally giving the defect location and type.

[0073] It should be noted that the image information and magnetic signal information of the steel wire rope are collected through a four-channel camera and a two-channel magnetic probe respectively. This multi-modal data acquisition method can obtain the state information of the steel wire rope from different angles. Compared with single-modal data acquisition, it can more comprehensively reflect the actual situation of the steel wire rope and reduce misjudgment caused by the limitations of a single data type. The image features collected by the camera and the magnetic signal features collected by the magnetic probe are fused to obtain multi-channel image samples. This fusion can comprehensively utilize the advantages of different modal data, extract more features related to steel wire rope defects, and improve the accuracy of defect recognition.

[0074] In this embodiment, considering the light environment and surface oil pollution problems of the steel wire rope, the image processing unit in the system can optimize the image resolution and display effect. This helps to obtain clear and usable image data even in a complex environment, ensuring that subsequent feature extraction and analysis can be carried out accurately.

[0075] In terms of model training, we made lightweight improvements based on the Yolov11 series target detection algorithm, and used GhostNetV3 to replace the original backbone network, reducing the complexity of parameters. This not only improves the operational efficiency of the model, but also speeds up the detection speed without losing too much accuracy, meeting the needs of real-time detection. Adding a neighborhood-aware self-attention mechanism can mine key information from feature maps, further improving the model's ability to extract wire rope defect features, and helping to identify defects more accurately.

[0076] Furthermore, the multimodal weight management unit can dynamically allocate weights according to the contribution of different modal data to the results through the built-in multimodal dynamic allocation algorithm. This dynamic allocation mechanism can adaptively adjust the importance of each modal data in decision-making and improve the adaptability of the entire system to different environments and wire rope states. Based on weight allocation, dynamic decision fusion of multi-channel image samples can comprehensively consider the analysis results of each modality, avoid the one-sidedness of single modal decision-making, and thus output wire rope defect location and identification results more accurately.

[0077] In this embodiment, the operation and maintenance personnel can perform auxiliary verification based on the two types of sensor data, and perform back-check control by calling the synchronous detection actuator to achieve secondary confirmation of the results. This multiple verification mechanism can effectively improve the reliability of defect identification results and ensure the accuracy of the detection results. Through the sample weight update formula, during the iterative training process, the weight of correctly classified samples is reduced, while the weight of incorrectly classified samples is increased. This allows the model to pay more attention to previously misclassified samples in subsequent training, gradually improve model performance, and continuously improve the accuracy of defect identification.

[0078] The above specific embodiments are only several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant inspirations of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A wire rope defect recognition and positioning method based on real-time target detection, characterized in that: The following steps are involved: S1. Use four high-definition cameras to obtain videos and pictures of the wire rope in four directions: front, back, left and right, and perform image preprocessing and feature extraction; S2. Obtain the leakage magnetic field signal and magnetic flux signal of the wire rope through two magnetic probes, and perform signal conditioning and image conversion to obtain the leakage magnetic field signal modal diagram and magnetic flux modal diagram; S3, feature fusion of the camera image and the magnetic probe signal image at the same time to obtain a multi-channel image sample; S4, input the multi-channel image samples into the NPU neural computing unit for training to obtain detection model A, detection model B and detection model C; S5. Using a multimodal weight management unit, assigning modal weights to detection model A, detection model B, and detection model C; S6. Perform dynamic decision fusion on multi-channel image samples based on modal weight allocation, and output wire rope defect location and recognition results; The S2 specifically includes: Use Hall and coil sensors to obtain the leakage magnetic field signal and magnetic flux signal of the wire rope; Condition the acquired signal and remove noise; The one-dimensional data is plotted and cut to obtain the magnetic field distribution, and then the leakage magnetic signal modal diagram and the magnetic flux modal diagram are obtained; In S4, the multi-channel image samples are input into the NPU neural computing unit, and the detection model A, the detection model B and the detection model C are obtained by training using a lightweight improved algorithm based on the yolov11 series target detection algorithm; The S5 specifically includes: According to the multimodal dynamic allocation algorithm, based on a custom dynamic allocation method, modal weights are allocated to detection model A, detection model B, and detection model C; The first ten principal components of the fusion modal graph of the wire rope camera have a contribution of The confidence level is θ i , the corresponding multimodal model weight is γ A ,but The first ten principal components of the modal diagram of the wire rope magnetic flux leakage probe contribute to The confidence level is θ j , the first ten principal components of the modal diagram of the wire rope flux probe contribute to The confidence level is θ k ; The corresponding multimodal model weights are γ B ,γ C ,but The final normalized weights are Among them, γ A is the camera weight, is the camera normalization weight, γ B is the magnetic flux leakage probe signal weight, is the normalized weight of the magnetic flux leakage probe signal, γ C is the flux probe signal weight, Normalize weights for flux probe signals.

2. The wire rope defect identification and positioning method based on real-time target detection according to claim 1 is characterized in that: The S6 specifically includes: According to the modal weight distribution results, dynamic decision fusion is performed on multi-channel image samples; Output wire rope defect location and identification results; The operating personnel conduct auxiliary verification of the two types of sensor data and call the synchronous detection actuator for back-check control to achieve secondary confirmation of the results.

3. The wire rope defect identification and positioning method based on real-time target detection according to claim 2 is characterized in that: It also includes model optimization, which includes the following steps: The middle position of the wire rope defect is the original predicted position middle = left + (right - left) / 2. By converting the prediction results into a sequence, it becomes: Among them, y i ∈{y0,y1,y2,…,y n-1 },y i is the label sequence, n is the total number of sequences; By expanding y i-1 and i+1 The probability of making the network quickly focus on the value near the label, and the loss of the wire rope target detection model is defined as: WireropeLoss(CE i WHAT i+1 ) =-((y i+1 -y)log(CE i )+(yy i-1 )log(CE i+1 )) Among them, CE is the complete cross entropy value, y is the predicted value, and y i-1 To predict the previous value, y i+1 To predict the next value, we expand the probability of the values ​​on both sides of the boundary, that is, expand y i-1 ,y i+1 Among them, CE i To predict the previous entropy value, CE i+1 To predict the next entropy value, ensure that the regression target is infinitely close to the true label y of the corresponding wire rope defect position; According to the sample weight update formula, the training sample weights are updated so that in the next round of training, the model will pay more attention to the samples that were previously misclassified, thereby gradually improving the model performance. The sample weight update formula is: in, is the weight of sample i in round t+1, is the weight of sample i in round t, t is the number of training rounds, Assign weights to the modes, y i is the true label, P t (x i ) is the predicted label of sample i, and exp is the exponential function.

Images